On April 15th, 2023, Blocknative’s internal MEV-relay alerting system began picking up an unusual pattern of missed slots. Upon further inspection, we found that these missed slots occurred due to unusual late block (proposer) reorgs. The reorgs appeared to involve blocks across the entire Ethereum network regardless of the builder or relay from which they originated.
As a relay operator, we understood the gravity of this situation and its impact on the network. Some validators were losing their proposer rewards. In addition, users faced longer confirmation times and increased uncertainty about the finality of their transactions.
We collaborated with the consensus layer teams, specifically Prysm and Lighthouse, to identify and rectify the issue.
Collaborating with consensus clients to identify the cause of the late block reorgs
After determining that these reorgs always happened on the second slot of an epoch, we began working with the Prysm and Lighthouse teams to pinpoint the cause.
Their analysis identified an issue in Prysm’s gossip protocol that slowed the propagation of the first slot in an epoch. The slow propagation of slot one resulted in the relays, validators, and builders in slot two never seeing the first slot. To them, slot one was a missed slot. Consequently, everything built, validated, and queried in slot two referenced the last slot of the previous epoch, instead of the first slot of the new epoch:
Importantly, both blocks in slots one and two were successfully validated and published on time. But, because two slots cannot share the same parent, slots one and two cannot coexist. Only one state can remain in the canonical chain, which ultimately resulted in a missed slot.
Finding a Fix: Next steps for validators
On June 15, 2023, Prysm released an update that changes the way the first two slots are produced and propagated throughout the network, fixing the issue that causes these late block reorgs. This fix can be found on the Prysm github here: https://github.com/prysmaticlabs/prysm/releases/tag/v4.0.6
This event underscores the importance of collaboration and adaptability in maintaining the health of the Ethereum network. These shared efforts and continuous learning help to ensure a reliable and secure blockchain ecosystem for all participants. We, at Blocknative, are committed to ensuring the robustness and stability of the Ethereum network.
If your validator currently runs on Prysm, we recommend updating to the newest release as soon as possible. As this incident highlights, your validator directly impacts other validators. Updating your instance of Prysm will help to improve Ethereum’s consensus layer across the board and can help prevent missed slots from these late block reorgs for your peers. It is important to note that because this issue stems from gossip protocol propagation, your validator can still be exposed to these missed slots if your peers have not upgraded.
Thank you to the Prysm and Lighthouse team, especially Potuz and Sproul, for their help with identifying and fixing the issue. If you have any questions, be sure to join our Discord to chat with our team.
