Missed slots caused by gossip issues on consensus layer: Addressing late block reorgs on Ethereum

Blocknative Ethereum Web3

On April 15th, 2023, Blocknative’s internal MEV-relay alerting system began picking up an unusual pattern of missed slots. Upon further inspection, we found that these missed slots occurred due to unusual late block (proposer) reorgs. The reorgs appeared to involve blocks across the entire Ethereum network regardless of the builder or relay from which they originated.

As a relay operator, we understood the gravity of this situation and its impact on the network. Some validators were losing their proposer rewards. In addition, users faced longer confirmation times and increased uncertainty about the finality of their transactions.

We collaborated with the consensus layer teams, specifically Prysm and Lighthouse, to identify and rectify the issue.

 

Collaborating with consensus clients to identify the cause of the late block reorgs

After determining that these reorgs always happened on the second slot of an epoch, we began working with the Prysm and Lighthouse teams to pinpoint the cause.

Their analysis identified an issue in Prysm’s gossip protocol that slowed the propagation of the first slot in an epoch. The slow propagation of slot one resulted in the relays, validators, and builders in slot two never seeing the first slot. To them, slot one was a missed slot. Consequently, everything built, validated, and queried in slot two referenced the last slot of the previous epoch, instead of the first slot of the new epoch: 

OFpP6rzeWZOgWtjuQocfyDuj6qujR5BXi5OeQ3DVBDwpsk_Oqb9JS8eAMVOlInPIzvOlh2n9MQFR_BuKUp1J4ULh-c4nR2aix-T2PtyDzkZXIUaCSNU7wwixSvSVYrkf5GUlwN0xMQwvr4UFN7nUB

Importantly, both blocks in slots one and two were successfully validated and published on time. But, because two slots cannot share the same parent, slots one and two cannot coexist. Only one state can remain in the canonical chain, which ultimately resulted in a missed slot.

 

Finding a Fix: Next steps for validators

On June 15, 2023, Prysm released an update that changes the way the first two slots are produced and propagated throughout the network, fixing the issue that causes these late block reorgs. This fix can be found on the Prysm github here: https://github.com/prysmaticlabs/prysm/releases/tag/v4.0.6 

This event underscores the importance of collaboration and adaptability in maintaining the health of the Ethereum network. These shared efforts and continuous learning help to ensure a reliable and secure blockchain ecosystem for all participants. We, at Blocknative, are committed to ensuring the robustness and stability of the Ethereum network.

If your validator currently runs on Prysm, we recommend updating to the newest release as soon as possible. As this incident highlights, your validator directly impacts other validators. Updating your instance of Prysm will help to improve Ethereum’s consensus layer across the board and can help prevent missed slots from these late block reorgs for your peers. It is important to note that because this issue stems from gossip protocol propagation, your validator can still be exposed to these missed slots if your peers have not upgraded.

Thank you to the Prysm and Lighthouse team, especially Potuz and Sproul, for their help with identifying and fixing the issue. If you have any questions, be sure to join our Discord to chat with our team.

Observe Ethereum

Blocknative's proven & powerful enterprise-grade infrastructure makes it easy for builders and traders to work with mempool data.

Visit ethernow.xyz

Want to keep reading?

Good choice! We have more articles.

cut-through-the-noise:-how-to-quickly-leverage-filters-to-find-real-time-insights-on-ethereum
Ethernow

Cut Through the Noise: How to Quickly Leverage Filters to Find Real-time Insights on Ethereum

With Ethernow, you get a real-time stream of what is happening right NOW on Ethereum. That means..

blocknative-supports-the-polygon-amoy-testnet
Polygon

Blocknative Supports the Polygon Amoy Testnet

The Polygon ecosystem is thriving, and Blocknative is excited to announce its support for the new..

announcing-the-blocknative-blob-archive-api:-preserving-the-ephemeral-data-layer
Blob

Announcing the Blocknative Blob Archive API: Preserving the Ephemeral Data Layer

Did you miss a blob? Do you want to know what was in a blob that went on-chain a month ago? We are..

Connect with us. Build with us.

We love to connect with teams who are building with Blocknative. Tell us about your team and what you would like to learn.

"After first building our own infrastructure, we appreciate that mempool management is a difficult, expensive problem to solve at scale. That's why we partner with Blocknative to power the transaction notifications in our next-generation wallet."

Schedule a demo