Missed slots caused by gossip issues on consensus layer: Addressing late block reorgs on Ethereum

Blocknative Ethereum Web3

On April 15th, 2023, Blocknative’s internal MEV-relay alerting system began picking up an unusual pattern of missed slots. Upon further inspection, we found that these missed slots occurred due to unusual late block (proposer) reorgs. The reorgs appeared to involve blocks across the entire Ethereum network regardless of the builder or relay from which they originated.

As a relay operator, we understood the gravity of this situation and its impact on the network. Some validators were losing their proposer rewards. In addition, users faced longer confirmation times and increased uncertainty about the finality of their transactions.

We collaborated with the consensus layer teams, specifically Prysm and Lighthouse, to identify and rectify the issue.

 

Collaborating with consensus clients to identify the cause of the late block reorgs

After determining that these reorgs always happened on the second slot of an epoch, we began working with the Prysm and Lighthouse teams to pinpoint the cause.

Their analysis identified an issue in Prysm’s gossip protocol that slowed the propagation of the first slot in an epoch. The slow propagation of slot one resulted in the relays, validators, and builders in slot two never seeing the first slot. To them, slot one was a missed slot. Consequently, everything built, validated, and queried in slot two referenced the last slot of the previous epoch, instead of the first slot of the new epoch: 

OFpP6rzeWZOgWtjuQocfyDuj6qujR5BXi5OeQ3DVBDwpsk_Oqb9JS8eAMVOlInPIzvOlh2n9MQFR_BuKUp1J4ULh-c4nR2aix-T2PtyDzkZXIUaCSNU7wwixSvSVYrkf5GUlwN0xMQwvr4UFN7nUB

Importantly, both blocks in slots one and two were successfully validated and published on time. But, because two slots cannot share the same parent, slots one and two cannot coexist. Only one state can remain in the canonical chain, which ultimately resulted in a missed slot.

 

Finding a Fix: Next steps for validators

On June 15, 2023, Prysm released an update that changes the way the first two slots are produced and propagated throughout the network, fixing the issue that causes these late block reorgs. This fix can be found on the Prysm github here: https://github.com/prysmaticlabs/prysm/releases/tag/v4.0.6 

This event underscores the importance of collaboration and adaptability in maintaining the health of the Ethereum network. These shared efforts and continuous learning help to ensure a reliable and secure blockchain ecosystem for all participants. We, at Blocknative, are committed to ensuring the robustness and stability of the Ethereum network.

If your validator currently runs on Prysm, we recommend updating to the newest release as soon as possible. As this incident highlights, your validator directly impacts other validators. Updating your instance of Prysm will help to improve Ethereum’s consensus layer across the board and can help prevent missed slots from these late block reorgs for your peers. It is important to note that because this issue stems from gossip protocol propagation, your validator can still be exposed to these missed slots if your peers have not upgraded.

Thank you to the Prysm and Lighthouse team, especially Potuz and Sproul, for their help with identifying and fixing the issue. If you have any questions, be sure to join our Discord to chat with our team.

Observe Ethereum

Blocknative's proven & powerful enterprise-grade infrastructure makes it easy for builders and traders to work with mempool data.

Visit ethernow.xyz

Want to keep reading?

Good choice! We have more articles.

blocknative-gas-estimator-extension-now-supports-multiple-blockchains!
Gas

Blocknative Gas Estimator Extension Now Supports Multiple Blockchains!

Big News: Blocknative Gas Estimator Extension Now Supports Multiple Blockchains! We're super..

understanding-blocknative's-gas-estimator:-why-do-some-predictions-show-the-same-values?
Gas

Understanding Blocknative's Gas Estimator: Why Do Some Predictions Show the Same Values?

Understanding Blocknative's Gas Estimator: Why Do Some Predictions Show the Same Values? If you've..

base-101:-how-gas-works-on-base
Gas

Base 101: How Gas Works On Base

Incubated and developed by the popular centralized exchange Coinbase, Base is an Ethereum Layer..

Connect with us. Build with us.

We love to connect with teams who are building with Blocknative. Tell us about your team and what you would like to learn.

"After first building our own infrastructure, we appreciate that mempool management is a difficult, expensive problem to solve at scale. That's why we partner with Blocknative to power the transaction notifications in our next-generation wallet."

Schedule a demo