Thank you to all those who provided input and review on this piece, including Julian Ma, Soubhik Deb, Barnabe Monnot, Max Resnick, Mallesh Pai, Davide Crapis, Ed Felten, Roberto Bayardo, Potusz, and the Blocknative team.
On Thursday, June 20, 2024, the Blob Base Fee spiked to 8,000 GWEI, making blobdata (type 3 transactions) more expensive than calldata (type 2 transactions) for the second time since the Dencun upgrade. During the inversion event, L2s incurred an estimated overpayment of ~166 ETH (approximately $550,000 USD) by including data in batches sent to Ethereum as blobdata instead of calldata.
What happened? Why did we see another spike similar to the blobscription craze?
The intertwining of the Blob Market and the MEV-Boost auction market makes answering these questions difficult, but the Blocknative team dug into the Blob Archive to attempt to shed light on this event and create working theories for the ecosystem to explore.
For this piece, we will refer to type 3 transactions as blobdata and type 2 transactions as calldata.
What Happened?
At the start of the event (block 20134150) the blob discount was nearly 90%. But by block 20134260, calldata was over 90% cheaper. This inversion took only 100 blocks - 20 minutes - for the entire market blob market to flip completely.
When it was clear that calldata was much cheaper than blobdata, Arbitrum and OP Mainnet implemented a dynamic batch posting process to switch between blobdata and calldata. Unfortunately, none of the other L2s switched during the spike causing them to overpay significantly to get transactions on-chain. OP Mainnet switched later than Arbitrum and therefore still overpaid on many batches.
In total, L2s overpaid by ~166ETH or ~$550K USD. The charts below show the breakdown of overpayment by major L2s and the total combined overpayment over the 350 blocks where blobdata was more expensive than calldata.
Why Did It Happen?
On June 20th, 2024 the Arbitrum LayerZero airdrop occurred increasing the transaction volume on Arbitrum. Users on Arbitrum were willing to pay the increasing fee, which in turn caused Arbitrum to post more batches to Ethereum. Below is a graph showing the increasing transaction volume over this period of time on Arbitrum.
To get the extra data on-chain, Arbitrum increased their batch submissions to Ethereum around block 20132500 going from ~20 batches per 100 Ethereum blocks to ~40 batches per 100 Ethereum blocks.
The increase in batch submissions resulted in an increase in confirmation time (the number of blocks during which the batch remained pending in the mempool). This started around block 20132500, with this particular batch being the first of several slow batches. At the particular batch, the priority fee was still 1 GWEI, and the blob base fee was still 1 wei. However, it took 35 blocks to confirm when typically, blobs take ~1.5 blocks to get confirmed on-chain.
Despite the significant increase in confirmation time, Arbitrum maintained a priority fee of 1-4 GWEI and did not significantly replace/speedup their batches before or during the blob base fee spike.
Interestingly, this caused some cascading reactions with other L2s, mainly OP Mainnet. In the graph below we can see that at around block 20132500, OP Mainnet had a big spike in batch transactions as well, going from ~5-10 batches per 100 Ethereum blocks to ~20-40 batches per 100 Ethereum blocks.
Despite posting 2-4x more batches per 100 Ethereum blocks, the pending time in the mempool for their batches remained low and very steady before the blob base fee spiked exponentially.
How did OP Mainnet maintain such low block pending time? They were rapidly replacing their batch transactions. Protocols do not take type 3 transaction replacements lightly: when they replace/speedup a batch type 3 transaction, they have to double their priority fee. Whereas type 2 transactions only require a 10% increase to issue a replacement. Speeding up type 3 transactions comes at a significant cost.
In the graph below we can see that OP Mainnet went from paying about 4-8 GWEI per batch transaction before block 20132500, to paying about 16-32 GWEI per batch transaction after. At the peak, they were paying a priority fee of 256 GWEI!
The data makes this look like a reaction to the increase in Arbitrum batch transactions. When Arbitrum started adding more batch transactions to the blob mempool (without changing their priority fee), OP Mainnet responded by dramatically increasing their speedup behavior, paying much higher fees in the process, to maintain their low confirmation time. Since Arbitrum did not increase their priority fee (or speedup their batch transactions), their confirmation time increased dramatically. Remember, Arbitrum had a priority fee set to about 1 GWEI, and OP Mainnet increased their fee (through speedups) to about 16-32 GWEI.
The below graphs show the interplay between Arbitrum and OP Mainnet.
When we zoom into the block period where the original increase in Arbitrum and OP Mainnet batches happen (blocks 20132000 - 20133000), we can see a normal rhythm between the two until block 20132500. At this point, Arbitrum increased their batch posting and then OP Mainnet reacted by increasing the amount of speedups.
What’s important to note is that Arbitrum posts 3 blobs per batch and OP Mainnet posts 5 blobs per batch. From a data perspective, not all blobs are created equal. When OP Mainnet started speeding up their blobs, they were heavy for the blob mempool and block builders to handle. Below is the blob count in the mempool over the block period in which this occurred: blocks 20132000 - 20133000.
In aggregate, the number of pending blobs in the blob mempool was pretty saturated around block 20132500, topping out at 878 blobs per 100 Ethereum blocks. The graph below shows the pending blob count per 100 Ethereum blocks from block 20131000 to block 20135000.
When we look at confirmed blobs only per 100 Ethereum blocks, we see it takes quite a while before the market consistently averages greater than 3 blobs per block (300 in the graphs). If we look at the graph above, we can see the spike in the mempool around block 20132500 kickstarted a spike in confirmed blobs (the below graph) around block 20132600, but it still wasn’t until later (blocks 20133500 and beyond) until we are clearly over the target blob space (3 blobs per block).
With the way the blob base fee works, it doesn’t take much for the exponential growth to take effect in a short time period. The graph below shows that even as the excess blob gas grew, it wasn’t until the final couple hundred blocks that the blob base fee became meaningful and increased from 1 GWEI to as much as 8,000 GWEI.
What About the Other L2s?
Interestingly, when we look at other L2s we do NOT see the same behavior as OP Mainnet and Arbitrum.
Let’s look at Base batch transactions overlaid with the blob base fee. Base showed very minimal signs of changing their batch posting behavior until the blob base fee completely spiked.
Base did a small increase in speedup behavior around block 20132500, but not to the extent that OP Mainnet did.
Because of this Base’s confirmation time did suffer, but later than Arbitrum and OP Mainnet.
When we look at Scroll, we see that they did do more speedups than normal but had a much later reaction than OP Mainnet. Interestingly, when the base fee spiked and blobdata became more expensive than calldata, Scroll stopped posting blobs altogether. This also explains why Scroll minimally overpaid compared to other L2s.
And finally, Taiko. Since they are a based rollup (ie. their transactions are confirmed on the L2 once they are confirmed on the L1), they must keep posting in order to keep the chain moving. This resulted in them overpaying by 25+ ETH in L1 fees. However, when the blob base fee spiked even Taiko slowed down their batch transactions by 30-50%, demonstrating that they do have some price sensitivity at extremely high prices.
Open Questions and Areas for Future Research
We are excited for the ecosystem to review our initial findings and provide feedback on this piece. As we look ahead, we wanted to pose some questions and thoughts for us to explore.
- The blob marketplace is only getting more saturated, increasing the frequency of these occurrences. L2s must have the appropriate infrastructure to deal with blob gas marketplace conditions and the ability to switch between blobdata and calldata transactions when appropriate to account for this. L2s should consider investigating this signal and look for ways to decision between calldata and blobdata (shill - Blocknative can help).
- Block builders should consider investigating how they handle blob data. This has a few implications:
- Today Builders do not have robust incentives for including blobs in blocks. We should consider research to investigate upgrades to EIP 4844, such as blob tip, to help drive the inclusion of blobs.
- The community should consider better monitoring of how block builders treat blobs. It appears Flashbots block builder was not including blobs at the time, so this may have contributed to the congestion.
- How did different validators and their respective clients react to the flood of blobs in the mempool? At times propagating the rapid increase in data P2P would have been quite expensive and may have been a subpar experience. It would be great to take a more granular view of the P2P layer with respect to understanding blob propagation.
- EIP 4844 was the 0 to 1 moment for the modular scaling thesis on Ethereum and there are kinks to iron out. Some areas of consideration:
- Adding a maxPriorityFeePerBlobGas or similar so that L2s can more effectively price on a per blob basis instead of using maxPriorityFeePerGas.
- Changing the replacement/speedup policies for type 3 transactions. Right now protocols must double their priority fee to do a speedup transaction. Perhaps we need a different penalty here or allow users to update the gas parameters only of a type 3 transaction without needing to repropagate the blob itself.
- Changes to the mempool policy for blobs. As Bert Kellerman points out, the mempool policies might be too restrictive and cause unnecessary delays in blob confirmation time.
- Increasing the supply side by accepting more than 6 blobs per block. This is already under active consideration.
- Changing the max amount of blobs per type 3 transaction. Perhaps allowing 5-6 blobs per type 3 transaction makes the knapsack problem for block builders too restrictive and actually hurts L2s chances of getting on-chain.
Next Steps for L2s
Are you an L2 that wants to improve your chain's performance during the next blob contention event? Reach out on telegram - we are happy to help.
Gas Extension
Blocknative's proven & powerful Gas API is available in a browser extension to help you quickly and accurately price transactions on 20+ chains.
Download the Extension