Mempool Archive Quickstart: How to use Blocknative's historical Ethereum mempool data to analyze private transactions, MEV, and OFAs

Blocknative Ethereum Web3

Blocknative offers the most exhaustive historical archive of Ethereum's mempool transaction events, encompassing >15 TB with over 5 billion transactions since November 2019. The dataset comprises 27 detailed data fields such as gas details, input data, time pending in the mempool, failure reasons, and regional timestamps for each instance seen by our global network of nodes.

Researchers can dive into the data for insights on major network events like massive surges in traffic, huge gas spikes, the launch of MEV-boost, the proliferation of Order Flow Auctions (OFAs) and private transactions, major hacks, and more. 

This quickstart will guide you through how to combine our mempool archive data with third-party datasets to dive deeper into private transactions for additional insights into OFA and MEV bot usage. 

Researchers can sign up for free access to our Blocknative Mempool Data Program here; for a commercial license, please reach out.

Understanding Private Transaction Volume Attribution

When a public transaction first enters the mempool, it is marked with a pending status. At block inclusion time, the public transaction status updates to confirmed.

Transactions bypassing the public mempool are known as private transactions. In our Mempool Archive, private transactions can be identified by filtering transactions that only have a confirmed status. The lack of a pending status signifies they have not been detected by our nodes and have likely bypassed the public mempool.

We can then further attribute this private transaction volume into OFAs, MEV bots, and the unknown thorough the use of third-party datasets.

For this quickstart, let’s focus on June 1, 2023!

Downloading the Data

For our examples below, we used the following datasets:

Mempool Archive Dataset

The dataset is partitioned by day and hour. For a detailed schema see: Blocknative Mempool Archive - Schema

MEV Bot Searcher Dataset

Searchers are a class of actors that bundle a group of transactions together and privately send them to Block Builders for MEV extraction. Before the rise of special MEV Protectioon RPCs, Searcher transactions comprised of nearly 70% of private transaction volume.

There are many different datasets that label various searcher MEV bots addresses you can choose from, such as BitWise, Frontier.Tech, and EigenPhi. We’ll use a combination of all of them for our analysis.

OFA (Order Flow Auctions) Dataset

Order Flow Auctions (OFAs) are special RPC endpoints that offer end users protection from the negative externalities of MEV by enabling them to submit transactions privately (often with the opportunity to be backrun by Seachers and receive a portion of the profits). 

OFA examples include Flashbots ProtectMEVBlocker, and Blocknative's Transaction Boost.

Data sources include:
Dune - MEVBlocker
MEV-Share Historical Hint Stream
Transaction Boost

⚠️
When an on-chain transaction is routed through several OFAs, it is not possible to determine the true source. We label any private volume as Seen on Special RPC

Not all OFAs share their order stream publicly, so the created dataset will always be a lower estimation of OFA market share.
⚠️

Exploring Private Transaction Data

Filtering out Private Transactions

Private transactions are transactions without a pending event in the Mempool Archive. To identify transactions without a pending event in the Blocknatve Mempool Archive, we will use the timepending column.

 The timepending field identifies the time a transaction has spent in the mempool and is  calculated by

in millisecond units.

All private transactions will have a timepending = 0, since there were no pending events for the transaction. For our analysis, we will also exclude any transactions with a status of cancel or speedup.


CREATE TABLE PRIVATE_TRANSACTIONS AS 
(
SELECT
  *
FROM
  MEMPOOL_ARCHIVE_SAMPLE
WHERE
  timepending = 0
  AND status not in ('speedup', 'cancel')
 )
 

Doing a count of total confirmed/failed transactions with timepending = 0, we see that 97% of private transactions were confirmed and 3% failed on January 06, 2023.

upload_31bf6c1f97c9680a2206d060279c09ce-1

 

Identifying if private transactions were from MEV Bots

We can identify if a transaction was sent by MEV bot or not by using our labeled MEV Bot Searcher dataset we downloaded previously.

By combining our Searcher data with the private transaction data from our Mempool Archive, we can see that 22% of private transactions on January 06, 2023 were sent by MEV bots.

 

Identifying transactions sent from OFAs

Similar to the above, we can use our downloaded OFA dataset to further analyze our private transaction data.

Doing so shows that nearly 50% of the private transactions in our sample can be attributed to OFAs!

upload_45066f19362d7e9933914dca8bae3a79

Beyond the Data

The share of private transactions has steadily increased with the introduction of OFAs. The adoption of OFAs will likely continue as they promise users speed and malicious MEV protection. End users will continue to seek privacy-preserving and guaranteed order execution free from MEV. For these reasons, private mempools will likely be a mainstay for the foreseeable future.

Continued research into private transactions is important to avoid centralizing forces in the ecosystem. We welcome the community to join in on this analysis! Feel free to reach out on Discord if you have any questions.

 
 

Observe Ethereum

Blocknative's proven & powerful enterprise-grade infrastructure makes it easy for builders and traders to work with mempool data.

Visit ethernow.xyz

Want to keep reading?

Good choice! We have more articles.

how-to-unstuck-a-transaction-with-metamask:-a-step-by-step-guide
Developer

How to Unstuck a Transaction with MetaMask: A Step-by-Step Guide

A stuck transaction is a transaction that you submitted to go on-chain but it is unable to be..

what-is-a-stuck-transaction?
Gas

What is a Stuck Transaction?

Understanding Stuck Transactions In the world of blockchain technology, transactions are the..

web3-onboard-supports-coinbase-smart-wallet-from-day-one
Developer

Web3 Onboard Supports Coinbase Smart Wallet from Day One

Web3 Onboard Supports Coinbase Smart Wallet We are thrilled to announce that Web3 Onboard now..

Connect with us. Build with us.

We love to connect with teams who are building with Blocknative. Tell us about your team and what you would like to learn.

"After first building our own infrastructure, we appreciate that mempool management is a difficult, expensive problem to solve at scale. That's why we partner with Blocknative to power the transaction notifications in our next-generation wallet."

Schedule a demo