Real-time blockchain applications. 

On-chain data tells you what has happened. Pre-chain data tells you why it is happening. For anyone who has ever felt transaction anxiety, scratched their head over unexpected slippage, or realized they have been rekt by a MEV bot, visibility into Ethereum’s ephemeral pre-chain layer is often the missing link. This is why we’re excited to introduce a preview release of Ethernow, a new class of real-time transaction explorers that enables end-users to see what is happening at the core of Ethereum, in real-time. We can all finally observe Ethereum's beautifully engineered yet-still-mysterious inner workings.

Blocknative offers the most exhaustive historical archive of Ethereum's mempool transaction events, encompassing >15 TB with over 5 billion transactions since November 2019. The dataset comprises 27 detailed data fields such as gas details, input data, time pending in the mempool, failure reasons, and regional timestamps for each instance seen by our global network of nodes. Researchers can dive into the data for insights on major network events like massive surges in traffic, huge gas spikes, the launch of MEV-boost, the proliferation of Order Flow Auctions (OFAs) and private transactions, major hacks, and more. This quickstart will guide you through how to combine our mempool archive data with third-party datasets to dive deeper into private transactions for additional insights into OFA and MEV bot usage. Researchers can sign up for free access to our Blocknative Mempool Data Program here; for a commercial license, please reach out. Understanding Private Transaction Volume Attribution When a public transaction first enters the mempool, it is marked with a pending status. At block inclusion time, the public transaction status updates to confirmed. Transactions bypassing the public mempool are known as private transactions. In our Mempool Archive, private transactions can be identified by filtering transactions that only have a confirmed status. The lack of a pending status signifies they have not been detected by our nodes and have likely bypassed the public mempool. We can then further attribute this private transaction volume into OFAs, MEV bots, and the unknown thorough the use of third-party datasets. For this quickstart, let’s focus on June 1, 2023! Downloading the Data For our examples below, we used the following datasets: Mempool Archive Dataset The dataset is partitioned by day and hour. For a detailed schema see: Blocknative Mempool Archive - Schema MEV Bot Searcher Dataset Searchers are a class of actors that bundle a group of transactions together and privately send them to Block Builders for MEV extraction. Before the rise of special MEV Protectioon RPCs, Searcher transactions comprised of nearly 70% of private transaction volume. There are many different datasets that label various searcher MEV bots addresses you can choose from, such as BitWise, Frontier.Tech, and EigenPhi. We’ll use a combination of all of them for our analysis. OFA (Order Flow Auctions) Dataset Order Flow Auctions (OFAs) are special RPC endpoints that offer end users protection from the negative externalities of MEV by enabling them to submit transactions privately (often with the opportunity to be backrun by Seachers and receive a portion of the profits). OFA examples include Flashbots Protect, MEVBlocker, and Blocknative's Transaction Boost. Data sources include: Dune - MEVBlocker MEV-Share Historical Hint Stream Transaction Boost ⚠️ When an on-chain transaction is routed through several OFAs, it is not possible to determine the true source. We label any private volume as Seen on Special RPC Not all OFAs share their order stream publicly, so the created dataset will always be a lower estimation of OFA market share. ⚠️ Exploring Private Transaction Data Filtering out Private Transactions Private transactions are transactions without a pending event in the Mempool Archive. To identify transactions without a pending event in the Blocknatve Mempool Archive, we will use the timepending column. The timepending field identifies the time a transaction has spent in the mempool and is calculated by firstConfirmation − firstDetection in millisecond units. All private transactions will have a timepending = 0, since there were no pending events for the transaction. For our analysis, we will also exclude any transactions with a status of cancel or speedup. CREATE TABLE PRIVATE_TRANSACTIONS AS ( SELECT * FROM MEMPOOL_ARCHIVE_SAMPLE WHERE timepending = 0 AND status not in ('speedup', 'cancel') )   Doing a count of total confirmed/failed transactions with timepending = 0, we see that 97% of private transactions were confirmed and 3% failed on January 06, 2023. Identifying if private transactions were from MEV Bots We can identify if a transaction was sent by MEV bot or not by using our labeled MEV Bot Searcher dataset we downloaded previously. By combining our Searcher data with the private transaction data from our Mempool Archive, we can see that 22% of private transactions on January 06, 2023 were sent by MEV bots. Identifying transactions sent from OFAs Similar to the above, we can use our downloaded OFA dataset to further analyze our private transaction data. Doing so shows that nearly 50% of the private transactions in our sample can be attributed to OFAs! Beyond the Data The share of private transactions has steadily increased with the introduction of OFAs. The adoption of OFAs will likely continue as they promise users speed and malicious MEV protection. End users will continue to seek privacy-preserving and guaranteed order execution free from MEV. For these reasons, private mempools will likely be a mainstay for the foreseeable future. Continued research into private transactions is important to avoid centralizing forces in the ecosystem. We welcome the community to join in on this analysis! Feel free to reach out on Discord if you have any questions.

The success of The Merge and the subsequent Shapella hard-fork showcased Ethereum's ability to smoothly ship significant network upgrades. However, this transition was not without its challenges and learning curves. Over the past year, Ethereum’s journey has been punctuated by encounters with malevolent entities, unintended software anomalies, and disparities in value distribution.