FLASH NOTE — Solana Network Issues

What Happened?

On Saturday, April 30, 2022, Solana’s Mainnet Beta cluster ceased creating new blocks as a result of network congestion. The network required a manual restart and was back up by Sunday, May 1. Overall, the outage lasted just over eight hours, significantly shorter than the previous network outage in September 2021 that lasted more than 17 hours. The cause of the congestion appears to have stemmed from bots attempting to win an NFT mint conducted on Metaplex’s Candy Machine. According to Solana Labs, these bots contributed to more than 6 million transaction requests per second, with some validators receiving over 100GB per second of traffic. Due to the massive amount of network spam, many validators ran out of the memory and crashed.

For reference, Metaplex is the largest NFT infrastructure provider on Solana having supported the minting of more than 10.6M NFTs worth over $2.5B in total sales. The company has a strategic partnership with Solana Labs and is integrated with more than 500 projects on the blockchain. Candy Machine V2 (“CMv2”) is Metaplex’s fully on-chain generative NFT distribution program, which is used to mint NFT projects in a reliable but random manner. A core solution provided by CMv2 is the ability to request a mint transaction without the risk of losing assets if the mint is completed. This capability, coupled with a fixed floor price for mints and naturally low transaction costs, resulted in the creation of a large community of bots with the primary goal of acquiring as many NFTs as possible from collections minting via CMv2.

As bots submitted massive amounts of transaction requests to CMv2-supported auctions, Solana validators became inundated and eventually crashed. Manual intervention was required as the number of abandoned forks, or versions of the Solana blockchain that have not received consensus from the validator network exceeded the capacity of multiple validators. Put simply, the number of transactions submitted by bots for pre-filled auction slots resulted in too many versions of the blockchain for validators to parse, overloading their memory and causing nodes to crash. This cascade failure prevented an agreement on the active chain — a breakdown of consensus.

The Solution

While many have proclaimed the death of Solana due to its inability to withstand a simple NFT minting process, solutions are being implemented that should resolve these issues and restore Solana’s stability, refocusing public sentiment on the network's high TPS and low transaction costs.

First, Metaplex has already implemented what they view as a solution to the bot exploitation of CMv2. As referenced in their recent GitHub post, Metaplex has implemented a “bot tax” of 0.01 SOL or the entire balance in the specific wallet, whichever is greater. This tax will be collected and then sent to the Token Metadata program account (aka the NFT issuer). As provided in the GitHub post, whenever a wallet attempts to mint an NFT prior to the start of a sale or after the total supply has been purchased, a tax will be levied on that wallet address. The provision is controversial in that both events have the potential to impact real users, however Metaplex rates this risk as relatively low. Since implementation, Metaplex has collected and distributed 1746 SOL from bots, worth ~$80,000 at current prices. According to the table below, the tax has yet to deter the blacklisted transactions, as taxes collected clearly fluctuate during periods of high NFT minting events (i.e. Friday – Sunday).