Here’s Why Solana Ceased Block Production for 7 Hours on Saturday

Don’t miss CoinDesk’s Consensus 2022, the must-attend crypto & blockchain festival experience of the year in Austin, TX this June 9-12.

Bots tied to a new non-fungible token (NFT) project built on Solana caused a seven-hour network outrage on Saturday, project developers confirmed in a post on Tuesday. No new blocks were produced by the network during that time.

Solana processes an average of 2,700 transactions per second (tps), blockchain explorers show, with an upper peak of over 710,000 tps on a standard network, as per developer documents.

However, on Saturday night millions of transactions flooded the network each second. This led to network validators – or entities that process transactions on and support the network – to run out of memory and crash, developers said.

“An enormous amount of inbound transactions (6 million per second) flooded the network, surpassing 100 Gbps of traffic at individual nodes,” developers wrote. “There is no evidence of a denial of service attack, but instead evidence indicates bots tried to programmatically win a new NFT being minted using the popular Candy Machine program.”

Validators ran out of memory while trying to clear up transactions in other, abandoned blocks. The number of total forks that validators had to evaluate exceeded their capacity to do so automatically, which led to a manual intervention to fix the issue.

Bots had swarmed the popular NFT minting tool known as Candy Machine on Saturday, as reported. This was mainly to take part in a new NFT project mint.

However, that mint used a fixed price for users instead of a Dutch auction – the usual process followed by crypto projects – which incentivized bots to send a huge number of transactions in hopes of winning the mint. This caused spam on the network.

Meanwhile, developers noted Solana has suffered intermittent congestion issues several times in the past months – with most of these resulting from bot activity targeted at NFT mints.

Story continues

Changes are coming to combat those problems. Developers said the upcoming v1.10 release, currently stabilizing on testnet, includes memory use improvements to prolong the time nodes can endure slow or stalled consensus, which would help mitigate issues like those seen on Saturday.

Fees for prioritizing transactions are coming to Solana as well, developers added. This could help mitigate the entirety of Solana getting affected because of a single project.

Solana’s SOL tokens trade at over $88 at writing time, as per CoinGecko data. Prices are down a nominal 1.8% in the past 24 hours.