Magnify #7 — Too hot to handle?

I know I missed the last week’s Magnify. I had my eyes zoomed in on some protocol developments that couldn’t be delayed. Everything’s on track now. Expect to be rocked soon. 😎

It’s a wonderful week, you have just seen Solana reaching all-time highs, the crypto fear and greed index is showing no signs of giving up, and the September winter scare has also not arrived yet. You open Sunny (the top yield aggregator on Solana), and transfer funds via your Phantom wallet. You wait.

And you wait.

Wait, wasn’t this supposed to be quick?

It was, but then the transactions stop processing. And as you scruffily open your Twitter at 8:38 AM ET, Solana Foundation’s tweet bogs you down.

Ugh, this is annoying. Just when you thought it was raining “scalability”, this happens.

Here’s how it all went down.

• At 8:38 AM ET, the Solana Foundation tweeted that the network was experiencing “intermittent instability”. About an hour later, the Phantom wallet tweeted that their wallet and other applications were having a hard time connecting.
• At 10:26 AM, the Solana Foundation cited resource exhaustion as the reason behind “denial of service”. A potential restart was suggested.
• The restart instructions were distributed at 3:21 PM ET. Meanwhile, the ballooning (roughly) $11B TVL on the chain fell into limbo.

The root cause: While the team behind the high-throughput offering L1 is yet to prepare a detailed report, it has been suggested that the disruption was caused by bot trading of Grape Protocol’s token offering. Since the TPS suddenly shot up to 400K, the network couldn’t handle the volume, the transaction processing queue got clogged up, and the lack of prioritization of “network-critical messaging” eventually caused the network to start forking. Restarting the network was the only option, but coordinating with 900+ validators over different timezones meant the issue was abhorrently prolonged.

You’d think that users would start cursing Solana after this on Twitter? Not really. Currently, Solana users are in their honeymoon phase — they are swooning over the network’s high throughput and snappy experience. While the outage was painful, the response to it has been largely natural.

Solana wasn’t alone in experiencing downtime. Arbitrum One, which was also launched just at the beginning of September also had minor problems with its Sequencer being down for about 45 minutes; however, the main network remained unaffected.

• In their report, they assured that users’ funds were never at risk. All transactions that had been accepted by the Sequencer were not re-ordered.
• In fact, the timestamps were re-assigned when the Sequencer returned online, roughly 45 minutes later.
• The team assured the users that the Sequencer is incapable of stealing funds or forging transactions — this is because each transaction is digitally signed by a user and the signatures are checked by the Arbitrum chain.

The root cause: There was a bug causing the Sequencer to get stuck when it received a very large burst of transactions in a short period of time.

Now, we could go all in to understand what the reasons behind these outages were — but that is not as essential as understanding who gets affected in instances like these? Is it the users? Is it the chain? Or the dApps?

While it is hard to determine the type of issues that users faced during the outage, there are ways that they could be harmed in such cases.

• For some users who are transacting, their transactions are stuck midway and are confused about the gas fees.
• For those users who are borrowing assets, their over-collateralized loans can soon get under-collateralized leading to liquidations.
• For those users who resume trading when the network does restore (after 17 hours on Solana, to be exact) they find that the prices on DEXes are all messed up. And a race between arbitrageur bots is on to make use of the opportunity. MEV bots are falling in line to bag their share of profits.

While these are still presumptions, had the market been more volatile, all of them could have turned out to be true.

All assurances about the networks being resilient aside, they are the ones who are bearing the brunt of these ramifications. And we mustn’t forget that these users are not forgiving. Just look at the plummet of TVL on Arbitrum and Solana and you’ll know.

While both the teams may be scrambling to find ways to ensure this downtime — if at all it happens in the future — is as short as possible, there are some major questions that we must find answers to.

• Do we have major fall-back options?
• What will we do as the ecosystem matures further and the number of users interacting with these platforms increases?

What do you think? Are these blockchains as immutable as they claim themselves to be? Or are these limitations inherent and cannot be dealt away with?

ICYMI

I did an under-the-hood thread of Solana, exploring its tech and explaining why it was getting all the craze. Read to find why I felt it was going to be successful.

Come hang out with me on Twitter @mohakagr