Solana network outage on December 4th, 2020
At approximately 1:46pm UTC on December 4th, 2020, the Solana Mainnet Beta cluster stopped producing blocks at slot 53,180,900, which prevented any new transactions from being confirmed. The Solana team is actively working with the validator community to restart the network. No funds are at risk, and the team’s primary focus is performing a smooth restart of the network. As information unfolds, we will continue to provide updates in this post, on Discord, and on our social platforms.
UPDATE: As of 9:30pm UTC 12/7/20, Solana Engineers have identified the issue and are working on a post-mortem. Again, no funds are at risk and the network is functioning normally. Once the issue is fully patched and validators update to the fix, a comprehensive post-mortem will be published.
While we work on the fix, we would like to thank the passionate and diligent global network of validators who identified this issue and coordinated to restart the network.
Live Updates
1:46pm UTC: The Mainnet Beta network halted new block confirmation. Validators began to work with the team in Discord to identify the issue.
2:02pm UTC: DevOps began their investigation into the root cause of the issue.
2:18pm UTC: The validator community began collecting logs in GitHub to assist with the diagnosis.
2:45pm UTC: Initial cluster restart plans were assembled.
4:22pm UTC: The engineering team began testing the network restart instructions.
4:50pm UTC: Network restart instructions were circulated to the validator community; the cluster restart began.
5:56pm UTC: 58% of active stake is prepared to restart.
6:19pm UTC: 63% of active stake is prepared to restart.
6:41pm UTC: 67% of active stake is prepared to restart.
7:14pm UTC: 73% of active stake is prepared to restart.
7:22pm UTC: 77% of active stake is prepared to restart. 80% is needed for the network to resume.
7:32pm UTC: 80% reached. The network has restarted and blocks are once again being produced.
7:46pm UTC: Network activity on explorer.solana.com has resumed. solanabeach.io is expected to resume functioning shortly.
