An article to understand the whole story of the "crash" event of the Medalla test network
source|What’s New in Eth2
Author | Ben Edgington
source|Author | Ben Edgington
Translator's Note: Users running the Prysm client should upgrade to
This issue is an update outside of the wnie2 plan, and will review and analyze the episodes that occurred on the Eth2 Medalla test network over the weekend.Previous period。
We launched Medalla almost two weeks ago, August 4th, as a large, public, multi-client testnet running the Eth2 mainnet spec. For an introduction to the Medalla testnet, please refer to
Previous period
Therefore, this article will review the incident, including its aftermath and next steps.
What happened?
first level title
What happened?We discovered that every validator running the Prysm client in the network suddenly disappeared. Since Prysm is the most used client, the consequences can be imagined.The Prysmatic team opened up a
document reportRoughtime, where incident details and team responses are continuously updated. Here are some highlights with my notes.
The cause of the event is a problem with clock sync. The Prysm client is configured using Cloudflare's
to count the time. The reason for this (in my opinion) is not very clear, but it is clear that Roughtime shifted the time four hours into the future, and it lasted for more than an hour. Prysm client validators suddenly found their time was four hours faster and continued to generate blocks and proofs for a blockchain that didn't exist yet.
On its own, not enough to be catastrophic. Even with many blocks lost and facing a large amount of proofs from the future, the remaining clients will still be able to build on the original chain. Gradually, as the Prysm nodes' clocks adjusted back, they started coming back into the network, and validator participation started picking up. The network appears to be returning to normal.
But a few hours later, the situation took a sharp turn for the worse.
Four hours after the initial time, two more things happened. First, proofs generated by all Prysm clients in the future start to have validity. Second, Prysm nodes rejoining the network started disappearing again because the slash protection mechanism was triggered to prevent them from generating any contradicting proofs.
Those two things happened at the same time, throwing the network into turmoil. The remaining clients were still struggling to process the information they received, and the beacon chain became a branching jungle. (Raul from the Prysmatic team told me that a bug in Prysm's first fix made things worse)
For a period of time, the information in the network is still under control. But over the next 24 hours or so, the memory and CPU required to navigate the increasingly complex and chaotic forks became overwhelming. I saw a Lighthouse client using 30GB of memory (about 100 times what it normally would), and for the Teku client, even with a 12GB Java memory heap and maxing out the processor, I was having trouble. Eth2StatsNote that this all happens on weekends. Thanks to all the customer teams who are fighting on the front line. In order for the nodes to cope with the chaotic network, they need to constantly optimize memory and efficiency.
So far, the network is gradually recovering. User reports have been mixed, but the newer versions of Prysm and Lighthouse just happened to be able to find the correct chain header and continue building the beacon chain.
Some nodes currently showing the chain head or nearby Lighthouse, Prysm and Teku nodes. We will continue to optimize Teku to reduce the resources it needs to sync.
No consensus failure occurred
One thing to be clear is that there is no consensus failure between clients, that is to say, when the network is restored, all clients can reach a consensus on the state of the chain head, which means that the beacon chain will not fundamentally fail, nor will it fail. Any hard forks are required.
Lessons experience
We will spend more time to fully reflect and summarize this episode, the following are some of my personal opinions.The Importance of Time Synchronization。
High dependence on third-party time services is a fatal point for the network. As it happens, Alex Vlasov of the ConsenSys TX/RX research team has previously written a detailed article explaining time synchronization and its importance in the Ethereum 2.0 network. His work is progressing rapidly, perhaps this is also an opportunity for everyone to pay attention to this aspect. here is his
Related articles and ethresear.ch posts
The significance of client diversity
Ideally, we would have four or more independent clients, with each client node accounting for no more than 30% of the network. That way, even if one client had a problem, it wouldn't have enough impact to get our attention.
Even if we can't achieve this ideal, reducing the extreme usage by individual clients will make the network more robust. Assuming only 50% of the validators go offline this time instead of 80%, the network will recover more easily. This is because when there is a problem with the client, it will affect the network's block generation, proof packaging, broadcast efficiency, point-to-point communication, and synchronization, and these factors will also have a knock-on effect on the remaining validators.
Effectiveness of backup plansSome stakers are able to switch signing keys to other clients' hot backup nodes. This certainly makes for a great security network, although one needs to be careful to avoid slashing: new validators may not know anything about the voting history of existing validators, and thus may vote contrary.In the future, once we've done
new API
Stakeholders' Sense of Responsibility
Haste makes waste
Participation in Eth2 is not a "one and done" thing at the moment. Stakers need to maintain a certain amount of attention, wandering between forums, providing feedback to developers and being able to update the client in a short time. I'm very supportive of people running their own personal validators, but only if they are aware of their responsibilities.
Why do things always go wrong on Friday evenings?incident reportEven at this time, the response from the Prysmatic team has been amazing. For more details, please refer to the team's
incident report
. What I say below is not meant to cast a negative light on the Prysmatic team, who have done a really good job, but to provide experience for the Teku team when faced with a similar situation.
When so many users are losing assets (even if it's just a test coin), and the network is under high pressure, it's natural to want to react quickly, but haste can sometimes be counterproductive.
Two things were avoidable in this incident. First, there was a bug in the initial fix release Alpha.21 that required users to rollback after 17 hours.
According to Raul of the Prysmatic team, this flaw was responsible for the ensuing network chaos. Second, the team inadvertently deleted its database of slash-proof records of 1024 validators while handling the situation, resulting in the majority of validators being slashed.
Similar situations can happen with any client. So even under high pressure, all of us, whether developers or users, must deal with it calmly and not blindly pursue speed. So when we tried to restore the network, we followed a slow and careful approach.
In the end, this episode was actually necessary. What's the point of a testnet if nothing is tested? It is obviously unrealistic to run smoothly all the time.
The Block in the article
Quoting my statement:
In the email, PegaSys engineer Ben Edgington wrote that Medalla "is the first testnet with the scale and configuration of the mainnet."to be honest,。
What's next?
What are you looking forward to?
What's next?
Website: ethereum.cn
For more Ethereum information, please pay attention to:
Public account: ETH Chinese website
Website: ethereum.cn
Disclaimer: ECN's translation work aims to deliver high-quality information and learning resources for the Chinese Ethereum community. The copyright of the article belongs to the original author. Reprints must indicate the source of the original text and ethereum.cn. For long-term reprints, please contactethereumcn@gmail.comAuthorize.
