Ethereum 2.0 Mainnet Accident Review

Source | Prysmatic Labs

secondary title

https://beaconcha.in/epoch/32302

Since epoch 32302, the beacon chain has lost a large number of block proposals. Since Prysm is the most used Eth2 client, the problem is most likely on Prysm. After some time, we reproduced the error locally. This is actually a known issue with eth1 data voting and validator deposits. Although this issue has been reported to us before, we have been unable to reproduce this bug and consider it an isolated incident. And this problem has never been widely propagated in any testnet or mainnet. This is the first time this issue has resulted in a block proposal failure.

During these 18 epochs, almost all Prysm beacon nodes were unable to produce new blocks. Epoch 32320 began to function normally again, at a time when the incident was widely believed to be over. About 24 hours later, however, the accident happened again with similar effects.

The official post-mortem report on this accident has been released, visit the link: https://docs.google.com/document/d/1nJr6_bd-UnLBxvhT8lcRYdAZr69QdVQ3zJNUr3LgW-0/edit

Influence

Some preliminary data suggest that each affected validator lost an average of 122,950 gwei ($0.3 at the time of writing) in the first incident. Within 24 hours of that incident, a second identical incident occurred, costing each affected validator approximately $0.22.

Some key facts:

➤ No validators are slashed

➤ Has no effect on beacon chain finalization

➤ The participation rate is still very high (84.8% at the lowest point) (Editor's note, this data is consistent with the latest issue written by Ben EdgingtonEth2 Progress UpdateThere are discrepancies. )

➤ Most validators lose 2 to 3 proofs regardless of client type

➤ This time does not appear to be a malicious or deliberate attack

secondary title

Will this accident weaken everyone's confidence in Eth2?

Won't. The incident did not cause a consensus failure, and the scale of the incident was very small compared to the size of the Eth2 mainnet (in the first incident, each affected validator lost an average of ~$0.3). Since genesis, Eth2 has been very strong with very high validator participation and finalization every epoch. From our perspective, the ability of the network to return to a perfectly functioning state after the fault is resolved has in turn strengthened the community's confidence in Ethereum's resilience.

Our response to and resolution of this incident was completely different from our previous efforts to deal with failures in the Eth2 testnet. Immediately after the incident, our team ruled out misinformation; quantified the impact; and laid out clear steps for validators while waiting for a resolution. Furthermore, after we have fully determined the solution, we will ask everyone to upgrade the client version. It is worth noting that since the Prysm client is the software with the largest proportion of users in the Ethereum 2.0 network, any bugs that appear may cause more serious problems.

secondary title

The Eth2 and Eth1 chains are loosely connected, and Eth2 only needs Eth1 for validator deposit verification. That is, the Eth2 PoS chain can continue to operate even if validators vote on garbage data. The only thing that will matter is that new validator deposits cannot be added until the PoS chain votes on the correct Eth1 data again. This "voting" is done during a "voting period", which is currently set to 64 epochs (roughly 6.8 hours) on mainnet.

The method of voting is a simple "supermajority" principle, which is explained in the Eth2 validator specification. Unfortunately, Prysm loses some validation when enforcing this principle (voting by supermajority). In this incident, due to a bug in Prysm, a block proposer created a completely invalid Eth1 deposit tree root, and other Prysm nodes first discovered the block proposal. They then voted validly on this, because the Prysm client follows a simple "supermajority vote" without explicit validation.

Then, all Prysm nodes "snowball" votes on invalid information, resulting in block proposers being unable to include blocks with deposits into the chain. This is because these deposits are not verified against the node's Eth1 deposit tree root, so block proposals will fail. After the voting period, the issue is automatically resolved, but if the bug is not fixed, the issue will recur.

secondary title

Note, technical details below! You can skip to the next section to read about the solution and the lessons learned from the incident.

Epoch 32302 began to have problems with lost block proposals.

Nishant notified the team and called an all-hands meeting. We then reproduced the incident via a local mainnet beacon node and started an investigation.

text

The initial suspicion was about how Prysm handles eth1 data votes in the validator proposal code path. In particular, we try to rule out some problems:

1. Is there any problem with packing deposits into blocks?

2. Is the deposit log information obtained mixed with eth1 information or not sure?

3. Is there a problem with our deposit Merkle tree?

For the next 16 hours or so, we spent a lot of time working together to diagnose the underlying problem. We combed through the lines of code, tried to reproduce the failure process through unit tests, and tried various methods. Even though we already had a potential solution, we were nervous about releasing a fix due to a lack of confidence.

We've learned some lessons from previous experience dealing with bugs in the Eth2 testnet that it's not enough to have confidence in the root cause. In high-stakes situations, we need to be 100% confident in the solution before releasing it to users. 28 hours after the incident, we sat down and asked ourselves, "What else do we not know? What questions can we ask that will bring us closer to the root cause of the failure?" Then we learned the following: point:

1. Our sparse merkle tree implementation is not seriously buggy as it uses mainnet and Prater testnet deposits, matching Lighthouse and Protolambda's Eth2 zrnt implementations.

2. The code path we use to retrieve Eth1 data from Eth1 nodes is not buggy nor returns incorrect data.

What we don't know are:

1. How Invalid Deposit Tree Roots Are Generated

2. Why this problem is reproducible in some nodes, but not in other nodes

3. Why did the Prysm node make an "off-by-one" error when determining the deposit amount in the block

To answer these questions, we looked at the code path that initializes our deposit tree. It turns out that a caching layer was added early on to avoid having to download all validator deposit records every time stakers boot up their nodes. Additionally, we've added a new feature - the ability to start Prysm from an embedded genesis state inside the client. A wrong default in our deposit tree caused information to be corrupted when populating the cache:

root of the problem

It turns out that the function len(items) will always return 1 if our deposit tree is empty. This means that we set the value of lastReceivedMerkleIndex to 0 when in fact we should have set it to -1. The above code will cause some Prysm nodes in the code path to skip embedding deposit 0 into the tree. The rest of our codebase all pointed to this weird part of our deposit tree implementation as the problem, not this code path.

secondary title

➤ Prysm persists eth1 data on disk to prevent users from having to make requests to the validator deposit contract log every time the process is restarted.

➤ If a node reboots and persists eth1 data on disk, we initialize our deposit cache from this data, but due to the way our sparse merkle tree (SMT) helper The code path for initializing this cache is different from the data above, we will skip embedding the 0th deposit into the deposit tree, resulting in an invalid deposit tree root. This code path only affected nodes that did not already have a database since genesis, and was later fixed.

➤ In the official specification, Prysm nodes follow the principle of "absolute majority" to implement an eth1 data voting algorithm, but Prysm does not fully implement some valid conditions of the algorithm. Prysm nodes vote with a supermajority eth1 data vote referencing an existing block root, which may cause a Prysm node to vote for a deposit tree hash generated by a problematic deposit tree because these deposits are not verified.

➤ Since the majority of the nodes in the network are Prysm nodes, the snowball effect of voting for faulty deposit stubs with a supermajority rule developed into a serious problem because Prysm nodes were unable to generate blocks on the mainnet for a period of time afterwards.

solution

At 13:00 Beijing time on Sunday, April 25th, after many hours of uncertainty, we released a fix for this issue. We are absolutely sure of this solution and are very confident that this issue will not recur in Eth2 after node upgrades.

We learned an important lesson about the value of good communication when we had the Eth2 Medalla testnet mishap. Every public comment and precise expression of language can have a serious impact on the outcome of events. In the case of the testnet, we thought an immediate solution was to tell everyone through public channels to "restart your nodes". This hasty decision caused most of the nodes on the network to go offline, and then scramble to find good peers among a bunch of bad peers in order to synchronize with the blockchain. Additionally, we quickly released a software upgrade hotpatch that we were not 100% confident would solve the problem. This introduces more confusion into the system and creates doubts among node operators about the solution.

In contrast, throughout the entire process of the mainnet new accident, we have always paid attention to careful and precise communication. Also, we don't release hotfixes until we are 100% confident about the source and solution of the problem.

➤ Patience and calm can help solve problems

Our team has spent the past few years building Eth2 and has learned how to stay calm in the face of adversity. We believe it is important to remain calm, communicate status reports frequently, and make sure the team feels supported and positive feedback is important during the problem-solving process. We were able to take the time to gather as much evidence as possible, and work meticulously with our users, and we will successfully resolve this issue. More importantly, we took the time to quantify event impact at the outset to reduce staker and lack of information anxiety. This lesson is very important for working in high stress and sleep deprived situations. Slow down, fix it with appropriate means, and avoid making the problem worse at all costs.

➤ Eth2 test network is not equal to the main network

For the Prysm client, we have extensively tested and listened to Prysm pre-production release candidates on the public Eth2 testnet. Both the Prater and Pyrmont testnets are great tools for users to test their setup before joining the Eth2 mainnet. However, these testnets assume that the proportion of the four production-grade Eth2 clients is close to the average score, that is, no client has a clear majority share among the validators. Unfortunately, this probably doesn't take into account the vulnerabilities that only arise when a certain client is used by most people. In the future, Prysmatic Labs will test in an internal testnet in an environment closer to the mainnet environment, or an environment with 50% of the Prysm network nodes.

secondary title

https://launchpad.ethereum.org

People choose to run Prysm because from the beginning our team has focused on making their Ethereum staking experience easier. I've spoken to our users many times, and many people choose one client not because of micro-optimizations or relatively small yield differences compared to other clients, but because we make their experience simpler - good documentation information, has been providing important assistance to all community members. For novices, Eth2 is scary, and staking is full of uncertainties and risks. Our team's mission is to let users know we're there for them and have our support no matter how small their problem is. In particular, we've been keeping an eye out for the average staker who might not be familiar with the command line, or the UNIX operating system.

In the future, you can expect the following from our team:

➤ Improve the accuracy of implementing specification conditions, ensuring that preset and valid conditions are fully reviewed and challenged before any code is written

➤ We will go the extra mile to improve this experience, making Prysm many times better than it is today, making it easier for stakers using our client to participate in the network, including improvements to the web interface.

➤ Prysm will double down on R&D,<>The pre-merge of eth2 provides key features and improvements.

➤ We believe healthy competition creates a strong incentive to drive ETH's Proof-of-Stake to be more involved and therefore more secure as all client teams continually improve their software

➤ Our team is committed to solving the problems that stakers may encounter with the highest professional standards. We believe we are ready to handle any issues that come our way and assure our community that we will make the staker experience our highest priority.

➤ Finally, we believe that there are many important features that can make Prysm a more attractive software for participating in Eth2, and we will continue to iterate towards this goal

secondary title

A common theme we've been hearing since the genesis of Eth2 is client diversity. Eth2 is a distributed system with people all over the world participating as validators. Different people use different software to participate in the blockchain consensus. If there is a serious problem with a certain software, if the clients running the network are distributed in a balanced manner, the impact will be smaller.

Leonardo Bautista-Gomez released a data analysis as early as January, and the results showed that Prysm nodes accounted for 65% of the network. This incident also shows that Prysm validators account for the majority today.

https://github.com/leobago/BSC-ETH2/tree/master/armiarma

We recommend that you take an objective look at each client: its software, its community, and its resilience, and decide which software and the team behind it is the best fit for your needs. If an Eth2 client is missing something that is important to you, or your reason for not choosing their client, we strongly encourage you to file a feature request. Prysmatic Labs continues to focus on helping you participate in the Ethereum network and pushing the boundaries of blockchain software.

refer to

• Incident communication https://www.reddit.com/r/ethstaker/comments/mxpz57/regarding_the_recent_beacon_chain_incident/

• Postmortem report https://docs.google.com/document/d/1nJr6_bd-UnLBxvhT8lcRYdAZr69QdVQ3zJNUr3LgW-0/edit?usp=sharing

• Medalla testnet incident https://medium.com/prysmatic-labs/eth2-medalla-testnet-incident-f7fbc3cc934a

https://medium.com/prysmatic-labs/tagged/blockchain)

Original link: https://medium.com/prysmatic-labs/eth2-mainnet-incident-retrospective-f0338814340c