Ethereum Prysm client experiences mainnet incident: resource exhaustion leads to large-scale missing blocks and witnesses.

According to Odaily Planet Daily, the Prysm team released a mainnet incident recap report stating that during the Ethereum mainnet Fusaka session on December 4th, almost all Prysm beacon nodes ran out of resources while processing specific attestations, resulting in their inability to respond to validator requests in a timely manner and causing a large number of missing blocks and witnesses.

The incident affected epochs 411439 to 411480, a total of 42 epochs. 248 blocks were missing out of 1344 slots, a missing rate of approximately 18.5%. Network participation dropped to 75% at one point, and validators lost approximately 382 ETH in witness rewards. The root cause was that Prysm received attestations from nodes that might have been out of sync with the mainnet. These attestations referenced the block root of the previous epoch. To verify their validity, Prysm repeatedly replayed the old epoch state and performed costly epoch transitions, causing nodes to exhaust resources under high concurrency. The related defect originated from Prysm PR 15965, which had been deployed to the testnet a month prior but did not trigger the same scenario.

The official temporary solution was to enable the `--disable-last-epoch-target` parameter in version 7.0.0. Subsequent versions 7.0.1 and 7.1.0 included a long-term fix that uses head state to verify attestations, avoiding repeated replays of historical states. Prysm stated that the issue gradually subsided after 4:45 UTC on December 4th, with network participation recovering to over 95% by epoch 411480.

The Prysm team pointed out that this incident highlights the importance of client diversity. If a single client accounts for more than one-third of the total, it may lead to a temporary inability to terminate; if it exceeds two-thirds, there is a risk of an invalid termination chain. They also reflected on the issues of unclear communication regarding feature switches and the failure of the test environment to simulate large-scale asynchronous nodes, and will improve their testing strategies and configuration management in the future.