Ethereum Merger: What Are the Risks of Running Multiple Clients?

Original translation: Block unicorn

Original translation: Block unicorn

Special thanks to Vitalik Buterin (V God), Hsiao-Wei Wang, Caspar Schwarz-Schilling for their feedback and review.

first level title

Why do we need multiple clients?

There is debate as to why a single-client architecture is preferable. And developing multiple clients creates considerable overhead, which is why we don't see any other blockchain network seriously pursuing a multi-client option.

So why is Ethereum aiming to be multi-client? Clients are very complex pieces of code that are likely to contain bugs. The worst of these are so-called "consensus bugs," errors in the logic of state transitions at the core of the blockchain. An oft-cited example is the so-called "infinite money supply" exploit, in which a vulnerable client accepts transactions that print an arbitrary amount of ether. If someone found such a bug, but wasn't stopped before they reached the exit (i.e. exploiting the funds through a mixer or exchange), it would cause the value of Ether to collapse dramatically.

If everyone is running the same client, human intervention is required to stop, as the chain, all smart contracts, and exchanges will continue to function as usual. Even a few minutes is enough time to execute a successful attack and disperse funds sufficiently that it would be impossible to just rollback the attacker's transaction. Depending on the amount of ETH printed, the community may coordinate a rollback of the chain to before the exploit (after the bug was identified and fixed).

Now, let's see what happens when we have multiple clients. There are two possible scenarios:

1.Vulnerable clients account for less than 50% of staking, the client will generate a block with a transaction that exploits the bug, printing ETH, let's call this chainA。

However, most stakers running a glitch-free client will ignore this block because it is invalid (for them, printing ETH operations is invalid). they will build

However, most stakers running a glitch-free client will ignore this block because it is invalid (for them, printing ETH operations is invalid). They will build an alternate chain B that does not contain invalid blocks.

Since the correct clients are in the majority, the B-chain will accumulate more proofs. Thus, even problematic clients will vote for chain B; the result is that chain B will accumulate 100% of the votes and chain A will die. The chain will continue as if the error never happened.

2.The majority of staked shares are using buggy clients, in this case the chainAA majority of votes will be accumulated. But due toBWith less than 50% of all proofs, offending clients will never see slave chainsAswitch to chainBreason. Therefore, we will see chain splits.

Case 1 is the most ideal case. Because this will most likely result in an orphaned block and most users won't even notice. Developers can debug the client, fix bugs, and you're good to go. In contrast, case 2 is clearly less ideal. But still better than just one client - most will detect chain splits pretty quickly (you can automate this by running multiple clients), exchanges will suspend deposits quickly, and Defi users can wait for the split to resolve Proceed with caution. Basically, this still gives us a blinking red warning light against the worst outcomes compared to a single-client architecture.

Case 2 would be worse if the faulty client was run by more than 2/3 of the stake. In this case, it will finalize the invalid chain. More on this later.

first level title

Incentivizing Client Diversity: Anti-Relationship Penalties

If the stake is spread across multiple clients, the best case scenario is that each client owns less than a third of the total stake, which is obviously beneficial to the network. This will make it resilient to errors in any single client. But why would stakers care? If the network doesn't have any incentives, they are less likely to incur the cost of switching to a minority client.

Unfortunately, we cannot make the rewards directly depend on the client that the validator is running on. There is no objective way to measure this that cannot be tricked.

However, you cannot hide when your client has errors. This is where anticorrelation penalties come in: the idea is that if your validators do something bad, the penalty will be higher if more validators are wrong at about the same time. In other words, you will be penalized for related failures.

In ether, you can currently get hacked for two behaviors:

1. Sign on two blocks of the same height.

2. Create a pair of prunable proofs (surround voting or double voting).

When you get slashed, you usually don't lose all your money. At the time of writing (Altair Fork), the default penalty is actually pretty small: you'll only lose 0.5ETH, or 1.5% of your staked Ether (which will eventually increase to 1ETH, or 3%).

However, there is a catch: there is an additional penalty that depends on all other slashes occurring within 4096 epochs (18 days) before and after your validator was slashed. The amount you are further penalized is proportional to the total amount slashed during this period.

first level title

Another anti-correlation penalty: quadratic inactivity leak

Another way for a validator to fail is offline, again, this has a penalty, but the mechanics are very different. We don't call this slashing, and it's usually small: under normal operation, validators who are offline are penalized the same as they would be if they were fully validating. As of this writing, that's an annual growth rate of 4.8%. If your validators are offline for hours or days, for example due to a temporary internet outage, then it probably isn't worth breaking a cold sweat.
Things get very different when more than 1/3 of validators are offline. The beacon chain then fails to finalize, threatening a fundamental property of consensus protocols, liveness.
To rejuvenate in situations like this, so-called "secondary inactivity leaks" come into play. If validators continue to go offline while the chain is incomplete, the total penalty increases quadratically over time. Initially it is very low; after about 4.5 days, offline validators will lose 1% of their stake. However, it increased to 5% after ~10 days and 20% after ~21 days (this is the value of Altair, which will double in the future).

The mechanism is designed so that in the event of a catastrophic event causing a large number of validator operations to fail, the chain will eventually be able to complete again. As offline validators lose more and more of their stake, their share of the total stake will get smaller and smaller, and when their stake drops below 1/3, the remaining online validators gain the required 2/3 majority, thus enabling them to finalize the chain.
first level title

How bad is running a majority client?

To understand what the dangers are, let's look at the three failure types:

1. Mass slashing event: Most client validators signed slashable proofs due to a bug.

2. Large number of offline events: due to a bug, all majority client validators went offline.

3. Invalid block event: Due to a mistake, most client validators proved that there was an invalid block.

first level title

Scenario 1: Dual Signature

This is probably the scenario most validator operators fear the most: a bug that causes a validator client to sign a deletable proof. An example is two provers voting for the same target epoch, but with different payloads. Because this is a client bug, the concern is not just one staker, but all stakers running this particular client. Once these ambiguities are discovered, it will be a bloodbath: all involved stakers will lose 100% of their staked funds. This is because we are considering multiple clients: if the client in question stakes only 10% of their stake, then "only" 20% of their stake will be slashed (in Altair; 30%, and set the final penalty parameter).

In this case, the damage is obviously extreme, but I also consider it highly unlikely. The conditions for deletable proofs are simple, which is why validator clients (VCs) were built to enforce them. The validator client is a small, well-vetted piece of software, and a breach of this size is unlikely.

first level title

Scenario 2: Large scale offline activities

For this scenario, we assume that most clients have a bug that, when triggered, causes the client to crash. The block in question has already been integrated into the chain, and whenever a client encounters that block, it is taken offline, preventing further participation in the negotiation. Most clients are now offline, so inactivity leaks start to appear.

Client developers will be scrambling to put everything back together. Realistically, within a few hours, at most a few days, they'll release a bug fix that removes the crash.

In the meantime, orderers can also choose to simply switch to another client. As soon as this is done enough to get more than 2/3 of the validators online, the quadratic inactivity leak stops. It's not impossible for this to happen until the buggy client is fixed.

first level title

Scenario 3: Invalid Data Block

For this scenario, we consider the situation where most clients have a bug that produces an invalid block and also accept it as valid -- that is, when other validators using the same client see the invalid block , they will consider it valid, thus proving it.
Let's call the chain containing the invalid block chain A, once the invalid block is produced, two things will happen:

1. All working clients will ignore invalid blocks and instead build on the latest valid heads that generate a separate chain B. All working clients will vote and build on chain B.

2. The faulty client thinks that both chains A and B are valid, so it will vote for whichever of the two chains it currently thinks is the heaviest.

We need to distinguish three cases:

1. Vulnerable clients held less than half of the total stake. In this case, all correct clients vote and build on the B chain, eventually making it the heaviest chain. At this point, even faulty clients will switch to chain B. Except for one or a few orphaned blocks, nothing bad will happen. This is the comforting situation, and why having only a majority of users is great.

2. Vulnerable clients held more than half but less than two-thirds of the stake.In this example, we will see two chains being built - A by the client with the bug and B by all other clients. Neither chain has a two-thirds majority, so they cannot be finalized. When this happens, developers will scramble to understand why there are two chains. When they discover that there are invalid blocks in chain A, they can proceed to fix the buggy clients. Once fixed, it will recognize chain A as invalid. So it will start building on the B chain, which will allow it to be finalized. This is very disruptive to the user. While hopefully the confusion over which chain is valid will be short-lived, less than an hour, the chain may not be finalized for hours or even a day. But for producers, even those running the client in question, the penalty remains relatively light. If they did not participate in chain B while building an invalid chain A, they would be penalized with an "inactivity leak". However, since this could be less than a day, we're talking about a penalty of less than 1% of stake.

3. The client in question holds more than two-thirds of the stake.In this case, the buggy client would not just build chain A — it would actually have enough stake to "finalize" it. Note that it will be the only client that will consider chain A complete. One of the conditions for finalization is that the chain is valid, and for all other correctly operating clients, chain A will be invalid. However, due to the way the Casper FFG protocol works, when validators finalize chain A, they can never participate in another chain that conflicts with A without being hacked, unless that chain is finalized (for any sense of detail For those interested, see Appendix 2). So once chain A is finalized, validators running buggy clients are in a dire situation: they have committed to chain A, but chain A is invalid. They can't contribute to B because it hasn't been finalized yet. Even a bug fix to their validator software won't help them - they've already sent offending ballots. What will happen now is very painful: the B chain that has not yet been finalized will enter a secondary inactivity leak. Over the course of a few weeks, the offending validators will leak their stake until enough stake is lost for B to finalize again. Let’s say they start out with 70% staked – then they will lose 79% staked, since that’s the amount they would need to lose to represent less than a third of the total staked. At this point, chain B will be finalized again and all stakers can switch to it. The chain will be healthy again, but the outage will last for weeks, with millions of ETH destroyed in the process.

Risk Analysis

Risk Analysis

So how do we assess these scenarios? A typical risk analysis strategy is to assess the likelihood of an event occurring (1-extremely unlikely, 5-very likely) and the impact (1-very low, 5-catastrophic). The most important risks to focus on are those that score high on both metrics, represented by the product of impact and likelihood.

With this in mind, Scenario 3 is by far the highest priority. When a client is in a two-thirds supermajority, the impact is quite catastrophic, which is also a relatively likely scenario. To emphasize how easy such a vulnerability can be, such a bug occurred recently on the Kiln Testnet (see Kiln Testnet Blocking Proposal Failed). In this case, Prysm did detect that the brick was defective after it was presented and did not prove it. If Prysm thinks that blocking is valid, and this happens on Mainnet, then we are in the catastrophic situation described in Case 3 of Scenario 3 - since Prysm currently has a 2/3 majority on Mainnet. So if you are currently running Prysm, there is a very real risk that you could lose all your funds and you should consider switching clients.

Scenario 1, which is probably the most feared, received a relatively low rating. The reason for this is that I think the chances of this happening are fairly low as I think the Validator client software is well implemented on all clients and it is unlikely that it will generate skewable proofs or blocks.

If I'm currently running multiple clients and I'm worried about switching, what options do I have?

Replacing clients can be a major undertaking, which comes with some risks. What should I do if the miter database is not properly migrated to the new setup? There might be a risk of getting hacked, which completely defeats the purpose.

I would suggest another option to anyone worried about this. It is also possible to leave your validator setup as is (no need to take out keys etc.) and just switch beacon nodes. This is very low risk because as long as the validator client is working as expected it will never re-sign and thus cannot be hacked. Especially if you have large operations where changing the validator client (or remote signer) infrastructure would be expensive and possibly subject to auditing, this might be a good option. It's also easy to switch back to the original client if your setup doesn't perform as expected, or try a handful of other clients.

first level title

What about the punished client?

What I wrote above applies to the Consensus clients - Prysm, LightTower, Nimbus, Loestar, and Teku, and at the time of writing, Prysm probably has a two-thirds majority on the network.
All of these apply in the same way to executing clients. Go-Etherum is likely to be the majority executing client after the merge, and if it produces invalid blocks, it may finalize, leading to the catastrophic failure described in Scenario 3.
secondary title

appendix

Addendum 1: Why aren't invalid blocks slashed?

In Scenario 3, we must rely on quadratic inactivity leaks to penalize validators who propose and vote for invalid blocks. That's weird, why don't we just punish them? It will seem faster and less painful this way.

In fact, there are two reasons why we don't - one is that we currently can't, but even if we could, we probably wouldn't:

1. Currently, it is almost impossible to introduce penalties ("slashes") for invalid data blocks. This is because neither the beacon chain nor the execution chain are currently "stateless" - i.e. in order to check if a block is valid you need a context ("state" ). This means that there is no "succinct proof" that a block is invalid. We need such proof to slash a validator: blocks that "slash" a validator need to include evidence that the validator has broken the law. In the absence of stateless consensus, there are ways around this problem, but it will involve more complex structures such as multi-round fraud proofs such as the one Arbitrum currently uses for aggregation.

2. The second reason we might be less eager to introduce this type of slashing, even if we could, is because generating invalid blocks is harder to prevent than the current slashing conditions. The current condition is so simple that validator clients can easily verify it with just a few lines of code. That's why I think Scenario 1 above is unlikely - so far, expungable messages have only been generated by operator error, and I think that's likely to continue. Adding the slash for producing invalid blocks (or proving them) increases the risk for coin holders. Now, even those who run minority clients can face severe penalties.

secondary title

Addendum 2: Why can't flawed clients switch to chain B after finalizing chain A?

This section is for those who want to understand in more detail why buggy clients cannot simply switch back and have to suffer the dreaded inactivity leak. For that, we have to look at how Casper FFG finalization works.

Each proof consists of a source checkpoint and a destination checkpoint. A checkpoint is the first block of an epoch. If there is a link from one epoch to another with a total of more than 2/3 of all votes at stake (i.e., so many proofs where the first checkpoints "source" and the second checks points are "targets"), we call them "supermajority links".

An epoch can be "reasonable" or "definite", which are defined as follows:

1. Epoch 0 is aligned.

2. An epoch is legitimate if it has a supermajority connection to a reasonable epoch.

3. If (1) epoch X is aligned, and (2) the next epoch is also aligned, and the source of the vast majority of links is epoch X, then epoch X is finalized.

Rule 3 is slightly simplified (there are more conditions to finalize an epoch, but they are not important for this discussion). Now, let's look at the conditions for a substantial cut. There are two rules for slashing proofs. Both compare a pair of proofs V and W:

1. V and W are hackable if they target the same epoch (i.e. same height), but they do not vote for the same checkpoint (double voting).

2. This means (1) V's source is earlier than W's source and (2) V's target is later than W's target (surround voting).

The first condition is obvious: it prevents simply voting for two different chains of the same height. But what does the second condition do?

Its function is to slash all validators involved in finalizing two conflicting chains (which should never happen). To see why, let's look again at our Scenario 3, where in the worst case there is an absolute majority of buggy clients (>2/3 of the stake). As it continues to vote for the faulty chain, it will finalize the epoch with invalid blocks, as follows:

The rounded boxes in this image represent epochs, not blocks. The green arrow is the last supermajority link created by all validators. Red arrows are supermajority links supported only by buggy clients. A working client ignores epochs with invalid blocks (red). The first red arrow will justify the invalid epoch and the second arrow will identify the invalid epoch.
Now let's assume that the bug is fixed, and the validators who finally determined the invalid epoch want to rejoin the correct chain B. To be able to terminate the chain, the first step is to adjust the epoch X:

However, in order to participate in the epoch X adjustment (which requires a supermajority link indicated by the dotted green arrow), they will have to "jump over" the second red arrow - the one that ultimately determines the invalid epoch. Upvoting those two links is an offense to chop.

This will continue to hold true for any subsequent era. The only way to fix it is through quadratic inactivity leaks: as chain B grows, locked validators will leak their funds until chain B can be justified and finalized by working clients.