Read the shuffling algorithm of Ethereum 2.0 in one article

Author |Ben Edgington

Introduction

Author |Introductionif you want to learn

ghost step dance

Shuffle list is a basic operation in Ethereum 2.0. It is mainly used to pseudo-randomly select verifiers to form committees in every 12-second slot, and to select proposers for beacon chain blocks in each slot.Hidden dangerShuffle seems fairly straightforward. although it has someFisher-Yeats shuffleHidden danger

up. So why don't we use it in Eth2? I will explain in detail at the end of the article, but in short - light client.paperpaperTekumiddle. I'm recently on the Eth2 clientmiddle

rewrite

Our implementation, so I wanted to write it out while it was still hot.

Swap-or-Not shuffling algorithm

round of operation

Shuffles are performed in rounds. The process is the same for each round, so I will only demonstrate the process for one round below, which is much simpler than it looks.

First, we choose a pivot index p, which is selected pseudo-randomly based on rounds and some other seed data. After the axis is selected, it is fixed in the round.

Based on this pivot point, we select a mirror index m1 at the midpoint between p and 0, that is, m1=p/2. (For the sake of explanation, we will ignore the troublesome one-by-one error rounding problem)

image description

Pivot point and first mirror index

2. From first mirror index to pivot point, replace or not

For each index between the mirror index m1 and the pivot index p, we randomly decide whether to replace those elements.

If we decide to replace, then we replace the list element on i1 with the one on i1', its image on the mirror index. That is, i1 is replaced with i1'=m1-(i1-m1), so that the distances from i1 and i1' to m1 are equal.

We make the same swap-or-not decision for each index between m1 and p.

image description

3. Calculate the second mirror index

After making all index decisions from m1 to p, we now find the second mirror index with m2 as the midpoint, i.e. the point that is equidistant from p and the end of the list. That is m2=m1+n/2.

image description

4. From the pivot point to the second mirror, replace or not

Finally, we repeat the swap-or-not process, considering all point-to-pivot p replacement decisions, that is, decisions from p to the second mirror m2. If we choose not to replace, we move on to the next one. If we choose to replace, then we replace the element on j1 ​​with its mirror image on j1' at mirror index m2.

image description

swap-or-not decision from pivot to second mirror index

At the end of a round, we have considered all indices between m1 and m2, i.e. half of all indices, and each index has a specific index in the other half, whether replaced or not. So, with regard to replacement or not, all indexes have been considered once.

The next round starts with incrementing (or decrementing) rounds, so that we have a new pivot index, and then begins to cycle through the process described above.

image description

The process of moving from one mirror to another in the same round

ingenious placeaccomplishWhen deciding whether to replace or not, the algorithm smartly chooses the higher of the candidate index or its mirror image. It means that when it is below the axis, i_1 is selected instead of i_1'; when it is above the axis, i_k' is selected instead of i_k. This means, that we can iterate over the indices in the list flexibly: we can separate 0 to m1 and p to m2 into two separate loops, or combine both in the same loop from m1 to m2, as I did above described in the text (and

accomplish

)of. The result is the same both ways: it doesn't matter whether I consider i_1 or mirror i_1'; the substitution or not yields the same result.RoundsIn Eth2, the above process will be performed 90 times.original paperIt is mentioned that it takes 6lgN rounds to "start to show a good security margin on selective password attack (CCA)", where N is the length of the list. at Vitalik

annotation specification

In , he said that "experts in cryptography suggest that we have 4log2N rounds to provide sufficient security". The absolute maximum number of validators in Eth2, which is the maximum number of times we need to shuffle the list, is about 222 (4.2 million). The estimated value given by Vitalik is 88 rounds, and the estimated value in the paper is 92 rounds (assuming lg is the natural logarithm). So we're now in a roughly correct range, especially since we very likely won't end up with this many active validators.Issue FAdjusting rounds based on list length could have interesting results, but we wouldn't do that, it might be an unnecessary optimization.

Interestingly, when the Least Authority audited the Beacon Chain specification, they initially found that there was a bias in the shuffling that selects block proposers (see

). But it turned out that they mistakenly used a shuffle configuration with only 10 rounds. When they increased the shuffle configuration to 90 rounds (the number we use on mainnet), the bias disappeared.

(pseudo)random

The shuffling algorithm requires us to randomly choose a pivot point in each round, and randomly choose whether to replace each element in each round.

In Eth2, we definitely generate randomness from a seed value, whereby the same seed will always produce the same shuffling result.

Pivot indicators are generated by 8-byte SHA2 hashing of the seed concatenated with the round,

The pivot index is generated from the eight bytes of the SHA2 hash of the seed value that is concatenated with rounds, so it usually changes every round.

The decisive bits used to decide whether to replace an element are extracted from the following elements: the SHA256 hash of the seed, the round, the index of the element on the list.

efficiency

This shuffling algorithm is much slower than the Fisher-Yates algorithm. If the Fisher-Yates algorithm requires N shuffles, our algorithm requires an average of 90N/4 shuffles. We also have to consider the generation of pseudo-randomness, which is the most expensive part of the algorithm. Fisher-Yates needs randomness close to Nlog2N digits, and we need 90(log2N+N/2) digits. According to the range of N values ​​we need in Eth2, the excess digits are quite a lot (when N is one million , Eth2 needs about twice N).

Why choose the swap-or-not algorithm?

If it's not efficient, why would you choose this implementation?

shuffling a single element

So, if we want to know where the element at index 217 has been shuffled, we can run the algorithm for that index only, without shuffling the entire list. Also, conversely, if we wanted to know what element was shuffled to index 217, we could run the algorithm in reverse to find element 217 (reverse means running the rounds from high to low, not low to high ).

existIn summary, we can calculate where element i is shuffled to, and where element i originated from (using the reverse operation) in constant time, and the calculation time does not depend on the length of the list. Fisher-Yates shuffle does not have this property, and cannot shuffle a single index, they often need to repeatedly shuffle the entire list.exist

Eth2 specification

Be a real "light" client

history

historyhereherehereCheck out the discussion on the original quest for a shuffle algorithm for Eth2,

hereIf you want to look at the swap-or-not shuffling algorithm from another angle, you can take a look at one published by Protolambda。 

at last

at last

Disclaimer: ECN's translation work aims to deliver high-quality information and learning resources for the Chinese Ethereum community. The copyright of the article belongs to the original author. Reprints must indicate the source of the original text and ethereum.cn. For long-term reprints, please contact ethereumcn@gmail.com Authorize.