4D Detailed Explanation of Danksharding: Or the Next Milestone Upgrade of Ethereum?
Author: Vitalik Buterin
Ethereum founder Vitalik Buterin recently answered questions related to Proto-danksharding (aka EIP-4844). Danksharding is a new sharding design proposed for Ethereum. What can this technology bring?
What is Danksharding?
Danksharding is a new sharding design proposed for Ethereum that introduces some significant simplifications compared to previous designs.
The main difference between all recent Ethereum sharding proposals since 2020 (including Danksharding and previous Danksharding) and most non-Ethereum sharding proposals is Ethereum’s rollup-centric roadmap: Ethereum Sharding Shards do not provide more space for transactions, but for data, which the Ethereum protocol itself does not attempt to explain.
Validating a blob simply checks that the blob is available, that is, can be downloaded from the network. The data space in these blobs is expected to be used by the Layer 2 (Layer 2) Rollup protocol that supports high-throughput transactions.
The main innovation introduced by Danksharding is the merged fee market: instead of a fixed number of shards, each shard has a different block and a different block proposer, in Danksharding only one proposer chooses to enter the slot All transactions and all data.
To avoid high system requirements for verifiers in this design, we introduce proposer/builder separation (PBS): a special class of participants called block builders bid for the right to select slots, and proposers only need to select bids The highest valid header is sufficient.
Only block builders need to process entire blocks (here, a third-party decentralized oracle protocol can also be used to implement distributed block builders); all other validators and users can sample data availability very efficiently Validate the block (remember: the "big" part of the block is just the data).
What is proto-danksharding (aka EIP-4844)?
Proto-danksharding (aka EIP-4844) is an Ethereum Improvement Proposal (EIP) that implements most of the logic and "scaffolding" (e.g. transaction format, validation rules) that make up the full Danksharding specification, but has not actually implemented any piece. In a proto-danksharding implementation, all validators and users must still directly verify the availability of the full data.
The main feature introduced by proto-danksharding is a new transaction type, which we call a transaction with a blob. A transaction with a blob is similar to a regular transaction, except that it also carries additional data called a blob. Blobs are very large (~125 kB) and much cheaper than a similar amount of calldata. However, the EVM execution cannot access the blob data; the EVM can only see the promises to the blob.
Because validators and clients still need to download the full blob content, the data bandwidth target in proto-danksharding is 1 MB per socket instead of the full 16 MB. However, since this data does not compete with the gas usage of existing Ethereum transactions, there is still a large scalability benefit.
Why is it okay to add 1 MB of data to a chunk that everyone has to download, instead of making calldata 10 times cheaper?
This has to do with the difference between average load and worst case load. Today we've run into situations where the average block size is around 90 kB, but the theoretically possible maximum block size (if all 30M gas in a block is used to call data) is around 1.8 MB. The Ethereum network used to process close to the maximum number of blocks.
However, if we simply reduce the calldata gas cost by a factor of 10, then although the average block size increases to a still acceptable level, the worst case becomes 18 MB, which is too much for the Ethereum network. too much.
The current gas pricing scheme cannot separate these two factors: the ratio between the average load and the worst case load depends on the user's choice of how much gas to spend on calldata vs other resources, which means that the gas price must be based on the worst case The likelihood setting of , causes the load average to be unnecessarily lower than what the system can handle.
However, if we change gas pricing to more explicitly create a multidimensional fee market, we can avoid the average-case/worst-case load mismatch and include in each block close to the maximum amount of data we can safely handle. Proto-danksharding and EIP-4488 are two proposals to do this.
How does proto-danksharding compare to EIP-4488?
EIP-4488 is an earlier and simpler attempt to solve the same average-case/worst-case load mismatch problem. EIP-4488 does this using two simple rules:
Calldata gas cost reduced from 16 gas per byte to 3 gas per byte
Limit of 1 MB per block plus an additional 300 bytes per transaction (theoretical max: ~1.4 MB)
A hard limit is the easiest way to ensure that a large increase in average-case load does not result in an increase in worst-case load. The reduction in gas costs will greatly increase the use of rollups, potentially increasing the average block size to hundreds of KB, but the hard limit will directly prevent the worst-case possibility of a single block containing 10 MB. In fact, the worst-case block size will be smaller than it is now (1.4 MB vs. 1.8 MB).
Proto-danksharding instead creates a separate transaction type that saves data cheaper in large fixed-size blobs and limits how many blobs each block can contain. These blobs are not accessible from the EVM (only commitments to blobs), and blobs are stored by the consensus layer (the beacon chain) rather than the execution layer.
The main practical difference between EIP-4488 and proto-danksharding is that EIP-4488 tries to minimize the changes required today, while proto-danksharding has a lot of changes today, so upgrading to full sharding in the future requires very few changes.
While implementing full sharding (using data availability sampling, etc.) was a complex task, and remains a complex task after proto-danksharding, this complexity is contained in the consensus layer. Once proto-danksharding is launched, no further work is required from the executive layer client team, rollup developers, and users to complete the transition to full sharding.
Note that the choice between the two is not either/or: we can implement EIP-4488 as soon as possible, and then follow it up half a year later with proto-danksharding.
What parts of complete darksharding does proto-danksharding implement? What else needs to be implemented?
Citing EIP-4844, the work that has been done in this EIP includes:
A new transaction type in the exact same format that needs to exist in "full sharding".
All execution layer logic required for full sharding.
All execution/consensus cross-validation logic required for full sharding.
Layer separation between BeaconBlock validation and data availability sampling blobs.
Most of the BeaconBlock logic needed for full sharding.
Self-adjusting independent gasprice for blobs.
The work to be done to achieve full sharding includes:
Low-level extension of blob_kzgs in the consensus layer to allow 2D sampling.
A practical implementation of data availability sampling,
PBS (proposer/builder split) to avoid requiring a single validator to process 32 MB of data in one slot.
Escrow proofs or similar in-protocol requirements for each validator to validate specific portions of shard data in each block.
Note that all remaining work is consensus layer changes and does not require any additional work to be performed by client teams, users, or Rollup developers.
What if all these very large chunks increase disk space requirements?
Both EIP-4488 and proto-danksharding lead to a long-term maximum usage of about 1 MB per socket (12 seconds). This equates to roughly 2.5 TB per year, well above the growth rate Ethereum needs today.
In the case of EIP-4488, solving this problem requires a history expiration scheme (EIP-4444), where clients are no longer required to store history beyond a certain period of time (persistent from 1 month to 1 year has been proposed time).
In the case of proto-danksharding, whether or not EIP-4444 is implemented, the consensus layer can implement separate logic to automatically delete blob data after a period of time (eg 30 days). However, regardless of the short-term data scaling solution, EIP-4444 is strongly recommended to be implemented as soon as possible.
Both strategies limit the additional disk load on consensus clients to at most a few hundred GB. In the long run, employing some historical expiration mechanism is essentially mandatory: full sharding adds about 40 TB of historical blob data per year, so users can only actually store a fraction of it for a while. So it's worth setting expectations for this early on.
How will users access old blobs if the data is deleted after 30 days?
The purpose of the Ethereum consensus protocol is not to guarantee the permanent storage of all historical data. Instead, the intention is to provide a highly secure real-time bulletin board and leave room for other decentralized protocols for longer-term storage.
The Bulletin Board exists to ensure that data posted on the Bulletin Board is available long enough that any user who wants that data, or any long-term agreement to back up the data, has enough time to get it and import it into their other application or protocol.
In general, long-term history storage is easy. While 2.5 TB per year is too large a requirement for a regular node, it's very manageable for dedicated users: you can buy very large hard drives for about $20 per TB, which is more than adequate for a hobbyist.
Unlike consensus, which has an N/2-of-N trust model, history storage has a 1-of-N trust model: you only need one of the data storers to be honest. Therefore, each piece of historical data only needs to be stored hundreds of times, instead of thousands of nodes that are doing real-time consensus verification.
Some practical ways to store the full history and make it easily accessible include:
Application-specific protocols such as Rollup may require their nodes to store portions of history relevant to their application. Lost historical data poses no risk to the protocol, only to individual applications, so it makes sense for applications to take the burden of storing data related to themselves.
Storing historical data in BitTorrent, eg. A 7 GB file containing the blob data from the block is automatically generated and distributed daily.
The Ethereum portal network (currently under development) can be easily extended to store history.
Block explorers, API providers, and other data services may store full history.
Individual hobbyists and academics engaged in data analysis may store complete historical records. In the latter case, storing the history locally provides them with significant value, since it makes it easier to compute directly on it.
Third-party indexing protocols such as TheGraph may store the full history.
At higher levels of historical storage (e.g. 500 TB per year), the risk of some data being forgotten becomes higher (in addition, the data availability verification system becomes more stressed). This may be the real limit of scalability for sharded blockchains. However, all currently proposed parameters are very far from this point.
What is the format of the blob data and how is it submitted?
A blob is a vector of 4096 field elements, numbers in the range:
0 <= x < 52435875175126190479447740508185965837690552500527637822603658699938581184513
A blob is viewed mathematically as representing a polynomial of degree < 4096 over a finite field with the above modulus, where the field element at position i in the blob is the evaluation of that polynomial at wi. w is a constant satisfying w=1.
The commitment to the blob is a hash of the KZG commitment to the polynomial. However, from an implementation point of view, it is not important to care about the mathematical details of polynomials. Instead, there will only be a vector of elliptic curve points (based on a trusted Lagrangian setup), and the KZG commitment to the blob will simply be a linear combination. Citing code from EIP-4844:
def blob_to_kzg(blob: Vector[BLSFieldElement, 4096]) -> KZGCommitment: computed_kzg = bls.Z1 for value, point_kzg in zip(tx.blob, KZG_SETUP_LAGRANGE): assert value < BLS_MODULUS computed_kzg = bls.add( computed_kzg, bls.multiply(point_kzg, value) ) return computed_kzg
BLS_MODULUS is the above modulus, and KZG_SETUP_LAGRANGE is a vector of elliptic curve points, which is a trusted setup based on Lagrange. It is now reasonable for implementors to think of it simply as a black-box specialized hash function.
Why use KZG's hash instead of KZG directly?
Instead of using KZG to directly represent a blob, EIP-4844 uses a versioned hash: a single 0x01 byte (indicating this version) followed by the last 31 bytes of the KZG's SHA256 hash.
This is done for EVM compatibility and future compatibility: KZG promises are 48 bytes, while EVM more naturally uses 32 byte values, and if we switch from KZG to something else (eg, for quantum-resistant reasons), KZG commitments can continue to be 32 bytes.
What are the two precompilations introduced in proto-danksharding?
Proto-danksharding introduces two kinds of precompilation: blob verification precompilation and dot evaluation precompilation.
The blob-validation precompile is self-explanatory: it takes as input a versioned hash and a blob, and verifies that the provided versioned hash is actually a valid versioned hash for the blob. This precompilation is intended to be used by Optimistic Rollup. Quoting EIP-4844:
Optimistic Rollup only needs to actually provide the underlying data when submitting a fraud proof. The fraud proof submission feature will require the entire contents of the fraud blob to be submitted as part of the calldata. It will use blob validation to validate data against a previously committed versioned hash, then perform fraud proof validation on that data as it does today.
Point-evaluation precompilation takes as input a versioned hash, x-coordinate, y-coordinate, and a proof (the blob's KZG commitment and the KZG evaluation proof). It verifies the proof to check that P(x) = y, where P is the polynomial represented by the blob with the given versioned hash. This precompilation is intended to be used by ZK Rollup. Quoting EIP-4844:
ZK rollup will provide two commitments for its transaction or state delta data: KZG in the blob and some commitment using whatever proof system ZK rollup uses internally. They will use Proof of Commitment of Equivalent Protocol, using point-evaluation precompilation, to prove that kzg (protocol guarantees pointing to available data) and ZK rollup's own commitment refer to the same data.
Note that most major Optimistic Rollup designs use a multi-round fraud-proof scheme, where the last round requires only a small amount of data. So, conceivably, Optimistic Rollup could also use point evaluation precompilation instead of blob verification precompilation, and it would be cheaper to do so.
What does a KZG trusted setup look like?
look:
A general description of how trusted settings work
https://vitalik.ca/general/2022/03/14/trustedsetup.html for powers-of-tau
Example implementations of all important trusted setup related calculations
https://github.com/ethereum/research/blob/master/trusted_setup/trusted_setup.py
Specifically in our case, the current plan is to run four sizes in parallel (n1=4096, n2=16), (n1=8192, n2=16), (n1=16834, n2=16) and (n1= 32768,n2=16) ritual (with a different secret). In theory, only the first one is needed, but running more of the larger size improves future applicability by allowing us to increase the blob size. We can't just have a larger setup, because we want to be able to have a hard limit on the number of polynomials that can be effectively committed, equal to the blob size.
A possible practical approach would be to start with a Filecoin setup, then run a ceremony to scale it. Multiple implementations including browser implementations will allow many people to participate.
Can't we use some other commitment scheme without a trusted setup?
Unfortunately, using anything other than KZG (such as IPA or SHA256) makes the sharding roadmap more difficult. There are several reasons for this:
Non-arithmetic commitments (such as hash functions) are not compatible with data availability sampling, so if we use such a scheme, when we move to full sharding, we have to change to KZG anyway.
IPA may be compatible with data availability sampling, but it leads to more complex schemes with weaker properties (e.g. self-healing and distributed block building become harder)
Both hashing and IPA are not compatible with cheap implementations of point evaluation precompilation. Therefore, a hash- or IPA-based implementation will not be able to effectively enable ZK Rollup or support cheap fraud proofs in multiple rounds of Optimistic Rollup.
So, unfortunately, the loss of functionality and increased complexity of using anything other than KZG far outweighs the risks of KZG itself. Additionally, any KZG-related risks are covered: a KZG failure will only affect Rollup and other applications that depend on blob data, not the rest of the system.
How "complex" and "new" is KZG?
KZG commitments were introduced in a 2010 paper and have been widely used in PLONK-style ZK-SNARK protocols since 2019. However, the underlying math that KZG promises is a relatively simple arithmetic on top of the underlying math of elliptic curve operations and pairings.
The specific curve used is BLS12-381, which was invented by Barreto-Lynn-Scott in 2002. Elliptic curve pairings, which are necessary to verify the KZG promise, are very complex mathematics, but they were invented in the 1940s and have been used in cryptography since the 1990s. By 2001, a number of encryption algorithms using pairings were proposed.
From an implementation complexity perspective, KZG is no more difficult to implement than IPA: the function to compute the commitment (see above) is exactly the same as in the case of IPA, just using a different set of elliptic curve point constants. Point-verification precompilation is more complicated as it involves pair evaluation, but the math is the same as what is already done in the EIP-2537 (BLS12-381 precompile) implementation, and is very similar to bn128 pair precompilation (see also: Optimized Python accomplish). Therefore, no complex "new work" is required to implement KZG verification.
What are the different software parts of the proto-danksharding implementation?
There are four main components:
1. The execution layer consensus has changed (see EIP for details):
New transaction type that includes blobs
Opcode that outputs the versioned hash of the ith blob in the current transaction
Blob validation precompilation
dot evaluation precompilation
2. Consensus layer consensus changes (see this folder in the repo):
List of blob KZG in BeaconBlockBody
"sidecar" mechanism where the full blob content is passed along with a separate object from the BeaconBlock
Cross-check between blob versioned hashes in the execution layer and the blob KZG in the consensus layer
3. Memory pool
BlobTransactionNetworkWrapper (see network section of EIP)
Stronger anti-DoS protection to compensate for large blob sizes
4. Block construction logic
Accept transaction wrapper from mempool, put transaction into ExecutionPayload, put KZG into beacon block and body in sidecar
Responding to the two-dimensional fee market
Note that for a minimal implementation, we don't need a mempool at all (we can rely on a second layer transaction bundle market), we just need a client to implement the block building logic. Only consensus changes in the execution layer and consensus layer require extensive consensus testing, which is relatively lightweight. Anything between such a minimal implementation and a "full" deployment where all clients support block production and mempool is possible.
What does a proto-danksharding multidimensional fee market look like?
Proto-danksharding introduces a multi-dimensional EIP-1559 fee market with two resources, gas and blob, with separate floating gas prices and separate limits.
Ethereum
The blob fee is charged in gas, but it's a variable amount of gas that adjusts so that in the long run the average number of blobs per block actually equals the target amount.
The two-dimensional nature means that block builders will face a harder problem: instead of simply accepting transactions with the highest priority fees until they run out of transactions or hit the block gas limit, they will have to avoid reaching both different restrictions.
Here is an example. Let's say the gas limit is 70 and the blob limit is 40. The mempool has many transactions, enough to fill up a block, and there are two types (tx gas includes per-blob gas):
Priority fee 5 per gas, 4 blobs, 4 total gas
Priority fee 3 per gas, 1 blob, 2 total gas
Ethereum
Do executing clients now have to implement complex multidimensional knapsack algorithms to optimize their block production? No, there are several reasons:
EIP-1559 ensures that most blocks do not hit either limit, so only a small number of blocks actually face the multidimensional optimization problem. In the usual case where the mempool doesn't have enough (enough paying) transactions to reach either limit, any miner can earn optimally by including every transaction they see.
In practice, fairly simple heuristics can come close to optimal. In a similar situation, see Ansgar's analysis of EIP-4488 for some data on this.
Multidimensional pricing isn't even the biggest revenue stream from specialization - MEV is. Dedicated MEV revenue extracted via dedicated algorithms from on-chain DEX arbitrage, liquidations, front-running NFT sales, etc. makes up a significant portion of total "extractable revenue" (i.e. priority fees): dedicated MEV revenue appears to average around 0.025 ETH per zone blocks, the total priority fee is usually around 0.1 ETH per block.
The proposer/builder separation is designed around highly specialized block production. PBS turns the block building process into an auction where professional participants bid for the privilege of creating a block. Regular validators only need to accept the highest bid. This is to prevent MEV-driven economies of scale from creeping into validator centralization, but it handles all the issues that could make optimizing block building more difficult.
For these reasons, more complex fee market dynamics do not greatly increase centralization or risk; in fact, the principles applied more broadly can actually reduce the risk of DoS attacks!
How does the Exponential EIP-1559 blob fee adjustment mechanism work?
Today’s EIP-1559 adjusts the base fee b to achieve a specific target gas usage level t, as follows:
where b(n) is the base fee for the current block, b(n+1) is the base fee for the next block, t is the target, and u is the gas used.
Ethereum
Although the average usage is equal to t, the base fee drops by 63/64. So the basefee is only stable if the usage is slightly above t; in practice it is apparently about 3% higher, although the exact number depends on the variance.
A better formula is exponential adjustment:
exp(x) is the exponential function e^x, where e≈2.71828. When the value of x is small, exp(x)≈1+x. However, it has a convenient property that is not related to transaction permutation: multi-step adjustment
Ethereum
Therefore, the same transactions included will result in the same final base fee, no matter how they are distributed among different blocks.
The last formula above also has a natural mathematical interpretation: the term (u1+u2+...+u/n-nt) can be seen as redundant: the difference between the total gas actually used and the total gas intended to be used.
The current base fee is equal to
The fact that the excess cannot exceed a very narrow range: if it exceeds 8t∗60, then the basefee becomes e^60, so ridiculously high that no one can pay it, and if it falls below 0, the resource basically is free, and the chain will be spammed until the excess returns above zero.
The adjustment mechanism works exactly in these terms: it keeps track of the actual total (u1+u2+...+u/n) and calculates the target total (nt), and calculates the price as an index of the difference. To make the calculation easier, instead of e^x, we use 2^x; in fact, we use an approximation of 2^x: the fake_exponential function in EIP. The false index is almost always within 0.3% of the actual value.
To prevent long periods of underutilization from causing long periods of 2x full blocks, we've added an extra feature: we don't let excess blocks go below zero. If actual_total is lower than targeted_total, we simply set actual_total equal to targeted_total. In extreme cases (blob gas all the way down to zero) this does break transaction order invariance, but the added safety makes this an acceptable tradeoff.
Note also an interesting consequence of this multi-dimensional market: when proto-danksharding is first introduced there are likely to be very few users initially, so for a while the cost of a blob will almost certainly be very cheap, even for "regular" Ethereum blockchain activity remains expensive.
The authors believe this fee adjustment mechanism is better than the current approach, so eventually all segments of the EIP-1559 fee market should move to using it.
For a longer and more detailed explanation, see Dankrad's post.
How does fake_exponential work?
For convenience, here is the code for fake_exponential:
def fake_exponential(numerator: int, denominator: int) -> int: cofactor = 2 ** (numerator // denominator) fractional = numerator % denominator return cofactor + ( fractional * cofactor * 2 + (fractional ** 2 * cofactor) // denominator ) // (denominator * 3)
Ethereum
The goal is to stitch together many instances of (QX), one of which is appropriately shifted and scaled for each [2^k,2^(k+1)] range. Q(x) itself is an approximation of 2^x for 0≤x≤1, chosen for the following properties:
Simplicity (it's a quadratic equation)
Correctness of the left edge (Q(0)=2^0=1)
Right edge correctness (Q(1)=2^1=2)
Smooth slope (we ensure that Q'(1)=2∗Q'(0), so each shifted+scaled copy of Q has the same slope on its right edge as the next copy has on its left edge)
The last three require three linear equations given three unknown coefficients, and Q(x) given above gives the only solution.
Ethereum
What issues are still being debated in proto-danksharding?
NOTE: This section is easily outdated. Don't trust it to give the latest thinking on any particular problem.
All major Optimistic Rollups use multi-round proofs, so they can use (much cheaper) point evaluation precompilations instead of blob verification precompilations. Anyone who really needs blob verification can implement it themselves: take a blob D and a versioned hash h as input, choose x=hash(D,h), compute y=D(x) using barycentric evaluation and predict Compile and verify that h(x)=y. So do we really need blob validation precompilation, or can we just remove it and just use point evaluation?
How capable is the chain at handling long-lasting 1 MB+ blocks? If the risk is too great, should the target blob count be reduced in the first place?
Should blobs be denominated in gas or ETH (burned)? Should there be other adjustments to the fee market?
Should the new transaction type be considered as a blob or as an SSZ object, in the latter case change the ExecutionPayload to a union type? (It's a "do more work now" vs "do more work later" trade-off)
The exact details of the trusted setup implementation (technically outside the scope of the EIP itself, since such a setup is "just a constant" to the implementor, but still needs to be done).
