Paradigm: Detailed load testing tool Flood
Original author: "Introducing flood: a load testing tool for benchmarking EVM nodes》
introduce
Compilation of the original text: Deep Tide TechFlow
introduce
Load testing is a key step in building and developing data systems with high resilience and excellent performance. However, load testing is not widely used in cryptocurrency infrastructure development. We're excited to introduce Flood, a benchmarking tool for RPC endpoint performance analysis, to fill this gap.
Initially, we used Flood as a tool to optimize Reth and understand its latency and throughput tradeoffs under different loads. Later, we discovered that Flood has significant utility in optimizing the performance of many types of cryptocurrency infrastructure in addition to Reth.
Let's take a look.
What is load testing and why is it important?
Load testing refers to the measurement of changes in performance characteristics of a system as it is subjected to different workloads. The key insight behind this approach is that performance metrics such as throughput, latency, and error rates typically degrade when a system is subjected to increasing load. Therefore, observing how the system behaves under different control loads can reveal information such as system bottlenecks, failure modes, and ultimate performance capabilities.
The information obtained through load testing can be used in many ways. While the system is being developed, load testing can highlight which system bottlenecks need to be prioritized for improvement. When comparing two systems, load testing can reveal which system is more performant or reliable. As a special case of this, load testing can compare two different hardware or software configurations of a single system. In each case, load testing enables highly optimized system development.
How to load test the blockchain nodes?
Our focus is on RPC, which is the communication protocol commonly used to extract data from blockchain nodes.
Currently, the most common way to measure RPC performance is not a load test, but a latency test: you send a request to an RPC node and measure how long it takes to get a response. Latency tests for various RPC providers can be found on various websites. Unfortunately, this type of testing provides a limited view into node performance because it reveals little about how the system behaves under load.
In a blockchain, workloads can vary in two important ways. The classic variable is size. A load of 10,000 requests per second stresses the system more than a load of 100 requests per second. Another load variable is the RPC method. There are different RPC methods for each type of data extracted from a blockchain node. For example, block vs transaction vs log vs trace. Each RPC method places a different type of load on the system. Some RPC methods are memory IO bound, while others are CPU bound.
What is Flood?
We developed a load testing tool called Flood based on these principles. Unlike latency testing, Flood provides an unprecedented view into the performance characteristics of RPC endpoints through load testing and extending test coverage to all relevant RPC methods.
Flood consists of three basic components:
Call Generation Engine: Flood generates a large set of parameterized RPC calls with random sampling distributions similar to different types of blockchain workloads. Flood utilizes Paradigm Data Portal datasets to ensure complete coverage of blockchain history.
Load testing engine: Flood then orchestrates Vegeta (a high-performance load testing tool written in Go) to use these calls to load test RPC endpoints.
Reporting Engine: After Flood executes a test, it summarizes the results using various charts, tables, and reports. These summaries are easy to integrate into scripts and data.
Each component is highly configurable, enabling Flood to cover a wide range of testing scenarios and environments.
What can Flood do?
During the normal operation of Flood, the user specifies the RPC methods to be tested and a list of RPC endpoints. For example, you might want to test the performance of two versions of Reth's eth_getLogs. Flood will run different control loads to test these RPC endpoints. For example, it might run eth_getLogs at 1,000, 2,000, 4,000, and 8,000 requests per second. Flood will then display tables and graphs summarizing how performance metrics vary with load. The output looks like this:
Specific ways in which performance metrics degrade under load provide rich insights that can reveal system bottlenecks and ultimate performance capacity.
Beyond that, Flood offers advanced features to suit every type of power user:
Flood can use different load testing plans, including: "stress testing" (gradually increasing the load over time), "spike testing" (a large sudden load followed by small loads) and "soak testing" (long-running load).
Flood can orchestrate load tests to run in native mode on each RPC node to eliminate noise caused by network bottlenecks.
Flood has an "equality" test mode that checks that each RPC endpoint returns the same response.
Why build Flood?
At Paradigm, we are developing a new node implementation called Reth, and performance is one of its main goals. We developed Flood to detail the performance characteristics of Reth. We have used Flood to reveal numerous Reth performance bottlenecks under various workloads and system configurations. Then, address those bottlenecks. With Flood, we've created a tight feedback loop that gives Reth developers high visibility into how any codebase change translates to end-to-end system performance.
In addition to Reth, we think Flood will be able to help solve many unanswered questions related to RPC nodes:
Which hardware specs are most important when running a node? How important is storage IO relative to RAM speed, RAM capacity, and CPU speed? Is RAID worth it?
What is the effective rate limit per RPC method per third-party RPC provider?
in conclusion
in conclusion
In this paper, we mainly introduce Flood, a load testing tool that provides an unprecedented view into the performance characteristics of blockchain nodes. While Flood was originally built to optimize the development of Reth, we believe it will become an important tool for the development of other types of high-performance cryptocurrency infrastructure. We look forward to seeing how others use Flood to build their own high-performance, reliable systems.
