first level title

TL;DR

• At present, there are two main directions for the combination of AI + Crypto: distributed computing power and ZKML; for ZKML, please refer to my previous article.This article will analyze and reflect on the decentralized distributed computing power network。 

• Under the development trend of AI large models,Computing power resources will be the big battlefield in the next decade, and also the most important thing for human society in the future, and not only stay in commercial competition, but also become a strategic resource for the game of great powers. In the future, investment in high-performance computing infrastructure and computing power reserves will increase exponentially.

• The decentralized distributed computing power network has the greatest demand for AI large model training, but it also faces the greatest challenges and technical bottlenecks. Including the need for complex data synchronization and network optimization issues. In addition, data privacy and security are also important constraints. Although there are some existing techniques that can provide preliminary solutions, they are still not applicable in large-scale distributed training tasks due to the huge computational and communication overhead.

• The decentralized distributed computing power network has a better chance of landing in model reasoning, and it can predict that the future incremental space is also large enough. But it also faces challenges such as communication delay, data privacy, and model security. Compared with model training, inference has lower computational complexity and data interaction, and is more suitable for distributed environments.

• first level title

1. Distributed Computing Power—Large Model Training

We are discussing the application of distributed computing power in training, and generally focus on the training of large language models. The main reason is that the training of small models does not require much computing power. In order to do distributed data privacy and a bunch of projects The problem is not cost-effective, it is better to solve it directly and centrally. The large language model has a huge demand for computing power, and it is now in the initial stage of the explosion. From 2012 to 2018, the computing demand of AI will double every 4 months. Now it is the concentrated point of computing power demand, which can be predicted Judging that the next 5-8 years will still be a huge incremental demand.

secondary title

 (NVIDIA NeMo Megatron Framework）

1. The overall training process

Take for example training a large model with 175 billion parameters. Due to the large size of the model, it needs to be trained in parallel on many GPU devices. Suppose there is a centralized computer room with 100 GPUs and each device has 32 GB of memory.

• data preparation: First of all, a huge data set is needed, which contains various data such as Internet information, news, books, etc. These data need to be preprocessed before training, including text cleaning, tokenization, vocabulary construction, etc.

• data segmentation: The processed data will be divided into multiple batches for parallel processing on multiple GPUs. Suppose the selected batch size is 512, that is, each batch contains 512 text sequences. Then, we split the entire dataset into batches, forming a queue of batches.

• Data transfer between devices: At the beginning of each training step, the CPU fetches a batch from the batch queue, and then sends the data of this batch to the GPU via the PCIe bus. Assuming the average length of each text sequence is 1024 tokens, the data size of each batch is approximately 512 * 1024 * 4 B = 2 MB (assuming each token is represented by a 4-byte single-precision floating-point number). This data transfer process usually only takes a few milliseconds.

• parallel training: After each GPU device receives the data, it starts to perform forward pass and backward pass calculations, and calculates the gradient of each parameter. Due to the large size of the model, the memory of a single GPU cannot store all the parameters, so we use model parallel technology to distribute the model parameters on multiple GPUs.

• Gradient Aggregation and Parameter Update: After the backpropagation calculation is completed, each GPU gets the gradient of a part of the parameters. These gradients then need to be aggregated across all GPU devices in order to compute the global gradient. This requires data transfer over the network, and assuming a 25 Gbps network, it would take about 224 seconds to transfer 700 GB of data (about 700 GB for 175 billion parameters, assuming single precision floating point for each parameter). Each GPU then updates its stored parameters according to the global gradient.

• Synchronize: After the parameters are updated, all GPU devices need to be synchronized to ensure that they all use consistent model parameters for the next step of training. This also requires data transfer over the network.

• Repeat training steps: Repeat the above steps until the training of all batches is completed, or the predetermined number of training rounds (epoch) is reached.

secondary title

2. The bottleneck of communication overhead:

It should be noted that the bottleneck of communication is also the cause of the current distributed computing power networkcan not do thisThe reason for large language model training.

Each node needs to exchange information frequently to work together, which creates communication overhead. For large language models, this problem is especially serious due to the large number of parameters of the model. The communication overhead is divided into these aspects:

• data transmission: Nodes need to frequently exchange model parameters and gradient information during training. This requires the transmission of a large amount of data in the network, consuming a large amount of network bandwidth. If the network conditions are poor or the distance between computing nodes is large, the delay of data transmission will be high, further increasing the communication overhead.

• sync problem: During training, nodes need to work together to ensure correct training. This requires frequent synchronization operations between nodes, such as updating model parameters, computing global gradients, etc. These synchronous operations need to transmit a large amount of data in the network, and need to wait for all nodes to complete the operation, which will cause a lot of communication overhead and waiting time.

• Gradient accumulation and update: During the training process, each node needs to calculate its own gradient and send it to other nodes for accumulation and updating. This requires the transmission of a large amount of gradient data in the network, and the need to wait for all nodes to complete the calculation and transmission of gradients, which is also the reason for a large amount of communication overhead.

• data consistency: It is necessary to ensure that the model parameters of each node are consistent. This requires frequent data checksum and synchronization operations between nodes, which results in a large amount of communication overhead.

Although there are some methods to reduce communication overhead, such as compression of parameters and gradients, efficient parallel strategies, etc., these methods may introduce additional computational burden or negatively affect the training effect of the model. Also, these methods cannot completely solve the communication overhead problem, especially in the case of poor network conditions or large distances between computing nodes.

As an example:

Decentralized distributed computing power network

The GPT-3 model has 175 billion parameters, and if we represent these parameters using single-precision floating point numbers (4 bytes per parameter), then storing these parameters requires ~700 GB of memory. In distributed training, these parameters need to be frequently transmitted and updated between computing nodes.

Assuming there are 100 computing nodes, each node needs to update all the parameters in each step, then each step needs to transfer about 70 TB (700 GB* 100 ) of data. If we assume that a step takes 1 s (very optimistic assumption), then 70 TB of data needs to be transferred every second. This demand for bandwidth already far exceeds that of most networks and is also a matter of feasibility.

In reality, due to communication delays and network congestion, the data transmission time may be much longer than 1 s. This means that computing nodes may need to spend a lot of time waiting for data transmission instead of performing actual calculations. This will greatly reduce the efficiency of training, and this reduction in efficiency cannot be resolved by waiting, but the difference between feasible and infeasible, which will make the entire training process infeasible.

Centralized computer room

Even in a centralized computer room environment, the training of large models still requires heavy communication optimization.

In a centralized computer room environment, high-performance computing devices are used as a cluster, connected through a high-speed network to share computing tasks. However, even when training a model with an extremely large number of parameters in such a high-speed network environment, the communication overhead is still a bottleneck, because the parameters and gradients of the model need to be frequently transmitted and updated between various computing devices.

As mentioned at the beginning, assume 100 compute nodes with 25 Gbps of network bandwidth per server. If every server needs to update all parameters every training step, it takes ~224 seconds to transfer about 700 GB of data per training step. By taking advantage of the centralized computer room, developers can optimize the network topology inside the data center and use technologies such as model parallelism to significantly reduce this time.

In contrast, if the same training is performed in a distributed environment, assuming there are still 100 computing nodes distributed all over the world, the average network bandwidth of each node is only 1 Gbps. In this case, it takes ~5600 seconds to transfer the same 700 GB of data, which is much longer than in the centralized computer room. Also, due to network delays and congestion, the actual time required may be longer.

However, compared to the situation in a distributed computing power network, it is relatively easy to optimize the communication overhead in a centralized computer room environment. Because in a centralized computer room environment, computing devices are usually connected to the same high-speed network, and the bandwidth and delay of the network are relatively good. In a distributed computing power network, computing nodes may be distributed all over the world, and the network conditions may be relatively poor, which makes the problem of communication overhead more serious.

secondary title

（Data used to train LLM models）

3. Why the distributed computing power network cannot do these optimizations

It can be done, but compared with the centralized computer room, the effect of these optimizations is very limited.

1. Network topology optimization: In the centralized computer room, the network hardware and layout can be directly controlled, so the network topology can be designed and optimized according to the needs. However, in a distributed environment, computing nodes are distributed in different geographical locations, even one in China and one in the United States, and there is no way to directly control the network connection between them. Although software can be used to optimize the data transmission path, it is not as effective as directly optimizing the hardware network. At the same time, due to differences in geographical locations, network delays and bandwidths also vary greatly, which further limits the effect of network topology optimization.

2. model parallelismsecondary title

4. Data security and privacy challenges

Almost all links involving data processing and transmission may affect data security and privacy:

1. data distribution: The training data needs to be distributed to each node participating in the calculation. Data in this link may be maliciously used/leaked on distributed nodes.

2. model training: During the training process, each node will use its assigned data for calculation, and then output the update or gradient of the model parameters. During this process, if the calculation process of the node is stolen or the result is maliciously analyzed, data may also be leaked.

3. Parameter and Gradient Aggregation: The outputs of individual nodes need to be aggregated to update the global model, and the communication during aggregation may also leak information about the training data.

What are the solutions to data privacy concerns?

• Secure Multi-Party Computation: SMC has been successfully applied in some specific, small-scale computing tasks. However, in large-scale distributed training tasks, due to its large computational and communication overhead, it has not been widely used yet.

• Differential privacy: Applied in certain data collection and analysis tasks, such as Chrome's user statistics. But in large-scale deep learning tasks, DP will have an impact on the accuracy of the model. At the same time, it is also a challenge to design an appropriate noise generation and addition mechanism.

• Federated Learning: Applied to some edge device model training tasks, such as vocabulary prediction for Android keyboards, etc. But in larger-scale distributed training tasks, FL faces problems such as high communication overhead and complex coordination.

• Summarize

Summarize

Each of the above methods has its applicable scenarios and limitations, and none of the methods can completely solve the data privacy problem in the large model training of distributed computing power network.

Can ZK, which has high hopes, solve the data privacy problem in large model training?

In theory, ZKP can be used to ensure data privacy in distributed computing, allowing a node to prove that it has performed calculations according to regulations, but does not need to disclose actual input and output data.

But in fact, the following bottlenecks are faced in the scenario of using ZKP for large-scale distributed computing power network training large models:

• Computational and communication overhead up: Constructing and verifying zero-knowledge proofs requires massive computing resources. In addition, ZKPs have a high communication overhead because of the need to transmit the proof itself. These overheads can become especially significant in the case of large model training. For example, if the computation of each mini-batch requires the generation of a proof, this can significantly increase the overall time and cost of training.

• Complexity of the ZK protocol: Designing and implementing a ZKP protocol suitable for large model training can be very complex. This protocol needs to be able to handle large-scale data and complex calculations, and it needs to be able to handle possible abnormal errors.

• Hardware and Software CompatibilitySummarize

Summarize

first level title

2. Distributed Computing Power—Model Reasoning

Another relatively large scenario of distributed computing power is model reasoning. According to our judgment on the development path of large models, the demand for model training will gradually slow down as the large models mature after passing a high point. Inference requirements will correspondingly increase exponentially with the maturity of large models and AIGC.

secondary title

(Power LLM inference with NVIDIA Triton)

1. Challenge

Communication delay:

In a distributed environment, communication between nodes is essential. In a decentralized distributed computing power network, nodes may be spread all over the world, so network latency can be a problem, especially for reasoning tasks that require real-time response.

Model deployment and update:

The model needs to be deployed to each node. If the model is updated, each node needs to update its model, which consumes a lot of network bandwidth and time.

Data Privacy:

Although inference tasks usually only require input data and models, and do not need to return a large amount of intermediate data and parameters, the input data may still contain sensitive information, such as users' personal information.

Model Security:

In a decentralized network, the model needs to be deployed on untrusted nodes, which will lead to the leakage of the model and lead to the problem of model property rights and abuse. This can also raise security and privacy concerns, if a model is used to process sensitive data, nodes can infer sensitive information by analyzing the behavior of the model.

QC:

secondary title

2. Feasibility

Computational complexity:

In the training phase, the model needs to iterate repeatedly. During the training process, it is necessary to calculate the forward propagation and back propagation of each layer, including the calculation of the activation function, the calculation of the loss function, the calculation of the gradient and the update of the weight. Therefore, the computational complexity of model training is high.

In the inference phase, only one forward pass is required to compute the prediction. For example, in GPT-3, the input text needs to be converted into a vector, and then forwarded through each layer of the model (usually the Transformer layer), and finally the output probability distribution is obtained, and the next word is generated according to this distribution. In GANs, the model needs to generate an image from an input noise vector. These operations only involve the forward propagation of the model, do not need to calculate gradients or update parameters, and have low computational complexity.

Data Interactivity:

During the inference phase, the model usually processes a single input rather than the large batch of data during training. The result of each inference only depends on the current input, not on other input or output, so there is no need for a large amount of data interaction, and the communication pressure is less.

Taking the generative image model as an example, assuming we use GANs to generate images, we only need to input a noise vector to the model, and then the model will generate a corresponding image. In this process, each input will only generate one output, and there is no dependency between outputs, so there is no need for data interaction.

Summarize

Summarize

first level title

3. Project

image description

1.Together

（RedPajama from Together）

Together is a company that focuses on the open source of large models and is committed to decentralized AI computing solutions, hoping that anyone, anywhere can access and use AI. Together just closed a $20m USD seed round led by Lux Capital.

Together was co-founded by Chris, Percy, and Ce. The original intention was that large model training required a large number of high-end GPU clusters and expensive expenditures, and these resources and model training capabilities were also concentrated in a few large companies.

From my point of view, a more reasonable entrepreneurial plan for distributed computing power is:

Step 1. Open source model

To implement model reasoning in a decentralized distributed computing power network, the prerequisite is that nodes must be able to obtain the model at low cost, that is to say, the model using the decentralized computing power network needs to be open source (if the model needs to be licensed in the corresponding If used below, it will increase the complexity and cost of the implementation). For example, chatgpt, as a non-open source model, is not suitable for execution on a decentralized computing power network.

Therefore, it can be speculated that the invisible barrier of a company that provides a decentralized computing power network needs to have strong large-scale model development and maintenance capabilities. Self-developed and open-sourced a powerful base model can get rid of the dependence on third-party model open source to a certain extent, and solve the most basic problems of decentralized computing power network. At the same time, it is more conducive to proving that the computing power network can effectively carry out training and reasoning of large models.

And Together did the same. The recently released LLaMA-based RedPajama was jointly launched by teams including Together, Ontocord.ai, ETH DS 3 Lab, Stanford CRFM, and Hazy Research. The goal is to develop a series of fully open source large language models.

Step 2. Implement distributed computing power in model reasoning

As mentioned in the above two sections, compared with model training, model inference has lower computational complexity and data interaction, and is more suitable for a decentralized distributed environment.

On the basis of the open source model, Together's R&D team has made a series of updates to the RedPajama-INCITE-3 B model, such as using LoRA to achieve low-cost fine-tuning, making the model run on the CPU (especially the MacBook using the M 2 Pro processor) Pro) to run the model more silky. At the same time, although the scale of this model is small, its ability exceeds other models of the same scale, and it has been practically applied in legal, social and other scenarios.

image description

(Schematic diagram of the computing power network of Overcoming Communication Bottlenecks for Decentralized Training)

In the medium and long term, although facing great challenges and technical bottlenecks, it must be the most attractive to undertake the computing power demand for AI large model training. Together began to lay out how to overcome the communication bottleneck in decentralized training at the beginning of its establishment. They also published a related paper on NeurIPS 2022:Overcoming Communication Bottlenecks for Decentralized Training. We can mainly summarize the following directions:

Scheduling optimization

When training in a decentralized environment, it is important to assign communication-heavy tasks to devices with faster connections because the connections between nodes have different latencies and bandwidths. Together builds a model to describe the cost of a specific scheduling strategy, and better optimizes the scheduling strategy to minimize communication costs and maximize training throughput. The Together team also found that even with a network 100 times slower, the end-to-end training throughput was only 1.7 to 2.3 times slower. Therefore, it is interesting to catch up the gap between distributed networks and centralized clusters through scheduling optimization.

Communication Compression Optimization

Together proposes communication compression for forward activations and backward gradients, and introduces the AQ-SGD algorithm, which provides strict guarantees for stochastic gradient descent convergence. AQ-SGD is able to fine-tune large base models on slow networks (e.g. 500 Mbps) with only 31% slower end-to-end training performance on centralized networks (e.g. 10 Gbps) without compression. In addition, AQ-SGD can be combined with state-of-the-art gradient compression techniques such as QuantizedAdam to achieve a 10% end-to-end speedup.

project summary

Together team configuration is very comprehensive, members have a very strong academic background, from large model development, cloud computing to hardware optimization are supported by industry experts. And Together did show a long-term and patient posture in path planning, from developing open source large models to testing idle computing power (such as mac) in the distributed computing power network using model reasoning, and then distributed computing power in large Layout on model training. — There is that kind of accumulation and thin hair feeling :)

image description

2.Gensyn.ai

 (Gensyn.ai)

From the technical path of Together, we can roughly understand the implementation process of the decentralized computing power network in model training and reasoning, as well as the corresponding research and development priorities.

Another important point that cannot be ignored is the design of the incentive layer/consensus algorithm of the computing power network. For example, an excellent network needs to have:

1. Make sure the benefits are attractive enough;

2. Ensure that each miner gets the benefits he deserves, including anti-cheating and more pay for more work;

3. Ensure that tasks are directly and reasonably scheduled and allocated on different nodes, and there will not be a large number of idle nodes or overcrowding of some nodes;

4. The incentive algorithm is simple and efficient, and will not cause excessive system burden and delay;

……

See how Gensyn.ai does it:

• become a node

verify

• verify

Solver generates multiple checkpoints while updating parameters (to ensure the transparency and traceability of work), and periodically generates cryptographic reasoning proofs (proof of work progress) about tasks;

When the Solver completes the work and produces a part of the calculation results, the protocol will select a verifier, and the verifier will also pledge a certain amount (to ensure that the verifier performs the verification honestly), and decide which part of the calculation results needs to be verified according to the proofs provided above.

• If solver and verifier diverge

Through the Merkle tree-based data structure, locate the exact location where the calculation results diverge. The entire verification operation will be on the chain, and cheaters will be deducted from the pledged amount.

project summary

The design of the incentive and verification algorithm makes Gensyn.ai not need to replay all the results of the entire computing task during the verification process, but only needs to copy and verify a part of the results according to the provided proof, which greatly improves the efficiency of verification. At the same time, nodes only need to store part of the calculation results, which also reduces the consumption of storage space and computing resources. In addition, potential cheating nodes cannot predict which parts will be selected for verification, so this also reduces the risk of cheating;

This method of verifying differences and discovering cheaters can also quickly find errors in the calculation process without comparing the entire calculation results (starting from the root node of the Merkle tree and traversing down step by step). Very effective for large-scale computing tasks.

In short, the design goal of Gensyn.ai's incentive/verification layer is: simple and efficient. However, it is currently limited to the theoretical level, and the specific implementation may face the following challenges:

• In the economic model, how to set appropriate parameters so that it can effectively prevent fraud without setting too high a threshold for participants.

• In terms of technical implementation, how to formulate an effective periodic encryption reasoning proof is also a complex issue that requires advanced cryptography knowledge.

• first level title

4. Thinking about the future

The question of who needs a decentralized computing power network has not been verified. The application of idle computing power to large-scale model training that requires huge computing power resources is obviously the most make sense and the most imaginative space. But in fact, bottlenecks such as communication and privacy have to make us rethink:

Is there really hope for decentralized training of large models?

If you jump out of this consensus, "the most reasonable landing scenario", whether to apply decentralized computing power to the training of small AI models is also a big scenario. From a technical point of view, the current limiting factors have been resolved due to the size and architecture of the model. At the same time, from the market point of view, we have always felt that the training of large models will be huge from now to the future, but the market for small AI models is Not attractive yet?

first level title

Reference

https://www.together.xyz/blog/neurips-2022-overcoming-communication-bottlenecks-for-decentralized-training-12 

https://www.together.xyz/blog/redpajama

https://docs.gensyn.ai/litepaper/

https://www.nvidia.com/en-in/deep-learning-ai/solutions/large-language-models/

https://indiaai.gov.in/article/training-data-used-to-train-llm-models

Disclaimer: All Foresight Ventures articles are not intended as investment advice. Investment is risky, please assess your personal risk tolerance and make investment decisions prudently.

Foresight Ventures bets on the innovation of cryptocurrency in the next few decades. It manages multiple funds: VC fund, secondary active management fund, multi-strategy FOF, special purpose S fund "Foresight Secondary Fund l", with a total asset management scale of more than 4 One hundred million U.S. dollars. Foresight Ventures adheres to the concept of "Unique, Independent, Aggressive, Long-term" and provides extensive support for projects through strong ecological forces. Its team comes from senior personnel from top financial and technology companies including Sequoia China, CICC, Google, Bitmain, etc.

Website: https://www.foresightventures.com

Disclaimer: All Foresight Ventures articles are not intended as investment advice. Investment is risky, please assess your personal risk tolerance and make investment decisions prudently.