NGC Ventures: September has arrived, is AI still worth starting a business with now?
Original Author: Cherry, NGC Ventures
Preface
This article was initially written during a leisure day at the end of August. After hasty publication, it received many comments and suggestions. Therefore, the author made some additions, changes, and deletions to avoid embarrassment.
The main content of this article is an evaluation of the AI industry from an investment perspective, a reflection and speculation on the technology/product roadmap of different companies, and an abstract summary of the strategies of AI industry companies. Therefore, there may be omissions in the specific technical aspects, so please forgive any shortcomings.
However, in the end, the few large companies that can publish papers are still engaged in fierce competition, and it seems that no one can judge the correctness of this article's content. It's like using GPT-4 to score GPT-3.5; it seems reasonable but is somewhat abstract upon careful consideration.
Therefore, the author suggests that this article be treated as a "judgment" formed after gathering information about an uncertain industry. Since it's a judgment, the stance must be clear and substantive. As for whether the judgment is correct or not, it will be tested by time.
The author always believes that in a new industry with a lot of noise, it is always right to think more and dare to make judgments. For multiple-choice questions, the accuracy of blind guesses is 50%, while the probability of guessing wrong three times in a row is 12.5%. Even making judgments at the level of flipping a coin still has significance. Making judgments is not scary; what's scary is having an accuracy rate lower than flipping a coin.
Before officially starting this article, it is necessary to thank the following works for providing valuable inspiration and data sources. Of course, since many inferences in this article are based on these works, if there are errors or misunderstandings on the author's part, the inferences in this article will also become less robust. This article does not constitute investment advice, nor is it easy to provide investment advice.
Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance (https://arxiv.org/abs/2305.17306)
LIMA: Less Is More for Alignment (https://arxiv.org/abs/2305.11206)
June 2023, A Stage Review of Instruction Tuning（https://yaofu.notion.site/June-2023-A-Stage-Review-of-Instruction-Tuning-f59dbfc36e2d4e12a33443bd6b2012c2）
GPT-4 Architecture, Infrastructure, Training Dataset, Costs, Vision, MoE（https://www.semianalysis.com/p/gpt-4-architecture-infrastructure）
Alright, let's officially start this article.
Large Models: Launching Cyber Rockets
The first step in discussing AI in 2023 is to discuss whether it is still possible to start a business with large models.
The pre-training of large models has now transformed into a problem of launching rockets. As long as the fuel is sufficient and the direction is correct, anyone can do it. It can be said that training large models is equivalent to launching cyber rockets.
An unconventional fact is that investors underestimate the difficulty of training large models, while overestimating the difficulty of launching real rockets. With the same cost of 60 million dollars, investors may think that there is a second chance if the rocket fails to launch into space, but training large models is considered a waste of funds if it fails.
GPT-4 still consumes 60 million dollars in GPU utilization efficiency at OpenAI (apparently equivalent to about 30%). This is a problem of {performance = efficiency × cost}, and performance is like a wall. If other startups cannot achieve a performance effect greater than 30% × 60 million = 18 million dollars, then users might as well use GPT-4 directly.
Currently, the funding rounds of many companies claiming to train large models are in the range of 1 million to 5 million dollars. In other words, even for companies with the highest funding amount, their ammunition is only sufficient for one launch. And even if the GPU utilization rate of this launch reaches 100%, it is still difficult to surpass GPT-4.
From this perspective, launching rockets is easier, because most of the current rockets are carrying rockets that carry satellites into space, and the payload for a single launch is limited. Therefore, small rocket companies can take over the satellites that others have not had time to launch.
The situation is different for large models. The marginal cost of horizontal expansion for large models is only the cost of computing power, and the cost of computing power can be elastic, which means that for large model companies, every profit from each order is a free profit, with almost no additional cost and a very high capacity to undertake. It is difficult for newly established and low-quality large model companies to receive overflow demand.
Unless the training cost drops significantly, many companies will find it difficult to develop marketable large models even if they know the full architecture of GPT-4.
Customization: Facing the "Winner Takes All" Problem
In the hardware industry, a common phenomenon is to achieve early profits through customization requirements, and then achieve technological breakthroughs (or catch up) through early profits. However, customization in the large model industry is difficult for newcomers to rely on.
The explanation for this judgment is very simple: the vast majority of fine-tuned models cannot catch up with GPT-4. Even if they catch up, the cost of using GPT-4 directly for generalization is lower, requires fewer personnel, requires less luck, and requires less data. As long as there is still a gap in performance between GPT-4 and the other models, customization cannot be the way out for large model companies.
A very typical example is Jasper, which serves enterprise customers using fine-tuned GPT-3. However, after OpenAI released ChatGPT (GPT-3.5), its customers quickly lost interest. This is because the outputs of Jasper can be obtained by inputting simple prompts into GPT-3.5, without the need to use a "backward version" with poor generalization capabilities and limited for internal enterprise use.
Compared to new companies, Jasper at least had a window of opportunity from GPT-3 to GPT-3.5 for development. However, current new companies need to face the squeeze of low-cost and high-speed GPT-3.5 and high-performance GPT-4 at the same time.
Therefore, the probability of survival is very low if one hopes to accumulate profits through customization to achieve technological breakthroughs.
Fine-tuning: Necessary, but Don't Overestimate It
The current AI industry has an unrealistic expectation of fine-tuning, which is overestimated in terms of specific technical implementation and macro technical rhythm.
What the industry currently discusses about fine-tuning mostly refers to "aligning" the pre-trained model to generate answers that align with human intent, rather than adding intelligence to large models.
According to the research results of multiple papers, the knowledge of large models should primarily come from pre-training, while fine-tuning is more for alignment.
Simply put, pre-training determines brain capacity, while fine-tuning determines native language. Fine-tuning pre-trained models is a process of "eliminating illiteracy".
However, in the industry, fine-tuning is often seen as a way to "increase intelligence" for models, that is, improving model performance and increasing model knowledge through fine-tuning, believing that this can achieve the "holy grail" of artificial intelligence. This line of thinking is somewhat biased.
Firstly, the performance of the model itself does not improve, it just aligns better with human intentions. If the complexity of the task exceeds the performance of the model, fine-tuning will not achieve the expected results. It's like asking the human brain to perform quantum computations, which is not an issue of education.
Secondly, "knowledge supplementation" in the "alignment of intentions" section has a more similar effect to "parroting". That is, the model only imitates experts' speech without understanding the meaning behind it. Although "parroting" has produced good solutions in many industries (after all, most industries are not complex...), it is obviously not the result we should strive for in the long run.
Finally, the training of "supplementing additional datasets, improving model performance, and increasing model knowledge" should be seen as the model's ability to "incremental learning/continuous learning", that is, the full parameters of the model can be optimized through incremental datasets. This is not the same concept as the so-called "instructional fine-tuning".
In summary, fine-tuning is very important, but it is wrong to have a "superstitious" attitude towards fine-tuning at the moment, especially the view that fine-tuning at the moment should be hailed as the holy grail. It has a sense of "only two dark clouds floating over the building of physics today".
On the other hand, if the demand for "increasing intelligence" can truly be solved through instructional fine-tuning, a simple vector search, directly stuffing knowledge into the context, and writing a few simple prompt templates can likely achieve the same or even better results.
Everyone likes fine-tuning, which may also be a revival of alchemy in modern times...
Outlook for Large Models: Four Arithmetic Operations
(Note that this part is based entirely on the data leaked by Dylan Patel, and its reliability cannot be verified at present.)
GPT-4 is trained on A Series cards from N, with a training efficiency of 30%, a training time of about 2 months, a cost of about 60 million, and a total parameter count of {17 trillion = 110 billion × 16 expert models}, with the parameters for handling individual questions at around 280 billion.
In other words, there are several key parameters that can change the pattern of training for large models.
Training Efficiency: Increasing from 30% to 60% can directly reduce the time by half.
Improved Computing Density: Switching from the A series to the H series and then to AI dedicated cards can effectively solve many efficiency issues in the architecture.
Reduced Computing Cost: Discounts on graphics cards by Huang (founder of Nvidia) significantly reduce costs.
Improved Parameter Efficiency: There is room for improving the parameter efficiency of models. Referring to the parameter efficiency of previous new models compared to old models, it is usually possible to achieve similar results with only 30% of the parameters of GPT-4.
In conclusion, there is a potential optimization space of 10 to 20 times the cost to train a model with GPT-4 level performance from scratch, compressing it to 3 million to 6 million US dollars. This cost is more acceptable for both startups and large companies in terms of budget control.
However, this change may take about 2 years to complete.
Currently, the technology of mainstream large models is still based on transformers, and the basic architecture remains unchanged. The idea of adding more parameters is still promising. The training of GPT-4 is constrained by computing power and the training time is not long enough.
If the parameters grow linearly with training time, the parameter limit for models with a similar architecture as GPT-4 may be around 10 trillion, that is, training time doubled (× 2), parallel graphics cards doubled (× 2), training efficiency increased by half (× 1.5), and parameter efficiency increased by half (× 1.5), resulting in a ten-fold improvement. According to the risk preferences of Silicon Valley, this parameter size is likely to be achieved within a year, regardless of whether the performance has improved.
However, after reaching 10 trillion parameters, whether LLM can still achieve remarkable performance improvement through increasing parameters is completely unknown.
If the model's performance improvement with parameter growth is marginally diminishing, then 10 trillion is likely a barrier. However, there is also a conjecture that the performance improvement with parameter growth is marginally increasing, similar to "a person who is smart enough learns everything quickly." The former is acceptable, but if the latter comes true, the model's performance may experience exponential improvement, and what will happen then will be completely unpredictable.
Predicting alchemy is difficult, but predicting the pace of corporate strategy is easy. For the vast majority of companies, whether they are giants like Google/MS/APPL or smaller ones like OpenAI, a model with 10 trillion parameters is a milestone-level end point, a position where they can pause and explore new technologies.
Enterprise/capital's preference for risk can be converted into a "tolerance time". If the entire tolerance time is spent on intense burning costs, it is difficult to exceed 6 months. The growth rate of human craftsmanship is not fast enough and usually takes around 5 years or even longer for a cycle. Therefore, within 5 years, the maximum parameter value of the model can be estimated to be around 20 trillion to 50 trillion. Unless there is another major breakthrough in craftsmanship/architecture, the probability of exceeding this magnitude is very low.
Multimodal: The Elephant in the Room
Multimodality is the elephant in the room that may significantly impact the race's landscape.
The simple definition of multimodality is the support for input and output of multiple modalities of information. This definition is quite loose. For example, some products on the market that claim to support multimodal input are actually ChatBots wrapped with OCR. There are also models that fully meet the definition of multimodality, but their performance is not commendable. Even GPT-4's image multimodal input capability has not been widely opened, indicating that this feature is not very stable.
However, the release of multimodality is not a distant event. GPT-5 is highly likely to be natively capable of supporting multimodality, which would require redesigning the structure and retraining. And according to the previous reasoning, there is still a growth space of 10 to 50 times for large models' parameters, which should be sufficient to incorporate multimodal capabilities. Therefore, it can be anticipated that high-performance and highly available multimodal models will appear within 2 years, with an optimistic estimate of around 1 year.
Multimodality is the elephant in the room that everyone knows will eventually exist, but many products/research/strategies overlook its existence, resulting in misjudgment in critical aspects.
For example, a single image model will theoretically be seriously oppressed by multimodal models, but currently, most research/investments ignore this problem, leading to overvaluing companies that focus on image models. These companies are likely to lose their technological barriers in the future and transform into service providers, and their valuation system should refer to service providers rather than technology companies.
If you want to tell a story of "investing based on people and how the same team can transform their business," just ignore what I said. Legends always exist, but research should not blindly believe in legends.
Who Can Train GPT-4: Anyone Can, But There's No Need
You don't need to spend so much time on alchemy, and big companies are all buying graphics cards. One very obvious thing is that a year later, large companies will have the capability to train models at the level of GPT-4. However, whether or not to train is another question.
In the field of gaming, there is a classic proposition called "play the original game if other similar games are not as good as the original game". This means that when players have the choice between playing the original game or a competing game, if the competing game is not as good as the original, they will play the original game.
This "winner takes all" mindset also applies to the field of large models. If a company follows OpenAI and, after six months of development, releases its own large model with 90% performance comparable to GPT-4 and hopes to bring it to the market, the company will face the following challenges:
OpenAI has the advantage of scale in cloud resources and lower costs
OpenAI's API has been widely used in product code, making it difficult to switch
The company's product performance still does not surpass GPT-4
OpenAI's next-generation product (possibly GPT-5) is about to be released
It can be seen that the company is under considerable pressure. Instead of training GPT-4, it might be better to directly invest in the next-generation model (like GPT-5). Then the problem will shift from "competition among similar products" to "technological innovation". This is a burden that small companies find difficult to bear.
Therefore, discussing "who can train GPT-4" is a strategically dead-end question. Instead of pondering this question, it is better to find a more certain and opportunistic direction.
Advice for AI startups: prioritize performance and avoid stagnation
I have written many articles criticizing langchain in the past, mainly because langchain did not leave room for developers to improve performance. It claims to be a "universal framework" and sacrifices many opportunities to boost the performance of large models, such as formatting control in multi-turn conversations and fine-tuning implementation. Similar cases include guidance/Auto-GPT/BabyAGI, all aiming to create a "framework for lifelong use".
An objective fact is that in May, OpenAI released Function Calling, which provided better solutions to many challenging aspects in the code. However, implementing these better solutions required significant refactoring of the product code. In August, OpenAI also granted permission to fine-tune GPT-3.5, opening up new potential solutions for precise output control.
Therefore, startups must face a key choice: whether to prioritize ① performance improvement and constant product refactoring, or ② reduce the use of new features and always use old features for development?
For startups applying new technologies, "development" not only represents the process of writing code, but also represents the "upper limit" of product functionality/strategy. The higher the manageable performance, the more theoretical functions the product can have and the higher the strategic flexibility.
The development of technology is unpredictable, and minor technological innovations can bring about highly sensitive changes in the competitive landscape. Startups should have the ability to be anti-fragile in the development of technology.
- In layman's terms: prioritize performance and avoid stagnation. In terms of development, use more new features; in terms of products, consider what functions can be achieved with new features; in terms of strategy, consider the impact of new features on strategy.
In "The Debate on Changsha", it was mentioned that after the establishment of the Qin Dynasty, all metal weapons in the world were confiscated and cast into twelve bronze men to eliminate the possibility of civilian uprisings. However, the Qin Dynasty was known for its short lifespan. It is more advantageous to value change rather than ignore it.
Advice for AI startups: Focus on application development
There is a common hidden danger for startups focusing on application development: the entry of big companies. Big companies here include not only application giants like Meta/ByteDance/Tencent, but also the upstream of the AI industry, such as OpenAI.
The reasons for the entry of big companies are usually twofold: seizing product opportunities and cutting down the upstream.
"Seizing product opportunities" is self-explanatory. Big companies believe that this direction is worth pursuing, so they pursue it.
"Cutting down the upstream" is often a helpless move. It may be because they have developed a large model similar to OpenAI, but due to the problem of winner-takes-all with large models, they lack users, resulting in burning costs, no revenue, and no data, leading to a gradual decline in performance. At this point, entering the downstream, developing specific applications, and using their own technology becomes the only option.
Based on historical experience, due to organizational structure issues, the closer a company is to the downstream, the more likely its technology will lag behind, and the more it falls behind technologically, the more it has to focus on the downstream. These so-called technology companies will eventually compete with application-level companies for the same ecological niche.
However, in the battlefield at the application level, AI technology has appeared relatively recently and there are no effective and reusable scale advantages. The starting point for big companies and startups is similar. Compared to big companies, startups have higher efficiency and deeper insights, making it easier for them to gain an advantage.
One thing worth noting is that almost all of MS Azure's promotional materials now revolve around OpenAI, which indirectly proves that startups have a natural advantage in the AI field, even for a giant like Microsoft.
Of course, some cloud providers may not be willing to be led by startups and want to dominate the entire market themselves. However, their high costs and slow speed are not imminent threats.
The fact is, there are indeed some AI applications that have a very short lifespan, but there are also many long-lasting tracks that have not been discovered yet, and AI applications are not winners in all cases. Extending from applications to platforms or technologies is also a more feasible path.
Therefore, we should rationally evaluate the ability of large companies to invade the application layer. Our recommendation is that AI startup companies can confidently focus on application development.
Advice for AI Startup Companies: Pay Attention to Product Lifespan
As mentioned earlier, AI startup companies can focus on application development, but they need to consider the performance issues of AI models and avoid stagnation. This situation directly manifests as AI products possibly losing their user base within a few months and gradually declining, and this situation may occur frequently.
AI applications require the use of services with large models, and the performance of these large models continues to improve. This improvement is not limited to "speed" or other single dimensions, but encompasses changes in output quality, length, and control. Each significant upgrade in technology will cause existing application layer products to become technologically outdated and create new opportunities and competitors.
We refer to the time period in which AI applications maintain advantages and necessity in terms of strategy, product, and technology as "lifespan".
Here are some examples of short lifespans:
When ChatGPT/Claude supports file uploads, ChatPDF becomes unnecessary
When Office 365 supports Copilot, products that use AI to create PowerPoint presentations lose their advantage
When GPT-3.5 is released, Jasper becomes unnecessary
Considering the rapid development in the AI industry, having a limited lifespan is the norm. Therefore, accepting the fact that lifespan is limited and choosing directions with longer lifespans as much as possible is conducive to maintaining long-term advantages and product necessity.
In general, the lifespan can be roughly divided into 3/6/12-month levels.
3 months: Functions that big companies don't have time to develop (such as unfinished functions in office/ChatGPT)
6 months: Functions with certain implementation difficulties that cannot be integrated into existing solutions, but the advantages/necessity will disappear as AI performance improves (such as general AI frameworks)
12 months: Advantages/necessity that can persist in the long term and are not easily affected by big companies/technological advancements (such as Hugging Face)
* The lifecycle of platform products may not necessarily be long, after all, the prompt store is also a platform
For startup companies, as long as there is a 6-month lifeline, they can proceed, while a 12-month lifeline is rare and hard to come by.
When the product's lifeline comes to an end, there are generally two situations. The first situation is when the advantages disappear and the product needs to be restructured and upgraded technically, please refer to the previous section on "prioritizing performance". The second situation is when the necessity disappears, and the product will gradually be replaced. At this point, the product still has several months of "operational lifespan", which is enough for startup companies to choose the next direction.
Advice for AI startup companies: Web3+AI is feasible
Currently, there are already many projects focused on entrepreneurship around the theme of Web3+AI. Considering the uncertainty of technological development and the early stage of the market, there are still many variables for the topic of Web3+AI in the future.
This article aims to find a high-probability correctness in uncertainty, so the author still hopes to stimulate discussion and propose some potential opportunities and directions for startup companies and interested researchers to consider.
Decentralization
Currently, the leaders in the AI industry only provide closed-source models, and the stability, transparency, and neutrality of their ongoing services are uncontrollable. Decentralization may become an important theme in the AI industry, which means providing stable, transparent, and neutral AI services based on a decentralized architecture.
Decentralization is an "alternative solution" and also a "deterrence" that can significantly increase the unethical costs of centralized/sovereign AI companies and prevent them from using AI models for military, cult, political purposes, etc.
In extreme cases, once centralized/sovereign AI services are no longer available/trusted for some reason, decentralized/distributed AI can provide sustainable and highly available services, preventing individual countries/regions or even humanity from losing AI services and falling into a paralyzed state.
Practicalization of computing power
The transition of ETH from PoW to PoS is driven by the criticized dilemma of "mining not creating value". By combining Web3 with AI, scenarios of practicalized computing power can be provided, achieving the digestion of existing computing power and promoting the overall growth of computing power.
Virtualization of assets
AI is an asset native to computing power and storage. The combination of Web3 and AI can provide a channel for transforming AI into virtual assets, realizing the value instantiation of the AI industry and creating true native virtual assets for Web3.
Variability of Web3 applications
The combination of Web3 and AI may bring new features and growth opportunities to Web3 applications, and existing Web3 applications can be completely revamped.
Final Thoughts: Is AI Still Worth Starting a Business in September?
First of all, the answer is yes, and it is highly probable that this conclusion can be extended until Chinese New Year.
People's perceptions of the situation often have biases, and the author is no exception. Some people are overly optimistic, while others are overly pessimistic. The author has had conversations with two teams, one of which believes they can develop an AI Agent in Q1 of next year, while the other team thinks AI is only suitable for knowledge base management work. Obviously, the former is too optimistic, while the latter is overly pessimistic.
When making long-term plans, being overly optimistic or overly pessimistic can lead to pitfalls, and widely circulated opinions often have significant biases. Independent thinking is precious. Therefore, regardless of whether readers can accept the viewpoints in this article, as long as they generate independent thinking and judgment during the reading process, the author will be extremely pleased.
Finally, let me advertise. If you have great AI startup ideas or already have a developed project, feel free to communicate with friends from NGC (like me) at any time.
