ChatGPT's Art of Web 3.0 Security Audit: Something, But Not Quite There
In November 2022, OpenAI launched an innovative artificial intelligence (AI) project.
In addition to being useful for writing and summarizing articles, creating jokes and poems, ChatGPT can also be used for debugging and code generation.
In the whole year of 2022, the loss of assets in the Web3.0 field due to incidents such as hacker attacks and fraud exceeded 3.7 billion US dollars. Such a huge loss made people in the industry think: whether new technologies such as ChatGPT can be used to improve the security of smart contract codes. safety.
ZKasino, a decentralized gaming platform, recently underwent a pre-audit through ChatGPT.
ZKasino hopes that while CertiK conducts a comprehensive audit, CertiK will conduct an initial review of the results of ChatGPT to test ChatGPT's ability to act as an AI "smart contract auditor".
So what are the test results of ChatGPT?
Is it ready to take over the job of human code audit experts?
Or is it still not enough to completely replace human labor?
On December 23, 2022, ZKasino "hired" ChatGPT to identify potential security risks in smart contracts. The tool does raise several ostensibly legitimate risk concerns.
However, although ChatGPT has undeniably provided some valuable services to the Web3. code that works fine. Here, we hope that the in-depth data and suggestions of CertiK security experts can help ChatGPT become a more powerful Web3.0 application security tool.
Below, we will introduce the two types of false discoveries of ChatGPT in this incident in detail.
What problems did ChatGPT find in the contract?
secondary title
Critical or critical vulnerability?
ChatGPT mentions several common security issues that exist in many smart contract implementations.
However, it fails to identify certain serious risks, including:
Project Specific Logic Vulnerabilities
Inaccurate mathematical calculations and statistical models
Inconsistency between code implementation and project design
Bug #1: Project-specific logic issues
ChatGPT failed to identify a critical vulnerability that would allow an attacker to continually win and drain users' funds in the Bankroll contract. Players can join the game by calling a Verifiable Random Function (VRF), and Chainlink's VRF will use random numbers to trigger the fulfillRandomWords() function to complete the game.
ZKasino's code allows users to get their funds back if the fulfillRandomWords() call fails.
During CertiK's review of the same smart contract code, a potentially harmful _transferPayout() call was discovered, a function designed to transfer winning funds to players' accounts.
However, the attacker can call _transferPayout() to roll back when losing, causing the entire fulfillRandomWords() call to fail-this will make it enter a waiting period of 100 blocks, and then trigger CoinFlip_Refund() to return money, which means that the attacker will never lose money.
Although ChatGPT was aware of the call failure problem, it failed to discover the attack method that exploited call failure in this project.
Therefore, ChatGPT did not see the impact of this vulnerability combined with the project logic. For a description of the specific attack flow, see ZKasino's full audit report.
Vulnerability #2: Inaccurate Mathematical Calculations and Statistical Models
Ensuring that randomness and outcomes meet reasonable expectations is critical in any game project. To confirm this, a full assessment of the randomness of all game outcomes will be required during the audit.
Although ChatGPT also "agreed" the importance of this, it did not find any unfair game results. It brings up the use of VRF, and the possible unfair outcomes if the VRF contract is broken or manipulated:
"If the VRF contract is insecure or manipulated, it could lead to unfair outcomes in the game."
However, this conclusion is only a conclusion and does not really address the root cause of the unfair game results. And we also found some potential risk issues about randomness during the audit process.
unfair randomness
One of the medium-level risks found with respect to randomness was the unfair use of random numbers in VideoPoker games, where players had fewer chances of getting certain cards.
decimal truncation
Another risk problem is found in dice-like games that allow players to choose specific multipliers that maximize their expected payoff.
Vulnerability #3: Inconsistency between code implementation and project design
ChatGPT can often understand the implementation of a single function, but cannot understand the root cause of it.
For example, it may understand how a certain function is executed at a technical level, but it cannot understand the purpose of the function in the entire smart contract.
In order to ensure that ChatGPT has no problems in coding, it needs to better understand the code logic of smart contracts.
As it stands, ChatGPT provides a superficial reading of the code. In order to make its audit work and level to the next level, it must learn to deduce its initial logic from a function in reverse - this is very important.
incorrect input validation
We found an input validation issue in the Plinko contract that caused the multiplier to be set incorrectly.
According to ZKasino, the number of lines used in Plinko should be 8 to 16. However, due to a bug in the following check, the Bankroll contract owner can set a line value outside the expected range via the function setPlinkoMultipliers().
The code shows that if neither numRows nor risk meet the condition, the process will be rolled back.
However, if only one of the two conditions is not met, the check will still pass and the code will not be rolled back.
ChatGPT gives a different answer to the second question: the function checks if the value of "numRows" is between 8 and 16, and if the value of "risk" is less than 3. If any of the above conditions are not met, the function will return the error "InvalidNumberToSet".
ChatGPT seems to understand the purpose of this function. However, it does not have the application knowledge to identify real vulnerabilities without additional information.
Inconsistencies in value updates
write at the end
write at the end
Despite the appropriate training, ChatGPT missed certain critical security issues in its audit. This is due to the limitations of artificial intelligence in fully understanding the complexities and nuances of code, as well as its lack of practical experience in real-world scenarios.
As stated on its official website, ChatGPT is a research version that relies on natural language processing for conversations. It often cannot understand the intent and logical reasoning behind the code in the same way that human audit experts can.
Therefore, it is important to supplement ChatGPT's analysis with manual audits by experienced security experts to ensure overall accuracy.
The figure below highlights the strengths and weaknesses of human-based services and ChatGPT on various criteria.
Just like we use Baidu translation, the accuracy of English translation results often depends on our Chinese wording - the effectiveness of ChatGPT answers depends to a large extent on prompt (prompt is a method in javascript language, the main use is to display a prompt dialog).
In this paper, we compare the results of a pre-audit of ZKasino interacting with ChatGPT with the results of a final audit performed by CertiK experts.
As the technology advances and a clearer understanding of prompt engineering emerges, engineers will be able to better utilize ChatGPT.
In the next release of the CertiK official public account, we will discuss with you more in-depth content of the prompt project on how to ask effective questions to ChatGPT.
But now, ChatGPT can even help us participate in the capture the flag competition: [I let AI write articles, analyze code, and even compete in CTF capture the flag for me!】Welcome to click to read!
