With the introduction of AI, will the newly launched LLM program become Dune's killer feature?

Recently, Dune, an encrypted data query website, announced to update its LLM (Large language model, Large language model) roadmap. The Query Explanations function supported by GPT-4 will be launched in the first phase, and more functions will be gradually added later, such as natural Language query (Natural Language Querying), SQL statement escape (Query Translations), optimized search, etc.

first level title

Dune's Data Query Gap

Thanks to the openness and transparency of blockchain data, anyone can directly access blockchain data, but raw data (Raw Data) is often difficult to identify, and it is difficult for non-professional programmers to understand its meaning, but the data on it It contains huge economic value, so various data analysis tools have emerged as the times require, providing indirect access and in-depth analysis tools for various analysts, researchers and ordinary retail investors.

For example, the official simplified"nft.trades "The query process includes nearly 200,000 lines of SQL statement conversion and 100,000 lines of test code, and is participated by 55 community members. A single user cannot handle such a large-scale task.

The second is that the SQL standards supported by Dune's V1 and V2 versions are not uniform. V1 and V2 use PostgreSQL and Spark SQL respectively, and Dune SQL is planned to complete the unification in the future.

Before upgrading the LLM function this time, Dune was already preparing a unified query engine, and planned to migrate to Dune SQL after July this year to ensure the unity of product logic. The updated Dune SQL is based on the implementation of the open source query engine Trino. Dune optimizes it to suit Dune's own needs. There is no essential difference from the popular Spark SQL, but more in terms of specific functions and syntax. Improve. For example, Dune SQL provides more operators to facilitate quick calculation of date and time, and to restrict management rights. All operations involving deletion and update of the original data itself cannot be performed to ensure data security.

However, the problem of the high threshold of SQL query cannot be solved by changing the SQL paradigm, which means that most users and programmers have to adapt to the new syntax format. For example, for specific query statements, new users and programmers have to to adapt. The demand for automation tools is not only of practical significance for ordinary users, but also of great benefit to programmers who are not familiar with the new syntax.

first level title

LLM: unlocking the sense of participation of ordinary users

The typical process of Dune is to deconstruct the data on the chain, professional users execute queries through SQL, and then share the dashboard (Dashboard) with users in need. In this process, the most critical thing is to execute the query, and most users cannot use the query function due to lack of code knowledge.

After introducing the query interpretation function, some fine-tuning of the above-mentioned process has occurred. In the SQL query interface written by professional users, an explanation page will appear, which intuitively tells the specific function of the user code in natural language format, which is equivalent to adding a SQL query. The explained patch will not change the current workflow. This is also the lesson learned by the team when merging SQL statements, that is, to reduce the interference with the user's existing habits, but to optimize the experience in the existing process as much as possible.

After the addition of LLM, the ability difference between professional users and ordinary users has been erased to a certain extent. With the help of GPT-4's ability to understand code, ordinary users can intuitively understand the role of query statements without mastering SQL knowledge. Before the introduction of this function, users could only passively read the charts on the Kanban page, but after introducing the query interpretation function, ordinary users can also understand the specific meaning of the SQL code.

For example, if Alice wants to inquire about LayerZero transaction information, she can directly find the Dune Kanban that Bob has made, and find that 49.4% of users are concentrated in 1 time, so Alice has reason to infer that this is for potential economic stimulation However, it is impossible to make a final decision, so Alice decided to go through the code to check whether the conclusion is reliable.

However, Alice finds that although the result is only 5 rows of data, the query code has 150 rows. Alice's SQL level is not enough to confirm the correctness of each statement. At this time, the query interpretation function will "translate" the specific description of the code, such as As shown in the figure, the query is divided into 3 parts: the collected data covers multiple chains such as Arbitrum, Avalanche, BNB, Ethereum, Optimism, Polygon and Fantom, and the second part is to calculate the number of transactions for each user (`sender`) . The third step is to classify users by setting thresholds for the quantity: 1 Tx', '2 ~ 5 Tx', '5 ~ 10 Tx', '10 ~ 20 Tx', and '>= 20 Tx'.

At this time, Alice can analyze and judge without understanding the meaning of the code written by Bob. In essence, the function of query interpretation is equivalent to an escape and translation between the code and humans. In turn, human language can also be translated into SQL statements. With the blessing of LLM, thanks to the massive amount of Query statement data, which is not difficult to achieve.

Natural Language Querying (Natural Language Querying) is an important direction for Dune's subsequent LLM improvement. Natural language query allows users to perform chart generation tasks by conveying instructions, which is more in line with the way of thinking of ordinary people than using SQL statements, drag and drop generation, and avoids paying attention to implementation details.

Moreover, natural language query is not a substitute for professional users, such as analyst groups, but a supplement. There are nearly 700,000 charts in existing Dune, and a considerable part of the analysis tasks are overlapping and conflicting. Using natural Language query also helps the system understand the relationship between graphs, thereby further improving the efficiency of the entire analysis work.

After the addition of LLM, the existing charts, SpellBook and document data will also be reintegrated. Following the example of OpenAI's chatbot, Dune will also develop a chatbot to help users understand and utilize the existing knowledge system in an easier way without requiring Be distracted by irrelevant information.

For example, Alice can reverse the above query of LayerZero user transaction volume distribution, first issue the query command in English, and explain the workflow of each step, and then Dune will help Alice write 150 lines of code, and then generate the chart.

Conclusion: Everyone can be a data analyst

The goal of Dune is not to build a purely on-chain data analysis platform, but to create a data pipeline that allows information to flow freely, allowing users to capture, transform, manage, query, visualize, and use data to take action.

The premise of data flow is that it must be modularized, can be combined and configured arbitrarily, and finally build a data set shared by the community, rather than concentrated in complex SQL statements or payment APIs, and finally achieve equal rights that everyone can interact with data picture.

In a nutshell, Dune's LLM plan is a "translation" and an assistant. The goal is to let ordinary users understand everything that the data represents, not only the display of the final result, but also go deep into the generation process, and finally everyone can do it. On-chain data analysis.