Panoramic overview of Web3 middleware and infrastructure
Original Author: Cam
Original compilation: saku, SeeDAO
Abstract: This article categorizes and details the web 3 middleware in zee prime's portfolio including: Storage/Data, Data Model and Validation, Indexers, Access Control, and Integration Platforms. It describes their relationship to the underlying blockchain, and how they combine with each other to form web 3 applications. In addition, this article explains why the next generation of developers should choose web3 middleware.
It has been a year since we published a middleware paper with Pocket Network as the main content. During this time, the middleware field has evolved and grown exponentially. In this article, we revisit middleware and infrastructure from a top-down perspective, trying to connect the changes taking place in this rapidly evolving field.
It is worth noting that this paper focuses on Zee Prime's portfolio. We explored our portfolio inwards as we delved deeper, taking individual studies forward and combining them into a middleware "meta thesis".
We apologize for using buzzwords like composability, Web 3, etc. But it's hard to paint its full picture without relying on a few memes. Memes play an important role.
Middleware Papers Volume 2
As higher-level applications on the blockchain continue to outperform early DeFi primitives, the need for robust infrastructure and middleware expands. We at Zee Prime have been loudly bullish on this "category" for years. Our first middleware article - "Infrastructure LEGO Middleware Paper" highlights the importance and economics associated with robust data relay in the development of sustainable decentralized platforms.
This time, we try to expand our thinking to the emerging middleware ecosystem, especially in the implementation of decentralized application middleware. While something like Dfinity exists for sure as a one-stop solution, end-to-end solution for all your troubles, until the day comes when a product like Dfinity is adopted on a mass scale, we'll have to piece things together Together, and gradually build independent solutions.
Meme said that DFINITY, whose goal is to become a "decentralized world computing cloud", is dead and cannot be rescued, so it must be buried.
Now, narrative is a tough sell.
Through the boom in application complexity (yay, real applications) we're seeing an explosion in the need for supporting infrastructure. From the early days of DeFi, massive amounts of network data, indexers, access control, and other middleware tools emerged — all of which are key glue pieces for next-generation applications.
The big picture about middleware can be difficult to paint. Middleware is a nebulous concept, like Web 3. Its boundaries are often blurred. Fundamentally, when we speak of middleware, we mean any item that supports other higher-level applications. The last time we discussed this topic, we pointed out that it will serve as a middle layer, providing connectivity between the upper and lower layers. However, the actual network interaction is very complex, and it is difficult to map the obvious hierarchical relationship between the upper and lower levels.
Is this the map for Web 3 middleware?
In this updated middleware paper, we describe other classes of middleware that we haven't covered before, while also clarifying their use cases and reasons for using them for builders. This is by no means an exhaustive list.
storage / data
A key element of the decentralized application stack, and an essential element of underlying computing: storage. The Cambrian explosion of activity and complexity of Web 3 also demands a storage solution that goes beyond simply recording account state at a base layer. More and more decentralized applications are seeking technology stacks that have no centralized points of failure or scrutiny associated with Web 2 solutions.
Every application will require the services provided by Web 3 middleware in some form. One of the challenges, however, is that Dev-Ops is complex, and not every builder has the expertise to implement Web 3 middleware in their projects. Therefore, we need abstraction layers so that it will be easier to assemble this infrastructure into new projects, just like assembling Lego components.
This reflects the challenges facing wider adoption of cryptocurrencies. Wallets, mnemonics, and the gas fees required for transactions are inherently user-unfriendly, and unless there is a higher level of abstraction, widespread adoption will face resistance. We have to assume that the average user doesn't have to deal with the complexities we see today.
Storage networks such as Arweave and Filecoin have been around for a while, providing distributed matching systems for excess storage and storage demand - this is probably well known and needs no introduction on our part.
Different solutions offer varying degrees of persistence and censorship resistance. They are the foundation of any decentralized technology stack. Storage can be broken down into two classes of protocols, the base storage layer, and aggregators that address scalability and facilitate wider adoption.
Banyan is a piece of the puzzle that we see as an important aggregation layer for storage networks. Banyan focuses on intermediary services for storage, improving the economic incentives of storage protocols currently used for bridging, Banyan ensures that applications can use Web 3 storage solutions in a network-agnostic manner with guaranteed provenance.
Banyan can also act as a bridge for Web 2 applications to integrate new Web 3 services such as decentralized storage solutions. Implementing these storage solutions is currently very complex, and Banyan's abstraction layer and marketplace lower the barrier to entry for these LEGO components.
Likewise, Spheron is intended as an abstraction layer for a wider range of middleware solutions. The protocol aims to be a one-stop solution for deploying and automating Web 3 projects. It has an "app store"-like interface where non-Web 3 native users can easily choose from decentralized infrastructure offerings — much like Web 2's digital ocean.
Data Model and Validation
A blockchain is a state machine that continuously generates data in the form of state transitions while performing computations. The number of accounts, states, and smart contracts grows rapidly over time. This can cause issues with everything from indexing to initial node synchronization and backup durability. These will affect the scalability and security of the underlying state machine.
KYVE takes a more granular approach to specific data validity issues, leveraging the Arweave network to provide critical support for applications and protocols. KYVE is a decentralized data lake (raw unstructured data all in one pool) solution for storing, validating and retrieving data streams.
KYVE's initial product-market fit largely helped with node synchronization issues. By providing easily retrievable, verified profile status data, the synchronization time of the initial node can be greatly reduced, ensuring that new validating nodes can be added to the network, and network security is maintained. In another case, if the initial node synchronization time is too long (and still growing), if the number of validators decreases and new validator nodes join difficulties, the network security will be threatened.
While we've discussed where and how data might be stored, we also need to consider the model and representation of that data. For applications built on these state machines, data generated during their activities will require flexible storage and computation, not just account balances.
Ceramic, at its most basic level, is a decentralized network of data models. Why this is important infrastructure for decentralized applications, in other words, "Layer 1" (L1) stores the state of account balances (accounting data). KYVE seeks to provide data validation for state changes at L1, while Ceramic provides application data state and model storage on top of the base layer. Users of the network can create IPFS data collections (streams), allowing static data (such as data on Filecoin or arweave) to become high-order variable content.
In addition, Ceramic has also established a trading market for open source data models to realize the composability of these data models. You may often see that Ceramic points out that doing so brings the same level of composability to data as ERC 20 brings to DeFi. It proposes a standard for data and allows them to be used in various applications. reuse. And just like that, the Lego of money meets the Lego of data. It's almost a throwback to open APIs like those built on social networks like Facebook.
Kwil takes a SQL-compliant approach to the web 3 data model. The biggest advantage of such a model is that there are a large number of developers who are familiar with SQL. Kwil uses a network of nodes to maintain a relational database. These databases (called database moats) are maintained by a subnet of nodes and kept updated by scanning for new writes and querying other nodes within the same moat. Most interestingly, these nodes can run an advanced request mesh gateway for efficient logic execution of database interactions.
indexer
With the proliferation of data generated by applications and networks, the need for an interpretation layer has increased. Just like in the early days of the web, people had to manually memorize addresses and maintain IP address books. DNS and search engines provide a human-readable indexing layer.
With the development of the Internet, the centralization of indexed data has grown with the development of economic scale and made the query of data more user-friendly. Similarly, in L1 blockchains and storage networks, indexes are also very important. Due to the nature of distributed systems, a piece of data may be divided and stored in multiple places, making it difficult to retrieve. An index layer helps speed up the query process and creates a normalized program.
Subsquid, a company invested by Zee Prime, is a case study on indexers. Subsquid takes a multi-tiered approach to indexing on-chain data in a decentralized manner. Their ultimate goal is to support the next generation of APIs for Web 3. The protocol supports both the Substra and EVM ecosystems, specifies the types, schemas and definitions of on-chain data, and subsequently enhances newly indexed data by switching it to their API-based call solution (rather than using RPC) retrieval capabilities.
The purpose of Subsquid is to obtain stronger retrieval capabilities for new index data at the same speed
The layer consists of two types of nodes: Squid classifies data and supports subsequent API queries, while Archive continuously ingests raw data from the underlying state machine and saves it in the database.
The above figure describes two types of Subsquid nodes. Archive ingests raw data from the underlying state machine (blockchain) and saves it in the database, while Squid classifies the data and supports subsequent API (such as GraphQL Gateway) queries.
Similarly, SolanaFM is an indexer and a block explorer that serves the Solana ecosystem by processing raw blockchain data into a queryable format, much like other ecosystem indexers . The immediate use case provided by SolanaFM is to use their API to power DeFi applications on Solana. If you've ever worked with Graph and Subquery, these solutions might sound familiar to you. Both solutions target a wide variety of end markets.
Glitter completely solves another problem: decentralized storage. It can be thought of as an indexing service for developers. As Web 2 applications seek opportunities to move into Web 3. They will bring huge data volumes to the new world. While increased data helps Web 3, it also confronts developers and the community with the daunting task of storing and indexing this data.
Glitter creates a win-win solution for developers and the community by providing hassle-free service in exchange for crowdsourced data. This model has proven effective in the collaboration of several social applications storing data on Filecoin.
Access control
Access control
One of the most important and historically underrepresented missing pieces of Web 3 application infrastructure is access control. Who can see things on the Internet? This is an important philosophical question, and access control becomes increasingly important when faced with security issues related to national/corporate/individual sovereignty. The semantic nature of public blockchain/Web 3 technologies allows us to better differentiate which users should be able to access what and how, and while these systems are inherently open, access control frameworks will allow Encrypt/decrypt.
The Lit protocol aims to solve this problem through threshold cryptography. At its core, the network can provide access to the resources and content of the entire network based on some public credentials (such as NFT in the wallet). The protocol runs a network of nodes that verify proofs and approve handshakes. This network can verify proofs presented and whether such proofs comply with previously set access control conditions, all within a computable security compartment. Once verified, the desired content can be exploited. Some people refer to the Lit protocol as a read-to-write solution to what Ceramic provides.
Guild.xyz also tries to tackle the access control problem from a different angle. Initially focused on creating token-permissioned discord environments, the project has now expanded to focus on multi-chain access portals based on similar principles.
integrated platform
To further integrate and piece together the blocks we imagined in the world of 3D bridges, Polywrap (formerly known as Web3 api ) is pushing the integration of Web 3 protocols to a higher level of efficiency. While Web 3 protocols are open and technically composable, truly achieving this composability in practice is much harder than it was in Web 2. This is because each protocol requires specific business logic to run in the application, and these business logics often compose an SDK in a specific language.
Integrating all these different SDKs is very inefficient due to lack of standardization. Furthermore, they all support specific languages, which means that protocol developers often release functionally duplicate SDKs for different languages, creating a maintainability nightmare.
Polywrap's solution alleviates this burden by leveraging standardized patterns and WebAssembly. Polywrap's integration makes it easy to call against an easy-to-read schema (think REST API), rather than pre-bundling SDKs for various protocols in your application. The wrapper will be downloaded at runtime and executed in the application. In short, this means that any application that integrates Polywrap can gain access to any Web 3 protocol.
The user experience of Web 3 applications is still not silky enough. As we highlighted earlier, entering gas creates friction for the user. By integrating Biconomy's API, an application can improve this user experience. Biconomy's platform provides a series of tools to achieve gas-free transactions, enjoy faster confirmation times, pay in ERC 20, and instant cross-chain transactions.
Gasless transactions are possible through the use of meta transactions (via ERC 2771) and some clever forwarding. The cross-chain function is realized by the liquidity pool on the supported layer/chain, and the off-chain server (executor node) is used to monitor the pool of funds flowing into the transaction, and then "release" the other party of the transaction.
This figure describes the operation mode of Glitter. The decentralized APP of Glitter ecology is responsible for uploading and receiving data stored on the network, and distributes tasks to Glitter network miners, and receives the results of their data inspection and indexing. The outside world can access the final index results through the centralized APP.
These kinds of tools are critical to making the user experience of the next billion encrypted apps smooth and silky. Our goal should be a continuous effort to achieve a seamless flow of interaction across web 3 systems.
While it doesn't fit neatly into a category, Sepana is building a search engine for web 3. Whether it is DeFi, social, DAO activities, or NFT, Sepana's solution will provide a full-text search engine that enables users to browse the entire web 3 . By utilizing the aforementioned indexers, as well as its own indexing of web 3 applications and data, the protocol will act as a gateway to the wider ecosystem. Furthermore, Sepana's transparent and open source algorithms can be used to drive other applications such as social media feeds based on social graphs stored in database solutions such as Ceramic and Kwil.
Long-term vision has very interesting branches. Imagine a world where you could tailor your social media experience (i.e. your feeds) to achieve a specific mood or outcome (such as less polarization or broader happiness as Sam Harris often discusses). All of this can be achieved through open source algorithms, so that the platform can be adjusted according to your needs, or the right to adjust can be handed over to users.
How does it all come together?
Most modern technology companies and applications can be distilled into some form of the following business model: data production/digestion, models built on said data, and control and distribution of data/models. The smooth user experience of modern web applications and the exploitation of user dopamine are built on these basic processes.
As a result of this workflow, we expect middleware solutions to propagate from the middle tier of these technical requirements and support them in the context of Web 3. We can see this by looking at the items/types of middleware described earlier. Every piece of Web 3 middleware fits into the generic workflow we describe.
The above figure summarizes the applications at all levels of the Web 3 middleware application stack layer by layer from top to bottom. Access control includes Lit Protocol and Guild.xyz, integration platforms include Polywrap, Gasless, and Sepana, data models include Ceramic, Kwil, and data is valid The nature includes KYVE, the indexer includes SolanaFM, SubQuery, Glitter, Subsquid, etc., and the storage network and solutions include Filecoin, Banyan, Arweave, and Spheron.
While the above categories can be used as an overview to be applied more broadly, in reality, many middleware functions may span multiple categories. In 2022, because of these overlaps, it is difficult for us to define these categories precisely. To make this question more realistic, let's take a common example, say, a social media network, and extend such a mental model to a broader Web 3 middleware application stack.
Our hypothetical social media network will be aptly named twatter. In practice, we can see the product of the platform — the social media experience — flowing through the components in the middleware stack in the diagram above. Note that we do not consider Web 3's social network to be "decentralized Twitter". We imagine Web 3 social networking as more of an emerging phenomenon, perhaps even citing Web 2 apps for proof (if they decide to open up the API) through something like Sismo.
Zee Prime's hypothetical social media network is called twatter, which aims to implement social networking applications through a middleware stack.
Starting from the most raw form, all the data of the platform (username, profile picture, historical activity, social graph, etc.) can be stored and indexed in IPFS format on one of the storage networks. The data models that give this structure and meaning are stored on Ceramic or Kwil, a database solution for Twitter accounts that would have models for each part of the platform mentioned earlier.
For example, if the platform requires you to hold a free NFT to access the platform (spam reduction mechanism or other purposes), then users will need to connect their wallets to the platform, and the access control protocol will verify and shake hands. The integration platform can be subsumed by the application layer so that it can natively enable other web 3 services, while Sepana's algorithms can be used to design social graph based pushes.
The above figure describes the workflow of the social network composed of the middleware stack. The data of the basic layer is sent to the data model through the state machine indexer. All the data of the platform are stored on the storage network, and the storage index is established. The data model of these indexes Called by the APP, and the access control layer is responsible for checking whether the user has access before presenting the platform to the end user. An integration platform may be included in the application layer.
Most interestingly, while writing this article, we stumbled upon Orbis Social, which has effectively built the social network described above, using an application stack almost identical to our description. The next generation of applications is in development, and we expect to see many more different use cases in the coming months.
An important point that must be explained here is that the middleware on the right on the diagram is more independent of the chain (Translator's Note: The dependence on the chain is less). Parts of this structure are interchangeable with their competitors, however, many of these business models lead to monopoly based on network effects. In contrast to the monopoly of Web 2, these platforms ultimately redistribute this pseudo-normalized compound value to the users of the platform.
Why should next-generation application builders choose Web3 middleware?
While the tools of Web 3 continue to emerge, we must continue to ask ourselves, why is this happening? Do they really bring more benefits than Web 2 solutions?
Web 3 middleware should be built on the same foundational principles as its early cryptographic predecessors. Teams should choose Web 3 middleware solely because of its benefits, because Web 3 middleware enables their applications to achieve their goals. Whether it's from a security, durability, or censorship-resistant perspective, the benefits of Web 3 middleware should stand on their own. Many new functions may be unlocked because of some inherent features of Web 3 middleware that we can't even imagine.
These infrastructure LEGOs enable deeper integrations often associated with the Semantic Web—Tim Berners-Lee's vision of an open and composable Internet—and offer cheaper hosting and Calculated solution. As Denis-Nazarov pointed out, if a complex computing system is to become a thriving ecosystem like a garden, it needs to be modularized and specialized, and in a Web 1/Web 2 world, users give up on Management of state to obtain the ability to implement connections. The giants of the Web 2 keep valuable state information private because having that information captures value as a result.
The public state machine allows this model to be subverted, and the state is maintained in an open manner. Moreover, the introduced token economics model can strengthen the union of both parties, allowing both parties involved to obtain better results. This is the nature of reflexive assets.
Zee Prime invests in middleware solutions before apps notice!
Zee Prime Viewpoint
In many ways, middleware is the B2B part of cryptocurrency. As such, good middleware solutions tend to be both highly technical and less intuitive to the end user (but never mind, the end user is not the target audience). Rather than keeping an eye on new DeFi protocols, NFT projects, or GameFi studios, we believe that helping to make sausage on the factory floor (Translator's Note: Performing invisible but important work behind the scenes to end users) is crucial to the continuous development of new applications. important.
Is the future all middleware? of course.
Taken together, this infrastructure (and future infrastructure) will:
Increase resistance to censorship
Improve efficiency
Improve efficiency
Implement new business models
An additional potential impact of this interchangeable infrastructure modules and abstraction layers is that applications will:
farther and farther away from the base layer, and
As a result, will become increasingly chain-agnostic
This is not a refutation of the fat protocol theory (the assumption of value accumulation at the base layer), but more of a marker for this fork of continued progress. In principle, this could be seen as reducing switching costs. The earliest on-chain applications (mainly DeFi) are characterized by extremely high binding to the base layer (that is, financial products built on financial accounts).
More complex non-financial applications will have looser affiliation with such chains, reducing switching costs (free NFT access controls are easily ported to new blockchains and wallets). Applications are already attracting users across chains.
We firmly believe that adding the transfer of value to the transfer of information is a meaningful step change, but realizing this potential, and the various applications and user experiences that enable it, requires a large number of infrastructure LEGO components.
The value capture of this "part" is one of the most debated topics when discussing middleware investments. In a way, really critical middleware looks a lot like public goods, although one could argue that this applies to some successful future applications as well (Twitter wants to be a public good).
Therefore, one might expect profit margins/expenses/revenues to converge towards the lowest possible bound. While we believe this will be the case to some extent, falling to a publicly acceptable rate is a more reasonable assumption.
While it might seem unattractive at first glance, this scale could easily be multi-billion dollar for these LEGO components in the context of an asset-light/resource-light business in the world's first truly global technological revolution. value capture. Since these middleware perform specific functions for applications, their total available market is independent of the blockchain below them, or the applications above them, at any one time. Rather than relying on specific upper and lower components, middleware provides functionality on a larger scale.
While middleware and DeFi do share the self-referential nature of token-based economic models, they ultimately differ in their ability to return value. Middleware projects typically benefit from a clear supply and demand drive for their tokens (like network nodes) used to deliver the services they provide. In contrast, most DeFi projects have less clear demand drivers for tokens, and regulatory considerations for cash flow distribution make the situation even more ambiguous.
It is for these reasons that we continue to search for new middleware solutions to enable more compelling next-generation applications for continued cryptocurrency adoption. We believe that a new generation of applications will unlock financial and online commerce/activity on a massive scale. What a16z-esque said about this is that we don't want knock-off apps, we want native apps.
If you're a founder, builder, or have an idea for the critical infrastructure that needs to be built to make this happen, you know where to find us.
ZeePrime doesn't care about MEV, ZeePrime only cares about middleware value!
Glossary
Web 3 - You can refer to the article: Beyond buzzwords
Apps - We don't use dAPPs (decentralized applications) because we think "how decentralized is your app?" is a spectrum. Applications may be driven by decentralized and centralized solutions. Does this mean they are not decentralized applications?
WASM - short for WebAssembly, can be thought of as a stack-based virtual machine that executes in a binary format but can be compiled in various languages such as Rust, C, Python, etc.
SDK - Stands for Software Development Kit. It is a set of software construction tools for a specific platform, usually in the form of libraries, frameworks, artifacts or debugging tools.
IPFS - Interplanetary File System. It is a distributed system that uses content addressing to store and access information.
Photorealistic Design - The official definition is as follows: Photorealistic design includes features that make a new thing look more like an old thing or more familiar. In practice, it means new ideas that are easy to digest. An immersive design app is the opposite of a native app.
API - Application Programming Interface. This is a software that can be exploited by other software.
RPC - Remote Procedure Call. In the context of public blockchains, this basically means that one part of a computer invokes a process (which can be considered an action) in another part of the network (i.e. another computer/application).
