Vitalik Buterin: Social media experiment Community Notes is very encrypted

Original author: vitalik

Compilation of the original text: Deep Tide TechFlow

The past two years have been arguably turbulent for Twitter (X). Last year, Elon Musk purchased the platform for $44 billion and then overhauled the companys staffing, content moderation, business model, and website culture. These changes may have more to do with Elon Musks soft power. rather than specific policy decisions. Amid these controversial actions, however, a new feature on Twitter has quickly become important and appears to be beloved by people across the political spectrum: Community Notes.

Community Notes is a fact-checking tool that sometimes appends background text notes to tweets, such as Elon Musk’s tweet above, as a fact-checking and counter-disinformation tool. It was originally called Birdwatch and first launched as a pilot project in January 2021. Since then, it has gradually expanded, with its most rapid expansion coinciding with the time Elon Musk took over Twitter last year. Community Notes appear regularly among the tweets that get the most attention on Twitter these days, including those dealing with controversial political topics. In my opinion, and my conclusion from talking to many people across the political spectrum, these Notes are informative and valuable as they appear.

However, what interests me the most is Community Notes, which although not a crypto project is probably the closest instance of crypto values weve seen in the mainstream world. Community Notes are not written or curated by some centrally selected experts; instead, anyone can write and vote, and which Notes appear or do not appear is determined entirely by open source algorithms. The Twitter website has a detailed and comprehensive guide describing how the algorithm works, and you can download the data containing published notes and polls, run the algorithm locally, and verify that the output matches what is visible on the Twitter website. While not perfect, its surprisingly close to the ideal of believable neutrality in rather controversial situations, and is very useful at the same time.

How does the Community Notes algorithm work?

Anyone with a Twitter account that meets certain criteria (basically: active for more than 6 months, no violations, verified mobile number) can sign up to participate in Community Notes. Currently, participants are being accepted slowly and randomly, but eventually the plan is to allow anyone who qualifies to join. Once accepted, you can first participate in rating existing Notes, and once your ratings are good enough (measured by seeing which ratings match the final results for that Note), you can also write your own Notes.

When you write a Note, the Note receives a score based on review by other Community Notes members. These reviews can be thought of as votes along the three levels of helpful, somewhat helpful, and not helpful, but reviews can also include other labels that play a role in the algorithm. Based on these reviews, Notes is assigned a score. If the score of Notes exceeds 0.40, then this Notes will be displayed; otherwise, this Notes will not be displayed.

Whats unique about the algorithm is how the scores are calculated. Unlike simple algorithms, which are designed to just calculate some kind of sum or average of user ratings and use that as the final result, the Community Notes rating algorithm explicitly tries to prioritize those who have received positive reviews from people with different perspectives Notes. That is, if people who normally disagree on ratings end up agreeing on a particular Note, that Note will be rated highly.

Let’s take a closer look at how it works. We have a set of users and a set of Notes; we can create a matrix M where cell Mij represents how the i-th user rated the j-th Notes.

For any given Note, most users have not rated that Note, so most entries in the matrix will be zero, but thats okay. The goal of the algorithm is to create a four-column model of users and Notes, assigning two statistics to each user, which we can call friendliness and polarity, and two statistics to each Notes, which we Call it usefulness and polarity. The model tries to predict the matrix as a function of these values, using the following formula:

Please note that here I introduce the terminology used in the Birdwatch paper, as well as my own terminology to provide a more intuitive understanding of the meaning of the variables without involving mathematical concepts:

• μ is a public sentiment parameter that measures how high users generally rate ratings.

• iu is the friendliness of the user, i.e. how likely it is that the user tends to give a high rating.

• in is the usefulness of the note, that is, how likely is that note to be highly rated. This is the variable we care about.

• fu or fn is the polarity of the user or Notes, ie their position on the dominant axis of political extremes. In practice, negative polarity roughly means left leaning and positive polarity means right leaning, but please note that the extreme axes are derived by analyzing user and Notes data, and the concepts of left and right are not hard-coded in.

The algorithm uses a fairly basic machine learning model (standard gradient descent) to find the best variable values ​​to predict the matrix values. The usefulness assigned to a particular Note is the final score for that Note. A Note will be displayed if its usefulness is at least + 0.4.

The core cleverness here is that polarity absorbs the characteristics of a Note that cause it to be liked by some users and disliked by others, while usefulness only measures the characteristics of a Note. These features lead to it being liked by all users. Selecting usefulness thus identifies Notes that are endorsed across tribes and excludes Notes that are cheered by one tribe but resented by another.

The above only describes the core part of the algorithm. In fact, there are many additional mechanisms added on top of it. Fortunately, they are described in public documentation. These mechanisms include the following:

• The algorithm is run multiple times, each time adding some randomly generated extreme fake votes to the vote. This means that the true output of the algorithm for each Notes is a range of values, and the final result depends on a lower confidence taken from that range and compared to a threshold of 0.32.

• If many users (especially those with a Notes polarity) rate a Note as Not Useful and they also assign the same tag (for example, argumentative or biased language, unusual source) Support Notes) as the reason for the rating, then the usefulness threshold required for Notes to be published will increase from 0.4 to 0.5 (this may seem small, but is very important in practice).

• For a Note to be accepted, its usefulness must be reduced to 0.01 points below the threshold required for the Note to be accepted.

• The algorithm performs more runs using multiple models, sometimes improving Notes with raw usefulness scores between 0.3 and 0.4.

All in all, you get some pretty complex Python code, totaling 6282 lines spread across 22 files. But its all open, and you can download Notes and the ratings data and run it yourself to see if the output matches whats actually happening on Twitter.

So what does this look like in practice?

Probably the biggest difference between this algorithm and simply taking an average score from peoples votes is the concept of what I call polar values. The algorithm documentation refers to them as fu and fn, using f for factor because the two terms multiply each other; the more general terminology is partly due to the eventual desire to make fu and fn multidimensional.

Polarity is assigned to users and Notes. The link between the user ID and the underlying Twitter account is intentionally kept secret, but Notes is public. In fact, at least for the English dataset, the polarity generated by the algorithm correlates very closely with left and right.

Here are some Notes examples with polarity around -0.8:

Note that Im not cherry-picking here; these are actually the first three rows in the scored_notes.tsv spreadsheet I generated when running the algorithm locally, and their polarity scores (called coreNoteFactor 1 in the spreadsheet) are less than -0.8.

Now, here are some Notes with a polarity of about + 0.8. It turns out that many of them are either people talking about Brazilian politics in Portuguese or Tesla fans angrily rebutting criticism of Tesla, so let me cherry-pick a little and find some Notes that dont fall into either category:

Just to remind you again,"Divide left and right"It is not hard-coded into the algorithm in any way; it is discovered computationally. This suggests that if you apply this algorithm to other cultural contexts, it can automatically detect their main political divisions and build bridges between those divisions.

Meanwhile, the Notes that get the most usefulness look like this. This time, since the Notes are actually showing up on Twitter, I can just take a screenshot:

And another:

For the second Notes, it deals more directly with highly partisan political topics, but its a clear, high-quality, and informative Notes, so it gets a high rating. Overall, the algorithm seems to work, and it seems feasible to verify the output of the algorithm by running the code.

What do I think about this algorithm?

What struck me most when analyzing this algorithm was its complexity. there is one"academic paper version", which uses gradient descent to find the best fit of five-term vector and matrix equations, and then the real version, a complex series of executions of the algorithm with many different executions and lots of arbitrary coefficients along the way.

Even the academic paper version hides the underlying complexity. The equation it optimizes is a negative fourth order (because there is a quadratic fu*fn term in the prediction formula and the cost function measures the square of the error). While optimizing a quadratic equation in any number of variables will almost always have a unique solution, which you can figure out with fairly basic linear algebra, optimizing a quartic equation in many variables usually has many solutions, hence the multiple rounds of gradient descent algorithms Different answers may be reached. Small input changes can cause the dip to flip from one local minimum to another, significantly changing the output results.

The difference between this and the algorithms I helped develop, like secondary financing, to me is like the difference between an economists algorithm and an engineers algorithm. Economists algorithms, in the best case, focus on simplicity, are relatively easy to analyze, and have clear mathematical properties, indicating that it is the best (or least bad) for the task to be solved, and ideally it can also be proved How much damage can someone do in trying to exploit it. An engineers algorithm, on the other hand, is derived through an iterative process of trial and error to see what works and what doesnt work in the engineers operating environment. Engineers algorithms are pragmatic and get the job done; economists algorithms dont completely lose control when faced with unexpected situations.

Or, as respected internet philosopher Roon (aka tszzl) puts it in a related thread:

Of course, I would say that the theoretical aesthetics aspect of cryptocurrencies is necessary to be able to accurately differentiate between those protocols that are truly trustless and those that look good and work well on the surface but actually require trust in some centralized actor, Or even worse, it could be a complete scam.

Deep learning is effective under normal circumstances, but it has unavoidable weaknesses in various adversarial machine learning attacks. If done well, technical traps and high-level abstraction ladders can combat these attacks. So I have a question: Can we turn Community Notes itself into something more like an economic algorithm?

To see what this means in practice, lets explore an algorithm I designed a few years ago for a similar purpose: Pairwise-bounded quadratic funding.

The goal of pairwise-bounded quadratic funding is to fill a loophole in regular quadratic funding, whereby even if two participants collude with each other, they can contribute very high amounts to a fake project, return the funds to them, and Receive large subsidies that drain the entire capital pool. In Pairwise-bounded quadratic funding, we allocate a limited budget M to each pair of participants. The algorithm iterates through all possible pairs of actors, and if the algorithm decides to add a subsidy to a certain project P because both actor A and actor B support it, then this subsidy is deducted from the budget allocated to the pair (A, B) . Therefore, even if k participants collude, the amount they steal from the mechanism is at most k * (k-1) * M.

This form of the algorithm does not work well in the context of Community Notes because each user casts only a small number of votes: on average, the number of votes in common between any two users is zero, so simply by looking at each pair individually User,algorithm cannot understand the user’s polarity. The goal of a machine learning model is precisely to try to populate a matrix from very sparse source data that cannot be directly analyzed in this way. But the challenge with this approach is that extra effort is required to avoid highly volatile results in the face of a small number of bad votes.

Can Community Notes really stand up to the left and the right?

We can analyze whether the Community Notes algorithm is actually able to resist extremes, that is, whether it performs better than a naive voting algorithm. This voting algorithm already resists extremes to a certain extent: a post with 200 likes and 100 dislikes will perform worse than a post with only 200 likes. But does Community Notes do better?

From an abstract algorithm perspective, its hard to say. Why wouldn’t a polarizing post with a high average rating not have strong polarity and high usefulness? The idea is that if those votes are conflicting, the polarity should absorb the features that caused the post to get a lot of votes, but does it actually do that?

To check this, I ran my simplified implementation for 100 rounds. The average result is as follows:

In this test, good Notes received a +2 rating among users of the same political affiliation, and a +0 rating among users of the opposite political affiliation, and good but more extreme Notes received a +0 rating among users of the same political affiliation. It received a +4 rating among users of the opposite faction and a -2 rating among users of the opposite faction. Although the average scores are the same, the polarity is different. And in fact, the average usefulness of good Notes seems to be higher than that of good but more extreme Notes.

Having an algorithm that is closer to the “Economist’s Algorithm” will have a clearer story about how the algorithm penalizes extremes.

How useful is all this in high-stakes situations?

We can learn some of this by looking at a specific situation. About a month ago, Ian Bremmer complained that a highly critical Community Note?item_id=888 was added to a tweet about Chinese government officials." target="_blank" rel="noopener noreferrer">see more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more? item_id= 888" target="_blank" rel="noopener noreferrer">View more, but the Notes have been deleted.

This is a difficult task. Its one thing to do mechanism design in an Ethereum community environment where the biggest complaint might just be $20,000 going to an extreme Twitter influencer. The situation is completely different when it comes to political and geopolitical issues that affect millions of people, where everyone often reasonably assumes the worst motives. However, interacting with these high-stakes environments is essential if mechanic designers want to have a significant impact on the world.

In the case of Twitter, theres an obvious reason to suspect centralized manipulation as the reason for Notes being removed: Elon Musk has a lot of business interests in China, so its possible that Elon Musk forced the Community Notes team to interfere with the algorithms output and remove it This particular Notes.

Luckily, the algorithm is open source and verifiable, so we can actually dig into it! Lets do this. The URL of the original tweet is https://twitter.com/MFA_China/status/1676157337109946369. The number at the end, 1676157337109946369, is the ID of the tweet. We can search for this ID in the downloadable data and identify the specific row in the spreadsheet that has the above Notes:

Here, we get the ID of Notes itself, 1676391378815709184. We then search for this ID in the scored_notes.tsv and note_status_history.tsv files generated by running the algorithm. We got the following results:

The second column in the first output is the current rating for the Notes. The second output shows the Notes history: its current status is in column seven (NEEDS_MORE_RATINGS), and the first status it previously received that was not NEEDS_MORE_RATINGS is in column five (CURRENTLY_RATED_HELPFUL). So we can see that the algorithm itself first showed the note and then removed it after its rating dropped slightly - there seems to be no centralized intervention involved.

We can also look at this issue another way by looking at the vote itself. We can scan the ratings-00000.tsv file to isolate all ratings for that Notes and see how many are rated HELPFUL and NOT_HELPFUL:

However, if you sort them by timestamp and look at the first 50 votes, youll see that there are 40 HELPFUL votes and 9 NOT_HELPFUL votes. So we came to the same conclusion: Notes was rated more positively by its initial audience and less by its later audience, so its rating started high and declined over time get lower.

Unfortunately, its hard to explain exactly how Notes change status: its not a simple matter of previously rated above 0.40, now rated below 0.40, so its removed. Instead, a large number of NOT_HELPFUL replies triggers one of the exception conditions, increasing the usefulness score that Notes needs to stay above the threshold.

This is another great learning opportunity that teaches us a lesson: making a trustworthy neutral algorithm truly trustworthy requires keeping it simple. If a Note goes from being accepted to not being accepted, there should be a simple and clear story explaining why this is the case.

Of course, there is another completely different way this vote can be manipulated: Brigading. Someone who sees a Notes they dont approve of can appeal to a highly engaged community (or worse, a legion of fake accounts) to rate it NOT_HELPFUL, and it probably doesnt take many votes to get the Notes removed from Useful becomes extreme. More analysis and work is required to properly reduce the algorithms vulnerability to such coordinated attacks. A possible improvement would be to not allow any user to vote on any Notes, but instead randomly assign Notes to graders using the for you algorithm recommendation, and allow graders to grade only those Notes they are assigned to.

Isnt Community Notes brave enough?

The main criticism I see against Community Notes is basically that it doesnt do enough. I saw two recent articles mentioning this. To quote one of the articles:

The process suffers from a serious limitation in that for Community Notes to become public, they must be generally accepted by a consensus of people across the political spectrum.

It has to have ideological consensus, he said. That means people on the left and people on the right have to agree that the note has to be attached to the tweet.

Essentially, he said, it requires cross-ideological agreement on the truth, which is nearly impossible to achieve in an increasingly partisan environment.

Its a tricky question, but ultimately Im inclined to think that its better to have ten tweets of misinformation go free than one tweet to be unfairly annotated. We’ve seen years of fact-checking that’s brave and from a “actually we know the truth, we know one side lies more often than the other” perspective. What will happen?

To be honest, there is a fairly widespread distrust of the very concept of fact-checking. Here, one strategy is to say: ignore those critics, remember that fact-checkers do know the facts better than any voting system, and stick with it. But going all-in on this approach seems risky. There is value in building intertribal institutions that are at least somewhat respected by all. Like William Blackstones dictum and the courts, I feel that maintaining that respect requires a system that errs by omission rather than by voluntary error. So it seems worthwhile to me that at least one major organization is taking this different path and seeing its rare inter-tribal respect as a precious resource.

Another reason I think its okay for Community Notes to be conservative is that I dont think every misinformed tweet, or even most misinformed tweets, should receive a corrective note. Even if less than one percent of misinformed tweets are annotated to provide context or correction, Community Notes still provide an extremely valuable service as an educational tool. The goal is not to correct everything; rather, the goal is to remind people that there are multiple viewpoints, that some of the posts that seem convincing and engaging in isolation are actually quite wrong, and that you, yes, you can usually do basic internet Search to verify it is wrong.

Community Notes cannot be, nor is it intended to be, a panacea for all problems in public epistemology. Whatever problems it doesn’t solve, there’s plenty of room for other mechanisms to fill in, whether it’s novelty gadgets like prediction markets, or established organizations hiring full-time staff with domain expertise to try to fill the gaps.

in conclusion

Community Notes is not only a fascinating experiment in social media, but also a fascinating example of an emerging type of mechanism design: mechanisms that consciously try to identify extremes and tend to foster crossover rather than perpetuate division.

Two other examples in this category that I am aware of are: (i) the paired quadratic funding mechanism used in Gitcoin Grants, and (ii) Polis, a discussion tool that uses clustering algorithms to help the community identify common Popular statements span people who often have different opinions. This field of mechanism design is valuable, and I hope we see more academic work in this area.

The algorithmic transparency provided by Community Notes isnt exactly the equivalent of fully decentralized social media - if you dont agree with the way Community Notes works, theres no way to view the same content through a different algorithm. But this is the closest that hyperscale applications will get in the next few years, and we can see that it already provides a lot of value, both in preventing centralized manipulation and ensuring that platforms that do not engage in such manipulation get their due. recognized.

I look forward to seeing Community Notes and many algorithms of a similar spirit develop and grow over the next decade.