"I'm panicking, what happened?" Cloudflare outage triggers global internet chaos.

Original author: Zhao Yuhe

Original source: Wall Street News

On Tuesday morning Eastern Time, internet infrastructure service provider Cloudflare stated that its global network was experiencing anomalies, causing numerous websites, including the social media platform X, to experience "internal server error" and other access problems. Users were unable to access many websites and services, including retail, e-commerce, social media, financial services, and transportation-related platforms. The company later claimed that the problem was resolved in less than four hours.

During the outage, some functions of X were interrupted, and several websites were also inaccessible. According to data from the fault tracking platform Downdetector, in addition to X, a large number of other sites were affected, and the number of related reports continued to rise. Users saw Cloudflare-related error messages when accessing websites such as X, ChatGPT, DoorDash, IKEA, and the Metropolitan Transportation Authority (MTA) of New York City.

Subsequently, a Cloudflare spokesperson stated that around 6:20 a.m. Eastern Time, one of their services experienced an unusual surge in traffic, causing errors in traffic passing through the company's network.

Another Cloudflare spokesperson, Jackie Dutton, said in an announcement that the issue was caused by an automatically generated configuration file used to manage threat traffic, and the fix took less than four hours. The company stated that core remediation measures have been deployed, but cautiously noted that the system "still needs time to fully stabilize."

Dutton stated:

"The file contained more entries than expected, triggering a crash in the software system responsible for handling traffic to part of Cloudflare's services."

The statement said there was no evidence that the incident was related to a cyberattack or malicious activity.

The impact of the outage was extremely widespread. Downdetector stated on its platform that during the Cloudflare outage, "more than 2.1 million reports of various affected services were received," indicating that this incident has become one of the most serious infrastructure-level outages in recent years.

Following the incident, Cloudflare's stock price initially plunged 7% at the opening on Tuesday before recovering.

The digital asset industry has also reacted. Binance co-founder and former CEO Changpeng Zhao posted on X, "Blockchain kept working," implying that decentralized systems were unaffected by the incident.

As of 12:15 PM Eastern Time, Cloudflare stated that the system was gradually recovering, but access errors, performance degradation, or login issues may still occur in some regions globally. The company will continue to update the recovery progress on the status page.

Over-reliance on a few companies

In recent years, problems with digital infrastructure providers have repeatedly paralyzed global internet access. Amazon Web Services (AWS), CrowdStrike Holdings Inc., and Microsoft have all experienced similar incidents, highlighting the significant reliance of the global internet on a few companies providing services.

Cloudflare and AWS services are almost "invisible" to ordinary users, but their tools power a large number of websites and services that consumers use every day.

Last month's AWS outage paralyzed parts of the internet, rendering millions of users' websites and applications unusable, disrupting retail sales, interfering with social media and financial services, and impacting numerous businesses. Last year, a vulnerability in a tool used by cybersecurity firm CrowdStrike caused widespread computer system crashes globally, triggering thousands of flight delays and cancellations, and throwing government agencies and large corporations into disarray.

Graeme Stewart, an expert at California-based cybersecurity firm Check Point Software, said the incident highlights the internet’s over-reliance on a few infrastructure providers.

He said:

"Many organizations still rely on the same path for all their critical services, without any truly effective backups. If that path fails, there are no backup options. This is the problem we keep seeing."

University of Surrey cybersecurity professor Alan Woodward said Tuesday's outage once again illustrates the internet's heavy reliance on a "few players." He described Cloudflare as "the biggest company you've never heard of."

"People have no choice but to rely on these few large companies."

Chief Technology Officer Apologizes

Cloudflare CTO Dane Knecht apologized for the incident. He wrote on X:

"When Cloudflare's network experiences issues that impact the massive amounts of traffic that rely on us, we let our customers down, and we let down the entire internet. The problem itself, the impact it caused, and the time it took to resolve it were unacceptable. We have already begun working to ensure that similar situations do not happen again, but I know that today has indeed caused inconvenience for everyone. Our customers' trust in us is paramount, and we will do everything we can to earn it back."

Cloudflare has experienced similar outages multiple times in the past few years.

In July 2019, a vulnerability in Cloudflare software caused some network modules to excessively consume computing resources, bringing thousands of websites worldwide that rely on Cloudflare (including Discord, Shopify, SoundCloud, and Coinbase) offline for up to 30 minutes. In June 2022, Cloudflare experienced an outage that affected traffic to 19 of its data centers, causing several major websites and services to be down for approximately one and a half hours.

Cloudflare software is used by hundreds of thousands of companies worldwide as a buffer layer between enterprise websites and end users, protecting websites from DDoS attacks or outages caused by sudden traffic surges.

Last year, a faulty software update released by cybersecurity company CrowdStrike caused millions of devices running Microsoft Windows to crash, creating widespread disruption in numerous industries, including aviation, banking, and healthcare.

CrowdStrike's outage stemmed from a bug in its product that ran at the lowest level of the customer's computer. Cloudflare, on the other hand, protects internet infrastructure such as websites and platforms; therefore, when Cloudflare goes down, many popular websites become inaccessible or experience malfunctions. Cloudflare primarily focuses on keeping websites online and fast, while CrowdStrike focuses on protecting computers and servers from attacks.