22.8 Cloudbleed

In 2017, Tavis Ormandy, a vulnerability researcher in Google’s Project Zero team, reported a security vulnerability in Cloudflare’s edge servers [76]. Cloudflare is a large Content Delivery Network (CDN) that operates a global network of servers that cache and deliver website content to end users from the server location closest to them.

Because of the vulnerability, the software running on Cloudflare’s edge servers – more precisely, an HTML parser – was reading past the end of a buffer and returning contents from the servers’ internal memory such as HTTP cookies, authentication tokens, and the bodies of HTTP POST requests.

Cloudflare reported that during the peak time, a period of about five days, 1 in every 3,300,000 HTTP requests to Cloudflare’s edge servers potentially resulted in a memory leak [76].

The proof of concept by Tavis Ormandy returned private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, and hotel bookings. Due to its similarity to Heartbleed, the bug was soon named Cloudbleed.

Numerous Cloudflare services need to parse and modify HTML requests and pages, for example, to insert Google Analytics tags, rewrite HTTP links to HTTPS links, exclude parts of a page from bad bots, or modify the HTML code of a web page. To achieve this, the services must parse HTML to identify elements that have to be changed.

To generate HTML parsers, Cloudflare initially used the Ragel state machine compiler tool. Every HTML parser was described in its own .rl file. Eventually, though, the company decided that Ragel-based parsers had become too complex to maintain and, therefore, started writing a new parser called cf-html to replace Ragel. At the time of the bug discovery, Cloudflare was migrating from the old Ragel to the new cf-html parser.

Both cf-html and the old Ragel-based parser were implemented as NGINX modules compiled into Cloudflare’s NGINX builds. These NGINX filter modules parse buffers (blocks of memory) containing HTML responses, make modifications as necessary, and pass the buffers onto the next filter [76].

Analyses performed after Tavis Ormandy’s report showed that the Cloudbleed bug was present in Cloudflare’s Ragel parser for many years but did not lead to memory leaks because of the way internal NGINX buffers were used. Unfortunately, introducing cf-html changed the buffering in a subtle way that, in turn, enabled the information leakage.

Once Cloudflare realized that the bug was caused by the use of the cf-html parser, they quickly disabled its multiple features using the corresponding feature flags they refer to as global kill switches. For example, the email obfuscation global kill switch was activated 47 minutes after Cloudflare received information about the security incident.