技术解读

以下内容由 GitRead 中文社区 编辑团队基于项目 README 与官方资料整理, 结合中文开发者的常见使用场景补充技术选型建议。具体以项目官方文档为准。

项目背景与痛点

“pirate/sites-using-cloudflare” 是一个存档在 CloudBleed 事件宣布时使用 Cloudflare DNS 的域名列表。CloudBleed 事件发生时,因 Cloudflare 的渲染系统 Bug,导致用户访问网站时可能泄漏其他用户的内存数据。这个问题严重到甚至影响了 Facebook、Twitter 等大网站的数据。这个问题背后,暴露出 DNS 服务提供商的脆弱性和隐私数据安全的重要性。因此,及时了解哪些网站正使用 Cloudflare 提供的 DNS 服务变得至关重要。“pirate/sites-using-cloudflare”项目应运而生,该项目提供了在事件爆发时所有使用 Cloudflare DNS 的网站清单,帮助用户识别并审慎选择自己经常访问的网站。

核心技术揭秘

该项目集成了多项技术,实现时采用了高效爬虫技术与数据爬取策略,不断抓取最新域名信息,以保持域名列表的更新。充分利用了开源社区的力量,采用社区提交的方式补充和修正域名列表。这背后的关键技术栈包括 Python 编程语言及相关爬虫框架(如 Scrapy 等),以及社区协作的 GitHub 仓库维护体系。

具体到技术实现上,该项目的核心架构可以分为以下几个步骤:

  1. 数据获取:通过爬虫技术不断抓取结果,锁定使用 Cloudflare 的网站。
  2. 数据清洗与存储:针对抓取的数据进行清洗、整理,并归档与记录。
  3. 版本控制:利用 GitHub 提供的版本控制功能,确保 数据有足够的回滚机制。

下面是一个简单描述该过程的架构图(使用 Mermaid):

graph TD A[数据源] --> B[解析 URL] B --> C{"解析结果"} C -->|"合格"| D[数据存储] C -->|"不合格"| E[数据丢弃] F[版本控制] --> G[GitHub 仓库]

功能亮点与差异

该项目与同类竞品相比,最大的亮点当属其基于社区的运作模式,允许用户随时更新和修正域名记录清单。通过社区的力量,该数据集能实现快速迭代和完善,保证数据的实时性和准确性。此外,它所记录的数据范围较广,包括但不限于知名网站、中小型网站和个人网站,尽可能全面覆盖了可能的隐私风险来源。

应用场景与落地建议

在生产环境使用此项目数据时,只将其作为重要的参考之一。每一个开发者都应进行独立的、更为深度的安全性核查,避免盲从数据。该仓收录的数据适合用于追踪和分析 Cloudflare 影响范围内的特定网站,尤其适用于寻求全面了解 Cloudflare 事件影响的安全研究者和网站管理员。在进行网站优化和安全评估时,主动验证网站的 DNS 记录服务至关重要。

综合评价

作为一个基于社区的项目,“pirate/sites-using-cloudflare”展现了优秀的开源社区协作模式,其数据的准确性和实时性极大便利了访问者的安全性评估。然而,该项目并非即时更新的数据流,其时效性和准确性依赖于社区成员的贡献,存在一定的局限性。尽管如此,它依然是一个极具价值的资源,值得尽可能多地传达给您身边每个对网络安全有关注的开发者和网站维护者。

关于

GitHub 原始描述(英文)

:broken_heart: Archived list of domains using Cloudflare DNS at the time of the CloudBleed announcement.

描述取自 GitHub 官方 API,保持仓库作者书写的原文。

简要分析

热度分
2548
价值分
1247
活跃状态
归档

README

List of Sites on Cloudflare DNS (archived)

This is an (archived) list of sites on Cloudflare DNS at the time of the CloudBleed HTTPS traffic leak announcement. Original vuln thread by Google Project Zero.

Cloudflare has posted a very detailed response, explaining exactly what the implications of this leak are. It thoroughly explains their language in earlier statements, and I highly recommend reading it before looking through this list for domains: https://blog.cloudflare.com/quantifying-the-impact-of-cloudbleed/

DISCLAIMER:

This list is archived and no longer under active maintenance. It may contain stale or inaccurate data that will not be corrected. Do not link to it from press releases, it is not intended for end-users. If people want to find it, they can Google it.

This list contains all domains that use Cloudflare DNS, not just the Cloudflare proxy (the affected service that leaked data). It’s a broad sweeping list that includes everything. Just because a domain is on the list does not mean the site is compromised, and sites may be compromised that do not appear on this list.

Cloudflare has not provided an official list of affected domains, and likely will not due to privacy concerns. I’ve compiled an unofficial list here so you know where to start searching for sessions to reset and passwords to change.

See issue #127 and issue #87 for additional info about which sites are likely to be affected.

Impact

Between 2016-09-22 - 2017-02-18 session tokens, passwords, private messages, API keys, and other sensitive data were leaked by Cloudflare to random requesters. Data was cached by search engines, and may have been collected by random adversaries over the past few months.

Requests to sites with the HTML rewrite features enabled triggered a pointer math bug. Once the bug was triggered the response would include data from ANY other Cloudflare proxy customer that happened to be in memory at the time. Meaning a request for a page with one of those features could include data from Uber or one of the many other customers that didn’t use those features. So the potential impact is every single one of the sites using Cloudflare’s proxy services (including HTTP & HTTPS proxy).

“The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests), potential of 100k-200k paged with private data leaked every day” – source

~~You can see some of the leaked data yourself in search engine caches: https://duckduckgo.com/?q=+{"scheme"%3A"http"}+CF-Host-Origin-IP&t=h_&ia=web~~ (2/25/2017) DuckDuckGo has removed this data

Confirmed affected domains found in the wild: http://doma.io/2017/02/24/list-of-affected-cloudbleed-domains.html

What should I do?

The most important thing you can do is ask your vendors and sites to reset all their session tokens, as more response data was leaked than request data, and responses generally contain session tokens rather than passwords. If websites you use have a button to “log out all active sessions”, use it. Since sites may be compromised this week due to data discovered in caches, it’s best to also do this again in a week or two after everything settles down. If websites you use don’t have an option to log out all active sessions, contact them and pressure them to rotate all their session tokens.

To be extra safe, you may want to check your password managers and change crucial passwords, especially those on these affected sites. Rotate API keys & secrets, and confirm you have 2-FA set up for important accounts. This might sound like fear-mongering, but the scope of this leak is truly massive, and due to the fact that all Cloudflare proxy customers were vulnerable to having data leaked, many of the extra cautious people out there would rather be safe than sorry.

Theoretically sites not in this list can also be affected (because an affected site could have made an API request to a non-affected one).

Methodology

This list was compiled from 3 large dumps of all Cloudflare customers provided by crimeflare.com/cfs.html, and several manually copy-pasted lists from stackshare.io and wappalyzer.com. Crimeflare collected their lists by doing NS DNS lookups on a large number of domains, and checking SSL certificate ownership.

I scraped the Alexa top 10,000 by using a simple loop over the list:

for domain in (cat ~/Desktop/alexa_10000.csv)
    if dig $domain NS | grep cloudflare
        echo $domain >> affected.txt
    end
end

The Alexa scrape, and the Crimeflare dumps were then combined in a single text file, and passed through sort | uniq. I’ve since accepted several PRs and issues to remove sites that were unaffected from the list.

Data sources:

I’d rather be safe than sorry so I’ve included any domain here that remotely touches Cloudflare. Don’t point end-users to this list please, it has too many false positives to be useful for non-analytical purposes. I’m no longer accepting PRs to remove sites from the list, our previous process to remove sites was error-prone and labor intensive. The list is now in archive mode, consider it defunct. If you think for some reason this will greatly impact you or your users, DM me on twitter.

Full List

Download the full list.zip (22mb)

4,287,625 possibly affected domains. Download this file, unzip it, then run grep -x domaintocheck.com sorted_unique_cf.txt to see if a domain is present.

Also, a list of some iOS apps that may have been affected.

Search Tools

There are several tools out there to search the list, I wont endorse any here due to them having greatly varying degrees of accuracy. Please do not make user-facing tools to search the list or cross-reference it with browser history, this list has too many false positives to use for that purpose. You will make users lose trust in many sites, despite there being less than a 1 in a million chance of them having data leaked.

Notable Sites

Alexa Top 10,000 on Cloudflare DNS:

评论

暂无评论