distributed-crawler 开源项目推荐（15 个）

SpiderClub /weibospider

：zap：微博的分布式爬虫，使用芹菜和请求构建。

brendonboshell /supercrawler

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

aaldaber /Distributed-Multi-User-Scrapy-System-with-a-Web-UI

基于Django的应用程序，允许以分布式方式创建、部署和运行Scrapy Spiders

A1014280203 /Ugly-Distributed-Crawler

基于Redis实现的简单到爆的分布式爬虫

fgksgf /DCVS

🎓 JD Distributed Crawler and Visualization System. 京东商品评论分布式爬虫

zezhix /distributed-crawler

第六届中软杯赛题 - 分布式爬虫系统（经纬度团队作品）

ORACLE128G /polaris

The Go Programming Language Learning.

RockyLOMO /rx-mimic

A distributed crawler.

strlexx /sina-real-time

Collect real-time stock data from Sina Finance using efficient Rust WebSocket connections with automatic reconnection and local CSV storage.

skyn9 /aiocrawler

自研 Python 异步爬虫框架：可插拔的调度器、下载器、中间件与存储后端

ChimangoScan /DITector

Large-scale measurement framework for the Docker ecosystem, extending the Dr. Docker (WWW '25) methodology. Implements distributed data collection, IDEA dependency graph construction, and supply-chain risk prioritization for OpenVAS.

liho98 /distributed-crawler

该项目在 GitHub 上没有填写描述

Khizarshah01 /distributed_crawler

A high-performance, PostgreSQL-backed distributed web crawler built with Node.js, featuring concurrent workers, fault recovery, host-level politeness, and measured horizontal scaling.

68110923 /suncent

亚马逊分布式爬虫.产品爬虫详情\排行\星级\价格\评论数据\QA数据等.亚马逊后台爬虫(多账号多服务器绑定的场景\支持 欧美日三个板块的站点):流量报告\库存\支付报告\提现功能\全部订单报告\出库报告\授权功能\业绩报告\广告SP\广告SD\广告付款记录等。web后台监控服务器状态\监控各种报告爬取状态\钉钉报警提醒\各种数据报告的展示\评论分析\产品分析\数据导出等功能

JiuyuParis /DushuwangSpider_Qianfeng

分布式爬虫代码