crawlers 开源项目推荐（209 个）

ai-robots-txt /ai.robots.txt

要阻止的 AI 智能体和机器人列表。

omrilotan /isbot

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

flathunters /flathunter

A bot to help people with their rental real-estate search. 🏠🤖

scrapfly /Antibot-Detector

Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

salimk /Rcrawler

An R web crawler and scraper

Norconex /crawler

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

SiENcE /astray

Astray is a lua based maze, room and dungeon generation library for dungeon crawlers and rougelike video games

hhuayuan /spiderbuf

Spiderbuf 是一个专注于 Python 爬虫练习的网站。提供丰富的爬虫教程、爬虫案例解析和爬虫练习题。Python爬虫开发强化练习，在矛与盾的攻防中不断提高技术水平，通过大量的爬虫实战掌握常见的爬虫与反爬套路。 引导式爬虫案例 + 免费爬虫视频教程，以闯关的形式挑战各个爬虫任务，培养爬虫开发的直觉及经验，验证自身爬虫开发与反爬虫实力的时候到了。

ArchiveTeam /wget-lua

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

narkhedesam /Proxy-List-Scrapper

Proxy List Scrapper

jonasjacek /robots.txt

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

behitek /social-scraper

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

howie6879 /hproxy

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

Potelo /laravel-block-bots

用Redis通过IP屏蔽爬虫和高流量用户

BaseMax /GooglePlayWebServiceAPI

Tiny script to crawl information of a specific application in the Google play/store base on PHP.

Symbolexe /Raven

Raven is a powerful and customizable web crawler written in Go.

herrbischoff /user-agents

User agent database in JSON format of bots, crawlers, certain malware, automated software, scripts and uncommon ones.

flulemon /sneakpeek

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

peterbencze /serritor

Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.

Intuned /cookbook

A cookbook of examples for building browser automations with Intuned