webscraper 开源项目推荐（1,561 个）

jaypyles /Scraperr

Self-hosted webscraper.

any4ai /AnyCrawl

AnyCrawl：一个 Node.js/TypeScript 爬虫，可将网站转换为可供 LLM 使用的数据，并从 Google/Bing/百度等提取结构化 SERP 结果；原生多线程支持批量处理。

anaskhan96 /soup

Web Scraper in Go, similar to BeautifulSoup

scrapfly /scrapfly-scrapers

Scalable Python web scraping scripts for +40 popular domains

benibela /xidel

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

rootVIII /proxy_requests

a class that uses scraped proxies to make http GET/POST requests (Python requests)

tech-engine /goscrapy

GoScrapy: High perfomance webscraping framework in Go, inspired by Python's Scrapy.

salimk /Rcrawler

An R web crawler and scraper

phoenixthrush /AniWorld-Downloader

AniWorld Downloader是一款跨平台应用程序，用于查找、流媒体和下载动漫、电影、连续剧和漫画。它具有基于浏览器的Web UI、交互式终端菜单以及用于脚本和无头设置的直接CLI。如果您喜欢使用它，请随时留下收件箱！

onepointAI /onepoint

An AI assistant tool that integrates coding, writing, and reading functions. For better alternatives see https://monica.im/desktop

intergalacticalvariable /reader

📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an LLM-friendly input with a simple prefix http://127.0.0.1:3000/https://website-to-scrape.com/

TBosak /mkfd

Self-hosted feed generation toolkit for turning webpages, email folders, APIs, and messy web sources into reliable RSS feeds.

toby-p /rightmove_webscraper.py

Python class to scrape data from rightmove.co.uk and return listings in a pandas DataFrame object

s-r-e-e-r-a-j /WebExtractor

WebExtractor is a powerful OSINT and ethical hacking tool developed in Python. It is used to extract email addresses, phone numbers, and links from a target website

serpapi /lego-ai-parser

Lego AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

MichaelYochpaz /iSubRip

A Python command-line tool for scraping and downloading subtitles from AppleTV and iTunes movie pages.

3xploitGuy /webscrape

A web scraper to scrape email's and phone numbers from Websites.

nmcassa /letterboxdpy

A letterboxd webscraper

mehmetozkaya /DotnetCrawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-w…

EchterAlsFake /unofficial-api-for-pornhub

Unofficial Python client for xvideos.com: search, metadata extraction, and video download (curl-cffi + selectolax)