scrape 开源项目推荐（707 个）

twintproject /twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

alirezamika /autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

d60 /twikit

Twitter API 抓取器 | 无需 API 密钥 | Twitter 内部 API | 免费 | Twitter 抓取器 | Twitter 机器人

Anorov /cloudflare-scrape

A Python module to bypass Cloudflare's anti-bot page.

any4ai /AnyCrawl

AnyCrawl：一个 Node.js/TypeScript 爬虫，可将网站转换为可供 LLM 使用的数据，并从 Google/Bing/百度等提取结构化 SERP 结果；原生多线程支持批量处理。

vasu-devs /JustHireMe

Local-first AI job intelligence workbench for scraping roles, ranking fit, and generating tailored application materials.

trevorhobenshield /twitter-api-client

Implementation of X/Twitter v1, v2, and GraphQL APIs

oxylabs /scrape-google-python

在本教程中，我们展示了如何使用Python和Oxylabs API抓取公共Google数据。

glebarez /cero

Scrape domain names from SSL certificates of arbitrary hosts

markowanga /stweet

Advanced python library to scrap Twitter (tweets, users) from unofficial API

drudge /n8n-nodes-puppeteer

n8n node for browser automation using Puppeteer

austinoboyle /scrape-linkedin-selenium

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

unixfox /pupflare

A webpage proxy that request through Chromium (puppeteer) - can be used to bypass Cloudflare anti bot / anti ddos on any application (like curl)

danieldotnl /ha-multiscrape

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

ScriptSmith /instamancer

Scrape Instagram's API with Puppeteer

yaroslaff /nudecrawler

Crawl telegra.ph searching for nudes!

JMousqueton /ransomware.live

🏴‍☠️💰 Another Ransomware gang tracker

Anonyfox /elixir-scrape

Scrape any website, article or RSS/Atom Feed with ease!

andrewstuart /goq

A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library

JaredLGillespie /proxyscrape

Python library for retrieving free proxies (HTTP, HTTPS, SOCKS4, SOCKS5).