web-scrapping 开源项目推荐（300 个）

tinyfish-io /agentql

AgentQL是一套用于将人工智能连接到网络的工具。具有查询语言和Playwright集成，用于与元素交互并快速、准确、大规模提取数据。包括REST API、Python和JavaScript SDK、浏览器调试器。

VarunS2002 /Python-NSE-Option-Chain-Analyzer

SEN有一个网站，可以近乎实时地显示期权链。此程序从NSO网站检索此数据，然后为指定指数或股票生成期权链的有用分析。它还不断刷新期权链，并直观地显示对技术分析有用的各种指标的趋势。

mldsveda /PyScrappy

Adaptive Python web scraping toolkit + MCP server for AI agents. Self-healing selectors that survive site changes, TLS-fingerprint stealth to bypass anti-bot filters, CSS/XPath parsing, and 24 built-in scrapers, clean, structured, LLM-ready data from any URL.

tinyfish-io /agentql-mcp

Model Context Protocol server that integrates AgentQL's data extraction capabilities.

narkhedesam /Proxy-List-Scrapper

Proxy List Scrapper

spyboy-productions /PhantomCrawler

PhantomCrawler 是一款基于 Python 的网页测试和研究工具，通过模拟多个代理IP地址的网站交互，分析不同网络条件下的流量行为、访问控制和响应模式。

sallamy2580 /python-web-scrapping

面向初学者的详细网络爬虫教程，涵盖 Reddit WallStreetBets、CME（期权与期货）、美国财政部、CFTC、LME、MacroTrends、SHFE 上的金融数据爬虫，以及 Tomtom、BBC、《华尔街日报》、半岛电视台、路透社、《金融时报》、彭博社、CNN、《财富》、《经济学人》上的另类数据爬虫

MrHacker-X /OsintifyX

OsintifyX: Powerful Open-source OSINT tool for extracting valuable information from Instagram profiles.

JarbasAl /PySychonaut

Unofficial python api for Erowid, PsychonautWiki and AskTheCaterpillar

raymelon /tagalog-dictionary-scraper

2016: Builds a Tagalog dictionary by collecting Tagalog words from tagalog.pinoydictionary.com

gugarosa /leaflyer

🍀 Scraps data from Leafly and collects amazing weed-based meta-data.

yashkathe /Download-ComicBooks-API

Unoffical api to get comic books from various publishers. Published on NPM !

tinyfish-io /agentql-integrations

AgentQL's integrations with workflow automation tools and AI agent frameworks let you extract structured data from web pages using queries or natural language and interact with the web with Playwright. Resilient, fast, and AI-ready.

didvc /jseval

Evaluate JavaScript on a URL through headless Chrome browser.

arv-anshul /campusx

Get all the resources like Notes and Notebooks provided in CampusX Courses.

Hrushi11 /Yellow-Pages-End-API

This repository contains the code for completely scraping the yellow pages website to get the details of all the categories from all the cities in America and then turning all of the data into an api to be used later.

Or-i0n /sololearnlib

Library that retrieves public data from sololearn.com

kingtroga /linkedin_scraper

This Python script automates interactions with LinkedIn Sales Navigator using Selenium WebDriver. It can log into LinkedIn, navigate through lists of prospects, send messages, and manage contacts based on specific criteria.

PB2204 /Covid-19

This Is A Web Scraping Projects With Covid-19 Data From 2 Very Popular & Authentic Websites

olucaslopes /BNMP-Scraper

Navega pela API do Portal BNMP coletando dados