trafilatura 开源项目推荐（86 个）

opendatalab /MinerU-HTML

MinerU-HTML：一个基于STM的HTML主内容提取器，输出干净的HTML正文。非常适合深度研究代理、RAG应用程序和训练数据生成。

vakovalskii /searcharvester

Self-hosted search + markdown harvester for AI agents. SearXNG (100+ engines) + FastAPI + trafilatura. Tavily-compatible /search plus /extract with size presets and pagination. One-command Docker Compose.

zyocum /reader

Extract clean(er), readable text from web pages via trafilatura.

michaelsoftmd /pebkac-chrome

pebkac Chrome Nonautomation - A Local LLM-Driven Web Co-Browser using Smolagents, Zendriver, Trafilatura.

Murrough-Foley /rs-trafilatura

Fast, accurate web content extraction in Rust. ML page-type classification, per-type extraction, confidence scoring. F1=0.966 on ScrapingHub (#1), F1=0.859 across 2,008 annotated pages (1,497 development + 511 held-out test

maximilianromer /Onion-Search-MCP

Tools for LLMs to anonymously search and browse the web

romek-rozen /cc-crawl4ai

Native non-MCP Crawl4AI and Trafilatura plugin for Claude Code

hec-ovi /websearch-skill

Keyless, uv-native web search + read for AI agents: ddgs multi-engine search with de-correlated rank fusion, Trafilatura extraction to paginated Markdown, keyless arxiv and github search. Self-hosted SearXNG without Docker, an optional Tor layer for .onion search and fetch, NordVPN/SOCKS5 egress, one-call init, and a doctor.

gorango /napi-rs-trafilatura

Fast and accurate web content extraction

agent-kreal /agent-web-search

Quota-aware web search for AI agents: 10 providers with local quota tracking (fallback before 429), intent routing, free local-first extraction cascade (gh CLI native for github.com, JSON API passthrough, Russian Trusted CA). Benchmarks included.

Gdi87 /Webscrapper

web Scrapper In Python

vedantvisoliya /Flutter-ChatGPT-Clone

ChatGPT AI Clone

clusy-io /clusy-crawler

Fast, bounded, self-hosted web extraction: URL to clean LLM-ready Markdown. Rust/Python, HTTP/2, conditional Chromium, PDF support. Apache-2.0; no model required.

brzvsk /longreader

Telegram Mini App that saves internet articles to read them later

bbulb /trawl

Selective web content extraction for AI agents — URL + query returns only the chunks that matter (Python library + MCP server)

khsarvar /siftdog

Open-source, self-hosted Tavily alternative: web search, extract, crawl and map API for AI agents, with an MCP server

JiamingShi-JsPhys /web-forager

Clean-room, robots-aware web information gathering toolkit for agents and analysts. Search 360/Bing/Sogou, read Bilibili, Zhihu, Xiaohongshu, Weibo and other public sites as Markdown, and browse with resumable sessions. CLI, MCP server and Claude Code skill.

Yasser03 /pipescraper

A pipe-based news article scraping and metadata extraction library for Python

janschachtschabel /Website-Textextraction-Selenium

Web pages and documents as Markdown over an HTTP API. Trafilatura for main content, MarkItDown for documents, and Chrome via Selenium only when a page needs JavaScript.

aiandcivilization /webtrawl

Agent-native web search and page extraction CLI: multi-provider discovery, a four-tier extraction cascade that reports paywalls instead of returning them, TOON output. Runs with zero API keys.