llm-crawler - 技术专题

Here are 7 public repositories matching this topic...

watercrawl / WaterCrawl

Transform Web Content into LLM-Ready Data

crawler scraper html2markdown crawling-python llm-scraper llm-crawler crawl4ai aicrawler

Updated Aug 17, 2026
TypeScript

lennyerik / crawl4ai-proxy

A simple proxy server to integrate crawl4ai with OpenWebUI

adapter crawler machine-learning ai proxy web-crawler inference gpt llm ai-crawler llm-ui openwebui llm-crawler crawl4ai

Updated Nov 20, 2025
Go

sonhm3029 / Auto-Crawl-Any-Page-with-Large-Language-Model

Auto Crawl Any Page with Large Language Model

crawler ai-agents large-language-models ai-crawler llm-crawler

Updated Oct 4, 2024
Jupyter Notebook

vipul-69-ax / Python-LLM-Crawl

An advanced web crawler powered by large language models, featuring adaptive rate limiting, content deduplication, dynamic content extraction, continuous learning, proxy management, and intelligent prioritization of links. It also respects robots.txt rules and parses sitemaps for efficient crawling

crawler scraper continuous-integration llm-crawler