anakin

AnakinScraper OSS

The open-source web scraping API for AI. Turn any website into LLM-ready markdown or structured data.

Self-host with a single command. No cloud dependencies. Powers RAG pipelines, AI agents, and data extraction at scale.

git clone https://github.com/Anakin-Inc/anakinscraper-oss.git && cd anakinscraper-oss && make up

# Scrape any website — one curl, full result:
curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq .markdown

Why AnakinScraper?

Features

• Handler chain with fallback — HTTP fetch → anti-detect browser → external API. Each handler tries in order; if one fails, the next picks up automatically. Most pages resolve on the free local HTTP handler — paid APIs are only called for the ~5% that actually need them. Docs →
• Custom API handlers — plug in any third-party scraping service as a chain fallback. Only invoked when local handlers fail — saves 90%+ on API costs vs routing everything through a paid service. Built-in anakin.io handler included. How to add your own →
• Domain configs — per-domain scraping strategies: choose which handlers to use, set timeouts, retries, custom headers, block domains, and validate content with pattern matching. Docs →
• Failure detection — define failure patterns and required patterns per domain. If the scraped content matches a failure pattern (e.g. CAPTCHA page) or misses a required pattern, the job auto-retries with the next handler. Docs →
• Anti-detect browser — Camoufox (anti-detect Firefox) with realistic fingerprints, not headless Chrome. Docs →
• Proxy auto-select — Thompson Sampling picks the best proxy per domain, learning from success/failure in real time. Docs →
• Structured JSON extraction — use Gemini AI to extract structured data from any page (bring your own API key)
• Sync + async + batch API — POST /v1/scrape for instant results, /v1/url-scraper for async with polling, batch up to 10 URLs
• LLM-ready markdown — automatic boilerplate removal, clean content extraction. Feed directly into RAG pipelines, Claude, GPT, or any LLM without preprocessing
• Web dashboard — built-in React UI for scraping, job tracking, domain config management, and proxy monitoring
• Zero-config mode — run with just Go, no database needed. Or use Docker for the full stack
• Self-contained — no Redis, no AWS, no message queues. Single Go binary. Optional PostgreSQL for persistence

Quick Start (no Docker, no database)

Just Go 1.25+. Two commands:

cd server && go run cmd/server/main.go

# In another terminal:
curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq .markdown

Jobs are stored in memory (lost on restart). For persistence, set DATABASE_URL. For JavaScript-heavy sites, add the browser service via Docker.

Self-Host (Docker — full stack)

Prerequisites

• Docker and Docker Compose

Start

git clone https://github.com/Anakin-Inc/anakinscraper-oss.git
cd anakinscraper-oss
make up

That’s it. Three containers start:

Web Dashboard

A built-in web UI is included for visual scraping, job tracking, and configuration:

cd webapp && npm install && npm run dev

Open http://localhost:3000 — the dashboard proxies API calls to the server on port 8080.

Pages: Dashboard (health + quick scrape) | Scrape (sync/async/batch with live results) | Jobs (tracked history with status filters) | Domain Configs (CRUD with handler chain management) | Proxy Scores (Thompson Sampling performance)

Scrape a URL

Synchronous (recommended for getting started):

curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq .

One request, full result back. No polling. Timeout: 30 seconds by default (configurable via the timeout request field, max 120 seconds).

Asynchronous (for long-running scrapes):

# Submit
curl -s -X POST http://localhost:8080/v1/url-scraper \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

# Poll for result
curl -s http://localhost:8080/v1/url-scraper/JOB_UUID | jq .

With AI-powered JSON extraction (requires GEMINI_API_KEY):

curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "generateJson": true}' | jq .generatedJson

No API keys required for the scraper itself. Just JSON in, results out.

Architecture

                    ┌─────────────────┐
                    │   Your App      │
                    │   (cURL / CLI)  │
                    └────────┬────────┘
                             │ HTTP
                             ▼
                    ┌─────────────────┐         ┌──────────┐
                    │     Server      │────────▶│  Gemini  │
                    │   (Go/Fiber)    │ optional│  (JSON)  │
                    │   Port 8080     │         └──────────┘
                    └──┬──────┬───┬──┘
                       │      │   │
            ┌──────────┘      │   └────────────┐
            ▼                 ▼                 ▼
      ┌──────────┐   ┌──────────────┐   ┌──────────────┐
      │ Storage  │   │   Browser    │   │ API Handler  │
      │ Postgres │   │   Service    │   │ (anakin.io   │
      │ or memory│   │  (Camoufox)  │   │  or custom)  │
      │(optional)│   │  (optional)  │   │  (optional)  │
      └──────────┘   └──────────────┘   └──────────────┘

The server is a single Go binary that runs with zero dependencies. Optionally add PostgreSQL for persistence, the browser service for JavaScript-heavy sites, and API handlers for hard-to-scrape sites. Workers execute the handler chain (HTTP → browser → API fallback), convert HTML to markdown, and optionally extract structured JSON via Gemini.

API Reference

See docs/API.md for the complete API reference. Quick overview:

Request Fields

Response

{
  "id": "550e8400-...",
  "status": "completed",
  "url": "https://example.com",
  "html": "<html>...</html>",
  "cleanedHtml": "<main>...</main>",
  "markdown": "# Page Title\n\nContent...",
  "generatedJson": {
    "status": "success",
    "data": {"title": "Page Title", "content": "..."}
  },
  "durationMs": 1234
}

generatedJson is only present when generateJson: true and GEMINI_API_KEY is configured.

Handler Chain

Each scrape job goes through the handler chain. On failure, it falls back to the next handler:

HTTP Handler (fast, ~200ms) ──fail──▶ Browser Handler (Camoufox) ──fail──▶ API Handler (optional)

HTTP Handler — direct HTTP GET with a browser user-agent. Handles static HTML, server-rendered pages. No browser overhead.

Browser Handler — connects to Camoufox (anti-detect Firefox) over WebSocket via Playwright protocol. Full JavaScript rendering, network-idle detection, realistic browser fingerprints. Handles SPAs, lazy-loaded content, and sites with anti-bot protection.

API Handler (optional) — delegates to an external scraping API when local handlers fail. Set ANAKIN_API_KEY to enable the built-in anakin.io fallback for hard-to-scrape sites (Cloudflare, DataDome, etc.). See Adding Custom API Handlers below.

Adding Custom API Handlers

The API handler pattern makes it easy to integrate any third-party scraping service as a chain fallback. The built-in anakin.io handler (server/internal/handler/api.go) is a working example — copy and modify it for your provider:

1. Copy api.go to my_provider.go
2. Add a constructor like NewAnakinHandler — set your provider’s URL, auth header name, and response format
3. Register in main.go:

   if cfg.MyProviderAPIKey != "" {
       handlers = append(handlers, handler.NewAPIHandler(handler.APIHandlerConfig{
           Name:       "my-provider",
           APIURL:     "https://api.my-provider.com/scrape",
           APIKey:     cfg.MyProviderAPIKey,
           AuthHeader: "Authorization",  // or "X-API-Key", "Bearer", etc.
       }))
   }
   

4. Add env var to config.go: MyProviderAPIKey: os.Getenv("MY_PROVIDER_API_KEY")

API keys always come from environment variables — never hardcoded. The handler only activates when its key is set.

Extending the Chain

Implement the ScrapingHandler interface:

type ScrapingHandler interface {
    Name() string
    CanHandle(ctx context.Context, req *HandlerRequest) bool
    Scrape(ctx context.Context, req *HandlerRequest) (*ScrapeResult, error)
    IsHealthy() bool
}

See examples/custom-handler/.

Configuration

All configuration via environment variables:

Project Structure

anakinscraper-oss/
├── server/                     # Go server (API + workers)
│   ├── cmd/server/             # Entry point
│   └── internal/
│       ├── config/             # Environment config
│       ├── models/             # Data types
│       ├── worker/             # Channel-based worker pool
│       ├── handler/            # Scraping handlers (HTTP, Browser)
│       ├── converter/          # HTML → Markdown
│       ├── gemini/             # Gemini AI JSON extraction
│       ├── domain/             # Domain configs + failure detection
│       ├── store/              # Job storage (PostgreSQL or in-memory)
│       ├── proxy/              # Proxy pool + Thompson Sampling
│       ├── telemetry/          # Anonymous usage telemetry
│       ├── processor/          # Job processing
│       └── http/
│           ├── handlers/       # API request handlers
│           └── router/         # Route registration
├── browser-service/            # Camoufox anti-detect browser server
├── webapp/                     # React web dashboard (Vite + Tailwind)
├── openclaw-skill/             # OpenClaw skill wrapper
├── examples/                   # Usage examples
├── docker-compose.yml          # Full stack (3 containers)
├── scripts/init-db.sql         # Database schema
└── .env.example                # Config template

Development

Running Locally (without Docker)

Minimal (Go only):

cd server && go run cmd/server/main.go

No database, no browser service. HTTP handler scrapes static sites. Jobs stored in memory.

Full local stack (Go + Python + PostgreSQL):

# Terminal 1: PostgreSQL
docker compose up postgres -d

# Terminal 2: Browser Service (for JS-heavy sites)
cd browser-service && pip install -r requirements.txt && python server.py

# Terminal 3: Server with persistence
cd server && DATABASE_URL="postgres://postgres:postgres@localhost:5432/anakinscraper?sslmode=disable" go run cmd/server/main.go

Running Tests

cd server && go test ./...

Building

cd server && go build -o server ./cmd/server

CLI

Use the Anakin CLI to scrape from your terminal. It works against both self-hosted and the hosted API:

# Install
pip install anakin-cli

# Scrape via your local instance (no API key needed)
anakin scrape "https://example.com" --api-url http://localhost:8080

# Or set it as your default
export ANAKIN_API_URL="http://localhost:8080"
anakin scrape "https://example.com"

# Extract structured JSON
anakin scrape "https://example.com" --format json --api-url http://localhost:8080

# Batch scrape
anakin scrape-batch "https://example.com" "https://httpbin.org/html" --api-url http://localhost:8080

The same CLI also supports AI web search and deep research via the hosted API — get a free API key to unlock those features.

See the anakin-cli repo for full usage.

Integrations

OpenClaw Skill

Use AnakinScraper as an OpenClaw skill:

cp -r openclaw-skill ~/.openclaw/workspace/skills/anakinscraper

See openclaw-skill/SKILL.md.

Self-Hosted vs Hosted

This repo gives you the full scraping engine. anakin.io adds the infrastructure you’d otherwise build yourself:

Already self-hosting? Switch to hosted with one line — same API, same CLI:

anakin login --api-key "ak-xxx"   # get your key at anakin.io/dashboard
anakin scrape "https://example.com"  # now routes through hosted

Try it free →

Community

Join our Discord for questions, discussion, and support.

Contributing

See CONTRIBUTING.md.

License

AGPL-3.0 — free to self-host. If you modify the code and offer it as a hosted service, you must open-source your changes.

Built by Anakin-Inc. If you find this useful, give us a star!