ScienceClaw

Skills and Operators that grow from replay-verified executions.

Quick start  ·  How it works  ·  Results  ·  Gateway  ·  Benchmark  ·  Paper  ·  Design

_{ScienceClaw-Eval spans 23 disciplines across the natural and social sciences, while ScienceClaw turns verified execution evidence into persistent Skill–Operator program updates.}

23 disciplines across the natural and social sciences

+16.45 % mean out-of-distribution gain over the frozen agent

98.23 % of instances pass every scientific hard constraint

91.30 OOD macro success rate after seven rounds (from 77.72)

ScienceClaw is an agent system for scientific work that gets better the more it is used. It solves each task as a typed, executable workflow. When a repair is reproduced under a clean replay, it becomes a linked Skill (strategy) and Operator (a typed, executable capability), and it is kept only if it still solves its source task and improves independent validation tasks.

✨ Highlights

🚀 Quick start

git clone https://github.com/beita6969/ScienceClaw.git
cd ScienceClaw

# One-click setup: Node, Python, the engine, its tools and weights, MCP servers, skills
chmod +x setup.sh && ./setup.sh

pnpm install && npx openclaw onboard
pip install -e packages/scienceclaw          # Python >= 3.11; extras: [all], vision, audio, nlp, forecast, materials, retrieval
python -m scienceclaw.cli setup              # install and verify every tool and pretrained weight (--profile light skips assets > 1.5 GB)
python -m scienceclaw.cli doctor             # which tool modules can run on this machine, and why not

export SCIENCECLAW_API_BASE_URL=<your OpenAI-compatible endpoint>
export SCIENCECLAW_API_KEY=<your key>
export SCIENCECLAW_MODEL=<your model name>      # or SCIENCECLAW_<POLICY|EXECUTOR|PATCH>_MODEL per role

🧠 How it works

The problem

LLM agents increasingly solve scientific tasks by connecting reasoning to data, domain tools and executable code. But a repair that works once rarely survives: it lives in a transient context, or in a single tool or prompt, so verified executions seldom become persistent improvements. Existing work evolves individual tools or Skills, and existing benchmarks treat tasks as independent episodes, so it has been hard to tell whether verified scientific executions turn into persistent, transferable improvements.

How can verified scientific executions drive persistent and transferable program-level self-evolution without updating foundation-model parameters?

_{Task formulation of verifiable program-level self-evolution for AI-for-Science agents.}

The answer: replay-gated, linked Skill–Operator evolution

1. A task for program-level self-evolution. The model Θ₀ is fixed. What evolves is an editable program A_r = (Skills, Operators). A task D_t = (D_T, D_V, D_E) specifies its objective and inputs, its evaluation protocol and hard constraints (units, feasibility, convergence, reproducibility), and its data, tools and environment.

2. Typed, execution-guided workflows. A solution is a directed graph whose ports carry a schema (type, shape, unit, provenance); an edge is valid only if types and shapes match and any unit conversion is explicit and recorded. The agent edits the graph one atomic action at a time, and the executor checkpoints every node by fingerprint and reruns only the affected descendants, so a long scientific workflow is repaired rather than regenerated.

3. Evidence you can replay. Every candidate result is regenerated from a reset environment and checked against the task’s evaluator and hard constraints. A replay that fails and a later replay that passes bracket the shortest reproduced repair, e⁻ → e⁺. Replay alone never authorizes persistence.

4. Linked Skill–Operator candidates, with no LLM judge. The reproduced repair is split by edit type. Control edits (topology, routing, configuration) become a Skill patch. Generated or repaired executable nodes are grouped into convex components and abstracted into typed Operators, each checked by boundary replay in isolation. Both are committed as one atomic bundle, so a strategy never arrives without the capability behind it, nor a capability without a strategy that selects it.

5. A replay-and-validation gate. A candidate persists only if source replay reproduces the repair and actually uses every new component (R_src = Pass ∧ Use), every hard integrity and scientific check holds on independent validation tasks, its cost stays within budget, and it strictly improves the validation score over the incumbent. A noise guard requires at least two improved and at most one regressed validation episode. Otherwise the incumbent program is kept.

_{Given task specification D_t and agent program A_r, ScienceClaw produces scientific solution Z_t and retains a candidate update only after source-task replay and independent program validation.}

📊 Results

Seven evolution rounds over a common stream of 23 disciplines, with 64 IID and 64 OOD instances per discipline, one fixed foundation model, and the same tools, source stream, validation data and update budget throughout. OOD means an independently sourced dataset of the same discipline; OOD results never generate or select updates.

• Consistent gains. ScienceClaw improves on the frozen agent in every discipline: +16.45 % on average out-of-distribution (+11.57 % to +23.34 %) and +12.25 % in-distribution. Its OOD score falls 11.73 % below its IID score on average, against 16.06 % for the frozen agent.
• It keeps improving. The OOD macro success rate rises from 77.72 to 91.30 over seven rounds (+13.59 pp). 70 of 161 submitted candidates are promoted (43.48 %), so the gate is selective without stalling.
• It transfers and retains. 18 of 20 cross-family pairs transfer positively (mean +1.58 pp, against +14.69 pp within a family). Average forgetting is 0.13 pp (maximum 0.51 pp), with 11.89 % negative transfer.
• It is reliable. Hard constraints pass on 98.23 % of instances and only 1.23 % of promotions are erroneous.
• The linkage and the gate are what matter. Committing Skills and Operators separately keeps only 60 % of the gain; removing Operator evolution keeps 59 %. Without independent IID selection only 15 % remains; without scientific constraints, source replay or the independent validator, 46 %, 55 % and 59 %.
• Execution structure is the base. The full system runs 4.84 planner rounds and 4.24 distinct Operators per task, repairs 71 % of failures from feedback, recovers 89 % after interruption and replays 96 % cleanly (a single-turn agent: 0 %, 31 %, 78 %).

_{Mean of IID and OOD task-native scores of the ablation variants across 23 disciplines. Colours are normalised within each discipline (darker is better); the two groups follow higher-is-better and lower-is-better metrics.}

_{(f) Gain over the frozen agent (%) of the linked-mechanism variants on Commerce (MASE) and Law (mAP). (g) Workflow mechanics: planner rounds, distinct Operators, Operator diversity, feedback repair, checkpoint recovery and clean replay. (h) Hard-constraint pass rate against cost per OOD gain (ScienceClaw = 1), coloured by planner wall-time share. (i) Promoted candidates and promotion rate.}

Reported trajectories are final snapshots, not uncertainty estimates over source orders or model configurations, and cost comparisons are relative to the protocol, not absolute. The paper has the full tables, ablations and the limitations discussion.

🔌 Use it from the gateway

Enable the plugin in ~/.openclaw/openclaw.json (see extensions/scienceclaw/README.md). It registers four agent tools:

A session, end to end

1. Declare the task. scienceclaw_canvas(operation=open, task={objective, inputs, required_output, constraints}). Each input becomes a read-only load_<name> tool restricted to the configured input roots; constraints (finite, shape, type, range, nonempty, len_eq_input, or an evaluator-only metric bar) are the acceptance test.
2. Build the workflow. The agent finds tools with scienceclaw_tools, then adds, modifies or removes one node or edge per act, reading the typed feedback after each edit.
3. Verify. replay re-executes the whole graph from a reset state; finish returns the verified deliverable.
4. Evolve (optional). Register independent validation tasks with val_add (the default noise guard needs at least two), then propose candidates from a finished, verified session and gate them. A session that contained a repair yields a linked Skill and Operator; one without a failed replay yields an Operator only. A candidate that passes becomes ready.
5. You decide. Review and promote with the CLI; every promotion is a new program version that can be rolled back.

python -m scienceclaw.cli live candidates
python -m scienceclaw.cli live show <candidate>
python -m scienceclaw.cli live promote <candidate>
python -m scienceclaw.cli live rollback <version>

The same engine speaks line-delimited JSON on python -m scienceclaw.rpc if you want to drive it from your own host.

🧰 Skills and tools

🛡️ Safety and governance

• Nothing persists without evidence. Source replay, validation, budget and strict improvement all have to hold, and the gate fails closed when the model is unavailable.
• Updates are your decision, and reversible. Candidates wait as ready until promoted; every version is a snapshot with a receipt and can be rolled back.
• Provenance everywhere. Ports record units and upstream transformations; operators record their source episode, steps and parent version.

ScienceClaw should support, not replace, experts. High-stakes use needs provenance, licensing and privacy safeguards, and independent review.

🧪 ScienceClaw-Eval

ScienceClaw-Eval benchmarks continual self-evolution rather than single-shot ability. Systems share the foundation model, the initial program, the source order, the tools and the budget, and are compared on:

• a source stream that supplies the only evolution evidence,
• an independent validation set used for candidate selection,
• held-out ID and same-discipline cross-dataset OOD sets,
• a replay set of earlier source tasks that measures retention.

An instance counts as solved only if execution completes within budget, the task-native metric meets its acceptance rule, and every scientific hard constraint holds; success is macro-averaged over disciplines. Each task has an isolated environment and a task-specific evaluator, reviewed by domain experts and verified by reset replay.

_{Construction of ScienceClaw-Eval: scientific-task collection, executable instantiation, validation and reproduction, and lineage-aware evaluation splits.}

The evaluation data (64 IID and 64 OOD records for each discipline) is on Hugging Face: beita6969/scienceclaw-eval.

🗂️ Repository

ScienceClaw/
├── packages/scienceclaw/    # the engine: typed workflows, runtime, Skill/Operator program, evolution, tool library
│   ├── scienceclaw/         #   core, runtime, agent, canvas, program, evolution, llm, tools, rpc, cli
│   ├── scilib/              #   42 scientific tool modules
│   ├── configs/             #   default run configuration (no model, no paths)
│   ├── scripts/             #   LLM serving, GPU-tool broker and environment scripts
│   └── docs/                #   DESIGN.md (the contract) and INTEGRATION.md
├── extensions/scienceclaw/  # gateway plugin: canvas, tools, program, evolve
├── skills/                  # 300+ skills, including the seed program and the 23 discipline skills
├── mcp-servers/             # arXiv-LaTeX and ChEMBL MCP servers
├── assets/                  # banner and the paper's figures
├── SCIENCE.md               # research protocol for the gateway agent
├── setup.sh                 # one-click setup
├── src/, ui/, apps/, ...    # the OpenClaw gateway, web UI and apps
└── docs/                    # gateway documentation

Documentation: DESIGN.md (the full contract, including the decisions the paper leaves open) · INTEGRATION.md (gateway, plugin, skill and evolution map) · plugin guide · SCIENCE.md

🛠️ Reporting a problem

For bugs or setup problems, open an issue with:

• The repository revision (git rev-parse HEAD), operating system, and relevant Python, Node.js, and pnpm versions.
• The command or gateway tool operation you ran, the expected result, and the actual result.
• The smallest reproducible example and relevant error output. For tool setup problems, include the relevant output from python -m scienceclaw.cli doctor.

Remove API keys, access tokens, private data, and local credentials from examples and logs before posting.

📖 Citation

If you use ScienceClaw or ScienceClaw-Eval, please cite the paper (arXiv:2610.08691):

@misc{zhang2026scienceclaw,
  title         = {ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences},
  author        = {Zhang, Mingda and Liu, Wenjin and Shen, Tiesunlong and Xiao, Zikai and Lin, Zhenghong and Xu, Qing and Cambria, Erik and Tang, Xiaoying and Luo, Haoran},
  year          = {2026},
  eprint        = {2610.08691},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2610.08691}
}

📬 Contact and license

[email protected] · MIT, see LICENSE.