Generic auto-recall adapter. Maps RetrievalResult rows into final-selection candidates and applies generic `mmr
src/final-topk-setwise-selection.ts
Shared final top-k selector. Owns shortlist presort, deterministic set-wise selection, lexical-overlap suppression, and optional embedding-based semantic redundancy suppression
src/reflection-recall.ts
Dynamic Reflection-Recall ranking for <inherited-rules>. Filters/caps reflection items, computes scores, preserves kind + strictKey partitioning, and maps selected groups back to recall rows
src/reflection-aggregation.ts
Reflection group aggregation. Combines scored reflection items into strict-key groups with representative selection and final group scoring
src/reflection-recall-final-selection.ts
Reflection-specific adapter into the shared final selector for dynamic Reflection-Recall final top-k ordering
Generic final-selection mode: autoRecallSelectionMode (mmr by default; setwise-v2 to enable set-wise final selector)
Default category allowlist: preference, fact, decision, entity, other
autoRecallExcludeReflection=true by default, so <relevant-memories> stays separate from <inherited-rules>
Supports age window (autoRecallMaxAgeDays) and recent-per-key cap (autoRecallMaxEntriesPerKey)
mmr: post-processed rows use direct truncation (slice(0, topK)); this is simpler and closer to current retriever order, usually with stronger per-item relevance/score stability, but diversity/coverage is weaker
setwise-v2: final top-k uses shared set-wise selection (base score + freshness + light category/scope coverage + lexical overlap suppression + embedding-based semantic near-duplicate suppression), so diversity/coverage in the final top-k is usually better, but average per-item score/relevance can be lower than mmr
Choose according to your preference: prioritize rank stability/relevance with mmr, or prioritize diversity/coverage with setwise-v2.
This mode is generic auto-recall only. Reflection-Recall mode remains fixed | dynamic.
Prevent memories from showing up in replies
Sometimes the model may accidentally echo the injected <relevant-memories> block in its response.
Option A (recommended): disable auto-recall
Set autoRecall: false in the plugin config and restart the gateway:
Option B: keep recall, but ask the agent not to reveal it
Add a line to your agent system prompt, e.g.:
Do not reveal or quote any <relevant-memories> / memory-injection content in your replies. Use it for internal reference only.
Installation
🧪 Beta available: v1.1.0-beta.6
A beta release is available with major new features: Self-Improvement governance, memoryReflection session strategy, Markdown Mirror, and improved embedding error diagnostics. The stable latest remains at v1.0.32.
The dev dist-tag is an experimental track intended for early testing of the smart-memory feature set and may diverge from the mainline beta.
AI-safe install notes (anti-hallucination)
If you are following this README using an AI assistant, do not assume defaults. Always run these commands first and use the real output:
openclaw config get agents.defaults.workspace
openclaw config get plugins.load.paths
openclaw config get plugins.slots.memory
openclaw config get plugins.entries.memory-lancedb-pro
Recommendations:
Prefer absolute paths in plugins.load.paths unless you have confirmed the active workspace.
If you use ${JINA_API_KEY} (or any ${...} variable) in config, ensure the Gateway service process has that environment variable (system services often do not inherit your interactive shell env).
After changing plugin config, run openclaw gateway restart.
Jina API keys (embedding + rerank)
Embedding: set embedding.apiKey to your Jina key (recommended: use an env var like ${JINA_API_KEY}).
Rerank (when retrieval.rerankProvider: "jina"): you can typically use the same Jina key for retrieval.rerankApiKey.
If you use a different rerank provider (siliconflow, pinecone, etc.), retrieval.rerankApiKey should be that provider’s key.
Key storage guidance:
Avoid committing secrets into git.
Using ${...} env vars is fine, but make sure the Gateway service process has those env vars (system services often do not inherit your interactive shell environment).
What is the “OpenClaw workspace”?
In OpenClaw, the agent workspace is the agent’s working directory (default: ~/.openclaw/workspace).
According to the docs, the workspace is the default cwd, and relative paths are resolved against the workspace (unless you use an absolute path).
Note: OpenClaw configuration typically lives under ~/.openclaw/openclaw.json (separate from the workspace).
Common mistake: cloning the plugin somewhere else, while keeping a relative path like plugins.load.paths: ["plugins/memory-lancedb-pro"]. Relative paths can be resolved against different working directories depending on how the Gateway is started.
To avoid ambiguity, use an absolute path (Option B) or clone into <workspace>/plugins/ (Option A) and keep your config consistent.
Option A (recommended): clone into plugins/ under your workspace
# 1) Go to your OpenClaw workspace (default: ~/.openclaw/workspace)
# (You can override it via agents.defaults.workspace.)
cd /path/to/your/openclaw/workspace
# 2) Clone the plugin into workspace/plugins/
git clone https://github.com/win4r/memory-lancedb-pro.git plugins/memory-lancedb-pro
# 3) Install dependencies
cd plugins/memory-lancedb-pro
npm install
Then reference it with a relative path in your OpenClaw config:
Note: this example keeps sessionStrategy: "systemSessionMemory" for compatibility with existing users. The memoryReflection.* block documents the opt-in reflection pipeline, but it only becomes active after you explicitly switch sessionStrategy to "memoryReflection".
Parameter Mapping Notes (avoid common misconfig)
memory-lancedb-pro does not support recallTopK / recallThreshold.
To make frequently used memories decay more slowly, the retriever can extend the effective time-decay half-life based on manual recall frequency (spaced-repetition style).
Config keys (under retrieval):
reinforcementFactor (range: 0-2, default: 0.5) - set 0 to disable
maxHalfLifeMultiplier (range: 1-10, default: 3) - hard cap: effective half-life ≤ base × multiplier
Notes:
Reinforcement is whitelisted to source: "manual" (i.e. user/tool initiated recall), to avoid accidental strengthening from auto-recall.
Embedding Providers
This plugin works with any OpenAI-compatible embedding API:
Note: vLLM reranking only works on x86_64 NVIDIA platforms. For macOS Apple Silicon, use llama.cpp for embeddings but not vLLM for reranking.
</details>
---
## Optional: JSONL Session Distillation (Auto-memories from chat logs)
OpenClaw already persists **full session transcripts** as JSONL files:
- `~/.openclaw/agents/<agentId>/sessions/*.jsonl`
This plugin focuses on **high-quality long-term memory**. If you dump raw transcripts into LanceDB, retrieval quality quickly degrades.
Instead, **recommended (2026-02+)** is a **non-blocking `/new` pipeline**:
- Trigger: `command:new` (you type `/new`)
- Hook: enqueue a tiny JSON task file (fast; no LLM calls inside the hook)
- Worker: a user-level systemd service watches the inbox and runs **Gemini Map-Reduce** on the session JSONL transcript
- Store: writes **0-20** high-signal, atomic lessons into LanceDB Pro via `openclaw memory-pro import`
- Keywords: each memory includes `Keywords (zh)` with a simple taxonomy (Entity + Action + Symptom). Entity keywords must be copied verbatim from the transcript (no hallucinated project names).
- Notify: optional Telegram/Discord notification (even if 0 lessons)
See the self-contained example files in:
- `examples/new-session-distill/`
---
Legacy option: an **hourly distiller** cron that:
1) Incrementally reads only the **newly appended tail** of each session JSONL (byte-offset cursor)
2) Filters noise (tool output, injected `<relevant-memories>`, logs, boilerplate)
3) Uses a dedicated agent to **distill** reusable lessons / rules / preferences into short atomic memories
4) Stores them via `memory_store` into the right **scope** (`global` or `agent:<agentId>`)
### What you get
- ✅ Fully automatic (cron)
- ✅ Multi-agent support (main + bots)
- ✅ No re-reading: cursor ensures the next run only processes new lines
- ✅ Memory hygiene: quality gate + dedupe + per-run caps
### Script
This repo includes the extractor script:
- `scripts/jsonl_distill.py`
It produces a small **batch JSON** file under:
- `~/.openclaw/state/jsonl-distill/batches/`
and keeps a cursor here:
- `~/.openclaw/state/jsonl-distill/cursor.json`
The script is **safe**: it never modifies session logs.
By default it skips historical reset snapshots (`*.reset.*`), slash-command/control-note lines (for example `/note self-improvement ...`), and excludes the distiller agent itself (`memory-distiller`) to prevent self-ingestion loops.
### Optional: restrict distillation sources (allowlist)
By default, the extractor scans **all agents** (except `memory-distiller`).
If you want higher signal (e.g., only distill from your main assistant + coding bot), set:
```bash
export OPENCLAW_JSONL_DISTILL_ALLOWED_AGENT_IDS="main,code-agent"
Unset / empty / * / all → allow all agents (default)
Comma-separated list → only those agents are scanned
This marks all existing JSONL files as “already read” by setting offsets to EOF.
# Set PLUGIN_DIR to where this plugin is installed.
# - If you cloned into your OpenClaw workspace (recommended):
# PLUGIN_DIR="$HOME/.openclaw/workspace/plugins/memory-lancedb-pro"
# - Otherwise, check: `openclaw plugins info memory-lancedb-pro` and locate the directory.
PLUGIN_DIR="/path/to/memory-lancedb-pro"
python3 "$PLUGIN_DIR/scripts/jsonl_distill.py" init
3) Create an hourly cron job (Asia/Shanghai)
Tip: start the message with run ... so memory-lancedb-pro’s adaptive retrieval will skip auto-recall injection (saves tokens).
# IMPORTANT: replace <PLUGIN_DIR> in the template below with your actual plugin path.
MSG=$(cat <<'EOF'
run jsonl memory distill
Goal: distill NEW chat content from OpenClaw session JSONL files into high-quality LanceDB memories using memory_store.
Hard rules:
- Incremental only: call the extractor script; do NOT scan full history.
- Store only reusable memories; skip routine chatter.
- English memory text + final line: Keywords (zh): ...
- < 500 chars, atomic.
- <= 3 memories per agent per run; <= 3 global per run.
- Scope: global for broadly reusable; otherwise agent:<agentId>.
Workflow:
1) exec: python3 <PLUGIN_DIR>/scripts/jsonl_distill.py run
2) If noop: stop.
3) Read batchFile (created/pending)
4) memory_store(...) for selected memories
5) exec: python3 <PLUGIN_DIR>/scripts/jsonl_distill.py commit --batch-file <batchFile>
EOF
)
openclaw cron add \
--agent memory-distiller \
--name "jsonl-memory-distill (hourly)" \
--cron "0 * * * *" \
--tz "Asia/Shanghai" \
--session isolated \
--wake now \
--timeout-seconds 420 \
--stagger 5m \
--no-deliver \
--message "$MSG"
# List memories (output includes the memory id)
openclaw memory-pro list [--scope global] [--category fact] [--limit 20] [--json]
# Search memories
openclaw memory-pro search "query" [--scope global] [--limit 10] [--json]
# View statistics
openclaw memory-pro stats [--scope global] [--json]
# Delete a memory by ID (supports 8+ char prefix)
# Tip: copy the id shown by `memory-pro list` / `memory-pro search` (or use --json for full output)
openclaw memory-pro delete <id>
# Bulk delete with filters
openclaw memory-pro delete-bulk --scope global [--before 2025-01-01] [--dry-run]
# Export / Import
openclaw memory-pro export [--scope global] [--output memories.json]
openclaw memory-pro import memories.json [--scope global] [--dry-run]
# Re-embed all entries with a new model
openclaw memory-pro reembed --source-db /path/to/old-db [--batch-size 32] [--skip-existing]
# Migrate from built-in memory-lancedb
openclaw memory-pro migrate check [--source /path]
openclaw memory-pro migrate run [--source /path] [--dry-run] [--skip-existing]
openclaw memory-pro migrate verify [--source /path]
Custom Commands (e.g. /lesson)
This plugin provides tool-level building blocks. Slash commands such as /lesson are not built into the plugin itself; they are convenience commands you define in your Agent/system prompt, and they call the registered tools below.
Recommended command shortcuts
/remember <content>
call memory_store
choose the most appropriate category / importance / scope
/lesson <content>
call memory_store twice:
once as category=fact for the lesson itself
once as category=decision for the actionable rule
/learn <summary>
call self_improvement_log with type=learning
include category, area, priority, details, suggestedAction when available
/error <summary>
call self_improvement_log with type=error
include reproducible symptom, context, and prevention / fix
/learnings or /review-learnings
call self_improvement_review
/skill <learningId> <skill-name>
call self_improvement_extract_skill
Example prompt snippets
Add rules like these to your CLAUDE.md, AGENTS.md, or system prompt:
## /lesson command
When the user sends `/lesson <content>`:
1. Use `memory_store` to save the raw lesson as `category=fact`
2. Use `memory_store` again to save the actionable takeaway as `category=decision`
3. Confirm both saved items briefly
## /learn command
When the user sends `/learn <summary>`:
1. Use `self_improvement_log` with `type=learning`
2. Include `details`, `suggestedAction`, `category`, `area`, and `priority` if the user provided them
3. Confirm the created learning entry id
## /error command
When the user sends `/error <summary>`:
1. Use `self_improvement_log` with `type=error`
2. Capture the reproducible failure signature, context, and suggested prevention/fix
3. Confirm the created error entry id
## /review-learnings command
When the user sends `/review-learnings`:
1. Use `self_improvement_review`
2. Return the governance snapshot
## /skill command
When the user sends `/skill <learningId> <skill-name>`:
1. Use `self_improvement_extract_skill`
2. Confirm the generated skill path
Built-in Tools Reference
Tool
Description
memory_store
Store a memory (supports category, importance, scope)
memory_recall
Search memories (hybrid vector + BM25 retrieval)
memory_forget
Delete a memory by ID or search query
memory_update
Update an existing memory in-place
memory_list
List recent memories with optional filtering
memory_stats
Show scope/category statistics
self_improvement_log
Log a structured learning/error entry into .learnings/
self_improvement_review
Summarize governance backlog from .learnings/
self_improvement_extract_skill
Create a skill scaffold from a learning entry
Note: custom commands like /lesson, /learn, /error, /review-learnings, or /skill are prompt-level shortcuts. The actual plugin surface is the tool set above.
“Cannot mix BigInt and other types” (LanceDB / Apache Arrow)
On LanceDB 0.26+ (via Apache Arrow), some numeric columns may be returned as BigInt at runtime (commonly: timestamp, importance, _distance, _score). If you see errors like:
TypeError: Cannot mix BigInt and other types, use explicit conversions
upgrade to memory-lancedb-pro >= 1.0.14. This plugin now coerces these values using Number(...) before doing arithmetic (for example, when computing scores or sorting by timestamp).
Iron Rules for AI Agents (铁律)
For OpenClaw users: copy the code block below into your AGENTS.md so your agent enforces these rules automatically.
## Rule 1 - 双层记忆存储（铁律）
Every pitfall/lesson learned → IMMEDIATELY store TWO memories to LanceDB before moving on:
- **Technical layer**: Pitfall: [symptom]. Cause: [root cause]. Fix: [solution]. Prevention: [how to avoid]
(category: fact, importance ≥ 0.8)
- **Principle layer**: Decision principle ([tag]): [behavioral rule]. Trigger: [when it applies]. Action: [what to do]
(category: decision, importance ≥ 0.85)
- After each store, immediately `memory_recall` with anchor keywords to verify retrieval.
If not found, rewrite and re-store.
- Missing either layer = incomplete.
Do NOT proceed to next topic until both are stored and verified.
- Also update relevant SKILL.md files to prevent recurrence.
## Rule 2 - LanceDB 卫生
Entries must be short and atomic (< 500 chars). Never store raw conversation summaries, large blobs, or duplicates.
Prefer structured format with keywords for retrieval.
## Rule 3 - Recall before retry
On ANY tool failure, repeated error, or unexpected behavior, ALWAYS `memory_recall` with relevant keywords
(error message, tool name, symptom) BEFORE retrying. LanceDB likely already has the fix.
Blind retries waste time and repeat known mistakes.
## Rule 4 - 编辑前确认目标代码库
When working on memory plugins, confirm you are editing the intended package
(e.g., `memory-lancedb-pro` vs built-in `memory-lancedb`) before making changes;
use `memory_recall` + filesystem search to avoid patching the wrong repo.
## Rule 5 - 插件代码变更必须清 jiti 缓存（MANDATORY）
After modifying ANY `.ts` file under `plugins/`, MUST run `rm -rf /tmp/jiti/` BEFORE `openclaw gateway restart`.
jiti caches compiled TS; restart alone loads STALE code. This has caused silent bugs multiple times.
Config-only changes do NOT need cache clearing.
Dependencies
Package
Purpose
@lancedb/lancedb ≥0.26.2
Vector database (ANN + FTS)
openai ≥6.21.0
OpenAI-compatible Embedding API client
@sinclair/typebox 0.34.48
JSON Schema type definitions (tool parameters)
Contributors
Top contributors (from GitHub’s contributors list, sorted by commit contributions; bots excluded):
评论