AGmind

AGMind

Private RAG platform for NVIDIA DGX Spark — one command, production-style, validated on real hardware

🔗 Running AMD Strix Halo / x86_64 instead? → AGmind64

Quick Start · Architecture · Cluster · Operations · Troubleshooting · 🇷🇺 Русский

[!IMPORTANT] Supported platform: aarch64 / NVIDIA GB10 (DGX Spark and equivalents). Since 2026-04-25, x86_64 support has been removed — install.sh exits 1 on other architectures. Override: AGMIND_ALLOW_AMD64=true (no guarantees — NGC vLLM and Docling cu130 ship arm64-only manifests).

📖 Overview

AGMind is a one-command installer for a private RAG platform on NVIDIA DGX Spark (GB10, 128 GB unified memory). It deploys 30+ containers via Docker Compose: Dify + vLLM + Weaviate/Qdrant + RAGFlow + Docling + monitoring, with an interactive wizard, hardware autodetection, and optional dual-Spark clustering over 200G QSFP.

sudo bash install.sh

Built for: DevOps engineers, ML teams, and IT departments running a private AI stack on DGX Spark hardware — no vendor lock-in, no cloud egress.

Proof of work

Not slideware — validated on real DGX Spark (GB10) hardware:

• 📋 Release validation — docs/release-validation/v3.2.0-spark-3eac.md: full v3.2.0 deploy validated on a GB10 Spark.
• 📊 Benchmarks — TTFT 183 ms · 23–24 tok/s single-stream · 50 tok/s @ 3 concurrent (gemma-4-26B, vLLM, fp8 KV); full table in the Benchmarks section below.
• ▶️ Reproducible demo — scripts/demo/agmind-quickstart-sample.md: end-to-end quickstart transcript.
• 🔒 Security model — docs/architecture/security-zones.md: LAN-only zones backing the zero-egress claim (UFW / fail2ban / SSRF proxy).

Why AGMind

• ⚡ One command, ~30 minutes to a working stack — wizard → configs → image pull → start → admin user → final URL screen. No manual YAML.
• 🔒 Local models, zero data egress — gemma-4-26B (vLLM) + bge-m3 + bge-reranker run on the GB10. Documents and queries stay on your hardware.
• 🛡️ Production hardening — UFW + fail2ban + Authelia (optional 2FA), 30+ Linux capabilities dropped, rate limiting, SSRF proxy, secret rotation.
• 🧠 GB10 unified memory aware — calibrated mem_limit and gpu_memory_utilization for the 121 GiB shared pool; mDNS via avahi for .local resolution; NAT-on-demand for air-gapped peer Spark.
• 🔧 Day-2 CLI — agmind status / health / backup / update / ragflow / docling bench / plugin-daemon / mdns-status — operations without docker knowledge.

💻 Hardware Requirements

[!WARNING] AGMind targets DGX Spark / GB10 unified memory only. Anything else is unsupported.

[!CAUTION] Do not upgrade NVIDIA driver past 580.x on Spark. Three independent regressions on GB10 unified memory: CUDAGraph capture deadlock, UMA memory leak (~80 GiB ghost), and Blackwell TMA bug in 595.58.03. NVIDIA staff: “we do not support new drivers past 580.126.09 on Spark”. Pin: apt-mark hold nvidia-driver-580-open.

🚀 Quick Start

git clone https://github.com/botAGI/AGmind.git
cd AGmind
sudo bash install.sh

The wizard asks 10–15 questions depending on choices (stack mode, LLM model, optional services, security toggles, monitoring). After ~25 minutes the stack is live.

Endpoints (mDNS — no DNS server needed)

[!NOTE] All credentials live in /opt/agmind/credentials.txt (chmod 600, root-only).

Language

The wizard is bilingual (English / Russian). Language selection works as follows:

1. Interactive: the wizard’s first question is “Language / Язык” — prefilled with the autodetected value. Answer en or ru.
2. Env override: set AGMIND_LANG=en or AGMIND_LANG=ru before running install.sh / agmind to force a language. Takes precedence over locale.
3. Autodetect: if AGMIND_LANG is not set, the system locale (LC_ALL → LC_MESSAGES → LANG) is checked; a value starting with ru resolves to Russian, everything else to English (default en).

# Force Russian
sudo AGMIND_LANG=ru bash install.sh

# Force English (also the default when locale is not ru_*)
sudo AGMIND_LANG=en bash install.sh

Non-Interactive Install

sudo NON_INTERACTIVE=true \
     LLM_MODEL=gemma-4-26b \
     EMBED_PROVIDER=vllm EMBEDDING_MODEL=bge-m3 \
     ENABLE_RAGFLOW=true \
     bash install.sh

📦 What’s Included

Core Stack

Service Registry (v3.2.0+)

• Source of truth: templates/services/registry.yaml — declarative catalog (image, profiles, group, healthcheck spec, mem_limit per service).
• Codegen artifact: lib/_registry.indexed.sh is generated from registry.yaml via make registry-codegen. Do not hand-edit — the registry-verify CI gate fails on drift. See ADR-0012.

RAGFlow Integration

• RAGFlow v0.24.1-spark — deep document parsing + retrieval, image ar2r223/ragflow-spark:v0.24.1-spark (cherry-picked TitleChunker / TokenChunker / 7 ingestion templates from upstream main + multilingual OCR Latin/Cyrillic/Chinese, file metadata in ES chunks, AVIF, Russian VLM prompts for image describe).
• Dify ↔ RAGFlow via witmeng/ragflow-api plugin from Dify Marketplace (8K+ installs).
• Storage: MySQL + Elasticsearch 9.x + MinIO (S3-compatible).
• Toggle: ENABLE_RAGFLOW=true in wizard or env.

Monitoring & Ops

• Prometheus + Grafana — 10 dashboards (overview, containers, GPU master, GPU worker, peer-worker, logs, alerts, audit, RAG, RAGFlow). Custom textfile collector for agmind_gpu_* metrics (NVML returns N/A on GB10 unified memory — dcgm-exporter does not work).
• Loki + Grafana Alloy (Promtail → Alloy migration, 2026-04). Searchable container logs.
• Alertmanager — Telegram / webhook channels.
• Portainer 2.39.1 — visual container management (master + auto-deployed agent on peer Spark).
• fail2ban + UFW — bruteforce protection, LAN-only firewall by default.

Optional Services

🏗 Architecture

                                Clients (LAN)
                                      │
                                      ▼  mDNS resolution (*.local → 192.168.x.x)
       ┌─────────────────────────────────────────────────────────────────────┐
       │  nginx — variable-form proxy_pass · agmind-*.local server-blocks    │
       │           :80  :443  :3000  :4001 LiteLLM                           │
       └────┬────────┬─────────┬───────────┬──────────────┬──────────────────┘
            │        │         │           │              │
   agmind-dify   agmind-rag  /litellm  /storage     agmind-chat (opt)
   .local (Dify) .local      .local    .local       .local (Open WebUI)
            │        │
            ▼        ▼
       ┌────────────────────────────────────────────────────────────────────┐
       │  Dify (api · worker · web · sandbox · plugin_daemon)               │
       │  RAGFlow (ragflow + mysql + ES) + Dify plugin witmeng/ragflow-api  │
       └────────┬───────────┬───────────┬──────────────────┬────────────────┘
                │           │           │                  │
                ▼           ▼           ▼                  ▼
            Postgres     Redis      Weaviate           MinIO (S3-compat)
            metadata    queues       vectors           agmind-storage.local

  ─── ML inference on GB10 unified memory (121 GiB pool) ───────────────────
   vLLM-embed (NGC 26.02-py3) :8001  bge-m3 1024-dim
   vLLM-rerank (NGC 26.02-py3) :8002  bge-reranker-v2-m3
   Docling-serve cu130        :8765  PDF/DOCX/PPTX → MD + OCR + VLM
   vLLM gemma-4-26B-A4B (cu130)
       single-Spark → shares GPU above, wizard asks ctx 32K/64K/128K
       dual-Spark   → peer 192.168.100.2:8000, dedicated GPU, 128K default

  ─── Monitoring (always on) ────────────────────────────────────────────────
   Prometheus :9090 → Grafana :3001 (10 dashboards)
   Loki + Grafana Alloy (Promtail migrated 2026-04)
   Alertmanager → Telegram / Webhook
   Portainer :9443 — master + auto-deployed agent on peer:9001
   node-exporter + cAdvisor — both nodes; agmind_gpu_* via textfile collector

  ─── Docker networks ───────────────────────────────────────────────────────
   agmind-frontend — nginx ↔ web UIs · Grafana · Portainer
   agmind-backend  — all services east-west
   ssrf-network    — isolated: Dify Sandbox ↔ Squid proxy

[!TIP] For rendered diagrams see docs/architecture/ — service topology, data-flow, and network/security zones (Mermaid). The ASCII above is the quick reference.

Repository Layout

agmind/
├── install.sh                   # Main orchestrator (11 phases)
├── lib/                         # 16 modules: wizard, config, compose, health, security, detect, …
├── scripts/                     # Day-2 CLI: agmind, update, backup, restore, mdns-status, docling-bench, gpu-metrics
├── templates/                   # docker-compose.yml, docker-compose.worker.yml, nginx.conf, env templates, versions.env
├── monitoring/                  # Prometheus, Grafana dashboards, Loki, Alloy, Alertmanager
├── tests/                       # unit + integration + compose manifest tests (run via tests/run_all.sh)
├── docs/                        # Detailed documentation
└── branding/                    # Logo + theme

Docker Networks

Install Phases

🌐 Cluster (Dual-Spark)

AGMind supports a two-node configuration: master + peer over QSFP 200G DAC.

  ┌─────────────────────┐                       ┌─────────────────────┐
  │  spark-master       │   QSFP 200G DAC       │  spark-peer         │
  │  (frontend + DB +   │ ◄──── direct link ───►│  (vLLM + heavy GPU  │
  │  Dify + RAGFlow +   │   192.168.100.0/24    │  workloads)         │
  │  monitoring)        │                       │                     │
  │  WAN: ethernet      │                       │  WAN: NAT via       │
  │  iptables MASQUERADE│ ────── default gw ────►│  master QSFP        │
  └─────────────────────┘                       └─────────────────────┘

[!TIP] Adding peer to master Portainer (one-time manual step): Open https://<master-ip>:9443 → Environments → Add → Agent → URL=<peer_ip>:9001, SECRET from credentials.txt.

⚙️ Configuration

All settings live in /opt/agmind/docker/.env (chmod 600). The wizard populates everything; no manual edits required.

LLM Provider

vLLM is the default and only first-class choice on GB10. Ollama exists as a hidden override (LLM_PROVIDER=ollama) but is gated behind a Compose profile.

The Spark wizard offers three paths for vLLM model selection:

1. Gemma 4 26B-A4B (NVIDIA playbook default — recommended)
2. Curated list — Qwen / Llama / Mistral / phi-4 with VRAM hints
3. Custom HuggingFace model — input field, e.g. meta-llama/Llama-3.1-70B-Instruct

[!NOTE] On dual-Spark the context question is skipped — peer has dedicated GPU → 128K default. On single-Spark the wizard asks 32K / 64K / 128K because vLLM shares GPU with docling.

Optional Service Toggles

ENABLE_OPENWEBUI, ENABLE_LITELLM, ENABLE_DOCLING, ENABLE_SEARXNG, ENABLE_DBGPT, ENABLE_CRAWL4AI, ENABLE_RAGFLOW, ENABLE_AUTHELIA, ENABLE_DIFY_PREMIUM, ENABLE_MINIO. All set by the wizard; override via env for non-interactive installs.

Image Versions

All image tags pinned in templates/versions.env. The :latest tag is forbidden. Each tag must have an arm64 manifest verified via docker manifest inspect.

bash tests/compose/test_image_tags_exist.sh   # CI test

🛠 Operations

agmind <cmd> is installed to PATH. Run agmind help for the full list.

Status & Diagnostics

agmind status [--json] [--watch] [--service <name>]   # Stack overview table
agmind doctor [--peer] [--json] [--fix [--dry-run]] [--bundle]   # System diagnostics
agmind health                        # Alias for doctor
agmind logs [-f] <service>           # Tail container logs
agmind mdns-status                   # Verify avahi publishing for *.local
agmind troubleshoot <topic>          # Print the matching docs/troubleshooting.md section
agmind security audit [--json]       # Read-only scan: exposed ports / privileged / docker.sock / weak secrets
agmind config validate [--json]      # Static check: .env / versions↔manifest / compose schema
agmind config diff                   # Pinned-vs-target update preview (read-only)

Access & Credentials

agmind open <svc>|--list             # Open a service URL (headless/SSH → prints the URL, pipeable)
agmind endpoints [--json]            # List all public service URLs (SERVICE | URL | STATE)
sudo agmind creds show [--show] [--json]   # Show stack credentials (root; masked unless --show)
sudo agmind creds rotate             # Regenerate passwords / keys (wraps rotate_secrets.sh)

Profiles & Sizing

agmind profiles [--json]             # The 8 named deployment profiles + the active one
agmind estimate [<profile>] [--json] # RAM / disk / GPU-mem estimate for a profile vs available

# Install controls (install.sh):
sudo bash install.sh --profile rag          # Pick a named profile non-interactively
sudo bash install.sh --dry-run              # Print the phase plan, change nothing
sudo bash install.sh --resume-from <phase>  # Re-run from a given install phase

Lifecycle

agmind stop                          # Stop all containers
agmind start                         # Start configured services
agmind restart                       # Restart all
agmind upgrade --diff                # Compare pinned versions vs running
agmind update [--check|--auto]       # Update stack from main branch

State management & upgrades

AGmind v3.2.0 introduces a versioned state store at /var/lib/agmind/state/ — schema-versioned, atomically-writable, with flock per-file locking and auto-backup before destructive operations. The agmind upgrade CLI is the operator interface.

# Read-only — report current schema, pending migrations, config diff
sudo agmind upgrade --check

# Apply pending migrations atomically (creates tar-backup at state.bak.<timestamp>/)
sudo agmind upgrade --apply

# Restore from auto-backup (specify target schema_version)
sudo agmind upgrade --rollback 0

Schema versions are monotonic integers. Migrations live under lib/migrations/NNN-<name>.sh and are applied in numeric order. The migration framework is idempotent (safe to re-run) and non-destructive of legacy state — old .preserved files and legacy ${INSTALL_DIR}/docker/.env remain in place for one full release cycle after migration. See ADR-0011 for the substrate design.

[!IMPORTANT] Re-running install.sh on an existing host no longer regenerates secrets (closes the v3.1.x BACKUP-01 regression). Existing secrets are preserved via the state store; only new keys (added in versions.env between releases) are generated.

GPU & Models

agmind gpu status                    # Loaded models, VRAM, utilization
agmind gpu assign <svc> <id>         # Pin service to GPU id
agmind model list                    # All loaded models (vLLM endpoints)

RAGFlow

agmind ragflow status                # 3 ragflow containers state
agmind ragflow query <text>          # Test retrieval
agmind ragflow es-health             # Elasticsearch cluster health

Plugin Daemon & Marketplace

agmind plugin-daemon status          # State + health
agmind plugin-daemon stop|start      # Toggle (root) — Dify plugins stop working when off
agmind plugin-daemon logs            # Tail logs

agmind plugins status                # ONLINE / OFFLINE
agmind plugins online                # Enable marketplace.dify.ai (default)
agmind plugins offline               # Local .difypkg only (supply-chain hardened)

Performance & Demo

agmind loadtest list                 # k6 scenarios
agmind loadtest chat --vus 8         # Concurrent chat load test
agmind docling bench <pdf>           # Cold/warm/per-page timing for any PDF
agmind demo install|ingest|ask       # ~5-min RAG demo: sample workflow + KB + bundled doc → answer with citations

Backup & Restore

sudo agmind backup create [--include-models]   # PostgreSQL + Redis + volumes (+ vLLM cache)
agmind backup list                             # DATE / SIZE / STATUS
agmind backup verify [latest|<dir>] [--json]   # Integrity check; exit 0 = valid, 1 = corrupt/incomplete
sudo agmind restore [latest|<dir>] [--dry-run] [--service <name>]   # Restore (--dry-run prints the plan)
sudo agmind rotate-secrets           # Regenerate passwords/keys
sudo agmind uninstall [--keep-models]  # Remove stack

📁 Server Layout

/opt/agmind/
├── docker/
│   ├── .env                         # Secrets and config (chmod 600)
│   ├── docker-compose.yml           # All services
│   ├── nginx/nginx.conf             # Reverse proxy
│   └── volumes/                     # Postgres, Redis, vectors, models, MinIO
├── credentials.txt                  # All passwords (chmod 600)
├── scripts/                         # Day-2 CLI + lib modules mirrored here (doctor/status/config/restore/security/…)
├── templates/                       # init SQL, env templates
├── monitoring/                      # Prometheus rules, Grafana dashboards
├── docs/                            # architecture/ · adr/ · compatibility & decision matrices · troubleshooting · operations
├── .install-phases.jsonl            # Per-phase install record (durations / errors)
├── install-report.json              # Machine-readable install summary
└── install.log                      # Full install transcript

And outside /opt/agmind/:

/var/lib/agmind/
└── state/                           # Versioned state store (v3.2.0+)
    ├── schema_version               # Monotonic integer (chmod 0600 root:root)
    ├── secrets.env                  # Migrated secrets (chmod 0600 root:root)
    ├── .locks/                      # flock per-key locks
    └── state.bak.<timestamp>/       # Auto-backups (3 generations retained)

/var/lib/agmind/state/ is 0700 root:root and tarballed by agmind backup create alongside /opt/agmind/. The state store survives agmind uninstall unless --purge-state is passed.

🩺 Troubleshooting

[!TIP] Detailed runbooks for known gotchas live in docs/. Open an issue if you hit something missing — we backfill the runbook.

✅ Definition of Done

A change is only complete when these checks pass green:

# Bash hygiene
shellcheck -S warning lib/*.sh scripts/*.sh install.sh

# Compose schema + image existence
cd /opt/agmind/docker && sudo docker compose config | grep 'image:' | sort -u
bash tests/compose/test_image_tags_exist.sh core/compose.yml

# Live health
sudo docker ps --format '{{.Names}} {{.Status}}' | grep -v 'healthy\|Up'   # must be empty
avahi-resolve -n agmind-dify.local                                          # must resolve
curl -sf http://agmind-dify.local/console/api/setup                         # must 200

All checks must pass before a PR is mergeable.

📊 Benchmarks

Results on NVIDIA DGX Spark (GB10, 128 GB unified memory):

Docling (5-page arxiv PDF, warm): 6.04s, 0.32s/page, ~1.6 GiB GPU memory.

🗺 Roadmap & Status

• v3.2.0 (current) — architecture release, infrastructure-only (zero new images, zero new daemons): versioned state store (/var/lib/agmind/state/), declarative service registry (templates/services/registry.yaml) with build-time codegen, byte-exact golden tests (tests/golden/), and namespace-reserving Go scaffolding. Closes the v3.2.0 milestone (58 REQ-IDs across 7 phases). See CHANGELOG.
• v3.1.2 — hotfix release: 9 critical/high-severity findings, each shipped behind a regression test in tests/unit/.
• v3.1.1 — release housekeeping: version strings synced v3.0.2 → v3.1.1 (RELEASE / install.sh VERSION / templates/release-manifest.json), versions.env header refreshed; ADR Status: Accepted lines confirmed; manifest digest regen → backlog 999.6.
• v3.1 — Day-2 UX + hardening + docs (9 phases): agmind doctor (preflight/health + --fix + sanitized --bundle), agmind status (overview table + --json/--watch), agmind open/endpoints/creds, agmind config validate/diff + install --dry-run/--resume-from + install-report.json, agmind backup verify/restore --dry-run, installer refactored into a phase engine (lib/phases.sh) + peer module, security hardening (docker-socket-proxy, airgapped mode, agmind security audit), onboarding docs (docs/architecture/ Mermaid diagrams, compatibility & decision matrices, docs/troubleshooting.md + agmind troubleshoot, docs/adr/ ADR-0001…0009, Makefile, agmind demo), profiles UX (named deployment profiles in the wizard, agmind profiles, agmind estimate).
• v3.0.2 — RAGFlow upgrade to v0.24.1-spark (TitleChunker / TokenChunker + 7 ingestion templates + Russian VLM prompts), Pipeline framework patches.
• v3.0.1 — mDNS hardening, dual-Spark cluster, master/worker wizard, NAT-on-demand peer, Portainer peer agent.
• 2026-04-26 — RAGFlow integrated via DockerHub ar2r223/ragflow-spark.
• 2026-04-25 — Yellow-zone version bumps (7/8 components arm64 re-verified). x86_64 path retired.
• Next — TUI in agmind update (toggle new tools introduced in updates), native AGmind chat plugin for Dify (file upload → KB, slash commands).

📚 Documentation

Quick navigation via CLI:

agmind troubleshoot vllm       # vLLM model not loading → docs/troubleshooting.md §1
agmind troubleshoot gpu        # CUDA not visible in container → §2
agmind troubleshoot dify       # Dify worker / tasks stuck → §4
agmind troubleshoot mdns       # mDNS / .local resolution → §6
agmind troubleshoot memory     # OOM / unified memory → §10
#  (no-arg `agmind troubleshoot` lists all topics; sections: 1 vllm · 2 gpu · 3 ragflow/es · 4 dify · 5 ports · 6 mdns · 7 model-download · 8 restore · 9 update · 10 memory)

🔮 Go Migration (roadmap — no Go code in v3.2.0)

AGmind v3.2.0 reserves the cmd/agmind/ and internal/ directory namespaces for a future Go port (v4.0+). No .go files, no go.mod, and no Go tooling ship in v3.2.0 — the placeholders are roadmap-only.

• Staged port plan and per-stage acceptance criteria: docs/ROADMAP-GO.md
• Migration intent + equivalence-proof preconditions: docs/adr/0010-go-migration-staged-port.md
• Single-binary layout (Q-07): docs/adr/0013-go-single-binary-internal-packages.md

Future Go release artifacts are aarch64-only (parent decision: ADR-0001).

🤝 Contributing

Development setup

One-time, on a contributor machine — install pre-commit hooks as a local guard that runs the same checks .github/workflows/test.yml enforces on push:

pip install pre-commit
pre-commit install --hook-type pre-commit --hook-type commit-msg

What runs (16 hooks after Phase 13):

• pre-commit stage (~3-8s warm cache): 14 existing — shellcheck, yamllint, gitleaks, markdownlint, trailing-whitespace, check-yaml, check-added-large-files, check-merge-conflict, plus the standard hygiene set; + 1 new — golden-update-guard triggers on commit-msg.
• commit-msg stage: golden-update-guard rejects any commit whose staged diff touches tests/golden/expected/** unless the message has a trailer golden-accept-reason: <≥10 graphic chars> explaining the snapshot bump. See tests/golden/UPDATE.md for the runbook.
• manual stage (opt-in): ascii-only-bash flags non-ASCII bytes in lib/*.sh, scripts/*.sh, install.sh, tests/golden/run.sh. Currently gated to manual invocation because the codebase legitimately uses RU comments (project policy) and UI glyphs in logger output; flip to pre-commit stage after a dedicated cleanup pass. Run on demand: pre-commit run --hook-stage manual ascii-only-bash.

Local regression suite (no CI required):

bash tests/run_all.sh                         # All categories — ~30s
bash tests/golden/run.sh --all                # 5 golden scenarios
make golden-test                              # alias for the above
make landmines-check                          # rendered-config landmine sweep
make help                                     # all Makefile targets

Contribution rules

• Work on main only — no feature branches, no merge commits. PRs are cut from main on demand.
• Use the Makefile task runner: make lint (shellcheck), make test (unit + integration), make compose-config, make manifest-check, make release-check. make with no target prints the list.
• Every PR must pass Definition of Done and make manifest-check (a.k.a. tests/compose/test_image_tags_exist.sh).
• Image tag bumps require docker manifest inspect <image>:<tag> | grep arm64 evidence in the commit message — LLMs hallucinate registry tags.
• Architectural decisions go in docs/adr/ as MADR-lite records; reference them from code comments instead of internal notes.

📜 License

Apache License 2.0

Copyright © 2024–2026 AGMind Contributors.

🇷🇺 Русская документация

sudo bash install.sh

git clone https://github.com/botAGI/AGmind.git
cd AGmind
sudo bash install.sh

sudo NON_INTERACTIVE=true \
     LLM_MODEL=gemma-4-26b \
     EMBED_PROVIDER=vllm EMBEDDING_MODEL=bge-m3 \
     ENABLE_RAGFLOW=true \
     bash install.sh

# Read-only — текущая схема, ожидающие миграции, diff против versions.env
sudo agmind upgrade --check

# Применить ожидающие миграции атомарно (создаёт tar-бэкап в state.bak.<timestamp>/)
sudo agmind upgrade --apply

# Откат к авто-бэкапу (указать целевую schema_version)
sudo agmind upgrade --rollback 0

# Статус
agmind status [--json]
agmind doctor [--peer]
agmind logs [-f] <service>

# GPU
agmind gpu status
agmind model list

# RAGFlow
agmind ragflow status
agmind ragflow query <text>

# Plugin daemon
agmind plugin-daemon status|stop|start|restart|logs
agmind plugins status|online|offline       # marketplace.dify.ai toggle

# Производительность
agmind loadtest chat --vus 8
agmind docling bench <pdf>

# Бэкапы
sudo agmind backup
sudo agmind restore <path>
sudo agmind rotate-secrets