AlphaAvatar Introduction
AlphaAvatar is a self-hostable Omni Personal Assistant framework designed to evolve into an intelligent personal butler — a continuous, personalized, and proactive assistant that can remember, understand, plan, and act on behalf of the user.
It is built around a plugin-based real-time Agent architecture, combining:
- 🧠 Memory for long-term user, assistant, and tool interaction history
- 🧬 Persona for user understanding, identity continuity, and personalization
- 💡 Reflection for self-improvement and long-term behavioral adaptation
- 📅 Planning for task decomposition, reminders, and future-oriented actions
- ⚙️ Behavior for response style, workflow policy, and proactive assistance
- 🧰 Tools through MCP, RAG, DeepResearch, and external integrations
- 😊 Virtual Character for real-time voice/avatar interaction
✨ Self-hostable and privacy-first — AlphaAvatar can run on your own infrastructure, giving you control over runtime configuration, memory and tools. Selected cloud model, embedding and remote-tool providers still require their own network access and credentials; self-hosting the framework does not make those providers local.
Runtime Architecture 🧠
AlphaAvatar follows a layered realtime multimodal architecture:
- 🎙️ Native Client Inputs: realtime text, voice, camera, and screen observations from first-party clients.
- 🏠 Application Host: assembles Agent execution and RTC adapters, coordinates their lifecycle, and hosts the application entrypoint.
- 🔌 RTC Adapter: provides LiveKit transport while keeping the application boundary replaceable; other backends are not yet implemented.
- 👁️ Core Perception: normalizes typed multimodal observations, source and segment lineage, ordered perception events, annotations, timelines, and historical snapshots.
- ⚙️ Agent & Runtime: owns Turn admission, native Loop execution, shared state, capability invocation and precise interruption. Router separately owns addressing and turn-taking decisions.
- 🧩 Plugin Ecosystem: adds Memory, Persona, RAG, MCP, Character, and other capabilities.
- 🧠 Foundation & Infrastructure: exposes ContextManager, Loop, ProviderService and VoiceBundle; structured extraction and native foreground streaming retain separate backend capabilities.
- 💾 Storage & Data: stores identity, memory, vectors, traces, artifacts, and media.
- 📤 Assistant Outputs: publishes validated EnvOutput content through OutputRuntime, with separate canonical-record and execution-observation streams. Router coordinates synthesis/transcripts; RTC owns delivery and playback feedback.
What AlphaAvatar Is Designed For
1️⃣ Personal Data & Life Metrics Management
- 📊 Track and analyze personal metrics such as health, fitness, sleep, and study progress
- 📈 Provide long-term insights and trend analysis
- 🎯 Suggest improvements based on historical patterns
|
2️⃣ Knowledge & Notes Management
- 📖 Organize personal notes, documents, and knowledge
- 🔍 Retrieve relevant information through RAG
- 🧠 Build a personal knowledge base over time
|
3️⃣ Task & Event Management
- 📅 Schedule tasks and reminders
- ⏰ Proactively notify based on context and priority
- 🔄 Break down long-term goals into actionable steps
|
4️⃣ Autonomous Planning & Execution
- 🧠 Plan multi-step workflows such as learning plans, projects, and research
- 🔧 Call tools automatically to complete tasks
- 📌 Maintain consistency across long time horizons
|
5️⃣ Personalized Companion & Context Awareness
- 🧬 Understand user preferences, habits, and personality
- 💬 Provide highly personalized responses
- 🤝 Maintain continuity across conversations and modalities
|
6️⃣ External World Interaction
- 🌐 Search, research, and summarize real-world information
- 🧰 Integrate with tools such as email, databases, APIs, and messaging apps
- 🔗 Act as a bridge between user intent and external systems
|
💡 AlphaAvatar is not just a chatbot.
It is a foundation for building stateful, proactive, multimodal, and self-evolving personal AI assistants.
AlphaAvatar Plugins
AlphaAvatar organizes plugins into three layers: shared foundation services, perception and interaction capabilities, and tools for external tasks.
Available means implemented in this source tree, not necessarily enabled by default or API-stable. Planned entries are not yet available as standalone plugins.
⚙️ Foundation
Shared components that other plugins use through FoundationRuntime.
| Plugin |
Capabilities |
Status |
| Voice |
Provides voice activity detection, speech recognition, and speech synthesis.
Shares reusable voice components with Router and other consumers. |
 |
| Provider |
Provides runtime-owned clients, structured task execution, native OpenAI Responses streaming and managed tracing.
VDB workers retain their Embedding ownership. |
|
| Context |
Owns native history and captured query preparation.
Composes State, perception, Memory and Persona without per-step prefix rewrites. |
|
| Loop |
Runs native model/tool execution with bounded limits, explicit authorization and canonical record publication.
Required by AvatarEngine. |
|
FoundationRuntime owns ContextManager, Loop, ProviderService and VoiceBundle. Memory and Persona borrow structured-task gateways while retaining processor-owned configuration and business prompts. The foreground Loop explicitly selects a streaming backend; the LangChain task backend is not made stream-capable by sharing the service.
🧠 Perception
Capabilities for understanding users, maintaining memory, and coordinating interaction.
| Plugin |
Capabilities |
Status |
| Router |
Coordinates multimodal routing, addressing, turn taking, and interruption.
Combines speech, transcripts, and attention evidence for realtime interaction decisions. |
|
| Memory |
Stores durable memories from conversations, tools, and environment observations.
Retrieves relevant information to maintain continuity across interactions. |
|
| Persona |
Builds user profiles and recognizes speakers and faces.
Maintains identity continuity and supplies context for personalization. |
|
| Character |
Native AIRI browser lifecycle with scoped transport readiness and separate frontend hosting.
Consumes existing Agent audio; examples keep it disabled. Renderer/RTC validation is pending. |
 |
| Status |
Publishes readiness, model/tool activity, and delivery outcomes.
Supports optional narration without forcing generic spoken waiting messages. |
|
| Reflection |
Will review memories, behavior, and interaction history.
Will derive insights to support ongoing adaptation and improvement. |
 |
| Planning |
Will turn goals into coordinated, multi-step plans.
Will track task progress and support longer-horizon execution. |
|
| Behavior |
Will manage response style, workflow policies, and proactive behavior.
Will adapt interaction strategies to user preferences and context. |
|
Integrations for research, document access, and external actions.
| Plugin |
Capabilities |
Status |
| MCP |
Discovers, retrieves, and invokes tools exposed by MCP servers.
Connects the assistant to external services and real-world operations. |
|
| DeepResearch |
Combines web search and content extraction for multi-step research.
Builds informed responses from external information. |
|
| RAG |
Ingests documents and retrieves relevant knowledge for model responses.
Connects uploaded content to retrieval-augmented workflows. |
|
| Sandbox |
Will provide isolated execution for external actions.
Will support controlled interaction with tools and other agents. |
|
See Plugin Layout and Boundaries for development details and the Roadmap for planned work.
Latest News 🔥
-
[2026/10] Released AlphaAvatar version 0.6.8: Completed the AlphaAvatar-owned native Host and streaming Agent runtime, removed the LiveKit Agents SDK, separated RTC access from execution ownership, introduced processor-owned Persona and Memory, runtime capabilities and committed-turn consumers, synchronized PCM speech and captions, structured diagnostics, and the Python 3.13 toolchain.
-
[2026/09] Released AlphaAvatar version 0.6.7: Added AlphaAvatar-owned multimodal turn taking, local Semantic Addressing with per-speaker conversation focus, typed addressing evidence and fusion, exact annotation-driven turn snapshots, full-duplex interruption, and removed LiveKit text turn detection and the legacy acoustic invocation path.
-
[2026/08] Released AlphaAvatar version 0.6.6: Added a unified time-aligned multimodal runtime, adaptive audiovisual ENV Memory, provider-neutral model input, runtime capability awareness, time-based perception retention, participant-scoped timezone context, and date-grouped session storage.
-
[2026/07] Released AlphaAvatar version 0.6.4: Added a transport-agnostic perception runtime with typed multimodal streams, shared timelines, annotated payload views, and online ENV memory extraction from live visual observations.
- Released AlphaAvatar version 0.6.5: Added shared realtime audio perception, the Interaction Router, AlphaAvatar-native VAD and STT, isolated per-runner inference processes, and migrated all AlphaAvatar VDB workloads away from LiveKit’s shared inference executor.
-
[2026/06] Released AlphaAvatar version 0.6.0: Added the Status plugin, sampled visual input support, and status-aware DeepResearch / RAG / MCP tool feedback.
- Released AlphaAvatar version 0.6.1: Added visual identity support for Persona, including face detection, face vector matching, speaker-face identity fusion, and several bug fixes.
- Released AlphaAvatar version 0.6.2: introduced the unified provider layer, nested configuration, provider tracing, and a cleaner session/runtime foundation for future multi-user multimodal memory.
- Released AlphaAvatar version 0.6.3: Added the first graph-aware Memory foundation, including multi-object memory items, session-scoped graph node mentions, LanceDB graph-node retrieval, alias-ready graph lookup, and cleaner session-content memory extraction prompts.
-
[2026/05] Released AlphaAvatar version 0.5.4:
- Added LanceDB-backed MCP tool retrieval, enabling AlphaAvatar to semantically search relevant MCP tools from Agent queries.
- Refactored system prompt and runtime prompt composition, improved Persona runtime state tracking, added temporary-user to real-user identity merging, and improved RAG runtime behavior.
- Released AlphaAvatar version 0.5.5: Fixed the inference runner registration lifecycle for production
start mode, ensuring plugins runners are registered after config parsing and before LiveKit creates the inference executor.
-
[2026/04] Released AlphaAvatar version 0.5.3:
- Added localized Markdown backup for the Memory plugin.
- Added LanceDB as the default local VDB option when Qdrant credentials are not provided.
-
[2026/03] Released AlphaAvatar version 0.5.0:
- Added the MCP plugin, enabling retrieval and concurrent invocation of MCP tools.
- Released AlphaAvatar version 0.5.1: Added the original WhatsApp bridge via Baileys. This is a historical milestone; external messaging bridges are removed from the v0.6.8 development line.
- Released AlphaAvatar version 0.5.2: Added the AlphaAvatar Voice plugin with Voice.ai TTS support.
-
[2026/02] Released AlphaAvatar version 0.4.0:
- Added RAG support through RAG-Anything.
- Optimized the Memory and DeepResearch modules.
- Released AlphaAvatar version 0.4.1: Fixed Persona plugin bugs and added a new MCP plugin.
-
[2026/01] Released AlphaAvatar version 0.3.0:
- Added DeepResearch support through the Tavily API.
- Released AlphaAvatar version 0.3.1: Added tool-call memory extraction during user–assistant interactions.
2025 Release History
Installation ⚙️
AlphaAvatar v0.6.8 uses an AlphaAvatar-owned native Host and targets standard CPython 3.13. Install from the matching release tag and its lockfile; older released packages may have different APIs.
git clone --branch v0.6.8 --recurse-submodules https://github.com/AlphaAvatar/AlphaAvatar.git
cd AlphaAvatar
uv python install 3.13
uv sync --all-packages --locked
source .venv/bin/activate
The alphaavatar command is provided by alpha-avatar-host. No LiveKit Agents installation or separate model-download command is required. Enabled model runners resolve their files during initialization. Native model backends may require additional platform-specific build tools.
When upgrading a previous Python 3.11 checkout, create a new environment and follow the development environment migration guidance.
Quick Start ⚡️
Start the native Host in dev mode against your configured LiveKit deployment. Application and execution-process hosting belong to the native AvatarHost; foreground generation belongs to AvatarEngine and Loop.
🧩 Step 1. Configure Environment Variables
cd AlphaAvatar
# Copy template
cp .env.template .env.dev
Edit .env.dev and provide credentials for the selected RTC, model, embedding and tool services. Example model/voice identifiers are deployment choices, not guarantees of account access.
✅ Step 2. Run the Agent
Enabled inference runners resolve their model files during initialization. Missing files are downloaded automatically; no separate download command is required.
Set ALPHAAVATAR_MODEL_CACHE to choose the model cache directory. Set ALPHAAVATAR_MODEL_OFFLINE=1 to require locally cached, validated files. See Model cache for downloads and offline use, and Avatar workspace for personal data, backups and cleanup.
ENV_FILE=.env.dev alphaavatar dev examples/agent_configs/voice/pipeline_openai_tools_minimal.yaml
# or
ENV_FILE=.env.dev alphaavatar dev examples/agent_configs/mm/pipeline_openai_tools.yaml
See the Host README for startup/lifecycle boundaries and Examples README for complete configurations and their service requirements.
Usage 🚀
AlphaAvatar focuses on first-party realtime interaction. The browser demo and Developer Playground provide client surfaces; dedicated App and OS experiences remain planned.
External messaging bridges are not included in the current development scope. This does not remove native text chat or tool-based integrations with external services.
🌐 Web Access
AlphaAvatar now provides a browser-based realtime demo interface built on LiveKit.
👉 Try the Web Demo: https://www.alphaavatar.ai/demo
The browser client surfaces include (availability depends on the deployed version and configuration):
- 🎙️ Real-time voice interaction
- 💬 Text chat with the Avatar
- 📷 Camera preview and video-ready interaction
- 🔊 Agent audio playback
- 😊 Optional native AIRI presentation with a separately hosted frontend; renderer/RTC integration must be validated for the deployment
- 🧠 Configurable Memory, Persona, RAG, MCP and DeepResearch, subject to enabled services and execution policy
- 🌍 Browser timezone metadata, enabling AlphaAvatar to understand local login time
Screenshots and hosted deployments may reflect another revision. In the current native examples, AIRI-named configurations retain a voice persona, not animated Character transport.
📲 Native Mobile App
A dedicated AlphaAvatar mobile application providing:
- 🎙️ Real-time voice communication
- 😊 Live2D / Virtual character visualization
- 🧠 Persistent memory & persona
🧪 Developer Playground
Developers can immediately access AlphaAvatar via the LiveKit Playground.
👉 https://agents-playground.livekit.io/
After starting your AlphaAvatar server:
- Connect to your LiveKit instance
- Configure the Agent name to match
avatar.name in the selected YAML (the supplied examples use AlphaAvatar) for Explicit Dispatch.
- Connect to the agent room
- Start testing real-time interaction
Supported capabilities:
- 🎙️ Voice interaction
- 🧠 Memory extraction
- 🔍 RAG retrieval
- 🧰 MCP tool invocation
- 😊 Character presentation is opt-in and requires the separate frontend setup; it is not enabled by the supplied configurations
💡 AlphaAvatar is currently developer-first, with a Web Demo available for realtime interaction.
More user-facing web and mobile experiences are under active development.
评论