kellyvv/PhoneClaw 是一个托管在 GitHub 上的开源项目，主要使用 Swift 编写，采用 Other 许可证，目前获得 1.3k 个 Star、173 次 Fork。
仓库全名
kellyvv/PhoneClaw
README 摘要
Turn your phone into a local AI agent runtime. Website · TestFlight · 中文 · Report an Issue · Request a Feature Core Features · Built-in Skills · Technical Notes · Quick Start · Mac Remote · Custom Skills · FAQ · Roadmap PhoneClaw is a local AI agent framework for phones and edge devices. It runs Gemma 4 E2B / E4B via LiteRT and MiniCPM-V 4.6 on device, uses native mobile Skills for Calendar, Reminders, Contacts,…
PhoneClaw is a local AI agent framework for phones and edge devices. It runs Gemma 4 E2B / E4B via LiteRT and MiniCPM-V 4.6 on device, uses native mobile Skills for Calendar, Reminders, Contacts, Clipboard, Health data, image understanding, voice, and LiveLand, and lets you explicitly choose Web Search, webpage reading, or LAN-based Mac Gateway inference when a task needs it.
PhoneClaw (this project) is unrelated to the Android automation repo “rohanarun/phoneclaw” or the “phoneclaw” GitHub organization. Android app-store search results may also show a separate app with the same name.
Added LiveLand: use PhoneClaw from Dynamic Island. The local AI listens, understands your request, runs the task, and shows the result right there
Launch LiveLand from Home Screen widgets, Lock Screen widgets, Shortcuts, Control Center widgets, or inside the app. It also works on non-iOS 27 systems
Task progress is easier to follow: PhoneClaw now shows when it received your command, is understanding it, is querying or executing, is preparing the result, and is done
More natural phone tasks: ask for health data, create reminders, schedule calendar events, find contacts, translate text, and more through natural language
Fixed a LiveLand widget launch issue that could incorrectly show “download model first” or “model loading” even after the model was already installed
Safer actions: unclear times, titles, contacts, deletes, and bulk operations ask for clarification or confirmation before running
LiveLand runs on local models by default, with a fully offline local path and local execution free of per-call token billing
Added the PhoneClaw Gateway Mac client: keep it running on your Mac, advertise it over Bonjour, then pair from the iPhone to use Mac-side Ollama, Codex CLI, or Antigravity CLI as a remote inference source
Added a Mac Remote page in iPhone settings: discover Macs on the same LAN, approve pairing on the Mac, choose a Mac-side model, then use it from the normal chat screen
Remote models are only used after you explicitly pair a Mac and select one. With Ollama, inference stays on your Mac; with CLI or other upstream providers, data handling follows that provider’s behavior
Rebuilt the overall Agent framework: improved Skill result semantics, Skill routing, tool-call chains, contextual follow-ups, and multi-step task handling, so the model can more reliably choose whether to answer directly, call tools, or continue from the previous result while reducing accidental tool triggers, repeated calls, and off-track follow-ups
Added Health range reports: summarize steps, distance, active energy, heart rate, sleep, workouts, weight, and HRV across a selected time range, then generate a local health summary, trends, and key metric explanations
Added English localization — the app automatically switches based on system language.
Refactored the download module: resumable downloads, background downloads, and automatic fastest-mirror selection based on current network conditions.
2026-04-23
Released v1.2.2 — added the ability to choose between GPU or CPU inference backend directly from the settings page; CPU is now the default to fit within Sideloadly-signed memory limits.
⚠️ Sideloadly-signed IPA usage note: due to the memory cap of sideload-signing, the E4B model only works on CPU (GPU will fail). We recommend using the E2B model — it’s fully featured and more stable under the cap.
💡 If you can, build from source with Xcode: Xcode-signed builds aren’t subject to the sideload memory cap — you can run E2B / E4B with GPU enabled for best performance. Download
2026-04-20
Released an unsigned IPA — sign and install to iPhone via Sideloadly. Download
2026-04-18
Added in-app download for LIVE mode voice models — download directly from the settings page and start using LIVE mode
2026-04-17
Added LIVE Mode: a new real-time voice interaction mode with natural conversation flow and barge-in interruption
LIVE Mode supports camera input: the model can recognize and understand the environment, objects, and scenes captured by the camera in real time, enabling multimodal “see and speak” interaction
2026-04-10
Added Health Skill: read HealthKit data including today’s/yesterday’s steps, weekly step trends, walking distance, active calories, resting heart rate, last night’s sleep, weekly sleep summary, and recent workouts — 9 tools total, all data processed locally
Improved multi-turn response speed: cross-turn KV cache reuse reduces time-to-first-token by ~3.5x for consecutive queries within the same skill
The 9 health tools are automatically selected by the model based on user intent
2026-04-09
Ongoing framework and infrastructure work. Major improvement to the multi-turn agent framework: the Router now correctly preserves skill context across turns, and even small models can reliably complete multi-turn tool calls.
2026-04-08
Model downloads now include a ModelScope mirror for faster Gemma 4 downloads in mainland China
Major rework of memory management: the inference budget is now dynamically derived from actual available memory, with improved long prompt, long answer, and multi-turn tool-call context handling
2026-04-07
Added voice input, with on-device audio analysis and recognition
Added Thinking Mode, available from the top-right corner in chat
Added chat history, with support for new sessions, switching, and deletion
Improved memory management and inference budgeting for long answers, multimodal requests, and model switching
2026-04-06
The default install flow is back to a shell app, with models downloaded on-device as needed
The settings page now includes model download, permission status, and bilingual display names
Contacts, reminders, calendar, device info, and clipboard flows have received a round of stability fixes
Core Features
Private Local Agent: Run inference and Skill calls directly on device. Use natural language to work with Calendar, Reminders, Contacts, Clipboard, Health data, and other local tasks.
Mac Remote Inference: Optionally pair with a Mac on the same LAN through PhoneClaw Gateway and use Mac-side Ollama, Codex CLI, or Antigravity CLI models while keeping the native iPhone chat and Skill experience.
Image Understanding and LIVE Vision: Ask questions about photos from the camera or photo picker, or enable the camera in LIVE mode so the model can understand the scene in real time.
Personal Data Analysis: Read local schedules, Health data, contacts, reminders, and clipboard content to generate summaries, availability analysis, and next-step suggestions. Personal data is processed on-device by default.
Realtime Information: When explicitly requested, search public webpages, fetch readable webpage text, and summarize live information into an actionable answer.
Voice Interaction: Supports voice input and LIVE real-time conversation for hands-free questions, notes, and actions.
Technical and Experience Features
File-Driven Skill System: Each capability is defined by a single Markdown file (SKILL.md). Adding or modifying a skill can happen through the Skill file. Skills are language-agnostic — anyone can write and share them.
Model Management and Resumable Downloads: Gemma main models and LIVE voice models can be downloaded, canceled, resumed, and retried directly on iPhone, or bundled into the app at build time.
Fully Offline First with Clear Data Flow: Inference and local Skill calls run on-device by default. Conversations, images, and personal data stay on iPhone. Web Search, webpage reading, and paired Mac remote inference are explicit user-triggered capabilities; Mac remote inference sends the current request to your paired Mac, and any further upstream access depends on the provider selected in the Mac client.
Mobile Memory Optimization: Includes model switching, system prompt editing, cache cleanup, and history trimming tuned for phone-scale on-device inference limits.
Bilingual Experience: Choose Auto, Chinese, or English in settings. The UI, default system prompt, built-in Skills, tool results, and permission text switch together.
Models/ is listed in .gitignore and stays outside commits.
Approximate file sizes: E2B ~2.4 GB, E4B ~3.4 GB.
In mainland China, set HF_ENDPOINT=https://hf-mirror.com to use the mirror, or download the same file from the ModelScope mirror.
LIVE Mode (voice interaction) additional models
If you want to use LIVE mode with voice recognition and synthesis, download the ASR and TTS models:
After downloading, add Models/sherpa-asr-zh and Models/vits-zh-hf-keqing as folder references to Copy Bundle Resources in Xcode. LIVE mode can also use system speech as a fallback.
4. Open the workspace
open PhoneClaw.xcworkspace
Always open .xcworkspace.
5. Configure signing and run
In Xcode, select the PhoneClaw target
Open Signing & Capabilities
Set your Team
Change the Bundle Identifier to a unique value
Connect your iPhone and press ⌘R
On first install, if prompted to trust the developer certificate: Settings → General → VPN & Device Management → Trust
6. First use
After opening the app:
Top-right puzzle icon: Skill management
Top-right slider icon: Model settings / system prompt / permissions
If you installed a shell-only app, tap Download in the model settings page first
Download a model first, then enable Calendar, Reminders, and Contacts in the permissions page, then try:
Remind me tonight at 8 to send the file
Save Wang's phone number 13812345678
Translate that last line into English
7. Use the Mac client for remote inference
The Mac client turns a Mac on the same LAN into an optional remote inference source for the phone runtime. PhoneClaw still uses its chat UI and Skill system, while model inference requests are sent to your paired Mac.
On first launch, allow the macOS Local Network permission prompt
If macOS blocks first launch, open Finder, right-click PhoneClawGateway.app, then choose Open
Gateway listens on port 18080 by default and advertises the _phoneclaw-llm._tcp Bonjour service.
Option B: build from source
cd MacGateway
bash build-app.sh
open PhoneClawGateway.app
The source-built app also needs macOS Local Network permission.
Configure the Mac runtime source
Open PhoneClawGateway.app
Choose a runtime source in the main window: Ollama, Codex CLI, or Antigravity CLI
If you use Ollama, install and start Ollama first, then pull a model, for example:
ollama pull gemma3:4b
Return to Gateway, scan, and confirm that the model appears in the list
Pair from the iPhone
Make sure the iPhone and Mac are on the same LAN
Open PhoneClaw → top-right slider → Mac Remote
Tap the Mac, then approve the request in the Mac client
After pairing, choose a Mac-side model and return to chat
Remote models appear under the Remote section in the model picker. For Mac discovery, check that the Mac client is running, Local Network permission is allowed, both devices are on the same Wi-Fi, and the macOS firewall allows PhoneClawGateway.app to accept LAN connections.
Default Install Flow and Model Bundling
Option A — Shell app + on-device model download
This is now the default recommended setup.
Advantages:
Much smaller install size from Xcode
Faster first-time app installation from the Mac
Users can choose E2B or E4B directly on the phone
By default, the project builds a lightweight app while model files download on device or get copied explicitly during custom builds.
In Xcode’s Project Navigator, delete the unused model file reference and choose Remove Reference
In PhoneClaw > Build Phases > Copy Bundle Resources, make sure gemma-4-E2B-it.litertlm is included as a single file resource
Edit allModels in LLM/Models/PredefinedModels.swift to only include the models actually shipped (otherwise the settings page will show options that don’t exist)
---
name: MySkill
name-zh: My Skill
description: What this skill does
version: "1.0.0"
icon: star
disabled: false
type: device # device = native API; content = prompt-only; network = public internet access
triggers:
- schedule planning
allowed-tools:
- my-tool-name
examples:
- query: "How a user might phrase it"
scenario: "What scenario triggers this"
---
# Skill Instructions
Tell the model when to call tools, how to structure arguments, and when to answer directly.
The type field controls routing: device calls native iOS APIs, content is prompt-only text processing, and network is for explicit live web search or webpage reading. If this skill needs to call native APIs or network tools, register the tool in Tools/ToolRegistry.swift (and add a handler under Tools/Handlers/). The framework validates allowed-tools against the registry at startup, so any typo will surface immediately in the console.
FAQ
How do I trigger permission dialogs after install?
Permission dialogs appear when a Skill reaches the related system API call. After a previous denial, re-enable the permission in system Settings.
Why does the model fail to load after switching?
Verify that the model file name matches allModels in LLM/Models/PredefinedModels.swift, that the model has finished downloading on-device if you are using the shell-only install flow, or that it was actually included in the app bundle if you are shipping it built-in, and that the device has enough memory.
Why does creating a reminder fail?
The latest code first attempts to reuse an existing writable reminder list. If none is found, it tries to automatically create a PhoneClaw list. If that also fails, the system reminder source itself is likely read-only.
How do I make the iPhone discover the Mac client?
Make sure PhoneClawGateway.app is running and macOS has allowed Local Network permission. The iPhone and Mac must be on the same LAN. If discovery still fails, check that the macOS firewall allows the app to accept incoming connections, or rebuild the app with bash MacGateway/build-app.sh and open the generated app again.
Roadmap
1. More iOS native APIs
File and directory access
Image picking, description, and Q&A
Photo library reading, organization, and search
Notes
Reminder due-time alerts
General local notifications
Maps and location
URL webpage reading and context passing
Safari / URL Scheme handoff to external apps
Contacts search, create, update, and delete
Calendar creation, schedule reading, busyness, and free-time analysis
Reminder creation
Read-only HealthKit analysis
2. More Skills
Continue breaking capabilities into focused Skills with clear model, tool, and permission responsibilities. Directions worth adding:
File management
Image understanding
Photo organization
Schedule creation, querying, and busyness analysis
Personal information management: contacts, calendar, reminders, clipboard, and Health data
Local knowledge base search
Voice input / text-to-speech
Web Search / webpage reading
Translation
3. More local models
Beyond the main chat model, suitable additions include:
Vision / multimodal model
OCR model
Speech recognition model
Speech synthesis model
Embedding / Reranker model
A smaller tool argument extraction model
A stronger planning model or multi-model pipeline
This moves PhoneClaw from “one big model doing everything” toward “multiple local models working together.”
4. Cross-app automation
PhoneClaw uses the cross-app capabilities that iOS actually provides:
System notification wake-up and cross-app orchestration
A realistic goal: pass content between apps, open a specific app to a specific screen, and compress multi-step operations into a single natural language command.
5. External hardware and visual input
LIVE camera real-time recognition
External video input
Screen understanding
External hardware integration
Explore connecting external video input and screen understanding with local models, so PhoneClaw goes beyond answering questions in isolation and develops stronger real-world perception and scheduling capabilities.
评论