graph TD
A[iPhone 屏幕画面] -->|"AirPlay 镜像流"| B[ESP32 接收端]
B -->|"H.264 视频帧"| C[主机侧 Agent 服务]
C -->|"截图与任务描述"| D[OpenAI 兼容视觉模型]
D -->|"动作 JSON"| E{"动作是否可执行"}
E -->|"可执行"| F[坐标映射与 HID 编码]
E -->|"不可执行"| G[重试或人工接管]
F -->|"USB HID 键鼠事件"| H[iPhone 输入栈]
H -->|"界面状态更新"| A
Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak.
Your AI looks at the iPhone screen, then taps, swipes and types for you,
through a $7 ESP32 board plugged into the phone. No computer, no changes to iOS.
It hears the phone. The board forwards AirPlay audio and the iPhone turns it into words, so listening
exercises and voice prompts work. How it works.
Free Jev line. Jev takes the plain steps in a fraction of a second and leaves the rest to the large model;
the agent loop is rebuilt around one model call per step.
Long tasks. No 40-step limit; any stopped run can continue. Every run can be replayed step by step on the phone.
Get it on TestFlight (App Store listing in review). Needs board firmware
v0.2.4. Full changelog.
Why a board
Most iPhone agents run on a computer. Before the first task you install Xcode, turn on Developer Mode, pair the
phone over USB and type commands into a terminal, and the computer has to stay on for every task after that.
Fine for a developer, a wall for everyone else. Here the board does the computer’s job, so the phone is all you need.
A Mac (often a recent macOS), full Xcode, developer tools
Setup
Flash in the browser, install from TestFlight, four guided steps in the app
Terminal: clone, build, install dependencies, sign the helper app
Changes on the iPhone
AssistiveTouch on (the app walks you through it)
Developer Mode, USB trust and pairing
While it runs
Just the phone. Take it anywhere
The computer stays on and connected
Apps it can use
Any app a person can use: it sees the screen and moves a pointer
What the debug and accessibility interfaces expose
Hears the phone
Yes (App 1.0.2)
Usually not
The honest costs: you buy a $7 board, each step takes a few seconds (most of it the model), and the phone uses
cellular data while it runs because the board’s hotspot takes the Wi-Fi.
Not a developer? Start here
Get the board. A Seeed XIAO ESP32S3, about $7: what to buy.
[!NOTE]
Free line busy or failing? The free model line is shared and can be busy or return errors (HTTP 503) for a
while, so a run may stall or fail every step. Switch to your own API key in Settings › Model to keep going, and
let us know on X: @iamhenryyuan.
iPhone ──Screen Mirroring (Wi-Fi)──▶ ESP32-S3: AirPlay receiver ──Wi-Fi TCP :7201──▶ ESP32 Agent app (same iPhone)
▲ own hotspot, H.264 passthrough decode → vision model → action
└──────── USB HID mouse + keyboard ◀──────────── BLE commands ◀──────────────────────┘
Eyes — AirPlay to the board. The board opens its own Wi-Fi hotspot and appears in the phone’s Screen Mirroring
list as ESP32 Agent. It keeps the H.264 stream (no decoding on the board) and forwards it to the app over Wi-Fi
(TCP, port 7201). BLE carries commands and status. The phone keeps using cellular data for the model while it is
on the board’s hotspot.
Hands — USB HID. The board is a standard USB mouse and keyboard. With AssistiveTouch on, the mouse moves an
on-screen pointer, so a click is a tap and a drag is a swipe.
Brain — the app. The app decodes the stream, sends screenshots to any OpenAI-compatible vision model, and runs
a planner and an actor: tap, swipe, draw, type, press keys, go home, finish. It runs in the background while it
works; the Dynamic Island shows what it is doing.
Closing the loop. The pointer is relative, so the app tracks it and re-homes it in a corner; before precise taps
it wiggles the pointer, finds it on screen and corrects the miss. After each action it waits for the screen to
settle and tells the model whether anything changed. A lost board, hotspot or mirroring link pauses the run and
resumes it from the same step.
Review. Every run is kept on the phone: replay it step by step with the taps and swipes drawn on top.
Export the screenshots, actions and saved annotations from Runs → open a run → ⋯ → Export trace for analysis
by another agent. See Export runs for agent review for batch export, USB copying and file details.
Left: the real phone. Right: what the agent sees through the board’s AirPlay mirror. The second shot is the pointer calibration.
The first version (v0.1.0) ran on a Mac/Linux computer with UxPlay and a Python server. It is kept as tag
v0.1.0 and branch
legacy; see docs/legacy.md.
Hardware
Supported boards and setup details: hardware guide.
Picking a board, cables and adapters, or stuck during setup: setup guide.
Part
Notes
ESP32-S3 board
Tested with a Seeed Studio XIAO ESP32S3 (plain version, 8 MB PSRAM; not Sense, not Plus). About $7 (Amazon*). Any ESP32-S3 with native USB on its USB-C port and PSRAM should work.
iOS 18 or later, with cellular data (the Wi-Fi is taken by the board’s hotspot). USB-C models plug in directly; Lightning models need a Lightning to USB Camera Adapter. The phone powers the board.
Vision model
Any OpenAI-compatible chat API that accepts images: a cloud API or your own server.
Mac with Xcode
Only to flash the board and build the app once. Not needed at run time. With the TestFlight app, Xcode is not needed for the app.
* Affiliate link. As an Amazon Associate I earn from qualifying purchases.
Quick start
Warning 警告
Only follow the install steps in this README. Never run commands from issues or comments (for example a “quick install” one-liner such as PowerShell iex(irm ...)); these are not from the maintainers.
1. Flash the board (plugged into the Mac)
Install arduino-cli, the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then:
Easiest: join the TestFlight beta on your iPhone and install
ESP32 Agent from there, then go to step 3.
Or build it yourself:
cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig # set DEVELOPMENT_TEAM and IA_BUNDLE_ID
open ios/IPhoneAgent.xcodeproj # run the IPhoneAgent scheme on your iPhone
The project is generated with XcodeGen from ios/project.yml
(cd ios && xcodegen). A free Apple ID can sign it (re-sign every 7 days). The one-tap hotspot join needs the
Hotspot Configuration capability (a paid account); without it the app shows the hotspot password to join by hand.
3. Set up (once, in the app)
Plug the board into the iPhone and open ESP32 Agent. Setup walks through four steps and checks each one itself:
Plug Me In — pair the board over Bluetooth.
Let Me See the Screen — join the board’s Wi-Fi, then Control Center → Screen Mirroring → ESP32 Agent.
Let Me Use the Pointer — turn on AssistiveTouch; the app calibrates the pointer by itself.
Give Me a Brain — enter your model endpoint and key (stored in the Keychain), then choose a vision model
from the endpoint’s model list (or enter its model ID).
4. Give it a task
Type a goal (for example “Open Settings and go to Wi-Fi”) and press Start. Stop it from the Dynamic Island,
or come back to the app. Siri, Shortcuts, widgets and Control Center can run saved tasks.
Use from a coding agent
The esp32agent CLI (npm, npx -y esp32agent mcp) is an MCP server, so a coding agent on your computer can see
and drive the iPhone through the board (board firmware v0.2.2 or later); see iPhone as an MCP server. This repository is also a plugin with that
MCP server and an agent skill (skills/handsy/SKILL.md). In Claude Code (walkthrough):
DeepSeek Harness (dsh): install the plugin in integrations/dsh, which adds the same
MCP server and skill (demo video):
dsh plugin --profile web add dsh-esp32agent
The CLI talks to the board from your computer over Bluetooth, or over your Wi-Fi once both are on it (esp32agent net join),
so it works without the iPhone app or TestFlight.
Commands and options are in the CLI reference and cli/README.md.
Model and prompts
Any OpenAI-compatible vision endpoint works (vLLM, SGLang, llama.cpp, Ollama, cloud APIs). The app lists the
endpoint’s models for you to choose one explicitly; you can also enter a model ID, and set a separate planner
and actor model.
Settings → Model → Advanced → Extra body JSON is merged into each chat request. DeepSeek’s API
(api.deepseek.com) thinks at high effort unless told otherwise, so the app sends
{"thinking":{"type":"disabled"}} there unless that JSON already contains thinking; use
{"thinking":{"type":"enabled"},"reasoning_effort":"low"} to keep a little reasoning.
The prompts are plain text in prompts/ and bundled with the app: edit them, rebuild, done.
Each finished run logs planner/actor call counts, requests with a screenshot, and token totals when the endpoint reports
usage. Missing usage is left absent (never coerced to 0); the run total is marked partial if any call
omitted it. A short task is typically a few planner calls plus one actor call per step — each actor step sends
a fresh screenshot — so image tokens dominate cost. Giving the planner a cheaper model while keeping a stronger
actor is usually the best tradeoff.
ios/ the app (SwiftUI): BLE link, H.264 decode, agent, setup, replay, Live Activity, widgets; tests
prompts/ planner / actor prompts used by the app
hardware/ ESP32-S3 firmware (Arduino): BLE GATT, hotspot, AirPlay receiver, USB HID; flash script,
host tests, Mac BLE test harness (hardware/tools)
docs/ BLE protocol, legacy version notes
harness/ wireless development loop: Mac → iPhone app (Wi-Fi install) → board (firmware through the app)
server/ legacy v0.1.0 computer-side server; not used by this version (see server/README.md)
With the board plugged into a test iPhone and the iPhone on the Mac’s Wi-Fi, everything goes through devicectl,
no cable: harness/app.sh builds and installs the Debug app, harness/fw.sh builds the firmware and copies it
into the app, which installs it on the board (rolls back by itself if the new image fails), harness/run.sh "<task>"
runs a task and harness/logs.sh pulls the log. BLE from the Mac is only for the one-time net sta. Setup
(Lightning iPhones: camera adapter), security: harness/README.md.
Roadmap
Drop the computer: the board mirrors and clicks, the app on the phone thinks
Landscape apps (coordinates and calibration assume portrait)
Chinese text input (typed as pinyin through the board’s keyboard; ctrl+space switches English and pinyin)
Faster steps: several actions per model call, pointer checks only when needed
A curated list of models that work best with this setup, shown inside the iOS app (suggested by @asharthinks)
Limitations
AirPlay source must be an iPhone. Screen Mirroring from macOS is not supported: the handshake can look healthy
while the board never receives video.
The screen is mirrored at 480×1040, up to 5 fps; each step takes a few seconds, most of it the model.
iOS does not mirror some screens (passcode entry, Apple Pay, DRM video); the agent cannot see them. Mirroring stops
when the phone locks.
While the agent runs, the phone’s Wi-Fi is the board’s hotspot (no internet over it); the model is reached over
cellular data.
Chinese is typed as pinyin through the board’s keyboard (ctrl+space switches English and pinyin). The clipboard is
not used for this; emoji are dropped. Landscape apps are not supported yet.
If something doesn’t work (no ESP32 Agent in Screen Mirroring, the board not connecting), see troubleshooting.
Responsible use
Only automate devices and accounts you own or are allowed to use, and respect the terms of the apps involved. The
agent acts with your hands: watch it, and keep the stop button within reach.
Contributing
Issues and pull requests are welcome; see CONTRIBUTING.md. Start from the latest main
and open pull requests against main. Before opening a pull request, update your branch from the latest
main (fetch and rebase, or merge in the latest main). Security reports: SECURITY.md.
The latest code is on main; release-1.0 is the version submitted to TestFlight and App Store review, and is no longer the branch for contributions.
MIT, except hardware/hid_bridge/src/airplay/ (ported from UxPlay), which is
GPL-3.0; the firmware binary that links it is therefore
distributed under GPL-3.0. Files under skills/ are MIT-0 (MIT No Attribution);
the rest of the repository keeps the license above.
评论