sim-use

sim-use

Give AI agents the ability to observe and act on iOS Simulator and Android emulator / device screens.

Observe — turn any screen into a token-efficient outline an LLM can reason about:

$ sim-use ui
App: Settings  402x874

[Top  y<120]
  @1  StaticText  "Settings"
[Content  y=120..754]
  @5  SearchField  "Search"
  @7  Button  "Sign in to your iPhone"
  @9  Button  "General"
  @10 Button  "Display & Brightness"
  @11 Button  "Wallpaper"
  ...
[Bottom  y>754]
  @43 TabBar

Act — tap any element by its alias, no coordinates needed:

$ sim-use tap @9
✓ Tap at (201.0, 452.0) completed successfully

Plan, code, verify, ship — teach this CLI to your agent and close the last gap in the agentic mobile development loop. Let agents verify what they built so you can focus on what matters.

sim-use is a cross-platform CLI that drives Apple’s Accessibility APIs, the iOS Simulator HID pipeline, and Android’s AccessibilityService through a single command surface. It emits a compact, agent-friendly screen description (ui) and an alias-cached tap shortcut (tap @N) so an LLM loop can observe → act in a few hundred milliseconds per round trip.

• The observe → act loop
• Why sim-use
• Install
• Platforms
• Commands
• Physical iOS devices
• Architecture
• Viewer
• Contributing
• Licence

The observe → act loop

Every interaction follows the same cycle — observe, act, verify:

sim-use ui                  # 1. read the screen
sim-use tap @9              # 2. act on what you see
sim-use ui                  # 3. verify the result

Multiple selector styles for different needs:

Selector	Example	Best for
@N alias	tap @9	Speed — cached from last ui
#<id>	tap #settingsButton	Stability — survives layout changes
--label	tap --label "General"	Scripted flows with --wait-timeout
-x -y / --point	tap --point 100,200	Last resort — no AX data

Selectors and explicit coordinates work in any orientation — on iOS, sim-use self-calibrates the current rotation on each command, so -x/-y / --point take the numbers ui prints. --coordinate-space native gives them in the device’s native portrait points instead.

Why sim-use

• Token-efficient. The outline representation is ~16x more compact than a raw JSON accessibility tree. An LLM can read and reason about an entire screen in a few hundred tokens.
• Nothing hidden. sim-use walks the full accessibility tree including WebViews, system overlays, and embedded content — no elements are silently skipped. When the frontmost app exposes an empty tree because a remote process owns the visible UI (a system document picker, for example), ui automatically retries with cross-process discovery and flags the recovered, flat hierarchy via the advisory envelope key.
• AI-native. Designed from day one for agent loops, not human testers. Alias-cached taps (@N), structured --json envelopes with actionable hint fields on errors, and a bundled agent skill (sim-use init --client claude) that teaches your AI client the full command surface.
• Fast. A per-device background daemon amortises init cost across calls. After the first command, each observe-act round trip completes in ~300 ms.
• Cross-platform. One command surface drives both iOS Simulator and Android emulator/device. Same verbs, same flags, same --json shape — write one agent loop that works on both.

Install

Homebrew (recommended)

brew tap lycorp-jp/tap
brew install lycorp-jp/tap/sim-use

On Homebrew 6.0.5+, if you see an “untrusted tap” error, run brew trust lycorp-jp/tap first.

Notarized releases keep their upstream Developer ID signature after installation. The ad-hoc signing fallback is only used for legacy, non-notarized release packages.

Build from source

sim-use is a Swift package targeting macOS 14+, built with the latest Xcode toolchain. It links against static XCFrameworks built from Meta’s idb, which are produced locally by the build script (they are large and not checked into the repository). The idb checkout generates its Xcode project with XcodeGen (brew install xcodegen).

git clone https://github.com/lycorp-jp/sim-use.git
cd sim-use

# Build the required XCFrameworks (first time only)
./scripts/build.sh dev

# Build sim-use itself
make build
.build/debug/sim-use --help

# Other Makefile targets
make test    # run tests
make clean

The XCFrameworks are built without library evolution, so their Swift modules are locked to the toolchain that produced them — re-run ./scripts/build.sh dev after switching Xcode versions.

For release packaging changes, scripts/verify-brew-install.sh --archive <tarball> --version <version> --require-developer-id tests the generated formula under a separate name without linking it into PATH. Add --download-url <release-asset-url> to test a real Homebrew download against the same archive checksum. The verifier checks that the original binary and Developer ID signature survive installation; it does not replace an existing sim-use installation.

Xcode 27 (beta)

Xcode 27 betas are fully supported from Beta 4 on (earlier betas ship no usable SimulatorKit.framework), including simulators booted while Device Hub is open — the HID transport is selected automatically per boot. Note that Xcode 27 no longer bundles Simulator.app; the one from an Xcode 26.x install still works, as does Device Hub itself.

Resize Mode and the iPhone Duo inner display are driven. Xcode 27 can put an iOS 27+ simulator into a resizable app session (Device Hub’s resize button, or xcrun devicectl device appResize start), which moves the frontmost app onto a virtual Resizable display at an arbitrary size. Xcode 27.1’s iPhone Duo has two integrated displays: the cover display (primary) and the inner display. sim-use reads the displays from xcrun devicectl device info displays and routes everything to the display that hosts the app: touch input (tap, long-press, swipe, touch, gesture, multi-touch, batch) goes to that display’s digitizer, accessibility hit-tests (ui --point, collapsed-children recovery) run on it, and screenshot captures it. The ui header names the Duo screen (App: Settings 951x669 (landscape-left, inner display), (cover display)) or Resize Mode (display Resizable 560x874). ios fold and ios rotate use the same inner/cover names. An active inner display does not prove a fully open hinge; half-open states can use it too. These aliases require the Duo device type and a valid two-panel topology; unknown layouts keep system names. --json keeps raw display names and ids; its display object appears only off the main display. Ordinary single-screen headers stay unchanged. On a display other than the main one the orientation comes from CoreDevice instead of hit-test probes. record-video and stream-video (h264, bgra) capture that display too — the inner panel in its native portrait orientation, like the main display; a Resize Mode scene cut from its canvas. A fold during a recording keeps the display it started on and says so at the end. The screenshot-based stream-video formats read the main display only and refuse there. Explicit -x/-y / --point coordinates are the numbers ui prints there too; with --coordinate-space native they are native to the display that hosts the app (the inner panel’s unrotated points; a Resize Mode scene’s points). sim-use ios fold closed|open|half|<degrees> folds and unfolds the Duo and confirms which display became active; sim-use ios fold alone reports the posture.

An app whose scene is smaller than its display — an iOS 27.0 SDK app keeps out of the iPhone Duo’s status bar strip, an older SDK app runs centred on the inner display — is placed by a few hit-test probes, so taps land on it in every orientation.

The display list costs one ~0.1 s devicectl query per daemon on a single-display simulator and at most one every 2 s on a multi-display one; the Resize Mode check is one devicectl device info appResize per device per 5 s. A ui (or a selector tap) whose accessibility root changed size re-reads both at once — after a fold, or when a session starts or ends. A raw -x/-y tap right after such a change, with no ui in between, can still see the previous answer for that long. SIM_USE_DISPLAY_TOPOLOGY_CHECK=0 and SIM_USE_RESIZE_SESSION_CHECK=0 turn the checks off (read by the per-device daemon when it starts). Background: issue #143, docs/ai/009-resizable-app-session/000-investigation.md and docs/ai/010-display-routing/000-plan.md.

Linux (Android only)

The Android backend also builds on Linux with a Swift 6 toolchain — iOS verbs, video capture, the Viewer and init are macOS-only. See docs/ai/008-linux-android-build/linux-guide.md for what is available, how to build and install, and reaching devices from WSL.

scripts/install-linux.sh

Agent skill

To install the bundled agent skill into your AI client’s skill directory:

sim-use init                        # auto-detect installed clients
sim-use init --client claude        # non-interactive
sim-use init --dest ~/.claude/skills
sim-use init --print                # print skill content without installing
sim-use init --uninstall --client claude

Platforms

sim-use drives both iOS Simulators and Android devices / emulators through the same command surface. The device ID shape decides which backend handles the call:

• 1A2B3C4D-... (UUID) → iOS Simulator
• emulator-5554 / R5CT1ABCD12 / 192.168.1.5:5555 → Android device
• 00008130-... (8-16 hex) / 40-hex → physical iPhone/iPad (restricted verb set)

For Android, run sim-use android init --device <serial> once to install the bridge APK. See AGENTS.md for Android toolchain setup. When ADB_SERVER_SOCKET points at an adb server on another machine (tcp:<host>:<port>), sim-use reaches the bridge on that host, since that is where adb forward listens; SIM_USE_BRIDGE_HOST overrides the host.

Physical iPhones and iPads (experimental) route through the same top-level verbs — sim-use ui, sim-use tap '#<id>' / --label and sim-use screenshot work against a plugged-in device’s UDID. The channel exposes no element geometry, so it trades coordinate taps, swipes and gestures for accessibility actions, and the remaining verbs reject with the reason and the nearest alternative — never assume capability parity; see the capability matrix. sim-use installs and signs no runner and needs no Developer Disk Image; ui/tap need the foreground app to be development-signed (get-task-allow=true), screenshot captures any screen.

Commands

All device-scoped commands accept --device <ID> (optional when only one simulator is booted). Three command layers:

• Top-level — cross-platform verbs: ui, tap, long-press, swipe, touch, multi-touch, type, paste, button, gesture, keyboard-state, screenshot, record-video, stream-video, app-state. Same flags on iOS and Android; physical iOS devices route through ui, tap and screenshot only (the rest reject with the reason — see the capability matrix).
• sim-use ios <verb> — iOS-Simulator-only: key, key-combo, key-sequence, batch, rotate, fold.
• sim-use android <verb> — Android-only: init, devices, ping, scroll.
• sim-use ios-device <verb> — physical-iOS-only: devices, ui, screenshot, tap (experimental). Peer of ios/android; also accepts ECIDs and the hierarchy tuning flags the top level doesn’t carry.

Run sim-use --help or sim-use <command> --help for the full flag set.

sim-use devices
# PLATFORM  KIND       STATE   NAME               UDID                                  RUNTIME
# ios       simulator  Booted  iPhone 17 Pro Max  B34FF305-5EA8-412B-943F-1D0371CA17FF  iOS 27.0
# ios       physical   Booted  My iPhone          00008130-00066D2A10EB8D3A             iOS 26.6
# android   emulator   device  Medium_Phone       emulator-5554                         Android
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"

One listing covers every target: iOS Simulators (simctl), Android devices and emulators (adb), and USB-attached physical iPhones/iPads (FBDeviceControl). KIND — simulator / emulator / physical — is orthogonal to PLATFORM and also appears as kind in --json; capabilities follow the kind (see Physical iOS devices for what physical iOS supports). --no-physical-ios skips the FBDeviceControl side entirely (~1 s saved when no device is attached) — the Viewer passes it, since it drives coordinate taps and video streaming, neither of which physical iOS supports.

Touch & gestures

sim-use tap @5 --device $UDID                                 # alias from the last `ui`
sim-use tap "#3" --device $UDID                               # 3rd cell of the dominant list ("#2@2": 2nd cell, 2nd list)
sim-use tap "#settingsButton" --device $UDID                  # AXUniqueId
sim-use tap --label "Safari" --device $UDID                   # also: --id, --value
sim-use tap -x 100 -y 200 --device $UDID                      # raw point (or --point 100,200)

sim-use swipe --from 100,300 --to 300,100 --device $UDID      # positional "100,300 300,100" works too
sim-use swipe --from 50,500 --to 350,500 --duration 2.0 --delta 25 --device $UDID

# Low-level touch control
sim-use touch -x 150 -y 250 --down --device $UDID             # then: touch ... --up
sim-use touch -x 150 -y 250 --down --up --delay 1.0 --device $UDID   # long press

# Gesture presets
sim-use gesture scroll-up --device $UDID
sim-use gesture swipe-from-left-edge --device $UDID
sim-use gesture scroll-down --pre-delay 0.5 --post-delay 1.0 --device $UDID

--pre-delay / --post-delay / --duration work on tap, swipe, and gesture alike for coarse timing control.

Single-finger presets (scroll-*, swipe-from-*-edge) name visual directions and are orientation-aware on iOS: their canvas size and rotation are auto-detected per command, so scroll-up scrolls the on-screen content up whether the device is portrait, landscape, or upside-down. Explicit tap/swipe/touch/multi-touch coordinates, paste --via-menu --target-x/--target-y and the pinch/rotate --center-x/--center-y are in the same visual space ui prints (orientation-calibrated per command; the split touch --down / --up form calibrates each half). --coordinate-space native gives them in the device’s native portrait points, as before 0.16.0. Calibration reuses the last ui snapshot when it can, so it costs a few milliseconds; with no snapshot it reads the tree once. On Android the flag is accepted and ignored — display coordinates already rotate with the UI.

Text input

sim-use type 'Hello World!' --device $UDID
echo "complex text" | sim-use type --stdin --device $UDID
sim-use type --file input.txt --device $UDID

Paste (IME-safe Unicode)

sim-use paste stages UTF-8 text, verifies exact clipboard readback and taps the native Paste action. On iOS, the standard HID Application key opens the focused field’s edit menu without moving its caret or selection. This works with Simulator.app and Device Hub, with or without a connected hardware keyboard. It bypasses IME composition and supports CJK, emoji and diacritics without keyboard-layout substitution. Clipboard staging uses verified simctl when CoreDevice ownership is inactive, otherwise verified CoreDevice.

sim-use paste 'ABC 日本語 🎉' --device $UDID             # at caret
sim-use paste 'new content' --replace --device $UDID   # Select All + Paste

printf '%s' "$CONTENT" | sim-use paste --stdin --device $UDID
sim-use paste --file body.txt --device $UDID

Focus the field first for default paste. To choose a field explicitly, use --via-menu: it long-presses an id or coordinates before invoking the edit menu. Both forms support --replace; neither changes keyboard or clipboard-sync preferences.

sim-use paste 'ABC 日本語' --via-menu --target-id chatTextField --device $UDID
sim-use paste 'at xy' --replace --via-menu --target-x 171 --target-y 513 --device $UDID

The control must support the native edit menu. --replace requires Select All unless the focused control exposes an explicitly empty accessibility value; otherwise it fails before Paste instead of appending. Existing menus require native-menu provenance to distinguish them from app controls. Anonymous modal menus need a verified native dismissal control in the same container; ambiguous or unsupported structures are refused instead of tapping app buttons. Native smart-paste formatting, single-line normalization and paste permission prompts still apply; sim-use does not approve prompts automatically. Verify the resulting field with sim-use ui. If completion cannot be confirmed after sending Paste, the command reports an ambiguous result without retrying. Check the field before you retry.

Clipboard sharing can also change the Mac and other devices’ clipboards. sim-use holds a host-wide lock for its own paste operations and verifies the clipboard again before Paste, but cannot lock out external apps. Do not run other clipboard writers during a paste. Clipboard mismatch errors never include unexpected contents.

sim-use ios batch supports the same default, --replace, --file and targeted --via-menu behavior. A paste step cannot use --stdin, which belongs to the batch input.

Keyboard state

Probe whether the software keyboard is visible. This is useful for keyboard-dependent UI checks; paste does not require branching on this result.

sim-use keyboard-state --device $UDID        # soft | hidden; non-zero means probe failure
sim-use keyboard-state --json --device $UDID # {"ok":true,"data":{"visible":true, ...}}

Hardware buttons

sim-use button home --device $UDID
sim-use button lock --duration 2.0 --device $UDID     # long press
sim-use button siri --device $UDID
# Also: side-button, apple-pay

Low-level keyboard (iOS-only)

These verbs speak USB HID keycodes, which have no Android counterpart — for Android text entry use sim-use type or sim-use paste.

# Individual key presses by HID keycode
sim-use ios key 40 --device $UDID                                     # Enter
sim-use ios key 42 --duration 1.0 --device $UDID                      # hold Backspace

# Sequences and modifier combos
sim-use ios key-sequence --keycodes 11,8,15,15,18 --device $UDID      # "hello"
sim-use ios key-combo --modifiers 227 --key 4 --device $UDID          # Cmd+A
sim-use ios key-combo --modifiers 227,225 --key 4 --device $UDID      # Cmd+Shift+A

Batch chaining (iOS-only)

Run multiple steps in a single invocation. Batch reuses one HID session and one AX snapshot across steps, cutting round-trip cost on multi-step flows (Android steps each round-trip through the bridge already, so batching would save nothing there).

sim-use ios batch --device $UDID \
  --step "tap --id SearchField" \
  --step "type 'hello world'" \
  --step "key 40"

# With element waiting — selector taps poll until the element appears
sim-use ios batch --device $UDID \
  --wait-timeout 5 \
  --step "tap --id LoginButton" \
  --step "tap --id WelcomeMessage"

# From file (one step per line)
sim-use ios batch --device $UDID --file steps.txt

Key semantics:

• Exactly one step source per run: --step, --file, or --stdin.
• Fail-fast by default; --continue-on-error switches to best-effort.
• A transient boot-readiness failure does not automatically replay the batch. Earlier steps may have completed, including when --continue-on-error reports multiple failures. Inspect the UI and completed steps before retrying.
• --wait-timeout <seconds> makes selector taps poll for the element to appear — primary mechanism for multi-screen flows.
• --ax-cache perBatch (default) reuses one AX snapshot for the whole run; --ax-cache perStep refreshes between steps when the UI changes; --ax-cache none disables snapshot reuse entirely. --wait-timeout polling always refetches.

Rotation and folding (iOS-only)

rotate takes the interface orientation the ui header prints and turns the device until the display that hosts the app shows it, then reads the result back from CoreDevice. It works on every iOS simulator (iPhone, iPad, iPhone Duo). An app that does not accept the orientation — a portrait-only app, SpringBoard on iPhone, portrait-upside-down on an iPhone-size screen — keeps its orientation, and the command reports that with an advisory (--json: rotate.confirmed: false) instead of failing.

sim-use ios rotate landscape-left --device $UDID      # portrait | landscape-left | landscape-right | portrait-upside-down
sim-use ios fold open --device $UDID                  # iPhone Duo: inner display
sim-use ios fold closed --device $UDID                # iPhone Duo: cover display
sim-use ios fold 90 --json --device $UDID             # any hinge angle; reports the display it observed
sim-use ios fold --device $UDID                       # report the posture only

Both verbs change the screen geometry, so they clear the cached ui snapshot: run ui again before tap @N. A fold also wakes a locked device to its lock screen. Single-display simulators rotate through a GSEvent sent to SpringBoard; the iPhone Duo rotates and folds through the vendor-defined HID reports that Device Hub’s posture controls send. Device Hub does not need to be running: the reports go to the simulator’s dtuhidd, which starts on demand. With Device Hub closed, sim-use also starts dtuhidd itself when touch input must reach the inner display or a Resize Mode scene, and rebuilds its HID connection once dtuhidd runs (an active dtuhidd disables the legacy keyboard path).

Screenshot

sim-use screenshot --device $UDID                                 # auto-named
sim-use screenshot --output ~/Desktop/shot.png --device $UDID     # specific file
sim-use screenshot --output ~/Desktop/ --device $UDID             # directory

The output path goes to stdout; progress messages go to stderr.

Video streaming & recording

# Native H.264 live stream (cross-platform) — the fastest path and the one to
# reach for. H.264 in MPEG-TS on both platforms, with no host-side codec pass.
# TS carries PTS, so players pace off the stream and stay in sync indefinitely.
sim-use stream-video --device $UDID --format h264 | \
  ffplay -f mpegts -probesize 32768 -i -

# Muxing the preview to a file works, but it is a live capture, not an
# archive: idle stretches are compressed (see below). For a faithful
# recording use `record-video`.
sim-use stream-video --device $UDID --format h264 | ffmpeg -f mpegts -i - -c copy out.mp4

# Screenshot-backed formats. DEPRECATED on iOS (h264 is ~6x the frame rate at
# an eighth of the bytes) and slated for removal there. Retained on Android:
# `screenrecord` is unavailable on some devices, and this loop is the only way
# to stream from those.
sim-use stream-video --device $UDID --fps 10 --format mjpeg > stream.mjpeg
sim-use stream-video --device $UDID --fps 30 --format ffmpeg | \
  ffmpeg -f image2pipe -framerate 30 -i - -c:v libx264 -preset ultrafast out.mp4

# Record MP4 directly (cross-platform)
sim-use record-video --device $UDID --output recording.mp4            # 30 fps default
sim-use record-video --device $UDID --fps 60 --output smooth.mp4      # up to 60 fps
sim-use record-video --device $UDID --quality 60 --scale 0.5 --output low-bw.mp4

# Record an animated GIF (cross-platform) — auto-plays inline in PRs / issues / chat
sim-use record-video --device $UDID --output demo.gif                 # format inferred from extension
sim-use record-video --device $UDID --format gif                      # sim-use-video-<timestamp>.gif
sim-use record-video --device $UDID --format gif --fps 15 --scale 0.4 --output demo.gif

On iOS, record-video and stream-video --format h264 share one set of H.264 encoder settings (--fps, --quality, --scale, keyframe interval) through a single factory, so the two verbs cannot drift apart on how the picture is encoded. They do differ downstream: recording takes Annex B through the host-side muxer into an MP4, while streaming takes MPEG-TS straight to stdout. h264 streams at a constant --fps (default 30, which is also the streaming cap); measured on a booted iPhone 17 Pro it runs ~220 KB/s at 30 fps where mjpeg manages ~4 fps at ~1.9 MB/s.

stream-video is a preview: its timeline advances by the real gap between pictures but never by more than 0.2 s, so an idle screen does not leave a hole a player has to sit through before it reaches the next picture. That is the right trade when the question is “what is on screen now”, and it means the stream is not a faithful record of elapsed time — piping it to a file turns a 60-second pause into 0.2 seconds of output. record-video keeps real arrival times and is what to use for an archive. Ctrl-C ends a stream, and so does quitting the player: a closed pipe is an orderly stop on both platforms.

-probesize 32768 is not decoration. ffplay identifies the stream by reading a bounded amount of it, and the default budget is measured in seconds of media. Android’s capture is variable-frame-rate: while the screen is still, screenrecord emits no pictures at all and the stream thins to just its program tables (a few KB/s), so a media-time budget can take arbitrarily long in wall-clock terms to fill. Bounding the probe by bytes instead lets it return after a picture or two. Two things do not help: -analyzeduration 0 (zero selects ffmpeg’s default window rather than disabling it, and the open measured identically with and without it) and -fflags nobuffer (it starves the probe of the data it needs and ffplay never opens the stream). One consequence to know about: a still Android screen sends no pictures, so the window appears only once something on the device moves. iOS streams at a constant rate and opens at once.

Both platforms carry h264 in MPEG-TS, and the reason is worth knowing: a bare H.264 elementary stream has no timestamps at all, so a player has to guess the frame rate — ffprobe reads such a stream as 25 fps whatever it really is — and when the guess is under the real rate the picture falls further behind the device every second, without bound. MPEG-TS carries a PTS per picture on a 90 kHz clock, so the player paces off the stream instead. iOS gets TS straight from the simulator’s encoder; on Android, screenrecord only emits Annex B, so sim-use re-containers it host-side (no re-encoding — the frames pass through untouched, only the wrapper is added). Frame rate differs by platform: iOS honours --fps as a constant rate, Android runs at the device’s native variable rate.

record-video captures a real H.264 stream and muxes it straight into the MP4 (passthrough — no per-frame screenshot re-encoding). iOS records at a constant --fps (default 30, max 60); Android records at the device’s native variable frame rate (--fps ignored, --quality → bitrate, --scale → size) and stitches across the screenrecord per-invocation limit on API < 34 automatically. Rotating the display mid-recording stops capture on Android (an MP4 track can’t change frame size). Press Ctrl+C to stop; sim-use finalises the MP4 before exiting.

--format gif (or a .gif --output extension) records the same H.264 stream to an intermediate MP4, then transcodes it into a looping GIF once recording stops — sampling at --fps (GIF default 10, capped at 50 by the format’s centisecond delay floor) and scaling at --scale (GIF default 0.5), since full-rate full-scale GIFs get enormous. Per-frame delays follow the source timestamps, so Android’s variable frame rate stays true to wall-clock. If the transcode fails, the intermediate MP4 is preserved and its path reported, so the footage is never lost. GIF is meant for short clips: the encoder holds every frame in memory until the file is written, so for sessions beyond a few hundred frames prefer a lower --fps or --format mp4. Because the GIF loops forever, --gif-markers can bracket the clip with START/END marker cards (~1 s each) so the boundary is visible.

Accessibility inspection

sim-use ui --device $UDID                      # compact outline (default)
sim-use ui --json --device $UDID               # structured envelope
sim-use ui --json --no-raw --device $UDID      # envelope without the raw tree (much smaller)
sim-use ui --point 100,200 --device $UDID      # specific point (same UI space as outline frames)

The --json envelope carries the raw accessibility tree under data.raw by default; --no-raw drops it while keeping outline / entries / lists intact — prefer it in agent loops where the raw tree is only debugging ballast.

The outline uses region banding ([Top] / [Content] / [Bottom] / declared Group regions; on an iPhone Duo, [Trailing x>=N] / [Leading x<=N] for the vertical bar of an iOS 27.1 SDK app) and @N / #N / #N@M / #<id> alias addressing. When the device is rotated, the App: header carries an orientation tag (e.g. (landscape-right)) and the --json envelope an orientation field.

A list cluster detector runs on every snapshot and attaches #N aliases to detected list cells — outline lines render as @N #M (dominant list) or @N #M@S (scope S>1). The --json envelope adds a lists array (one summary per cluster, ordered by detector score, dominant first) and per-cell membership under entries[*].aliases.list.

App state & crash detection

sim-use app-state --device $UDID                              # list running apps
sim-use app-state --bundle-id com.example.app --device $UDID  # running | not_running
sim-use app-state --reset --device $UDID                      # re-baseline crash detection

While the daemon drives a device, it watches for the target process disappearing between commands and surfaces a banner on the next ui call. The signal is process liveness (not foreground identity), so backgrounding for a permission dialog or share sheet never false-fires. On Android, ui also detects the AOSP system crash dialog directly from the accessibility tree. Call app-state --reset after an intentional relaunch; SIM_USE_NO_CRASH_DETECT=1 disables detection entirely.

Daemon

UDID-scoped commands auto-spawn a per-UDID background daemon on first use and reuse it on subsequent calls, amortising FBSimulatorControl / accessibility init (~200 ms per ui-shaped call). Scripts do not need to manage the daemon.

sim-use daemon status
sim-use daemon stop --device $UDID
sim-use daemon stop --all

# Force in-process execution for a single call (diagnostics)
SIM_USE_NO_DAEMON=1 sim-use ui --device $UDID

Daemons self-exit after 600 s of idle and log to /tmp/sim-use-<uid>/<UDID>.log. Streaming commands (screenshot, record-video, stream-video) always run in-process regardless.

Physical iOS devices

Experimental: this surface intentionally supports development-signed target apps only. Its commands and compatibility may change while the device matrix grows.

A connected iPhone or iPad is driven through its accessibility audit daemon over usbmux lockdown. sim-use installs no XCUITest runner, performs no signing and needs no Developer Disk Image. The foreground target app must already be signed with a Development provisioning profile whose final code-sign entitlements contain get-task-allow=true, and the device must be paired, trusted, unlocked and in Developer Mode. A Release-configuration build remains supported when installed with a Development profile.

Distribution/Ad Hoc, TestFlight, App Store and system apps do not expose the hierarchy or actions required by this channel. ui and tap fail with an entitlement-oriented diagnostic instead of reporting an empty tree or a successful action.

Verify the app before installing it when in doubt:

codesign -d --entitlements :- /path/to/MyApp.app
# ... <key>get-task-allow</key><true/> ...

The top-level verbs route a physical UDID automatically — the same observe → act → verify loop as every other target:

sim-use devices                      # physical devices appear with kind `physical`
# ios  physical  Booted  My iPhone  00008140-000210603A40801C  iOS 27.0
UDID="00008140-000210603A40801C"

sim-use ui --device $UDID
# Button  "Chats Button, Selected"
# Button  "Friends"
# Button  "Settings"  #settingsButton
# ...
# 117 elements (316 nodes) in 6647 ms

sim-use tap --label "Friends" --element-type Button --device $UDID
# Sent Activate to 'Friends' [Button]

# Dynamic labels can use the regular substring selector vocabulary.
sim-use tap --label-contains "Reply" --element-type Button --device $UDID

# Or target the stable accessibility identifier shown as #id.
sim-use tap '#BackButton' --device $UDID

# Screenshot — any screen, not limited to development-signed apps. Prints the
# absolute saved path on stdout (plus a confirmation on stderr).
sim-use screenshot --device $UDID
# /Users/me/Device Screenshot - My iPhone - 2026-08-27 at 09.34.10.png

# --json everywhere, shared {ok, data} envelope.
sim-use tap '#BackButton' --json --device $UDID
# {"ok":true,"data":{"action":"Activate","identifier":"BackButton","kind":"physical","label":"sim-use Playground","role":"Button"}}

The sim-use ios-device namespace remains as the physical-only peer of ios/android — same three verbs plus devices, and additionally accepts ECIDs (shape-based routing can’t recognise a bare ECID) and hierarchy tuning flags (--fast, --concurrency, --connections).

A device is addressed by UDID or ECID, and --device is optional only when exactly one is attached (top-level auto-resolution still picks booted simulators only). A freshly attached device may be listed by ECID until a session has opened (AMDevice publishes the lockdown UDID lazily). Run ui again after every action: accessibility actions are fire-and-forget, so the follow-up read is the authoritative verification.

Capability matrix

Never assume capability parity with the simulator. Only three verbs route; everything else fails with the reason and the nearest alternative (as a hint in --json):

Verb	Simulator	Android	Physical iOS
ui / describe-ui	✅	✅	✅ outline + #ids only — no @N aliases, frames, or --point
tap	✅	✅	✅ #<id> / --id / --label / --label-contains / --element-type only
screenshot	✅	✅	✅ any screen (CoreDevice)
swipe, gesture, touch, multi-touch, long-press	✅	✅	❌ no coordinate input or geometry
tap -x/-y/--point, @N / #N aliases, --value, --label-regex, --frame, --duration, --wait-timeout	✅	✅	❌ rejected per form
type, paste	✅	✅	❌ no text-input channel yet
button, keyboard-state	✅	✅	❌
record-video, stream-video	✅	✅	❌ use screenshot
ios key / key-combo / key-sequence / batch	✅	❌	❌
app-state	✅	✅	❌

Pass a physical UDID to a sim-use ios <verb> (simulator-only by contract) and it fails fast with a pointer back to the routed verbs.

This channel deliberately differs from the simulator backend:

• No element geometry. There is no coordinate tap, swipe, gesture or multi-touch. Only the exposed tap accessibility action is currently supported; the other verbs reject a physical UDID with the reason and the nearest alternative rather than degrading silently.
• No @N aliases, but stable #ids. Element handles encode a live pointer and expire with their DTX connection, so — like the missing geometry — the cross-invocation @N alias cannot be backed faithfully and the outline advertises none. The stable accessibility identifier can: the outline shows each element’s #id, and tap accepts it as a positional #<id> or --id (mirroring the simulator), alongside --label / --label-contains / --element-type. Prefer the #id when a label is dynamic — a navigation-bar back button is labelled with the previous screen’s title but keeps #BackButton, and is an ordinary, tappable row in the outline.
• Screenshots go over CoreDevice, not the audit daemon. screenshot shells out to xcrun devicectl device capture screenshot, a separate channel with different rules: it is not limited to development-signed foreground apps and captures whatever is on screen, SpringBoard and system apps included. Screen recording exists on the same channel (devicectl device capture screen-record) but is capability-gated per device (CoreDevice can report “Screen Recording is not supported by this device”) and is not exposed yet.
• No recording yet. Screen recording is not exposed (see the CoreDevice capability gate above). --json is supported on every ios-device verb, with the same {ok, data} envelope as the rest of the CLI: devices returns unified device rows (the deviceId / kind / runtime schema of top-level sim-use devices --json), ui returns the outline text plus structured rows (depth / role / label / #identifier) with element/node counts and timing, tap returns the matched element, and screenshot returns the saved path. Errors carry a machine-readable hint alongside the message.
• Slower snapshots. A full tree costs a few seconds. ui --fast stops at labelled elements and is roughly 40% quicker, at the cost of about a quarter of the elements.
• Reading order, not screen order. With no frames to sort by, the outline follows accessibility nesting and reading order.

Architecture

sim-use drives iOS Simulators through the lower-level XCFrameworks of Facebook’s idb (statically linked), Apple’s Accessibility APIs, and the simulator HID pipeline. Android devices are driven through an on-device bridge APK that exposes the AccessibilityService tree and input injection over HTTP, tunnelled via adb forward. Physical iOS devices go through a third path: idb’s FBDeviceControl opens a lockdown service connection, over which sim-use speaks Apple’s DTX message protocol to the accessibility audit daemon (screen capture instead shells out to Xcode’s devicectl). Everything ships as a single binary, and every surface — simulator, Android and physical iOS — supports --json with the shared {ok, data} envelope.

Viewer

A local web app that renders sim-use ui --json onto a scaled SVG canvas — see which elements the accessibility tree exposes, spot blind spots, and tap directly from the browser.

sim-use viewer

No Node or npm needed — the SPA is bundled into the binary. Opens your browser automatically. For front-end development on the Viewer itself, see Tools/Viewer/README.md.

Contributing

See CONTRIBUTING.md for development setup, coding conventions, and the DCO sign-off every contribution needs.

Licence

sim-use is licensed under the Apache License, Version 2.0 — see LICENSE and NOTICE.

sim-use began as a fork of cameroncooke/AXe (MIT, © 2025 Cameron Cooke), cut from AXe v1.6.0 in April 2026 and substantially modified since. It also links against XCFrameworks built from Meta’s idb (MIT). The MIT License of both works permits this Apache-2.0 redistribution; their original notices are reproduced in THIRD_PARTY_LICENSES.