Update to Autonom

{
  "description": "Understand and control what is on an Android or iOS screen for AI agents using compact accessibility/UI trees, semantic find/tap, gestures, screenshots, and key/text input via the Autonom CLI.",
  "included_files": [
    {
      "relative_path": "agents/openai.yaml",
      "size_in_bytes": 291
    }
  ],
  "name": "mobile-screen",
  "skill_md_contents": "---\nname: mobile-screen\ndescription: Understand and control what is on an Android or iOS screen for AI agents using compact accessibility/UI trees, semantic find/tap, gestures, screenshots, and key/text input via the Autonom CLI.\n---\n\n# Mobile Screen (Android + iOS Simulator)\n\n## Purpose\n\nAnswer **what is on screen** and **act on it** without relying on pixel-only guessing.\n\n1. Prefer a **compact UI tree** (UI Automator on Android, accessibility tree via idb on iOS).\n2. Confirm with a **screenshot** when visual layout matters or the tree is ambiguous.\n3. Act with **semantic selectors** first; coordinates only as fallback.\n\nThe compact node schema is identical on both platforms, so the same reasoning works\neither way. What differs is **which field carries the visible label** — see below.\n\n## CLI\n\n```bash\n# Compact tree (meaningful nodes by default)\npython3 <autonom-root>/scripts/autonom.py ui tree\npython3 <autonom-root>/scripts/autonom.py --platform ios --target <UDID> ui tree\npython3 <autonom-root>/scripts/autonom.py ui tree --format outline --interactable   # one indented line per actionable node\n\n# Wait until the screen stops changing (animations, spinners, loading lists)\npython3 <autonom-root>/scripts/autonom.py ui wait --settled --timeout-ms 5000       # settled, snapshots, changes, elapsed_ms; exit 1 if never settled\n\n# Find\npython3 <autonom-root>/scripts/autonom.py ui find --text \"Login\" --mode contains     # Android\npython3 <autonom-root>/scripts/autonom.py ui find --desc \"Log In\" --mode exact       # iOS\npython3 <autonom-root>/scripts/autonom.py ui find --resource-id com.apple.settings.general\n\n# Act\npython3 <autonom-root>/scripts/autonom.py ui tap --desc \"Continue\"\npython3 <autonom-root>/scripts/autonom.py ui tap --x 540 --y 1600\npython3 <autonom-root>/scripts/autonom.py ui type \"user@example.com\"\npython3 <autonom-root>/scripts/autonom.py ui swipe --from 200,600 --to 200,200\npython3 <autonom-root>/scripts/autonom.py ui key KEYCODE_BACK      # Android\npython3 <autonom-root>/scripts/autonom.py ui key HOME              # iOS\n\n# Evidence\npython3 <autonom-root>/scripts/autonom.py screenshot --out /tmp/screen.png\npython3 <autonom-root>/scripts/autonom.py record start --name login\npython3 <autonom-root>/scripts/autonom.py record stop\n\n# Offline parse of a dump (CI / fixtures) — the platform is detected from the file\npython3 <autonom-root>/scripts/autonom.py ui tree --dump tests/fixtures/ui_dump.xml\npython3 <autonom-root>/scripts/autonom.py ui tree --dump tests/fixtures/idb_describe_all_sample.json\n```\n\n## Compact node schema (identical on both platforms)\n\n```json\n{\n  \"ref\": \"n5\",\n  \"role\": \"button\",\n  \"text\": null,\n  \"desc\": \"General\",\n  \"resource_id\": \"com.apple.settings.general\",\n  \"bounds\": [16, 380, 386, 432],\n  \"clickable\": true,\n  \"long_clickable\": false,\n  \"checkable\": false,\n  \"enabled\": true\n}\n```\n\n## Where the visible label lives\n\n**This is the one asymmetry that matters.**\n\n| Platform | Visible label lands in | Select with |\n| --- | --- | --- |\n| Android | `text` (and `desc` for content-description) | `--text`, `--desc`, `--resource-id` |\n| iOS | **`desc`** (from `AXLabel`); `text` comes from `AXValue` and is often `null` | **`--desc`**, `--resource-id` |\n\nOn iOS a control labelled \"General\" has `desc: \"General\"` and `text: null`, so\n`ui find --text \"General\"` returns **zero matches**. Read the tree first and select\nby `--desc`, or by `--resource-id` when the app sets accessibility identifiers.\nFlutter on Android puts its labels in `desc` (content-description) too. On both\nplatforms, when a `--text` miss would have matched a `desc`, the response says so\nwith a `label_is_in_desc` warning whose hint is the `--desc` retry. A `ui find` or\n`ui tap` with no selector at all is `selector_required`. Whitespace is normalised\n(iOS joins a merged Flutter label with a newline, Android with a space), except\nin `--mode regex`.\n\n## What is on screen, and which match is picked\n\niOS lists Flutter's scroll cache — nodes laid out but not on screen — at a 0x0\nframe. Such a node stays in `ui tree` (it is counted) marked `visible: false`,\nand so does any node entirely outside the app's frame, live or from `--dump`.\n\n- `ui tap` and `ui find` resolve **on-screen matches first**: duplicates and\n  `--index` are counted among them, exactly as a flow counts, so a selector\n  that is unique on Android is not ambiguous on iOS because of an off-screen\n  copy. Only when no match is on screen is every match counted.\n- `ui tap` on a node that was never laid out on screen refuses with\n  `element_offscreen` (with `ref`, `bounds`, `match_count`) and taps nothing —\n  it used to tap the screen's origin and report success. Swipe it into view,\n  then select it again.\n- `ui find --all` lists on-screen matches first, then off-screen ones. Every\n  match `ui find` reports, with or without `--all`, carries `index` — the\n  `--index` that selects it (`null` for an off-screen match no index reaches\n  while another match is on screen). Position *i* of the `--all` list is what\n  `--index i` taps.\n- `ui wait --settled` says the tree was \"still changing\" only when its last\n  snapshot was a change. If it changed and then held still for less than\n  `--quiet-ms`, it says that (`stable_ms`); if only one dump fit in\n  `--timeout-ms` (an Android dump can take ~2 s), it says settling could not\n  be confirmed and how far to raise it.\n\n## Which tool touched the screen (`backend`)\n\nEvery input payload (`ui tap`, `ui swipe`, `ui type`, `ui key`) names its\n`backend`: `adb` on Android; `idb` or `axe` on iOS. The iOS tree always comes\nfrom idb. On Xcode 27 an idb companion built before the fix cannot load\nSimulatorKit, so the tree works while input fails with\n`ios_hid_framework_missing`; with AXe installed\n(`brew install cameroncooke/axe/axe`) the default `--ios-hid auto` sends\ninput through AXe instead. `--ios-hid idb|axe` (or `AUTONOM_IOS_HID`) pins a\nroute, and `--axe PATH` (or `AUTONOM_AXE`) names the binary; both go before\nthe verb. `autonom doctor` reports `ios_tree` and `ios_hid` separately with\nthe fix for each. On Xcode 27 with AXe carrying input, `ios_hid.ready` stays\n`false` (idb's own HID) while `ios_hid.axe_ready` and `ios_ui.ready` are\n`true`: read `ios_ui` and `axe_ready` — input works, do not give up on it.\n\n## Gestures and keys by platform\n\n| Verb | Android | iOS |\n| --- | --- | --- |\n| `ui tap`, `ui swipe`, `ui type` | yes | yes |\n| `ui pinch`, `ui rotate`, `ui shake` | refused with `unsupported_on_platform` | refused with `unsupported_on_platform` |\n| `ui key` | `KEYCODE_*` | `HOME`, `LOCK`, `SIDE_BUTTON`, `SIRI`, `APPLE_PAY`, or a numeric HID code |\n\n**No platform has pinch, rotate, or shake.** Android's `input` cannot express\nthem and idb has no such command either, so both are refused rather than faked.\nUse `ui swipe` for anything reachable by a drag; rotation and shake need a hand\non the Simulator window (Device > Rotate).\n\niOS has no global Back button; tap the navigation bar's back control instead.\n\n**Typing needs focus.** `ui type` sends keystrokes to whatever has keyboard\nfocus; with no focused field the characters vanish while the command still\nexits 0. The response carries `focused` (the node that will receive the text) with\n`focus: verified` on Android or `unverified` on iOS — idb's dump has no focus\nattribute, so on iOS the check degrades to \"a text field is on screen\" — or a\n`no_focused_field` warning. Tap the field, wait for it, type, then confirm\nwith `ui find` on the typed text. In flows, `inputText` refuses to type into\nnothing (`flow_no_focused_field`).\n\n## Coordinate space\n\niOS accessibility frames and `idb ui tap` both use **points**, not pixels. Never\nmultiply by the display scale. A tap computed outside the target's reported screen\nrectangle is refused with `coordinate_space_mismatch` rather than dispatched — on a\n3x device a pixel mix-up would otherwise land silently in the wrong place and make\nthe agent report a defect that does not exist.\n\n## Deterministic captures\n\nTwo screenshots of the same screen differ by the battery glyph and the\nsignal bars unless the status bar is pinned — and a before/after comparison\nthen reports a change the app never made. Pin it once per session, on either\nplatform, before the first capture you intend to compare. The clock stays\nreal, because evidence should say when it was taken; the marketing 9:41 is\none key away:\n\n```bash\npython3 <autonom-root>/scripts/autonom.py simulator status-bar pin        # full battery, full signal, no notification icons, real clock\npython3 <autonom-root>/scripts/autonom.py simulator status-bar pin --value hhmm=0941   # Android: 9:41 via demo mode (clock frozen); iOS: time=9:41\npython3 <autonom-root>/scripts/autonom.py simulator status-bar clear      # restore the live bar when done\n```\n\nOn Android the default pin never enters SystemUI demo mode, because demo mode\nfreezes the clock at the moment it is entered; `hhmm=`, `wifi=`, `mobile=`, or\n`mode=demo` opt into it when a fixed clock or shaped Wi-Fi bars matter more\nthan a ticking one. `values.mode` in the response says which path ran.\nThe emulator's signal cannot be read back, so a live Android pin carries a\n`signal_unstable` warning and does not claim the signal as verified. The\nemulator's bars and battery icon drift within a minute of any pin, so\n`autonom screenshot` and flow captures re-send the live pin right before the\nframe (`repinned: true` in the response) — take evidence through them, not a\nraw `adb screencap`. `status-bar clear` restores the battery and signal state recorded before the\nfirst pin.\n\nA pin is bound to the emulator it was set on (AVD name and boot id): an\nemulator that took the same serial later — another AVD, or a reboot — never\nreceives it. The record is dropped instead (`stale_pin_dropped: true` and a\nwarning), and `devices shutdown` / `devices boot` drop it outright; pin again\non the new device. On API 36 emulators SystemUI may not draw the battery\nglyph at all even though `dumpsys battery` reports the pinned level, so a\nmissing battery icon in a capture is the emulator's, not the pin's or the\napp's.\n\nOn Android, turn system animations off before comparing captures, and wait\nfor the screen to settle:\n\n```bash\npython3 <autonom-root>/scripts/autonom.py simulator animations pin        # window, transition, animator scales to 0\npython3 <autonom-root>/scripts/autonom.py ui wait --settled\npython3 <autonom-root>/scripts/autonom.py simulator animations reset      # restores what the first pin replaced\n```\n\niOS has no host-level animation switch (`unsupported_capability`).\nEvery `simulator` control answers `verified: true` only after reading the\nstate back; otherwise `verified: false` with the reason in `verification`.\n\n`ui type` on iOS is at the mercy of autocorrect: a non-English keyboard\nrewrote \"Sync conflicts when editing offline\" into something else mid-flow.\nPin the keyboard and locale on the **shut-down** simulator before typed text\nmust be exact (`reboot=true` lets the verb cycle a booted one):\n\n```bash\npython3 <autonom-root>/scripts/autonom.py simulator keyboard pin --value locale=en-US --value reboot=true\npython3 <autonom-root>/scripts/autonom.py simulator keyboard show          # pinned: true|false, per key; backup: true while pinned\npython3 <autonom-root>/scripts/autonom.py simulator keyboard reset         # restores the values pin replaced\n```\n\n`pin` snapshots the keys it overwrites (a region-override locale, a keyboard\nsetting a human chose) and `reset` puts them back, so a shared simulator\nleaves a test the way it entered it. Always pair a pin with a reset.\n\nAndroid has no host-level keyboard store (the settings live inside Gboard), so\n`simulator keyboard` refuses there with `unsupported_capability`; turn\nsuggestions off in the field or the keyboard app instead.\n\nEvery `screenshot` (and `shots show`) reports `width` and `height` from the\nPNG itself. On iOS that is **pixels** while the tree and `ui tap` speak\npoints; on Android it is whatever the AVD's screen is. State the space you\ncomputed in when you report a coordinate.\n\n## Agent workflow\n\n1. `session start` so the target and artifacts are explicit.\n2. `ui tree` → read labels, roles, enabled state. Note which field holds the label.\n3. `ui find` / `ui tap` for the next action; after a navigation, `ui wait\n   --settled` before the next read.\n4. `screenshot` after state changes that need visual proof — **compare before/after**;\n   an exit code of 0 does not prove the screen changed. Pin the status bar first\n   so the diff shows only what the app changed.\n5. Re-dump the tree after navigation; never reuse stale refs.\n6. Report **measured** on-screen text and tree facts separately from hypotheses.\n\n## Boundaries\n\n- The iOS tree is the **accessibility** hierarchy, not the SwiftUI/UIKit view tree.\n  Its quality depends on the app's labelling.\n- A tree with fewer than three meaningful nodes is reported with\n  `sparse_accessibility_tree`. That means \"the app exposes little\", not \"the screen\n  is empty\" — add `Semantics` (Flutter) or `.accessibilityLabel` /\n  `.accessibilityIdentifier` (SwiftUI), or fall back to a screenshot.\n- When no element carries an identifier, `no_accessibility_identifiers` says so, so\n  a zero-match `--resource-id` query is not mistaken for a missing control.\n- System dialogs (permissions) need the same selectors or coordinates.\n- Browser mirror (`android-emulator-browser`) is visual support, not a substitute\n  for the tree.\n\n## Related\n\n- `mobile-session`\n- `mobile-network` — correlate on-screen errors with HTTP traffic\n- `android-debugger-agent`, `ios-debugger-agent`\n"
}