What a device tool costs your coding agent.
We gave Claude Code five bugs that only show up in a running React Native app, and asked it to find the cause, fix it and check the fix on the simulator. The app, the bugs, the prompt and the model stayed the same; only the device tool changed.
Every setup fixed every bug. On these bugs the device tool did not decide whether the agent succeeded; it decided what success cost. Parallax gives the agent the screen, the component tree, logs and the app's stored values, and costs about what a bare shell costs.
Run on 2026-11-04
- runs; every setup fixed every bug, each fix checked on the device
- 68runs; every setup fixed every bug, each fix checked on the device
- Parallax’s median cost per run; shell only: $0.07
- $0.07Parallax’s median cost per run; shell only: $0.07
- Parallax’s median time per run; shell only: 40 s
- 36 sParallax’s median time per run; shell only: 40 s
- Parallax’s median tokens per run; shell only: 195k
- 205kParallax’s median tokens per run; shell only: 195k
Results
The five runtime bugs, 15 runs per setup, medians per run. Cost is the API list price Claude Code reports for the run, not a bill. Cost per fix is the total cost of every run divided by the number of fixes (a mean); every other figure is a median.
| Setup | Fixed | Time | Tokens | Cost | Cost per fix | Model calls |
|---|---|---|---|---|---|---|
| Shell only | 15 / 15 | 40 s | 195k | $0.07 | $0.09 | 16 |
| Parallax 0.1.0-beta.3 | 15 / 15 | 36 s | 205k | $0.07 | $0.08 | 16 |
| agent-device 0.21.36 | 15 / 15 | 36 s | 204k | $0.09 | $0.13 | 11 |
| Argent 0.31.1 | 15 / 15 | 71 s | 772k | $0.22 | $0.24 | 29 |
What each setup did well
- agent-device had the fastest median (35.6 s) and the fewest model calls (11). It was the fastest setup on three of the five bugs (the race between two requests, the invisible overlay and the missing Intl API) and level with Parallax on a fourth (the decimal-comma price).
- Argent reaches into the app’s JavaScript runtime: on the storage bug it read the stored value through the debugger instead of navigating to it.
- The bare shell is a strong baseline: it read the app’s storage files straight from the simulator’s container.
- Parallax had the lowest median cost per run ($0.07) and the lowest cost per fix ($0.08, the mean: every run’s cost divided by the fixes), and its context stayed the smallest by the last call (23k tokens).
Where the tokens go
The model reads the whole conversation again on every call, so a run costs roughly its calls times its context. The table has the medians per run. Claude Code loads each MCP tool's full description only when the agent asks for it, so a long tool list barely shows up.
| Setup | Model calls | Context, first call | Context, last call | Tools offered |
|---|---|---|---|---|
| Shell only | 16 | 17k | 26k | – |
| Parallax | 16 | 17k | 23k | 19 |
| agent-device | 11 | 18k | 32k | 62 |
| Argent | 29 | 25k | 55k | 77 |
Bug by bug
Fixed runs and median cost for each bug. The two controls have their cause in plain sight in the code.
| What the user sees | Shell only | Parallax | agent-device | Argent |
|---|---|---|---|---|
| Some products on the Products list show '—' instead of a price. | 3/3 · $0.07 | 3/3 · $0.07 | 3/3 · $0.06 | 3/3 · $0.28 |
| Every product on the Products list says 'Out of stock', but the warehouse says most of them are in stock. | 3/3 · $0.06 | 3/3 · $0.06 | 3/3 · $0.08 | 3/3 · $0.22 |
| Pressing Save on Settings does nothing.control | 1/1 · $0.23 | 1/1 · $0.09 | 1/1 · $0.13 | 1/1 · $0.16 |
| The Add to cart button on a product page does not respond to taps. | 3/3 · $0.07 | 3/3 · $0.06 | 3/3 · $0.10 | 3/3 · $0.20 |
| The Checkout button is missing on the Cart screen. | 3/3 · $0.17 | 3/3 · $0.16 | 3/3 · $0.32 | 3/3 · $0.25 |
| On a product page the 'Restocked' line shows no date: it only says 'Restocked'. | 3/3 · $0.08 | 3/3 · $0.08 | 3/3 · $0.09 | 3/3 · $0.20 |
| The cart total is wrong when the same product is added twice.control | 1/1 · $0.11 | 1/1 · $0.10 | 1/1 · $0.13 | 1/1 · $0.21 |
What this benchmark changed in Parallax
The first round ran Parallax 0.1.0-beta.2. On the stale-storage bug it was slower and costlier than the bare shell (92 s and $0.20 against 76 s and $0.17): without our SDK in the app, Parallax could not read the app's storage, so the agent tapped through screens to find the value. 0.1.0-beta.3 reads stored values from the simulator: short, non-sensitive values are shown as they are, everything else is masked. On the same bug it took 63 s and cost $0.16. Both rounds are in the raw results.
Since then, 0.1.0-beta.4 added a built-in navigate command that takes the agent to a screen through the app's own navigation, instead of a chain of taps. We ran 0.1.0-beta.4 on the storage bug only, 3 runs: the agent used navigate in 3 of 3 runs, and the median run took 57 s and cost $0.12 (0.1.0-beta.3: 63 s, $0.16). The other bugs were not rerun; the tables above are 0.1.0-beta.3's.
How it was run, and its limits
- Agent: Claude Code 2.1.318 with Claude Sonnet 5.5, headless. Every setup got the same prompt, the same allowed tools and a fresh configuration.
- App: a small shop app on React Native 0.86.3 (new architecture, Hermes) with a local mock API, on an iPhone 17 Pro simulator (iOS 26.3, Xcode 27.0), one Mac.
- Every run, for every setup, started cold: the simulator was rebooted, the app folder and the app’s storage were reset, and no tool’s background process was left from an earlier run (agent-device’s daemon was stopped with
daemon stop --cleanand Argent’s tool server withargent server stopafter each of their runs). - A fix counts when the bug’s own check passes after the agent has finished: a script relaunches the app and drives it with real taps, using none of the tools under test.
- The causes were graded blind by a separate model, with every tool name masked.
- “Shell only” means no MCP server. On this Mac the shell had simctl and AXe on the path, and the agent used them to look at the simulator in every run.
- Three runs per setup on each runtime bug and one on each control: 68 runs. With three runs per bug a median is a fair middle, not a precise figure.
- Limits: one model, one app, one machine, seven bugs. This compares what the setups cost on the same work. It does not rank how capable they make an agent, and it does not test what a human and an agent do together.
Each tool, installed by its own setup
- agent-device 0.21.36
- npm i -g agent-device, claude mcp add agent-device -- agent-device mcp, and its skill with npx skills add callstack/agent-device
- Argent 0.31.1
- npm i -g @swmansion/argent, then argent init --yes --no-telemetry
- Parallax 0.1.0-beta.3
- npx -y @parallax-rn/mcp@0.1.0-beta.3 setup claude
Raw results and corrections
The 68 counted runs of this comparison are in runs.csv, which also has the 17 0.1.0-beta.2 runs and the 3 0.1.0-beta.4 runs: bug, setup, run, fixed, cause, seconds, tokens, cost, model calls, device calls. If you maintain one of these tools and we set it up or used it wrongly, or if you find an error, write to zemantservices@gmail.com. We will fix the page and rerun if needed.
Code: parallax-rn/benchmark on GitHub, with the harness, the app, the bugs with their answers and the raw runs.
The counted runs used Parallax 0.1.0-beta.3. The current version is 0.1.0-beta.11.