Marionette Flutter Drive App

marionette_flutter: drive an app

Marionette lets an AI agent drive a running Flutter app (debug or profile — never release) the way a real user would: read the screen, tap, type, scroll, screenshot, read logs, hot reload. It has two halves:

• marionette_flutter — the package you add to the app. It registers a VM service extension per action (ext.flutter.marionette.*).
• The bridge — either marionette_mcp (MCP tools, e.g. connect, tap, get_interactive_elements) or marionette_cli (shell commands, e.g. marionette tap --key ...). Nearly the same capabilities, different transport — the CLI additionally offers record-video, with no MCP equivalent (see the capability table, below). Prefer the MCP tools when they're in your tool list; fall back to the CLI in restricted environments (enterprise policy, a shell-only agent) — run marionette help-ai once at the start of such a session and follow its reference for exact syntax.

Everything shared between the two applies to either transport — action names differ only in casing (get_interactive_elements vs get-interactive-elements).

Preparing the app

connect (or any CLI command) only works if the target app initialized MarionetteBinding. If you're not sure whether it has, check main.dart (or ask) — skip the rest of this section if it's already there and you just need to drive the app, and jump to When to use this.

flutter pub add marionette_flutter

void main() {
  if (kDebugMode) {
    MarionetteBinding.ensureInitialized();
  } else {
    WidgetsFlutterBinding.ensureInitialized();
  }
  runApp(const MyApp());
}

This is all a standard Material-widgets app needs. It runs inside main(), so any time you add or change it, hot restart — a hot reload doesn't re-execute main(), so the change will look like it did nothing. Gate every piece of Marionette wiring behind kDebugMode (or !kReleaseMode — see Device-config sweeps, below) so none of it ships in a release build: both are compile-time constants the release branch survives cleanly.

Custom design system? Teach Marionette about it

Zero-config setup only recognizes stock Material widgets (ElevatedButton, TextField, Switch, …) and extracts text only from Text, RichText, EditableText, TextField, and TextFormField. If the app wraps its own buttons, fields, or text — true of most production apps — get_interactive_elements will look sparse and tap(text: ...) will fail to find things that are clearly on screen. That's not a Marionette bug; it just doesn't know your widgets yet:

MarionetteConfiguration(
  // Recognize your custom interactive widgets.
  isInteractiveElement: (element) => switch (element.widget) {
    MyButton() || MyTextField() => true,
    _ => false,
  },
  // Extract their visible text for text-based matching and discovery.
  extractText: (element) {
    final widget = element.widget;
    if (widget is MyText) return widget.data;
    return null;
  },
)

Pass it as MarionetteBinding.ensureInitialized(MarionetteConfiguration(...)). For a non-trivial design system, extract the configuration to its own file (e.g. lib/debug/marionette_config.dart) imported only behind kDebugMode, rather than growing it inline in main.dart — that keeps release builds free of design-system introspection helpers even before the compiler strips them.

isInteractiveElement and extractText receive the Element, not just the Widget. Match with an is check or pattern on element.widget — that covers generic widgets (MySelect() matches MySelect<String>) and subclasses (MyButton() matches MyPrimaryButton extends MyButton), and can read the widget's fields to decide per instance. extractText can also walk the subtree when a label is itself a widget rather than a plain string. If the app still uses the deprecated isInteractiveWidget: (type) => ... or shouldStopTraversal: (type) => ..., move it to isInteractiveElement / shouldStopTraversalAtElement: a Type only compares with exact ==, so it silently misses generic and subclassed widgets. While both are set, their results are combined with OR. Custom-painted text and badges reach no Text widget at all, and a WidgetSpan is only a problem when its embedded content isn't itself built from Text/RichText (an icon, a custom-painted chip) — plain text nested inside one is still its own discoverable element. For genuinely non-text content, annotate with Semantics(label: ..., value: ...) instead; Marionette surfaces that as a Semantics element with the joined 'label: value' string, and it costs nothing if label/value are absent (unlabeled Semantics nodes are silently skipped).

Not every custom widget needs an entry: add one when it's a primary interactive primitive (button, field, toggle, tab, chip) or when get_interactive_elements is noticeably missing it; skip widgets that are only ever decorative, or ones that already wrap a primitive Marionette sees (a MyCard built on GestureDetector is already covered). The list doesn't need to be exhaustive up front — extend it as real use surfaces a missing target.

One more MarionetteConfiguration field, shouldStopTraversalAtElement, is worth naming only to warn against it: it's tempting to add a scroll container there to "reduce traversal cost," but on a real production app that measurably dropped widget coverage from 25.8% to 17.8% — the agent lost visibility into everything nested below the cut, including the content it needed to reach. Leave it null unless a profiler has shown a real, measured cost, and never point it at a scrolling container.

Logs for get_logs

Without a LogCollector, get_logs returns a message explaining how to add one rather than actual logs. Pick whichever matches how the app already logs:

• Uses the logging package → flutter pub add marionette_logging, pass LoggingLogCollector() as logCollector.
• Uses the logger package → flutter pub add marionette_logger, pass LoggerLogCollector() (it doubles as a LogOutput for that package too).
• Anything else → PrintLogCollector (ships in marionette_flutter) exposes a manual addLog(message) to call from wherever logs already flow — routing Flutter's own debugPrint through it is a common choice, tee'd into the existing implementation rather than replacing it, to keep debugPrintThrottled's throttling.

Screenshot size

take_screenshots downscales captures to fit within 2000×2000 physical pixels by default, to keep base64 payloads manageable. Override maxScreenshotSize on MarionetteConfiguration if a screen needs more detail (Size(3000, 3000), say), or set it to null to disable resizing entirely — but keep the default unless there's a concrete reason to raise it, since larger screenshots mean larger payloads on every call.

Device-config sweeps

set_device_config — sweeping text scale, bold text, or light/dark appearance without touching real OS settings — is the one capability that needs a widget in the tree, since Marionette won't insert one into an app behind its back:

void main() {
  if (!kReleaseMode) {
    MarionetteBinding.ensureInitialized();
    runApp(const MarionetteDeviceConfig(child: MyApp()));
  } else {
    WidgetsFlutterBinding.ensureInitialized();
    runApp(const MyApp());
  }
}

Without it, set_device_config responds with these setup instructions instead of succeeding. Because it lives in main(), adding it also needs a hot restart. !kReleaseMode (rather than kDebugMode) is the better gate here because it additionally covers profile builds, where the VM service is still reachable — use the same gate for both the binding and this widget, since gating the widget more loosely than the binding gains nothing (it degrades to a no-op child-passthrough if the binding isn't there, so a missed gate costs one inert element, never a crash).

The single-binding rule

Flutter allows exactly one WidgetsBinding per process. If something else claims it first, MarionetteBinding.ensureInitialized() throws a clear error — except with one real offender, where it doesn't:

• flutter test running main() under kDebugMode collides with AutomatedTestWidgetsFlutterBinding. Guard against it — Platform.environment.containsKey('FLUTTER_TEST') — or give tests a separate entrypoint that skips MarionetteBinding entirely.
• Sentry is the known silent offender: SentryFlutter.init() always calls WidgetsBinding.ensureInitialized() inside its own appRunner-wrapping zone before your code runs, and it swallows the resulting binding error as if it were a reportable crash — so the app just hangs on the splash screen forever with no exception, no crash, and no log output. The fix is initialization order: call MarionetteBinding.ensureInitialized() before SentryFlutter.init(), not inside its appRunner. Sentry checks for an existing binding first and reuses it, so going first avoids the conflict entirely. The same ordering fix applies to any other plugin whose init() touches WidgetsBinding ahead of your appRunner.

Version alignment

The MCP connect tool checks that marionette_mcp and the app's marionette_flutter are on the same version, and fails with a clear message rather than a confusing runtime error if they aren't. The CLI does not perform this check — a stale marionette_flutter there can fail in less obvious ways instead of a clear mismatch error. Either way, don't chase a mismatch by retrying — align the versions: flutter pub add marionette_flutter in the app, and dart pub global activate marionette_mcp/marionette_cli for the bridge (or dart pub add dev:marionette_mcp/dev:marionette_cli if you'd rather pin it as a dev dependency instead of a global tool).

When to use this

Reach for Marionette any time the fastest way to know whether something works is to actually run the app and look, rather than reason about the code:

• After implementing a feature. Connect, navigate to the new screen, and exercise the happy path — a working build isn't the same as a working feature.
• After a UI or design-system change. Confirm the affected screens still render and respond as expected, not just that the app compiles.
• Reproducing a bug from a ticket. Follow the reported steps, capture a screenshot and get_logs output as evidence the bug is real and what it looks like — before touching any code.
• Verifying a bugfix. Reproduce with the same steps, apply the fix, hot_reload, and replay those exact steps again to confirm the symptom is gone (not a different, adjacent symptom).
• Walking a "behaviour-neutral" refactor. Claims that a refactor changes nothing observable are exactly the claims worth checking — a quick regression walk of the touched screens catches what review alone misses.
• An accessibility sweep. Use set_device_config to try text_scale around 2.0–3.0 (real devices top out near 3.0) and bold_text: true on the screen in question, then check get_logs for overflow errors — that combination is where cramped layouts usually break first. (Needs the app to have opted in via MarionetteDeviceConfig — see Device-config sweeps.)
• Form and validation work. Enter deliberately invalid input and read the error strings back via get_interactive_elements/take_screenshots — it's the only way to confirm the user-visible message is right, not just that a validator returns an error code.
• Evidence for a PR or ticket comment. A quick before/after take_screenshots pair is often more convincing, and faster to produce, than a written description of a UI change. For a multi-step flow rather than a single static change, record-video (CLI-only, no MCP equivalent — see the capability table) captures the whole interaction as a short clip, which shows off a new feature or a fixed flow better than a screenshot pair can.

When not to use this

• CI or a repeatable regression suite. Marionette's gestures are best-effort simulations of user input, not deterministic instrumentation — results can vary across platforms, custom widgets, and overlays. For a suite that needs to pass or fail the same way every time, use Patrol instead.
• Release builds. Not a limitation to work around — Marionette relies on the Dart VM Service, which doesn't exist in a release binary. Debug and profile only.
• Performance work. Marionette can trigger the interactions you want to profile, but measuring them is Flutter DevTools' job, not this one.

Connecting

Every action needs a VM service WebSocket URI (ws://127.0.0.1:PORT/TOKEN=/ws) — the developer running the app has it in their flutter run console output, or in a DevTools link. Ask for it and pass it to connect (MCP) or --uri / a registered -i <name> (CLI) before anything else.

If more than one running instance is already in play — several apps registered with the CLI, or the user's message mentions more than one device or simulator — don't guess which one to use. Either infer the right one from what's already been said in the conversation (e.g. "the iOS one" when only one registered instance is iOS), or ask which to target. Getting the wrong instance is easy to miss until several steps later, so it's worth the one question up front.

For the CLI, prefer --uri for a one-off session and marionette register <name> <uri> + -i <name> when you'll issue many commands against the same app in a row. When you're done for the session (MCP), call disconnect — it's not required for correctness (a fresh connect implicitly replaces it), but leaving a session dangling makes it easy to act on a stale connection later without noticing.

Give connect a session_title (a short description of what you're testing, e.g. "profile validation" — keep it under ~60 characters, longer is truncated) — if the app enables session reports (MarionetteConfiguration(enableSessionReports: true)), it opens a fresh session directory that carries this run's step log and screenshots. One Marionette run is one session directory, always: there's no resuming, even if you pass the same title again on a later connect — that just opens another, separate directory. If a run gets interrupted partway through (a compaction, a dropped connection), its session is simply abandoned; don't try to pick up where it left off, and don't reuse its title. Always give it in English, regardless of the prompt's own language (see The report contract, below) — it gets slugified straight into the session directory's name, and that name should stay predictable and filesystem-friendly rather than following the run's language. See Reporting what you found for what that directory is for and what you're expected to do with it before disconnecting.

What you can do once connected

Good practices

• Look before you act. Call get_interactive_elements (or take_screenshots for a visual check) before the first gesture on a new screen. Don't infer what's on screen from memory of the source code — the whole point of Marionette is to check the running state. The listing is compact by default: it drops rendering details and reports visible only when an element is not visible. Pass compaction: "none" (--compaction=none in the CLI) when you need the full property dump.
• Selector priority: key > identifier > text > type/coordinates. Keys (ValueKey<String>) and Semantics identifiers survive copy changes, localization, and refactors; text breaks the moment a label is edited or translated; coordinates break on any layout shift, font-scale change, or screen-size difference. Treat coordinates as a last-resort diagnostic, not the normal way to target something — if nothing else works, that's a sign the widget needs a key or Semantics(identifier: ...), not a reason to keep using coordinates.
• Repeated keys: scope the match with ancestor_keys. Every tool takes the first match in the tree, so on a screen that repeats the same subtree (grid cells, repeated cards, a dev screen embedding several copies of the app) a plain tap(key: "cell.joinButton") always hits the first copy. Pass the keys of the wrappers around the one you mean, outermost first: {"key": "cell.joinButton", "ancestor_keys": ["session_2", "grid.cell_3"]}. Each key is looked up strictly inside the previous one, which is what lets you reach a wrapper whose own key also repeats (grid.cell_3 exists in every session), and the target itself must be inside the last one. A single key is often not enough — if the wrapper repeats too, go one level up.
  • Finding the chain: get_interactive_elements is a flat list in tree order, so a keyed wrapper, when it is listed at all, appears just before the elements inside it. A wrapper whose centre catches no hit test isn't listed, so if a key you expect is missing, read it from the source or ask for it. To confirm a chain, list only that subtree: get_interactive_elements(ancestor_keys: [...]). A scope alone is enough there, the wrapper itself is listed too, and a wrong key fails by name.
  • A key with no element fails the call, naming it and where it was looked for (Scope element with key "grid.cell_9" (ancestor_keys[1]) not found inside "session_2"). It never falls back to searching the whole screen, so a typo can't silently act on the wrong copy.
  • scroll_to works into lazy lists: a scope that isn't built yet (e.g. ancestor_keys: ["row_20"] in a long ListView) is looked for again after every drag, and the list that scrolls may be inside the scope or above it. A scope key that exists nowhere is reported only after the list has been scanned, like a missing target.
  • coordinates and focused_element ignore ancestor_keys, and a scope on its own is not a selector for the gesture tools.
  • Prefer this over tapping by coordinates or baking indices into keys: the chain survives layout changes and keeps production keys clean.
• An element "not found" is usually a setup gap, not a bug. Before concluding a widget can't be reached, suspect it's a custom widget type Marionette doesn't recognize yet, or custom-painted content with no Semantics annotation — both fixed under Preparing the app, above, not something to work around here.
• If coverage looks low across a whole screen rather than one missing element, don't reach for shouldStopTraversalAtElement. It's tempting to suggest filtering a scroll container out of traversal to "reduce noise," but that's the one config change measured to make things worse (25.8% → 17.8% widget coverage on a real app) — see Custom design system?, above.
• Confirm side effects with get_logs, not just UI state. A tap that looks like it worked isn't the same as the network call actually firing.
• Give the app's flows explicit context. Marionette can see the widget tree and act on it, but it has no idea what the product's flows, naming conventions, or edge cases are. State which screen to reach, the expected keys/labels, any precondition the flow assumes (e.g. "assume the user is already logged in"), and the interaction goal in one sentence, rather than assuming that's inferable from the UI alone.
• Treat every gesture as best-effort. Focus, text entry, and scrolling are simulations of user input and can behave differently across platforms, custom widgets, or overlays. A flow that's consistently flaky (not just occasionally) is a signal to expose a clearer key/identifier, flatten a deeply nested GestureDetector hit target, or extend MarionetteConfiguration — not to retry harder.