Anki Gui Automation

Agentic GUI development for HyperTTS

Use this whenever you add or change a HyperTTS dialog. pytest-qt tests prove the logic; this harness proves the dialog is actually reachable, laid out correctly, and correct against a real Anki collection (real notes, real update_note, real undo).

Everything runs on a virtual display against an isolated Anki base folder and profile, so the developer's own Anki profile and collection are never touched. Both service ports are deliberately non-default (AnkiConnect on 8766, not 8765) so the harness can never talk to a real Anki session that happens to be open.

You must run scripts/gui_automation/teardown.sh when you are done. Leaving Xvfb, x11vnc, websockify and a headless Anki running is the main failure mode of this workflow.

The loop

cd scripts/gui_automation

./start_anki.sh                 # Xvfb + openbox + x11vnc + noVNC + Anki with HyperTTS
./ankiconnect.py seed --reset   # deck + note type + 5 notes with assorted sound tags
./ankiconnect.py browse         # open the browser on those notes
./gui_probe.py browser-select-all

./gui_probe.py actions --text Audio                   # what menu entries exist
./gui_probe.py trigger --text 'Remove Audio (Collection)...'   # open the dialog
./gui_probe.py windows                                # confirm it opened, and is modal
./gui_probe.py tree --window hypertts_remove_audio_dialog      # read the whole dialog as text
./gui_probe.py table --object-name hypertts_remove_audio_preview_table
./gui_probe.py screenshot --path /tmp/hypertts-gui-automation/artifacts/dialog.png \
    --window hypertts_remove_audio_dialog             # then Read the png

./teardown.sh                   # ALWAYS

After editing any HyperTTS python file, run ./start_anki.sh --restart. The add-on is symlinked into the profile, but Anki only imports it at startup. --fresh additionally throws away the collection.

Read text first, screenshot second. tree costs a few hundred tokens and tells you state (enabled, checked, current index, combo items); a screenshot costs thousands but is the only way to catch layout problems. Both matter — the two bugs found while building the Remove Audio dialog (a QLabel rendering <b> as literal text, and preview columns truncated by an even stretch) were invisible in the widget tree and obvious in the screenshot.

Watch it live in a browser at http://localhost:6099/vnc.html while it runs.

Widget naming convention (do this first, it unblocks everything)

Give every widget you create a stable objectName, prefixed hypertts_<screen>_:

self.field_combobox = aqt.qt.QComboBox()
self.field_combobox.setObjectName('hypertts_remove_audio_field')

Name the dialog too (self.setObjectName('hypertts_remove_audio_dialog')) so --window can target it. Without object names you have to address widgets by class + index path (RemoveAudioDialog/QGroupBox[0]/QComboBox[0]), which breaks the moment the layout changes. hypertts_addon/component_remove_audio.py is the reference implementation.

Older components (component_batch.py, component_voiceselection.py, …) have no object names yet. For those, address widgets by --class + --text, or by the path printed by tree.

scripts/gui_automation reference

Runtime state lives under /tmp/hypertts-gui-automation: anki_base/ (throwaway collection and add-ons), logs/anki.log, logs/hypertts.log (HyperTTS debug logging is on), artifacts/ (screenshots), pids/.

gui_probe.py

Inspect:

• windows — every top-level window: class, title, modal, active, geometry
• tree [--window W] [--named-only] [--all] [--max-depth N] — indented widget tree
• info --object-name X — one widget in detail
• table --object-name X — dump a QTableView/QTreeView model as rows (QVariant unwrapped)
• actions [--text substring] — every QAction, i.e. every menu entry
• undo-status — the label of the next undoable operation
• note-fields --note-id N — a note's fields straight from the collection

Drive:

• click --object-name X [--no-wait] — also --class QPushButton --text Cancel
• set-text --object-name X --value '...'
• combo --object-name X --text '...' (or --index N)
• check --object-name X [--off]
• select-row --object-name X --row N
• trigger --text 'Menu entry...' — opens dialogs; does not wait, by design
• close --title '...' [--class ...], undo, browser-select-all, browser-search --query ...
• screenshot --path P [--window W] — QWidget.grab() of one window
• raw '{"action": "eval", "params": {"expression": "mw.pm.name"}}' — escape hatch, main-thread eval with aqt, qt and mw in scope

Modal dialogs block the Qt main thread. Any action that opens one must not wait for the main thread, or the request times out. trigger defaults to not waiting; pass --no-wait to click when the click opens a dialog (including a QMessageBox such as HyperTTS's "Save changes to current preset ?" confirmation) or closes the window it lives on. When a probe call returns {"status": "pending"} that is not an error: a modal is open and holding the main thread — call windows to see it and deal with it.

Anki's own dialogs are reachable the same way, which is how you dismiss things like the add-on startup error box:

./gui_probe.py tree --window QMessageBox
./gui_probe.py click --class QPushButton --text '&No' --window QMessageBox --no-wait