Live Debug

Live debugging

just run_local and just stack up start the LGTM collector by default and the agent browser with --with-chrome (both global: one per machine, shared across instances, left running):

--traces off disables the collector; --traces jaeger|datadog swaps it. All collectors bind 4317/4318 — exactly one runs at a time.

Every Rust service exports spans AND its tracing events (as correlated log records) over OTLP. The frontend exports browser spans through the proxy and propagates traceparent, so one trace covers browser → proxy → services. Wiring happens at stack start: if you start a collector by hand, restart the stack to pick it up.

Query telemetry (prefer APIs over the Grafana UI)

The grafana MCP server (.mcp.json / opencode.json / .cursor/mcp.json, Docker mcp/grafana on the host network) is pointed at this Grafana. Prefer its tools:

• Logs: query_loki_logs, list_loki_label_values, find_error_pattern_logs
• Traces: tempo_traceql-search, tempo_get-trace, tempo_get-attribute-values, tempo_docs-traceql (proxied from Tempo's own MCP server; every tempo_* call needs datasourceUid: "tempo")
• Metrics: query_prometheus; plus search_dashboards, generate_deeplink

Datasource UIDs are stable: loki, prometheus, tempo, pyroscope.

Typical calls — service names are the binary names (email_service, document-storage-service); match on route, duration, or any span attribute:

• tempo_traceql-search {datasourceUid: "tempo", query: '{resource.service.name="email_service" && status=error}'} (also '{span.http.route="/documents" && duration>500ms}'), then tempo_get-trace with the returned trace ID. Searches default to the past hour; widen with RFC3339 start/end.
• query_loki_logs {datasourceUid: "loki", logql: '{service_name="email_service"} |= "error"'} — log lines carry trace_id/span_id for correlation. Discover services with list_loki_label_values on service_name.

Timing quirks: Tempo's search index flushes every ~30s — a trace you just produced is fetchable by ID immediately but may not show in search yet.

Everything above is also plain HTTP: TraceQL search at http://localhost:3200/api/search?q=<traceql>, trace fetch at /api/traces/<id>, LogQL at http://localhost:3100/loki/api/v1/query_range?query=<logql> (start/end are unix epoch nanoseconds). Use the HTTP form when there is no MCP (pi), and for bulk retrieval you want to reduce before reading — a full trace can be 50+ spans, so curl /api/traces/<id> | jq (filter to slow spans, compute offsets) beats dumping tempo_get-trace output into context. docker compose -p macro logs -f <service> still works for raw stdout, but Loki is queryable and survives restarts.

Verbosity knobs (set in the shell before just run_local, or per service in Doppler): RUST_LOG filters console + Loki output; OTEL_TRACE_FILTER independently filters exported spans (default info). Lowering RUST_LOG never silences traces.

Drive the shared headless Chrome

--with-chrome runs a headless Chrome in Docker with CDP on 9222 (if 9222 doesn't answer, start it: the compose command is in the headless-chrome comment in docker/docker-compose.yml). The chrome-devtools MCP server (.mcp.json / opencode.json / .cursor/mcp.json) is already pointed at it — prefer its tools (navigate, snapshot, click, evaluate, console, network) for browser work. State (cookies, login) persists across agent sessions until the container restarts.

• The container uses host networking, so plain localhost URLs work: the app is http://localhost:3000/app, the proxy https://localhost:8090 (named instances remap these ports; read the stack summary).
• A human can watch the browser live at http://localhost:6080/vnc.html (noVNC over the Xvfb display) — work in the visible window, not isolated contexts, when someone may be watching.
• Login is passwordless: any email works, and the login API returns the code in its response (also visible in Mailpit at http://localhost:8025).
• From Playwright instead: chromium.connectOverCDP('http://localhost:9222'). For token-injection and route-interception recipes see apps/web/docs/playwright-debugging.md.

Correlate a browser repro with backend traces: note the time, then search Tempo for that window — the browser's traceparent means the frontend action and the Rust handler share one trace ID.

chrome-devtools technique

• Snapshot-first: take_snapshot after every navigation or pane change; act only on uids from the latest snapshot (uid prefixes bump on re-render). Macro snapshots are huge — split panes duplicate the doc text — so save big ones to a file (filePath param) and grep them.
• navigate_page can time out while the SPA actually loaded (cold Vite compile); follow with wait_for on expected text instead of re-navigating.
• wait_for matches any text presence, including placeholders. For "AI finished"-style conditions, poll in evaluate_script for the Stop button's absence — the only reliable completion signal.
• fill works on plain inputs but NOT contenteditable: click to focus, then type_text (Enter splits paragraphs/sends). Combobox token fields need type → wait for the "N options available" live region → Enter to tokenize.
• If a radio/tab control won't click ("did not become interactive"), click its adjacent label text node instead.
• On any error dialog or blank state: list_console_messages + list_network_requests (filter xhr/fetch), then get_network_request for the failing request's body — pairing console error with failing request localizes the fault in one step.
• Verify editor state with evaluate_script (e.g. query [contenteditable] strong to confirm an AI edit) — cheaper and more precise than screenshots.