Verify Lemon

Verify Lemon

A runbook for verifying lemon behavior end to end. Work in three tiers and always prefer the lowest tier that can answer the question — most claims about routing, sessions, runs, delivery, and transport plumbing never need real Telegram.

If $ARGUMENTS is provided, scope the verification session to that feature.

Tier 1 — DIRECT (no Telegram)

Attaching to a running instance

Discover the node name and cookie from the beam process args (or fall back to bin/lemon defaults: node lemon, cookie lemon_gateway_dev_cookie, control plane 4040, web 4080, sim UI 4090):

More than one beam can be running (production + a disposable instance you started, e.g. via scripts/product_smoke_local). NEVER pick one blindly — list them, check the registered names, and select the node you actually mean:

epmd -names                    # every distributed node on this machine
ps -eo pid,args | grep beam.smp | grep -v grep | grep -o '\-sname [^ ]*'

Then extract the cookie from the args of that specific node's process:

# -sname and -setcookie are adjacent "<flag>\n<value>" pairs in the beam args
NODE=lemon_bot   # <- the node you chose above, not just "the first beam"
ARGS=$(ps -eo args | grep beam.smp | grep -v grep | grep -- "-sname[= ]$NODE" | head -1 | tr ' ' '\n')
COOKIE=$(echo "$ARGS" | grep -A1 '^-setcookie$' | tail -1)   # NEVER echo/print this

The read-only-on-production rule below depends on this selection being right: if two nodes are up and you are about to mutate state, double-check you are attached to the disposable one (its name will look like product_smoke_<pid> or whatever --sname you passed).

Non-interactive one-shot eval on the node (the expression runs in a fresh process on the remote node, so a subscribe + receive in one expression works):

elixir --sname "probe_$$" --cookie "$COOKIE" \
  --rpc-eval "$NODE@$(hostname -s)" \
  'LemonCore.RunStore.list_sessions() |> length() |> IO.inspect(label: "sessions")'

For interactive introspection use iex --sname attach_$$ --cookie "$COOKIE" --remsh "$NODE@$(hostname -s)".

Read-only by default. Against the user's production node, restrict yourself to inspection (:sys.get_state/1, store reads, Bus.subscribe). Anything that starts runs or sends messages belongs on a disposable instance.

Control-plane WebSocket (port 4040)

An unauthenticated local connect gets the operator role with all scopes (parse_role(_) -> :operator in LemonControlPlane.Auth.Authorize), so a plain WS client can drive everything. Frames are {"type":"req","id":...,"method":...,"params":{...}}; the server answers hello-ok to connect and {"type":"res","id":...} to requests. Useful methods (exact names from LemonControlPlane.Methods.Registry):

• connect → hello-ok handshake (send {"client":{"id":...,"name":...}})
• chat.send — params sessionKey (required), prompt, optional agentId, queueMode
• agent — params prompt (required), session_key, model, idempotency_key; returns run_id
• agent.wait — params runId, timeoutMs; blocks until the run completes
• events.subscribe — params topics (allowed: all, system, cron, nodes, presence, exec_approvals, channels, goals, plus run:<id> / runId)
• sessions.list — params limit, offset, agentId
• logs.tail — params limit (max 1000), level

Minimal probe with python-websockets (no install needed via uv; websocat works too if present):

uv run --with websockets python - <<'PY'
import asyncio, json, websockets

async def main():
    async with websockets.connect("ws://127.0.0.1:4040/ws") as ws:
        async def req(id_, method, params):
            await ws.send(json.dumps(
                {"type": "req", "id": id_, "method": method, "params": params}))

        async def recv_until(pred):
            while True:
                frame = json.loads(await ws.recv())
                if pred(frame):
                    return frame

        await req("c1", "connect", {"client": {"id": "verify", "name": "Verify"}})
        await recv_until(lambda f: f.get("type") == "hello-ok")

        await req("a1", "agent", {"prompt": "ping",
                                  "session_key": "agent:verify:main"})
        run = await recv_until(lambda f: f.get("type") == "res" and f.get("id") == "a1")
        run_id = run["payload"]["run_id"]

        await req("w1", "agent.wait", {"runId": run_id, "timeoutMs": 10000})
        done = await recv_until(lambda f: f.get("type") == "res" and f.get("id") == "w1")
        assert isinstance(done["payload"]["answer"], str), done
        print("native run ok:", run_id)

asyncio.run(main())
PY

Observing runs on the bus

From an attached node (or --rpc-eval), subscribe to LemonCore.Bus topics — "run:<run_id>" and "session:<session_key>" carry typed events (LemonCore.Event structs; catalog in docs/platform/bus-events.md):

elixir --sname "probe_$$" --cookie "$COOKIE" --rpc-eval "$NODE@$(hostname -s)" '
LemonCore.Bus.subscribe("session:agent:verify:main")
receive do
  %LemonCore.Event{} = event -> IO.inspect({event.type, event.ts_ms}, label: "event")
after
  30_000 -> IO.puts("no event in 30s")
end
'

Observing outbound channel delivery (ChannelDelivery)

Every LemonChannels.Dispatcher.dispatch/1 (success and failure) now emits:

• telemetry [:lemon, :channels, :dispatch] — measurements %{count: 1, duration: native}, metadata %{channel_id, account_id, kind, intent_id, run_id, session_key, ok}
• a typed LemonCore.Events.ChannelDelivery broadcast as :channel_delivery on the "channels" bus topic (fields: intent_id, run_id, session_key, channel_id, account_id, peer_kind, peer_id, thread_id, kind, text_preview ≤200 chars, ok, error, duration_ms, ts_ms)
• a control-plane WS event channel.delivery (camelCase payload), received by clients subscribed via events.subscribe with topics: ["channels"]

So "did lemon actually deliver a reply to the channel?" is answerable without touching the channel:

elixir --sname "probe_$$" --cookie "$COOKIE" --rpc-eval "$NODE@$(hostname -s)" '
LemonCore.Bus.subscribe("channels")
receive do
  %LemonCore.Event{type: :channel_delivery, payload: p} ->
    IO.inspect({p.channel_id, p.kind, p.ok, p.text_preview}, label: "delivery")
after
  30_000 -> IO.puts("no delivery in 30s")
end
'

Injecting synthetic channel-shaped inbound

To exercise the router pipeline with a channel-shaped message (bypassing any transport), build a %LemonCore.InboundMessage{} (enforced keys: channel_id, account_id, peer, message) and hand it to LemonChannels.Runtime.submit_inbound/1 — on a disposable/test instance only, since this starts a real run and a real outbound delivery:

LemonChannels.Runtime.submit_inbound(%LemonCore.InboundMessage{
  channel_id: "telegram",
  account_id: "default",
  peer: %{kind: :dm, id: "310001", thread_id: nil},
  sender: %{id: "310001", username: "probe", display_name: "Probe"},
  message: %{id: "1", text: "hello from probe", timestamp: System.system_time(:second), reply_to_id: nil},
  raw: %{},
  meta: %{}
})

Deterministic agents-under-test

• Local provider fixture — scripts/product_smoke_local boots a disposable native executor against a deterministic Responses API fixture with no live provider credentials.
• LemonPlatformTest.FakeLLM — scripted stream function for LemonAgent loops in ExUnit: FakeLLM.script([{:tool_call, "name", %{...}}, {:text, "answer"}]) yields a conforming provider stream, letting you assert tool-call handling without a network.

Spinning a disposable instance

scripts/product_smoke_local is the reference recipe (dev boot; --release for CI-parity release boot). It is safe next to the production node: temp HOME (never the real ~/.lemon), unique --sname product_smoke_$$ + random cookie, dynamic free ports, isolated store/dotenv, kills only what it started. Run it as-is for a green/red product check, or copy its isolation levers for a custom instance:

scripts/product_smoke_local            # PASS/FAIL + .lemon/proofs/product-smoke-local-latest.json
PRODUCT_SMOKE_KEEP_WORKDIR=1 scripts/product_smoke_local   # keep workdir for debugging

Port-collision env knobs (all honored by config/runtime.exs): LEMON_CONTROL_PLANE_PORT, LEMON_WEB_PORT, LEMON_SIM_UI_PORT, LEMON_GATEWAY_HEALTH_PORT=0, LEMON_ROUTER_HEALTH_PORT=0 (0 = ephemeral bind, avoids the running instance's fixed 4042), plus LEMON_STORE_PATH, LEMON_DOTENV_DIR, LEMON_GATEWAY_NODE_NAME, LEMON_GATEWAY_NODE_COOKIE.