Camera

Camera

Quick Start

Accesses the device's built-in camera at http://127.0.0.1:5001 to take snapshots or check the environment. Only use when the user explicitly asks you to look at something.

Already-captured frame (reuse, don't re-snapshot)

If the incoming turn contains a line like:

[vision-image] <absolute-path-to-a.jpg> (a photo was JUST captured ...)

a photo was already taken for this exact request by the realtime voice layer (it captured the frame, then handed the turn to you — e.g. it timed out mid-answer), and the OS layer delivers it with this very message — either as an [image description] line (when the main model is text-only, a vision model has already analyzed the photo for you) or as an attached image. Answer the visual question from that description/attachment. Do NOT call /api/vision/look or /camera/snapshot again — re-snapshotting wastes time and may capture a different moment than what the user asked about. Do NOT read the path with a file tool — it is there for traceability only, and on text-only models a file-read image is silently dropped.

Use the capture protocol below when no current image or description was supplied for the request.

Capture Protocol

One tool call. It checks that the camera can actually deliver a frame, and only then takes the photo, sizes it, and gives you back what is in it:

c=$(curl -s http://127.0.0.1:5001/camera); case "$c" in
  *'"disabled":true'*) echo "CAMERA_OFF" ;;
  *'"has_frame":false'*) echo "CAMERA_UNAVAILABLE" ;;
  *) curl -sX POST http://127.0.0.1:5000/api/vision/look \
       -H 'Content-Type: application/json' \
       -d '{"question":"<what the user asked>"}' ;;
esac

Read the output:

• CAMERA_OFF → the user turned the camera off (privacy). Say so in one sentence and that they can say "camera on" to turn it back on. Stop. Do not look, do not enable it yourself.
• CAMERA_UNAVAILABLE → the hardware is not delivering frames (not connected or not detected). Say the camera is not working right now. Stop.
• {"data":{"description":"...","path":"..."}} → answer from description. It is the only thing in this turn that actually saw the frame. No description field (the main model can read images itself) → open path with your file/image tool.
• Any other error → tell the user you couldn't see it this time. Do not guess.

Keeping the check inside the same shell command costs no extra tool round: a separate GET /camera turn would add one more model call (~5s) for every look. The server handles servo freeze, frame wait, and image sizing. No preparatory aim or sleep is needed unless the user explicitly requested a movement (see Move first, then snapshot).

For raw-frame export rather than a visual answer, use GET http://127.0.0.1:5001/camera/snapshot?save=true&width=768&quality=75 (HAL, port 5001). It writes a file and returns its path, without a description; the path alone is not visual evidence.

Never describe the view without an image

If you are about to say what you see, this turn MUST contain either a [vision-image] line or a /api/vision/look call whose answer you actually looked at, or a /api/vision/look description. Describing the room from memory, from an earlier turn's photo, or from a plausible guess ("same view — the desk, your screen…") is a fabrication, even when the guess happens to be close. No image → say you'll take a look and take one; never invent.

Move first, then snapshot

When the request combines a movement and a visual question ("turn right, hold it there, and tell me what you see"), fire the servo calls with curl during the turn (POST /servo/aim, POST /servo/hold), then call /api/vision/look. [HW:...] markers are executed only after your reply is composed, so a marker-based aim would move the device after the photo — you would describe the old view.

Workflow

1. Run the Capture Protocol command (the /camera check and the look are one shell call). CAMERA_OFF / CAMERA_UNAVAILABLE → answer from that word and stop.
2. Respond from the returned description, or inspect the returned path with an image tool when the server provides only a path (see Capture Protocol).

You also receive camera snapshots automatically as part of sensing events ([sensing:*] messages with images). You do not need the camera API for those — just look at the attached image.

Examples

Input: "What do you see right now?" Output: POST /api/vision/look → say: "I can see your desk with a laptop and a coffee mug. Looks like a productive setup!"

Input: "Is anyone in the room?" Output: POST /api/vision/look → say: "I can see one person sitting at the desk."

Input: "Take a photo" or "Send me a photo" Output: POST /api/vision/look → say what description reports.

Input: (sensing event with image already attached) Output: Do NOT call the camera API. Just look at the attached image and react.

Tools

Bash with curl — http://127.0.0.1:5000 for /api/vision/look, http://127.0.0.1:5001 for HAL camera control.

Look at the scene

curl -sX POST http://127.0.0.1:5000/api/vision/look -H 'Content-Type: application/json' -d '{"question":"..."}'

Returns {"data":{"description":"...","path":"..."}}. See Capture Protocol.

Live stream

curl -s http://127.0.0.1:5001/camera/stream

Returns an MJPEG stream (multipart/x-mixed-replace). Only use when continuous video is needed. Prefer snapshot for one-time checks.

Camera On/Off (Privacy Control)

Users can toggle the camera via voice or chat. Use HW markers — no curl needed.

Disable camera

[HW:/camera/disable:{}]

The user wants privacy. Camera stays off until the user explicitly re-enables it (voice or web toggle).

Enable camera

[HW:/camera/enable:{}]

Enabling is not seeing. The marker only flips the privacy switch; you have not captured a frame, and the hardware may not even be delivering one (a camera that is unplugged still "enables" fine). Never say "I can see you again", "there you are" or describe anything after an enable — say the camera is on and stop, or, if the user wants to be seen, run the Capture Protocol in the same turn and answer from what it returns.

Trigger phrases (MANDATORY — must call HW marker, not just reply with text)

Any phrase meaning "stop looking" or "camera off" MUST trigger [HW:/camera/disable:{}]. Any phrase meaning "look at me" or "camera on" MUST trigger [HW:/camera/enable:{}]. Do NOT just acknowledge — you MUST include the HW marker.