Pensieve Search

Pensieve Search Skill

Pensieve (memos) continuously captures screenshots and indexes them. This skill teaches you to query that index over its HTTP API.

Where the data lives

• Server: http://127.0.0.1:8839 by default (config: server_host / server_port).
  • Override with MEMOS_SERVER_HOST / MEMOS_SERVER_PORT env vars or ~/.memos/config.yaml if the user has a non-default setup. Confirm with the user if the default doesn't respond.
• Screenshots on disk: ~/.memos/screenshots/YYYYMMDD/<entity>.webp. The API returns filepath directly so you don't have to compute it.
• Web UI (same server, browser-friendly):
  • Search page (mirrors API filters as URL params): http://127.0.0.1:8839/?q=...&start=...&end=...&app_names=...&library_ids=... — you should always offer this URL in your reply so the user can click and verify results visually.
  • Entity detail page: http://127.0.0.1:8839/entities/<id> — opens one specific screenshot in context with its metadata. Offer this for any hit you cite.
  • Config page: http://127.0.0.1:8839/config — surface this if the user asks about settings.

Pre-flight check

Before the first query in a session, verify the server is up:

curl -s -o /dev/null -w "%{http_code}\n" "http://127.0.0.1:8839/api/search?q=test"

200 means good. 000 or connection-refused means memos isn't running — tell the user to start it (memos serve or system-service).

Always quote the URL — the ? and & are zsh glob characters and will fail with no matches found if unquoted. Every curl example below assumes a quoted URL.

The endpoint

GET /api/search

Parameters:

Returns SearchResult { hits: [...] }. Each hit:

{
  "document": {
    "id": "12345",
    "filepath": "/Users/.../screenshots/20260423/00001.webp",
    "file_created_at": "2026-04-23T04:18:21+00:00",
    "tags": [],
    "metadata_entries": [
      {"key": "active_app", "value": "iTerm2", "source": "..."},
      {"key": "active_window", "value": "✳ Debug memos service background job issue", "source": "..."},
      {"key": "ocr_result", "value": "...long JSON of OCR boxes...", "source": "ocr"},
      {"key": "structured_vlm_v1_qwen3_6_35b", "value": {"primary": {"app": "Claude Code", "title_or_topic": "...", "what": "...", "workspace": "memos"}}, "source": "structured_vlm"},
      ...
    ]
  }
}

The metadata_entries is the goldmine — the VLM-extracted structured fields (primary.app, primary.title_or_topic, primary.what, primary.workspace) live inside the structured_vlm_v1_* entry's value. In /api/search hits the value is a parsed JSON object, not a string — access fields directly with .value.primary.what, and do NOT pipe through fromjson (it errors with "only strings can be parsed"). Note the inconsistency: /api/entities/{id} and the /context endpoint (Strategy 6) return the same value as a JSON string, where you do need fromjson. The portable branch is (.value | if type=="string" then fromjson else . end). The exact key suffix (e.g. qwen3_6_35b) varies per install — match with select(.key | startswith("structured_vlm")).

Important: that JSON metadata is also fully tokenized into the FTS index. So q="Claude Code Mastra workspace" will match a screenshot whose VLM said app=Claude Code, what="...Mastra...", workspace=... — even if "Mastra" doesn't appear in the OCR text.

How to query well

Strategy 0 — look at raw results before narrowing

Resist the urge to pre-anchor on what you think the user means. A "find 最终幻想" question sounds like it's about the game, but the same brand lives across YouTube videos, Bilibili clips, wallpaper engines, wikis, store pages, and game launchers. The user's casual phrasing names the topic, not the surface — only the corpus knows where they actually consumed it.

So run one wide query first and let the results tell you what surfaces exist:

curl -s --get 'http://127.0.0.1:8839/api/search' \
  --data-urlencode 'q=最终幻想' --data-urlencode 'facet=true' --data-urlencode 'limit=20' > /tmp/raw.json
jq -r '.hits[].document
  | "\(.file_created_at)  \([.metadata_entries[] | select(.key=="active_app") | .value][0] // "?")  \([.metadata_entries[] | select(.key=="url") | .value // "-"][0])  \([.metadata_entries[] | select(.key=="active_window") | .value // "-"][0] | .[0:80])"' /tmp/raw.json

Sample output (verified on a real install):

2026-04-03T15:26Z  Google Chrome  about:blank                                    (55) When Tifa played「Tifa's T…Rebirth🖤 Ru's Piano - YouTube
2026-04-02T15:47Z  Google Chrome  about:blank                                    (55) Final Fantasy X「To Zanark… Medley | Ru's Piano - YouTube
2026-02-15T15:59Z  Google Chrome  https://www.bilibili.com/video/BV1K2HQzbEAy…  “蒂法的审视 手机动态壁纸 wallpaper engine_哔哩哔哩_bilibili”🔊
2026-01-30T05:17Z  Google Chrome  https://www.bilibili.com/                      哔哩哔哩 (゜-゜)つロ 干杯~-bilibili

Now you know: the user doesn't play Final Fantasy — they watch Ru's Piano play FF themes on YouTube, and 蒂法的审视 wallpaper engine clips on Bilibili. If you'd anchored on q=最终幻想 game or guessed app_names=Final Fantasy you'd have found nothing. The corpus revealed both the surfaces (YouTube + Bilibili) AND the user's actual relationship to the topic (music + wallpapers, not gameplay).

Now you can build per-surface queries anchored on each surface's stable identifier and merge the timelines.

Anchor on stable identifiers, not free-text names, because free-text is noisy — short or common substrings match the library_ids= parameter inside memos's own URL bar, and a CJK brand name often matches OCR'd filenames of unrelated downloaded videos. Stable identifiers come from the URL or the exact app name:

• Website: domain in the URL (guancha.cn, youtube.com/watch?v=…)
• Bilibili channel: UID in space.bilibili.com/<uid> (e.g. 10330740)
• YouTube channel: handle in URL (@mastra-ai, @RusPiano)
• App / brand on macOS: exact active_app value (iTerm2, Google Chrome, 企业微信)

Cross-check every match by filtering hits client-side on the URL or app fields — never trust the FTS hit list alone for "did the user actually visit / consume X" questions. The skill's job is to use multiple raw queries + client-side filtering to give the user a true answer, not to take the first FTS rank as gospel.

Strategy 1 — content keywords first

curl -s 'http://127.0.0.1:8839/api/search?q=Mastra+roadmap&limit=15' | jq '.hits[].document | {id, filepath, file_created_at}'

Joining terms with + (URL-encoded space) gives an AND query (jieba splits, then FTS5 does AND-of-tokens). Mix English + Chinese freely: q=理赔+Demo+演示.

Strategy 2 — narrow with app

The user's active_app is recorded literally per OS. Common values:

• macOS: Google Chrome, iTerm2, Claude, Cursor, WeChat, 企业微信
• Windows: msedge.exe, chrome.exe, WeChat.exe, claude.exe

curl -s 'http://127.0.0.1:8839/api/search?q=mastra+roadmap&app_names=Google+Chrome&limit=10' \
  | jq '.hits[].document | {id, filepath, file_created_at}'

Strategy 3 — narrow with time

Times are UNIX seconds, UTC. Compute with date:

# "last 7 days"
SINCE=$(date -v-7d -u +%s 2>/dev/null || date -d '7 days ago' -u +%s)
NOW=$(date -u +%s)

curl -s "http://127.0.0.1:8839/api/search?q=mastra&start=$SINCE&end=$NOW&limit=20"

For specific local dates: parse the user's date words (e.g. "上周三" → resolve to absolute → 00:00 local → UNIX UTC) and use a 1- or 24-hour window.

Strategy 4 — combine

The fastest path to a precise hit is q + app + tight time window. For "find the YouTube Mastra roadmap livestream from last week":

SINCE=$(date -v-10d -u +%s 2>/dev/null || date -d '10 days ago' -u +%s)
NOW=$(date -u +%s)
curl -s "http://127.0.0.1:8839/api/search?q=mastra+roadmap+youtube&app_names=Google+Chrome&start=$SINCE&end=$NOW&limit=10" > /tmp/hits.json
jq '.hits[].document | {id, filepath, file_created_at}' /tmp/hits.json

Tip: pipe the response to a tmp file before complex jq filters. Inline shell-quoted jq with nested \"...\" escapes is fragile — use a file.

Strategy 5 — facet to discover

When q returns hundreds of results, set facet=true to see what apps and time range dominate the matches:

curl -s 'http://127.0.0.1:8839/api/search?q=mastra&facet=true&limit=5' \
  | jq '{date_range, bucket_unit, date_buckets: .date_buckets[:10], facet_counts: .facet_counts[0].counts[:10]}'

• date_range (top-level): {earliest, latest} ISO timestamps spanning all matched entities under the current filters. Use it to suggest a tighter start/end window.
• date_buckets (top-level): [{date, count}] grouped by day or month — the bucket unit is adaptive based on the matched span (≤ 60 days → day, else month). The chosen unit is reported in bucket_unit. Single-bucket results are suppressed (returned as empty + bucket_unit=null) since they're not useful for narrowing.
• facet_counts[0].counts: list of {value, count} for active_app, sorted desc. Use it to suggest an app_names= filter, or to ask the user which app they meant.

All three are populated only when facet=true (or when settings.facet=true server-side).

To drill into a bucket, re-issue with ?date=YYYY-MM or ?date=YYYY-MM-DD. The server intersects date with the existing filters; if you also set start/end, the effective range is the overlap. After drilling into a month, the next response's date_buckets automatically adapts to days within that month.

Strategy 6 — context around an entity (grep -C for the timeline)

Search pinpoints a moment; it doesn't show what surrounds it. When a hit lands on the right frame but the answer isn't on that frame — the working directory a command ran in, what led up to it, what happened next — pull its temporal neighbors. Think grep -C N for the screenshot timeline, especially the before side where the lead-up lives (the cwd, the command typed, the app you came from). This is what the web UI's context strip does.

GET /api/libraries/{library_id}/entities/{entity_id}/context?prev=N&next=N

• prev / next are counts (0–100 each), ordered by file_created_at — true temporal neighbors. Robust where id arithmetic is not: entity ids are sparse and non-contiguous, so "fetch ids around X" silently misses frames. Always use this endpoint, never id math.
• Returns { "prev": [entity…], "next": [entity…] }. The target is not included — you already have it (the id you passed), sitting chronologically between the last prev and the first next.
• library_id comes from the entity itself: curl …/api/entities/{id} | jq -r .library_id (or reuse LIB). Frontend default (web/src/lib/api/entities.ts → fetchEntityContext) is a symmetric ?prev=12&next=12 (grep -C 12) — match it.

Three gotchas, all differing from /api/search:

1. It does NOT strip ocr_result. A prev=12&next=12 call is ~230 KB; prev=100&next=100 is multi-MB. Project only the fields you need (jq below) — never dump the raw response.
2. Timestamps are naive UTC — 2026-06-07T01:51:51, no Z/offset. They are UTC; append Z (or convert) before you cite a time, or you'll report the wrong hour.
3. structured_vlm value is a JSON string, not a parsed object — fromjson it to reach .primary.what.

Recipe. Get library_id, then project just the fields the question needs — that projection is the jq's only job (it keeps the bloated response small):

B=http://127.0.0.1:8839; id=1748373
lib=$(curl -s "$B/api/entities/$id" | jq -r .library_id)
curl -s "$B/api/libraries/$lib/entities/$id/context?prev=12&next=12" | jq -r '
  .prev[], .next[] | "\(.id)\t\(.file_created_at)Z\t"
    + ([.metadata_entries[]|select(.key=="active_window").value][0] // "-") + "\t"
    + ([.metadata_entries[]|select(.key|startswith("structured_vlm")).value][0] // "{}" | fromjson.primary.what // "-")'

Don't capture the body with $(curl …) — OCR/VLM text mangles JSON in command substitution; pipe straight to jq (or a tmp file).

Worked example — the directory a codex command ran in. Search pinpointed the codex launch (entity 1748373), but that frame was full-screen with the version-update prompt — no directory visible. Pull its neighbors (the recipe above, id=1748373):

1748372  2026-06-07T01:51:45Z  you@host:~/projects/memos/pensieve-rpa  在终端中执行 'z pensieve' 命令切换目录并准备运行…
1748374  2026-06-07T01:51:56Z  ✳ Review blocking feature project for…  …

The frame 6 s before the launch (1748372) shows the shell prompt …:~/projects/memos/pensieve-rpa and a VLM note that the user ran z pensieve to jump directories — recovering the answer (~/projects/memos/pensieve-rpa) that wasn't on the target frame at all. (Target 1748373 itself isn't in the strip; you supply it.)

Widening (mirrors clicking an edge thumbnail in the UI). If the lead-up frame isn't in the window yet: bump the counts (prev=30&next=30 → 60 → 100, the per-side max), or re-center on the oldest frame returned and pull its context — the same walk-further-back the UI does, keeping each response small so you can scan back indefinitely.

Citing it in a reply. When context resolves the answer, cite the lead-up frame's entity URL next to the pinpoint hit, e.g. "ran in ~/projects/memos/pensieve-rpa — the prompt is visible 6 s earlier → http://127.0.0.1:8839/entities/1748372". The user clicks through to verify the exact frame.

Handling large result sets

/api/search caps limit at 200 per request, but the response now exposes the real total under filters in the found field. Use it directly — no heuristics needed.

Reading found and out_of

Every response includes:

{ "found": 2883, "out_of": 1464297, "hits": [ ... 200 items ... ], ... }

• found — unbounded count of FTS matches under your filters (q, start/end, app_names, library_ids). This is the truthful answer to "how many things matched my keywords?". If found > limit, there are more matches you didn't get back.
• out_of — total entities in the collection scope (library_ids only — q/start/end/app_names are dropped). Typesense convention: "found N out of M total". Surface this only when the user asks dataset-level questions ("how many screenshots do I have?"); for normal "find X" replies use found.

Edge case: very rarely, found < len(hits) when vector search contributes hits whose FTS rank was zero (e.g. queries with no good keyword match but semantic neighbors). Treat found as the keyword-match count, with the understanding that hits may include a few extra semantic neighbors.

Decision flow

resp  = search(q, start, end, limit=200)
hits  = resp.hits
found = resp.found

if found <= len(hits):
    return hits                              # complete, done

if user wants "top match" / "first few":
    return hits[:N]                          # top-ranked are already here

if user wants comprehensive scan ("show me everything ..."):
    return time_slice(q, start, end)         # recipe below — still needed (no offset yet)

else:
    tell user: f"{found} matches, narrow with app or shorter time window"

Recipe: time-window slicing

For "show me every screenshot of X in the last week", recursively halve the time window when truncation hits. Always use a tmp file for the response body — bash command substitution $(curl ...) mangles JSON that contains literal newlines or control chars (which OCR / VLM metadata frequently does).

# bash function — paste into agent shell or save as ~/bin/pensieve-paged-search.sh
paged_search() {
    local q="$1" start="$2" end="$3"
    local tmp
    tmp=$(mktemp)
    curl -s "http://127.0.0.1:8839/api/search?q=$(echo -n "$q" | jq -sRr @uri)&start=$start&end=$end&limit=200" > "$tmp"
    local n
    n=$(jq '.hits | length' "$tmp")
    if [ "$n" -lt 200 ] || [ $((end - start)) -lt 60 ]; then
        jq -c '.hits[]' "$tmp"
        rm "$tmp"
        return
    fi
    rm "$tmp"
    local mid=$(( (start + end) / 2 ))
    paged_search "$q" "$start" "$mid"
    paged_search "$q" "$mid" "$end"
}

# Usage: collect all 'memos' hits in the last 7 days
SINCE=$(date -v-7d -u +%s 2>/dev/null || date -d '7 days ago' -u +%s)
NOW=$(date -u +%s)
paged_search "memos" "$SINCE" "$NOW" > /tmp/all_hits.ndjson
wc -l /tmp/all_hits.ndjson    # raw hit count (may include boundary dups)

Boundary duplicates: the mid second may match in both halves if multiple captures share that timestamp. Dedupe by id:

jq -s 'unique_by(.document.id)' /tmp/all_hits.ndjson > /tmp/dedup.json
jq 'length' /tmp/dedup.json

In testing, a "last 24h" query for q=memos produced 2888 raw hits → 2883 unique. Boundary dup rate is small but real.

Floor: 60-second window

If a 60-second window still saturates 200 hits, the burst is too dense to enumerate (e.g. continuous scrolling capturing every 4 s = 15 captures/min × 5+ apps multiplied = saturation). Stop recursion there and report: "this minute hit the cap; the screen activity was too dense to enumerate, please narrow further".

When NOT to slice

• Casual lookup → top hits are already most relevant; no slice needed
• User said "first 10" or "best match" → respect that intent, don't expand
• Truncation in a query that already has tight time + app filters → suggest different keywords instead of slicing

How to interpret results

Order matters — hybrid_search ranks by reciprocal rank fusion of FTS (weight 0.7) + vector embedding similarity (0.3). The top hit is most likely the right one for direct lookups; for "find all" queries, walk all hits.

Per hit, extract the human-readable summary. Save to a tmp file first to keep the jq filter readable:

curl -s 'http://127.0.0.1:8839/api/search?q=mastra&limit=10' > /tmp/hits.json

# Per-hit one-liner: timestamp + app + topic + truncated 'what'
jq -r '.hits[].document |
       "\(.file_created_at)  \([.metadata_entries[]
                               | select(.key | startswith("structured_vlm"))
                               | .value.primary
                               | "app=\(.app // "?")  topic=\(.title_or_topic // "?")  what=\((.what // "?")[0:80])"][0])"' /tmp/hits.json

Sample output (verified):

2026-04-30T01:35:21Z  app=Google Chrome  topic=mastra roadmap  what=在 Google 搜索框中输入并搜索 'mastra roadmap'
2026-04-30T01:35:26Z  app=Google Chrome  topic=mastra roadmap  what=在 Google 搜索框中输入并搜索 'mastra roadmap'
2026-05-03T13:03:00Z  app=iTerm2         topic=Agent 检索       what=在终端内运行 Claude Code 进行代码开发...

primary.what is in the user's interface language (Chinese for Chinese-locale machines). Don't translate unless asked — show it as-is.

Timestamp is file_created_at (ISO 8601 UTC). Convert to user-local for display.