Fetch Tweets

<!-- autoresearch: variation B — sharper output via clustering + signal scoring + insight extraction. Merged HUB: absorbs tweet-digest, tweet-roundup, list-digest, refresh-x, agent-buzz behind a `source:` selector. -->

${var} — <source>:<arg> where <source> ∈ keyword | topic | account | list | agent-buzz. The <arg> is source-specific (a query, a topic, a handle, comma-separated list IDs, or an optional focus). If no source: prefix is given, the source is inferred from the shape of <arg> (see Source selector). Required for keyword and list; optional for topic, account, and agent-buzz.

Today is ${today}. This skill fetches X/Twitter content along one of five source axes and produces a curated digest — clustered by sub-narrative, ranked by signal, one insight per item — never a flat chronological dump.

Source selector

Parse ${var} into SOURCE and ARG before doing anything else.

Explicit form (recommended): <source>:<arg>

• keyword:$SOL OR solana OR "solana network" — raw X search query, passed to Grok verbatim (OR/AND honored).
• topic:brain-computer interfaces — a single topic roundup. topic: (empty arg) → resolve a topic list from MEMORY.md, then built-in defaults.
• account:vitalikbuterin — one account's recent tweets. account: (empty arg) → digest every handle in memory/topics/tracked-accounts.yml.
• list:1953536336675365173,1937207796270829766 — one or more numeric X list IDs. Append |<topic> for a topic booster: list:195...,193...|AI agents.
• agent-buzz — the curated AI-agent-ecosystem preset. agent-buzz:MCP protocol prioritizes a project/topic within the preset.

Implicit form (back-compat with migrated bare-var configs): when ${var} has no recognized source: prefix, infer SOURCE in this order:

1. ${var} is empty → topic (default multi-topic roundup).
2. ${var} is all-digits, or comma-separated all-digits (optionally with a |<topic> suffix) → list.
3. ${var} is @handle or matches ^[A-Za-z0-9_]{1,15}$ (a bare handle) → account.
4. Anything else → keyword.

Note: agent-buzz has no distinct implicit shape (its arg looks like a keyword/topic), so it is only selectable via the explicit agent-buzz / agent-buzz:... prefix.

Once SOURCE and ARG are set, jump to the matching branch below. Only one branch runs per invocation.

Shared preamble (all branches)

1. Read memory/MEMORY.md for context and the recent memory/logs/ (each branch specifies its lookback window — 2 or 3 days) to dedup already-reported tweets.

2. Load the dedup set SEEN_TWEETS by unioning two sources:

   • The branch's persistent seen-file (per-mode path below), if it exists — read all URLs.
   • The branch's log lookback window — grep each memory/logs/*.md file in range for lines matching https://x.com/.

   Per-mode seen-files (kept at their legacy paths so dedup history survives the merge):

   mode	seen-file	log lookback
   keyword	memory/fetch-tweets-seen.txt	3 days
   topic	memory/tweet-roundup-seen.txt	3 days
   account	(logs only — see branch)	2 days
   list	memory/list-digest-seen.txt	2 days
   agent-buzz	(logs only — 3-day status/<id> set)	3 days

3. Formatting invariants shared by every branch's notification:

   • Use x.com/handle (never @handle) so Telegram doesn't ping/tag users. (Exception: the account-digest and agent-buzz formats below historically use @handle in-body; keep their documented format but prefer x.com/handle when practical.)
   • Every surviving tweet gets a tappable Markdown link — [View](url) / [View tweet](url). If a URL is unavailable, drop the link and say "(link unavailable)".
   • Never fabricate engagement counts. Missing → 0, not a guess.
   • Notify only on signal. A legitimately empty or all-duplicate run logs its status and sends nothing.

Voice

Used by the account and agent-buzz branches for one-line takes/insights. If soul/SOUL.md and soul/STYLE.md are populated, read both and match the operator's voice. If they are empty templates or absent, write in a clear, direct, neutral tone — state what the tweet says, no hedging or editorializing beyond the tweet itself.

---

Branch: keyword (source:keyword)

Search X for tweets matching ARG and produce a curated digest grouped by sub-narrative.

Seen set: memory/fetch-tweets-seen.txt + last 3 days of logs (loaded in preamble).

1. Build the search prompt. Pass ARG to Grok verbatim as the query — do NOT narrow it to a single angle; broad coverage is the goal. Ask for at least 15–20 candidate tweets (you'll cull to ~7–10). Always require explicit engagement counts (likes, retweets, replies) so ranking is data-driven.

2. Fetch tweets. Record SOURCE_PATH=api|websearch for the log.

   Path A — X.AI API (primary; see the Fetching (all branches) contract — attempt this, set the Bash tool timeout ≥180000, capture the HTTP status):

   FROM_DATE=$(date -u -d "yesterday" +%Y-%m-%d 2>/dev/null || date -u -v-1d +%Y-%m-%d)
   TO_DATE=$(date -u +%Y-%m-%d)
   PROMPT="Search X for tweets about: ${ARG}. Date range: ${FROM_DATE} to ${TO_DATE}. Return at least 15-20 candidate tweets — mix of high-engagement posts and smaller accounts that add a distinct angle. For each tweet include: @handle, the full text, date posted, exact engagement counts (likes, retweets, replies — never N/A; if unknown, say 0), and the direct link (https://x.com/handle/status/ID). Return as a numbered list."
   jq -n --arg p "$PROMPT" '{model:"grok-4.7", input:[{role:"user",content:$p}], tools:[{type:"x_search"}]}' > /tmp/xai-ft-keyword.json
   HTTP=$(./secretcurl -s -o /tmp/xai.json -w '%{http_code}' --max-time 150 -X POST "https://api.x.ai/v1/responses" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer {XAI_API_KEY}" \
     -d @/tmp/xai-ft-keyword.json)
   echo "xai http=$HTTP bytes=$(wc -c </tmp/xai.json)"

   On HTTP=200, parse /tmp/xai.json with: jq -r '.output[] | select(.type == "message") | .content[] | select(.type == "output_text") | .text' and mark SOURCE_PATH=api.

   Path B — WebSearch fallback (only if the key is KEY_UNSET, or Path A gave a non-2xx / empty / timeout per the contract): use the built-in WebSearch tool with site:x.com "<query terms>" after:${FROM_DATE}. Note at the top of the log the true reason (http-<code> / timeout / empty, never "unavailable" when the key was set) and "results compiled via WebSearch — quality lower than usual". WebSearch favours high-engagement older tweets — prioritise results dated within the last 48 hours. Mark SOURCE_PATH=websearch.

3. Empty vs. error handling (distinguish):

   • Legitimate empty (0 tweets): log FETCH_TWEETS_EMPTY (source=${SOURCE_PATH}) and stop — no notification.
   • API/cache error (HTTP error, malformed JSON, all paths failed): log FETCH_TWEETS_ERROR (last_path=${SOURCE_PATH}, reason=...) and stop — no notification.

4. Deduplicate each candidate URL against SEEN_TWEETS. If ALL are dupes: log FETCH_TWEETS_NO_NEW: all results already reported and stop — no notification.

5. Curate (the core step): a. Cluster survivors into 2–4 sub-narratives by what they're claiming/discussing (e.g. for a token: "price action", "team announcement", "criticism/FUD", "ecosystem integration"). Name the angle, not the topic. b. Rank within each cluster by signal (not raw engagement): signal = likes + 2×retweets + replies, but demote pure replies, generic shilling, and near-duplicate paraphrases. Drop tweets with <5 total engagement unless they add a unique angle. c. Cap each cluster at 2–3 tweets, total 7–10. Quality over quantity — if only 5 pass, send 5. Don't pad. d. Extract the claim/signal per tweet — what's new or interesting, not a literal paraphrase. Bad: "User says token is going up." Good: "Calls out the team's silence on the postponed unlock — first major holder to do so publicly." e. Compute a one-line signal for the top of the notification — one observation about the shape of the conversation (e.g. "Sentiment split — 4 bullish on the launch, 3 critical of the unlock terms.").

6. Save + update seen-file (see Log). Append each kept tweet URL (one per line) to memory/fetch-tweets-seen.txt (create if missing).

7. Notify via ./notify with the clustered output:

   *Top Tweets — ${ARG} (${today})*
   _${signal_one_liner}_
   
   *${cluster_1_name}*
   1. x.com/handle — [insight summary]
   Likes: X | RTs: Y | Replies: Z
   [View tweet](https://x.com/handle/status/ID)
   
   2. x.com/handle — [insight summary]
   Likes: X | RTs: Y | Replies: Z
   [View tweet](https://x.com/handle/status/ID)
   
   *${cluster_2_name}*
   3. x.com/handle — [insight summary]
   ...

   The signal one-liner is italic (_..._) directly under the title; cluster headers are *bold*.

Status codes: FETCH_TWEETS_OK (notified) | FETCH_TWEETS_EMPTY | FETCH_TWEETS_ERROR | FETCH_TWEETS_NO_NEW.

---

Branch: topic (source:topic)

Gist of the latest X chatter on one or more configurable topics.

Seen set: memory/tweet-roundup-seen.txt + last 3 days of logs.

1. Resolve the topic list (priority order):

   1. ARG set → TOPICS=("$ARG") (single-topic mode).
   2. Else if MEMORY.md has a ## Tweet Roundup Topics section → use its bulleted lines, one query per line.
   3. Else built-in defaults:
      • artificial intelligence OR AI agents OR LLM
      • crypto OR bitcoin OR DeFi
      • technology OR startups OR open source

2. Fetch per topic — track SOURCE ∈ {api, websearch, failed} per topic.

   Path A — direct X.AI curl (primary): for each topic, call Grok's x_search.

   FROM_DATE=$(date -u -d "yesterday" +%Y-%m-%d 2>/dev/null || date -u -v-1d +%Y-%m-%d)
   TO_DATE=$(date -u +%Y-%m-%d)
   PROMPT="Search X for recent tweets about: ${TOPIC}. Date range: ${FROM_DATE} to ${TO_DATE}. Return up to 8 substantive tweets. For each: @handle, full text, date, exact engagement counts (likes, retweets, replies; 0 if unknown), and the direct link https://x.com/handle/status/ID."
   jq -n --arg p "$PROMPT" '{model:"grok-4.7", input:[{role:"user",content:$p}], tools:[{type:"x_search"}]}' > /tmp/xai-ft-topic.json
   ./secretcurl -s -o /tmp/xai-topic-out.json -X POST "https://api.x.ai/v1/responses" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer {XAI_API_KEY}" \
     -d @/tmp/xai-ft-topic.json

   Parse with the standard jq extractor. If it yields text, SOURCE=api. Extract each tweet's @handle, text, engagement counts, and permalink.

   Path B — WebSearch fallback (only if XAI_API_KEY unset, or Path A errors/empty): site:x.com "<topic keywords>" after:<YESTERDAY>. Always include the word "today" and ${today} to force fresh results. Discard any result whose visible date is older than 48h. Collect up to 5 candidates per topic. Mark SOURCE=websearch. If both paths return nothing, mark SOURCE=failed.

3. Score and filter. Require: a known @handle; a https://x.com/<handle>/status/<id> URL (if missing, keep but mark "link unavailable"); posted within 48h; URL not in SEEN_TWEETS. Compute signal_score = likes + 2×retweets + replies (on WebSearch path with no counts, use result rank as a weak proxy). Demote −50%: replies to a parent tweet; near-duplicates of a higher-scoring tweet (>70% text overlap or same linked URL).

4. Curate per topic:

   • 0 survivors → drop the topic. Do NOT pad.
   • 1–3 survivors → list ranked by signal_score, highest first.
   • 4+ survivors → group into 2–3 sub-narratives (shared keywords/entity/claim); label each, surface the top-1 tweet per narrative as exemplar. Write an insight per reported tweet (what it asserts/reveals, not a headline paraphrase). Write a one-line conversation shape per topic ("bullish momentum, dissenters quiet", "split opinion on X's launch", "single story dominating — Y").

5. Notify. If every topic dropped: log TWEET_ROUNDUP_EMPTY and stop — no notify. Otherwise send via ./notify (≤4000 chars):

   *Tweet Roundup — ${today}*
   _Source: api:X websearch:Y failed:Z_
   
   *[Topic 1]* — _conversation shape_
   - x.com/handle — insight (signal: 12.3k) [View](https://x.com/handle/status/ID)
   - x.com/handle — insight (signal: 4.1k) [View](https://x.com/handle/status/ID)
   
   *[Topic 2]* — _conversation shape_
   - x.com/handle — insight (signal: 8k) [View](https://x.com/handle/status/ID)

   Show signal: <score> only when engagement counts were available (api path); omit silently on WebSearch.

6. Persist + log (see Log). Append each reported URL (one per line) to memory/tweet-roundup-seen.txt (create if missing).

Constraints: never notify an empty roundup (silence beats filler); never @handle anyone; never report a URL already in SEEN_TWEETS. Status codes: TWEET_ROUNDUP_OK | TWEET_ROUNDUP_EMPTY.

---

Branch: account (source:account)

Two sub-modes: single handle (decision-ready gist of one account) vs. all tracked accounts (theme-grouped digest of a watchlist). Choose by ARG.

Seen set: last 2 days of logs — extract every https://x.com/ URL under a prior ### fetch-tweets account entry into SEEN_URLS.

account — single handle (ARG is one @handle)

1. Normalize ARG. Strip leading @, https://x.com/, https://twitter.com/, https://nitter.net/, trailing slash / /status/.... Lowercase. Reject if empty, contains whitespace, or >15 chars. On reject → REFRESH_X_NO_VAR: send ./notify "fetch-tweets: REFRESH_X_NO_VAR — set an X handle" and exit 0. Store the cleaned handle as ACCOUNT.

2. Load tweets:

   • Path A — X.AI API (primary): search this account's recent tweets via Grok's x_search.

     PROMPT="Search X for the latest tweets, replies, and quote tweets from @${ACCOUNT} in the last 2 days. Return each with full text, timestamp, type (original|reply|quote), what it replies to/quotes if any, exact engagement counts (likes, retweets, replies; 0 if unknown), and the permalink https://x.com/${ACCOUNT}/status/ID. Skip retweets of others. Return chronological."
     jq -n --arg p "$PROMPT" '{model:"grok-4.7", input:[{role:"user",content:$p}], tools:[{type:"x_search"}]}' > /tmp/xai-ft-account.json
     ./secretcurl -m 30 -s -o /tmp/xai-account-out.json -X POST "https://api.x.ai/v1/responses" \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer {XAI_API_KEY}" \
       -d @/tmp/xai-ft-account.json

     Parse with the standard jq extractor. Record source=api.
   • Path B — WebFetch fallback (only if XAI_API_KEY unset, or Path A errors / parsed text has zero x.com status URLs): WebFetch https://x.com/${ACCOUNT} with prompt: "List every tweet, reply, and quote tweet visible on this profile with its full text, timestamp, engagement counts (likes/retweets/replies) if shown, and the permalink https://x.com/handle/status/ID. Return a chronological list." Record source=webfetch.
   • Path C — degraded: if XAI_API_KEY unset and WebFetch returns nothing → skip to step 8 with status REFRESH_X_NO_API_KEY (key missing) or REFRESH_X_ERROR (key set but both paths failed).

3. Parse into structured tweets: url, text, timestamp, type (original/reply/quote), reply_to, quoted_text, likes, retweets, replies. Drop retweets of others. Missing counts → 0. Compute signal_score = likes + 2*retweets + replies − (3 if type=reply else 0).

4. Dedup and gate: drop any tweet whose url is in SEEN_URLS (deduped_count). If fewer than 3 tweets survive AND no thread is detectable (step 5) → skip to step 8 with REFRESH_X_NO_NEW (everything deduped) or REFRESH_X_EMPTY (account posted nothing).

5. Detect threads: a thread = 2+ tweets by ACCOUNT within 30 minutes where later tweets reply to earlier ones OR share ≥2 meaningful keywords with the opener. Thread tweets are atomic units regardless of individual score. Record {opener_url, tweet_count, combined_signal}.

6. Cluster and extract insights: group survivors (threads = one unit) into 2–4 sub-narratives by topic overlap; if <2 emerge, use one cluster. Per cluster: Title (3–8 words), Top tweet(s) (1–3 excerpts ≤200 chars each, with permalink + engagement), Insight (one sentence — what the cluster reveals about the author's stance/claim/shift; not a paraphrase — if you can't beat paraphrase, drop the cluster). Per thread: a 1–2 sentence landing summary + opener URL.

7. Write the verdict (pick exactly one) + a ≤20-word lede:

   Verdict	When
   ANNOUNCEMENT	launch, hire, policy, or product drop
   ARGUMENT	majority signal from contrarian takes or fights
   BUILDING	ships/code/tech-progress clusters dominate
   SHITPOST	jokes, memes, low-stakes banter dominate
   CONTEXT	mostly reacting to a news cycle, not driving one
   QUIET	<3 originals and no thread

8. Save gist (see Log). On empty/no-new/error/no-var statuses, write only the account header + status footer, skip cluster sections.

9. Update MEMORY.md (conditional): only if a cluster carries an announcement, specific claim, named project, or stance shift — add one bullet under a ## Tracked X Accounts section (create if missing): - @ACCOUNT YYYY-MM-DD: [one-sentence claim] — [permalink]. No paraphrases/memes/generic opinions.

10. Notify via ./notify. On REFRESH_X_OK:

    x refresh — @ACCOUNT ([VERDICT])
    [lede]
    top cluster: [title] — "[≤80 char excerpt]" ([likes]❤)
    [N tweets, T threads, K deduped]

    On REFRESH_X_EMPTY / REFRESH_X_NO_NEW: skip notify (write the log entry only). On REFRESH_X_NO_API_KEY / REFRESH_X_ERROR / REFRESH_X_NO_VAR: notify with the status code + a one-line hint (e.g. "fetch-tweets: REFRESH_X_NO_API_KEY — set XAI_API_KEY in workflow secrets").

Constraints: never fabricate engagement; never include a SEEN_URLS URL; an insight that only paraphrases is not an insight (drop the cluster); MEMORY.md updates are one line each. Status codes: REFRESH_X_OK | REFRESH_X_EMPTY | REFRESH_X_NO_NEW | REFRESH_X_NO_API_KEY | REFRESH_X_ERROR | REFRESH_X_NO_VAR.

account — all tracked accounts (ARG empty)

Use this to answer "what did these specific people post" across a watchlist.

1. Read config memory/topics/tracked-accounts.yml. If missing or accounts: [] → log TWEET_DIGEST_NO_CONFIG and exit (no notification). Schema:

   accounts:
     - handle: vitalikbuterin
       why: ethereum core thinking      # optional — grouping/context label
     - handle: balajis
       why: macro + tech narratives

2. Fetch recent tweets per account. For each handle:

   • Path A — live curl (primary, XAI_API_KEY is injected and set):

     PROMPT="Search X for the latest tweets from:${HANDLE} in the last 3 days. Return the 5 most interesting or substantive tweets. For each: full text, date, direct link (https://x.com/${HANDLE}/status/ID). Skip retweets of others."
     jq -n --arg p "$PROMPT" '{model:"grok-4.7", input:[{role:"user",content:$p}], tools:[{type:"x_search"}]}' > /tmp/xai-ft-acct1.json
     ./secretcurl -m 30 -s -o /tmp/xai-acct1-out.json -X POST "https://api.x.ai/v1/responses" \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer {XAI_API_KEY}" \
       -d @/tmp/xai-ft-acct1.json

     Parse with the standard jq extractor. If XAI_API_KEY is unset, log TWEET_DIGEST_NO_KEY: skill requires XAI_API_KEY and exit (no notification). Dedup: drop any candidate URL already in SEEN_URLS (last 2 days of logs).

3. Group by theme, not by account. Walk the full candidate set; identify 2–4 themes (e.g. "L2 design decisions", "macro / rates", "AI model releases", "regulation"). Each tweet maps to one theme; a why: label can seed theme naming for single-topic feeds.

4. Write a one-sentence take per notable tweet — what the tweet says, not your opinion of it. Voice per the Voice section.

5. Notify via ./notify:

   *Tweet Digest — ${today}*
   
   *Theme: <theme>*
   @handle: <one-sentence summary> — [link](url)
   @handle: <one-sentence summary> — [link](url)
   
   *Theme: <theme>*
   ...

   If no notable tweets across all accounts: log TWEET_DIGEST_OK and end (no notification).

Status codes: TWEET_DIGEST_OK (notified or clean) | TWEET_DIGEST_NO_CONFIG | TWEET_DIGEST_NO_KEY.

---