Subscription Videos Metadata

Subscription Videos Metadata → Second-Brain Ingest

Pulls video metadata from a real YouTube account's subscriptions (via OAuth) using scripts/youtube_subscriptions.py, and turns it into a real knowledge graph in a second-brain vault — not just a searchable list. Replaces manually checking the YouTube subscription feed.

Not hardcoded to any one project or vault. The script asks where to put things on first run (see "First-time setup" below) and remembers — portable to any machine/checkout, any Brain-Matter-style vault. For Jeffrey's own setup specifically, it's currently configured with --raw-dir "Brain Matter/raw/youtube-videos" and --wiki-root "Brain Matter", but nothing in the script assumes that path — treat every path in this document as "wherever it's configured," not a hardcoded constant.

This is the working copy — the one Claude actually loads each session in this project. The canonical, publicly-released source lives at https://github.com/SomewhereSimulated/youtube-subscriptions-ingest (public repo, MIT license, generalized from this project's version 2026-08-11 — first of hopefully many standalone releases from here). That repo's SKILL.md/README.md/GUIDE.md/scripts/youtube_subscriptions.py are the source of truth for the portable version; this copy is specifically wired for Jeffrey's own Brain Matter vault (frontmatter, allowed-tools, and behavior are otherwise identical). When either copy changes in a way that should apply to both — a bug fix, a new capability, a taxonomy improvement — port the change to the other manually and note it in decisions/log.md here. Don't let them silently drift: Jeffrey-specific config/paths stay local-only, but logic/behavior fixes belong in both places.

Three-phase, mirroring Brain Matter's own raw/ → wiki/ convention (Brain Matter/CLAUDE.md — read that file if you haven't; this skill follows its schema, doesn't replace it):

• Fetch — YouTube → <raw_dir>/. Verbatim metadata only, no interpretation. Raw source material is immutable per Brain Matter's rule.
• Transcript (added 2026-08-14) — for each raw video, pulls its caption track via scripts/fetch_transcript.py (youtube_transcript_api, cached/throttled/retried per video — a straight caption scrape, no audio transcription, no fallback if captions are off) and appends it directly onto the raw .md file as a ## Transcript section, right after the description (plus a plain-text .transcript.txt convenience copy — see Archive structure). A video with no captions gets a ## Transcript section too, reading "Unavailable" — recorded as unavailable, not an error.
• Ingest — <raw_dir>/ → <wiki_root>/wiki/sources/, <wiki_root>/wiki/concepts/, <wiki_root>/wiki/entities/, <wiki_root>/index.md, <wiki_root>/log.md. Auto-detects concept/entity matches via keyword taxonomy and writes real [[Page Name]] Obsidian links between a video's source page and the concepts/entities it touches — creating stub pages for new ones, appending to existing ones (hand-written or previously auto-created) without disturbing their prose. Carries the raw file's ## Transcript section (if the transcript phase already ran for that video) onto the source page in the same position — directly after ## Description, before ## Concepts/## Entities.

This is deliberately the lightweight, automated tier of Brain Matter ingest — real links and auto-created stubs, but no hand-written synthesis prose (that's the full manual process: "read it, talk through takeaways, write real analysis," which doesn't scale to hundreds of videos per fetch). Full manual treatment for one specific video that actually matters is still just a normal conversation — ask for it directly, video by video, following Brain Matter/CLAUDE.md's real ingest workflow instead of this one.

Archive structure

Relative layout under wherever configure points — shown here with Jeffrey's actual current configuration (raw_dir = Brain Matter/raw/youtube-videos, wiki_root = Brain Matter):

<raw_dir>/                                            e.g. Brain Matter/raw/youtube-videos/
    <Channel Name>/<published-date>-<video-id>.md               verbatim — written by fetch; transcript phase APPENDS a
                                                                  "## Transcript" section directly onto this file, right
                                                                  after the description — this is the canonical copy
    <Channel Name>/<published-date>-<video-id>.transcript.txt   plain-text convenience copy of the same transcript,
                                                                  written alongside it (may not exist: unavailable or
                                                                  transcript phase hasn't run yet) — not the source of
                                                                  truth, just grep-able without frontmatter parsing
    _fetch-index.tsv                                  internal: fetch dedup (not a real source, don't ingest it)
    _transcript-index.tsv                             internal: transcript dedup — only ok/cached/unavailable are terminal; errors retry next run
    _ingest-index.tsv                                 internal: ingest dedup
<wiki_root>/                                          e.g. Brain Matter/
  wiki/
    sources/youtube-videos/
      <Channel Name>/<Video Title> [<Video ID>].md    one page per video — written by ingest
    concepts/<slug>.md                                auto-created/updated by ingest (or hand-written)
    entities/<slug>.md                                auto-created/updated by ingest (or hand-written)
  index.md                                            concept/entity catalog — updated by ingest
  log.md                                               one entry per ingest RUN — updated by ingest

Gitignored (bulk data, local-only): raw/youtube-videos/, wiki/sources/youtube-videos/. Tracked in git (the actual synthesized knowledge, small and worth versioning): wiki/concepts/, wiki/entities/, index.md, log.md.

The [<Video ID>] filename suffix on source pages is load-bearing, not decorative — titles are not unique (a real collision: two different Greg Isenberg videos, different IDs, identical title, one a rebroadcast — silently overwrote one note with the other before the ID suffix was added). Never drop it for a cleaner-looking filename.

First-time setup

Two separate one-time steps — neither is specific to Jeffrey's machine or this particular project, so a fresh checkout (or a different user entirely) needs both before the skill does anything.

A. Folder configuration (only if not yet configured)

The script has no hardcoded path to any particular vault — check whether scripts/.youtube_subscriptions_config.json exists. If it doesn't, this is a genuine first run: ask via AskUserQuestion before running fetch, refresh, or ingest (all three will error out clearly if this step is skipped, so it's safe to attempt them and let the error prompt this rather than pre-checking every time — but if you already know it's unconfigured, ask up front instead of waiting for the error).

Two things to ask, each needs a real folder path — check for an existing likely default first (e.g. a Brain Matter/ folder already in this project) and offer it as the recommended option, with "Other" (built into AskUserQuestion) as the free-text path for anything else:

• Where should raw video files be stored? → --raw-dir
• Where should the wiki output go? (the root of a second-brain vault — the script creates wiki/sources/youtube-videos/, wiki/concepts/, wiki/entities/, index.md, and log.md under whatever folder is given here, same relative layout Brain Matter itself uses) → --wiki-root

Then run:

python scripts/youtube_subscriptions.py configure --raw-dir "<path>" --wiki-root "<path>"

This is safe to run against an existing vault — it only scaffolds index.md/log.md if they're genuinely missing, never overwrites real content. Confirm the resolved absolute paths back from the JSON it prints.

B. YouTube auth (only if test reports not authenticated)

1. Check auth: python scripts/youtube_subscriptions.py test
2. If it errors with "Not authenticated yet" and GOOGLE_YOUTUBE_CLIENT_ID/GOOGLE_YOUTUBE_CLIENT_SECRET are empty in .env: tell the user they need to create a Google Cloud OAuth Desktop-app client first (console.cloud.google.com — enable "YouTube Data API v3", then Credentials → Create Credentials → OAuth client ID → Desktop app) and paste the client ID/secret into .env. Do not attempt this for them — it requires their Google login.
3. Once .env is filled in: python scripts/youtube_subscriptions.py auth — opens a local browser window to consent. Token caches to scripts/.credentials/youtube_token.json (gitignored) and auto-refreshes after that; this is a one-time step.
4. Re-run test to confirm — it prints the account's channel name and subscription count.

Workflow

1. Ask: Fetch, Transcript, Ingest, or Show Local?

Use AskUserQuestion with four options:

• Fetch Latest — hits the YouTube API, pulls new videos into raw/. Does NOT make anything searchable in the wiki sense by itself.
• Transcript — pulls caption-track transcripts for raw videos that don't have one yet, as sidecar files. Does NOT touch the wiki by itself — run ingest after so the transcript actually lands on the source page.
• Ingest — processes whatever's in raw/ and not yet ingested into the real wiki (wiki/sources/, wiki/concepts/, wiki/entities/). Folds in a transcript sidecar if one already exists for that video.
• Show Local — query what's already in the wiki. No API call.

If Jeffrey just says "pull my videos" without specifying, fetch alone is the literal ask — but mention that transcript + ingest are separate steps needed before the new videos are actually searchable (with transcript) in the wiki, and offer to run them too.

A subtlety worth knowing: ingest dedups by video ID like every other phase — a video already ingested before its transcript was fetched will NOT automatically get the transcript folded in on a later ingest run. There is no ingest --force/backfill path yet (flagged as a known gap, not built — see decisions/log.md, 2026-08-14). If Jeffrey wants the transcript backfilled into an already-ingested video's source page, that's a manual ask for now, not something a normal ingest run will pick up.

2. Fetch Latest

Every time, before running — not just on a first ever run — ask via AskUserQuestion: "How far back should this fetch look?" Options (max 4 + the built-in "Other" free-text slot):

• New only (recommended) — just what's posted since the last fetch. Normal incremental behavior — don't pass --days at all.
• 90 days
• 6 months
• 1 year
• (Other, free text — covers "10 days", "30 days", or anything custom)

Convert a chosen/typed window to a day count and pass --days N explicitly (90 → 90, 6 months → 182, 1 year → 365; parse free text by judgment — a bare number is days, "N weeks"/"N months"/"N years" scale accordingly). Any explicit --days value overrides the normal incremental cutoff, even on an already-established archive — that's what makes "go back further than normal" a real, repeatable option (e.g. to backfill a channel that was missed before), not just a first-run-only setting. Dedup by Video ID makes this safe to run repeatedly.

Run:

python scripts/youtube_subscriptions.py fetch [--days N]

The script handles everything: paginating subscriptions, resolving each channel's uploads playlist, walking it newest-first with an early stop at the cutoff date, pulling full metadata in batched calls, deduping against raw/'s fetch index by Video ID, and writing one verbatim file per new video into raw/youtube-videos/<Channel>/. No per-channel video cap by default — the archive's value is being a comprehensive local search corpus, not a small curated feed, so a prolific channel gets its full history within the day window. --max-per-channel N exists only as an explicit override for deliberately bounding one run — never assume it.

refresh (no args) re-fetches metadata for every video already in raw/ and overwrites those files in place (e.g. to pick up updated view counts). Touches raw/ only — does not re-run ingest.

Report back from the JSON it prints: videos added, channels touched, total channels checked. If added: 0, say so plainly — could mean the archive's already current, or that there are no subscriptions. Mention that transcript (optional) and ingest are the next steps if anything was added.

3. Transcript

Run:

python scripts/youtube_subscriptions.py transcript [--limit N]

For each raw video without a terminal transcript result yet, calls scripts/fetch_transcript.py (importable — same folder), which pulls the caption track via youtube_transcript_api, caches it (yt_<id>.txt in the configured cache dir, C:\tmp by default on Windows), throttles between requests, and retries with backoff on IP/rate blocks.

On success, the transcript text is written to two places: appended onto the raw .md file itself as a ## Transcript section directly after the description (the canonical copy — write_transcript_into_raw_file, idempotent, replaces rather than duplicates on a re-run), and a plain-text <date>-<id>.transcript.txt sidecar next to it (convenience copy only). refresh re-writing a raw file's metadata preserves an existing Transcript section rather than wiping it.

Three outcomes per video, tracked in _transcript-index.tsv:

• ok / cached — written to the raw file and sidecar, counted in the report.
• unavailable — captions genuinely off or region-blocked. Still gets a ## Transcript section on the raw file (reading "Unavailable"), so this is distinguishable at a glance from "not attempted yet". Terminal (indexed, not retried) — there's no audio-transcription fallback.
• error (e.g. a transient IP block that outlasted the retry budget) — not written to the index or the raw file, so it's retried automatically on the next transcript run rather than silently given up on.

Use --limit N before a big run, same as ingest. This can be slow at archive scale (thousands of videos, throttled ~1.5s apart plus request time) — don't run it unprompted against the full untried backlog; confirm the scope with Jeffrey first if it's more than a handful.

Report back from the JSON: fetched, already_cached, unavailable, errors_this_run, remaining_untried.

Not yet built (Phase 2, only after Phase 1 is proven — see decisions/log.md, 2026-08-14): chapter-based slicing for 3+ hour videos via yt-dlp chapter metadata, and bulk concurrent fetch via ThreadPoolExecutor for faster large runs. fetch_transcript.py also supports an optional Webshare rotating proxy (YT_WEBSHARE_USER/YT_WEBSHARE_PASS in .env) if IP blocks turn out to be a real problem in practice — off by default, not yet needed.