Research

Research

The user asks a question. This finds the sources, reads them, and writes down what they say with a link to each. You do not need a URL to start — that was the whole problem with the version this replaces, and it is the thing a person asking for research is least able to supply.

cd <the skill directory>/scripts

# The usual case: one command, question in, dossier out.
python3 research.py "how close is the Kessler syndrome" --out notes.md
cat notes.md

Then answer from notes.md, citing the URLs in it.

The four scripts

research.py — question in, dossier out

python3 research.py "how does a tokamak confine plasma" --out notes.md
python3 research.py "ggml quantisation formats" --sources papers,web --pages 6 --out notes.md
python3 research.py "battery degradation" --site nature.com --out notes.md
python3 research.py --out notes.md https://a.example/docs https://b.example/spec
python3 research.py "rust async" --follow 1 --same-site --out notes.md

Discovers sources, reads the best few, and writes notes.md plus notes.json (the same stem) for analyze.py. --pages is how many it reads (5 by default), --per-host stops one site supplying all of them, and --delay is the pause between fetches — leave it at 1 unless the run is short, because a burst from one address is what turns a working index into a challenge page for the next question.

--budget (90 seconds by default) is the wall-clock the whole run may spend. It exists because a tool call here has a timeout, and a run killed at that timeout writes nothing at all — no dossier, no sources, nothing to tell the user. When the budget runs out it stops reading and writes down what it has; the pages it did not reach are listed by name under "Could not be read". Raise it with --budget 0 only if you know the call has room.

The dossier holds, per source: the title, the URL, the publication date if the page states one, the sentences that mention what was asked about, and an excerpt. Read those passages first. Reading five whole pages into your context spends most of it on navigation menus.

discover.py — where would the answers come from

python3 discover.py "why is the sky blue" --count 8
python3 discover.py "quantised attention" --sources papers --count 6
python3 discover.py "http caching" --site developer.mozilla.org
python3 discover.py "rust web frameworks" --sources all --json hits.json

Nine services, all keyless, asked at once and merged:

Name a group, several groups, or individual providers: --sources papers,github. --sources all asks nine services in turn, which is slow: use it when auto came back with nothing, and lower --timeout if the call is at risk of being cut off.

Results are ranked by how much of the question the title and snippet actually cover, then by how many independent indexes named the same page. Agreement between two indexes that share no crawler is the only quality signal available here, and relevance is what stops one index's mistake being promoted by its own confidence.

fetch_page.py — one page, as text

python3 fetch_page.py https://example.com
python3 fetch_page.py https://example.com --links --out page.txt
python3 fetch_page.py https://example.com/stats --tables numbers.csv

Markup, scripts and styles removed; the page's own publication date printed when it states one. --tables writes the page's tables out as CSV (biggest first) — a research answer is very often a number in a table, and a table read as prose is a row of words with the columns gone. Hand that CSV to the documents skill's analyze_table.py to compute over it.

analyze.py — what the sources say together

python3 analyze.py notes.json
python3 analyze.py notes.json --claim "the syndrome has already begun" --quotes 2
python3 analyze.py notes.json --terms --top 25
python3 analyze.py notes.json --numbers
python3 analyze.py notes.json --timeline

--claim is the one to reach for before writing an answer. It sorts the sources into those that state the claim plainly, those that state it with a hedge or a denial nearby (marked ⚠), and those that never mention it — and it prints the sentence and the source for every one, so the judgement stays with you rather than with a count. --numbers groups every figure by the figure, which is how you notice that two sources say 27,000 and one says 2,700.

Doing this well

1. Search before you ask for a URL. research.py "the question" is the first move. Ask the user for a link only when a run comes back with nothing.
2. Answer from the dossier, and cite. Every claim you repeat came from a source in the list. Say which. Quote the passage where it matters.
3. Check agreement before asserting. analyze.py --claim exists for the moment before you write "X is true". One source is not corroboration, and a ⚠ line is worth more than three plain ones.
4. Say what you could not read. The dossier lists every page that failed and every index that did not answer. A summary that silently omits three unreadable sources misrepresents its own coverage.
5. Say how old it is. The dossier carries each page's stated date, and analyze.py prints the range. If a source states no date, say so rather than implying it is current.
6. Narrow with --site rather than with more words. For "what does the MDN say about CORS", --site developer.mozilla.org beats any phrasing.