Forecasting Expert Knowledge

What it does

When your agent uses it

• Estimating the probability that an event will happen by a given date
• Weighing conflicting evidence before committing to a forecast
• Reviewing why past predictions were wrong

Example prompts

• “Estimate the chance that our competitor ships a rival product this year, and explain the reasoning.”
• “Weigh these three signals and give me a calibrated probability.”
• “Do a post-mortem on last quarter's forecasts and tell me where we were overconfident.”

Workflow steps

10 steps, taken from the first numbered list in SKILL.md.

1. 1Triage: Focus on questions that are hard enough to be interesting but not so hard they're unknowable
2. 2Break problems apart: Decompose big questions into smaller, researchable sub-questions (Fermi estimation)
3. 3Balance inside and outside views: Use both specific evidence AND base rates from reference classes
4. 4Update incrementally: Adjust predictions in small steps as new evidence arrives (Bayesian updating)
5. 5Look for clashing forces: Identify factors pulling in opposite directions
6. 6Distinguish signal from noise: Weight signals by their reliability and relevance
7. 7Calibrate: Your 70% predictions should come true ~70% of the time
8. 8Post-mortem: Analyze why predictions went wrong, not just celebrate the right ones
9. 9Avoid the narrative trap: A compelling story is not the same as a likely outcome
10. 10Collaborate: Aggregate views from diverse perspectives

What it can do on your machine

Read from SKILL.md and the folder at commit acf2587. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are json).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Forecasting Expert Knowledge loads about 2.5k tokens when it runs. Until then it costs about 42 tokens; SKILL.md has 790 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~42

When it runs · the whole SKILL.md, loaded when a task matches

~2.5k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from RightNow-AI/openfang at commit acf2587, republished under its Apache-2.0 licence (© RightNow-AI). 790 words, ~2,502 tokens.

name

predictor-hand-skill

description

Expert knowledge for AI forecasting — superforecasting principles, signal taxonomy, confidence calibration, reasoning chains, and accuracy tracking

version

1.0.0

runtime

prompt_only

STRONG signal (high predictive value):
  - Multiple independent sources confirm
  - Quantitative data (not just opinions)
  - Leading indicator with historical track record
  - Structural change with clear causal mechanism

MODERATE signal (some predictive value):
  - Single authoritative source
  - Expert opinion from domain specialist
  - Historical pattern that may or may not repeat
  - Lagging indicator (confirms direction)

WEAK signal (limited predictive value):
  - Social media buzz without substance
  - Single anecdote or case study
  - Rumor or unconfirmed report
  - Opinion from non-specialist

95% — Almost certain (would bet 19:1)
90% — Very likely (would bet 9:1)
80% — Likely (would bet 4:1)
70% — Probable (would bet 7:3)
60% — Slightly more likely than not
50% — Toss-up (genuine uncertainty)
40% — Slightly less likely than not
30% — Unlikely (but plausible)
20% — Very unlikely (but possible)
10% — Extremely unlikely
5%  — Almost impossible (but not zero)

Brier Score = (predicted_probability - actual_outcome)^2

actual_outcome = 1 if prediction came true, 0 if not

Perfect score: 0.0 (you're always right with perfect confidence)
Coin flip: 0.25 (saying 50% on everything)
Terrible: 1.0 (100% confident, always wrong)

Good forecaster: < 0.15
Average forecaster: 0.20-0.30
Bad forecaster: > 0.35

PREDICTION: [Specific, falsifiable claim]

1. REFERENCE CLASS (Outside View)
   Base rate: [What % of similar events occur?]
   Reference examples: [3-5 historical analogues]

2. SPECIFIC EVIDENCE (Inside View)
   Signals FOR (+):
   a. [Signal] — strength: [strong/moderate/weak] — adjustment: +X%
   b. [Signal] — strength: [strong/moderate/weak] — adjustment: +X%

   Signals AGAINST (-):
   a. [Signal] — strength: [strong/moderate/weak] — adjustment: -X%
   b. [Signal] — strength: [strong/moderate/weak] — adjustment: -X%

3. SYNTHESIS
   Starting probability (base rate): X%
   Net adjustment: +/-Y%
   Final probability: Z%

4. KEY ASSUMPTIONS
   - [Assumption 1]: If wrong, probability shifts to [W%]
   - [Assumption 2]: If wrong, probability shifts to [V%]

5. RESOLUTION
   Date: [When can this be resolved?]
   Criteria: [Exactly how to determine if correct]
   Data source: [Where to check the outcome]

{
  "id": "pred_001",
  "created": "2025-01-15",
  "prediction": "OpenAI will release GPT-5 before July 2025",
  "confidence": 0.65,
  "domain": "tech",
  "time_horizon": "2025-07-01",
  "reasoning_chain": "...",
  "key_signals": ["leaked roadmap", "compute scaling", "hiring patterns"],
  "status": "active|resolved|expired",
  "resolution": {
    "date": "2025-06-30",
    "outcome": true,
    "evidence": "Released June 15, 2025",
    "brier_score": 0.1225
  },
  "updates": [
    {"date": "2025-03-01", "new_confidence": 0.75, "reason": "New evidence: leaked demo"}
  ]
}

ACCURACY DASHBOARD
==================
Total predictions:     N
Resolved predictions:  N (N correct, N incorrect, N partial)
Active predictions:    N
Expired (unresolvable):N

Overall accuracy:      X%
Brier score:           0.XX

Calibration:
  Predicted 90%+ → Actual: X% (N predictions)
  Predicted 70-89% → Actual: X% (N predictions)
  Predicted 50-69% → Actual: X% (N predictions)
  Predicted 30-49% → Actual: X% (N predictions)
  Predicted <30% → Actual: X% (N predictions)

Strengths: [domains/types where you perform well]
Weaknesses: [domains/types where you perform poorly]

© RightNow-AI, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file in crates/openfang-hands/bundled/predictor of RightNow-AI/openfang.

• SKILL.md
• HAND.toml

Open the folder on GitHubat commit acf2587