Build Knowledge Graph

Build Knowledge Graph

Recursively decompose ML algorithms and mathematical concepts into a directed knowledge graph. Each node is a precisely definable concept -- an algorithm, theorem, mathematical object, or operation (e.g. "eigenvalue", "transformer layer", "decision tree", "directed acyclic graph"). Nodes are never fields, disciplines, or vague terms.

The final graph must be a DAG (no directed cycles).

Node format

{
  "id": "a0237bba",
  "label": "singular value decomposition",
  "to": ["9f9d6c30", "d2155e0e"],
  "from": ["bf594ee2", "1c77c3c7"],
  "category": "Linear and multilinear algebra; matrix theory",
  "definition": "A factorization M = U S V^T where U, V are orthogonal and S is diagonal with non-negative entries...",
  "long_description": "The singular value decomposition (SVD) expresses any m × n real (or complex) matrix A as A = UΣV^T ...",
  "_depth": 3,
  "_pagerank": 0.00024,
  "_degree_centrality": 0.00120,
  "_betweenness_centrality": 0.0,
  "_descendant_ratio": 0.99759,
  "_prerequisite_ratio": 0.0,
  "_reachability_ratio": 0.99519
}

Tool

python3 .claude/skills/build-knowledge-graph/scripts/graph.py <subcommand> [args]

Workflow

1. Initialize

GRAPH="knowledge_graph.json"
GR="python3 .claude/skills/build-knowledge-graph/scripts/graph.py"

$GR init "$GRAPH"
$GR add-seeds "$GRAPH" "concept 1" "concept 2" "concept 3"

If a graph file already exists and the user wants to resume/extend, skip init and just run pending to see what remains.

2. Iterate: decompose pending concepts

Repeat until pending prints NO_PENDING:

Step A -- Get the next batch:

$GR pending "$GRAPH" --limit 10

Step B -- For every concept in the batch, produce a decomposition. Analyze them all at once. For each concept determine:

1. Definition: precise mathematical/algorithmic definition (1-3 sentences).
2. Long description: an extended mathematical description (200-300 words) covering the formal definition, key properties and theorems, relationship to other concepts, computational considerations, and intuition. Write at the level of a graduate mathematics or ML textbook. Use LaTeX-style notation for formulas.
3. Prerequisites: specific concepts directly used in the definition. Each must be a single, precisely definable concept. Use canonical lowercase names. Decompose deeply into mathematical foundations -- if a concept relies on "vector", "function", "real numbers", "limit", "supremum", etc., list them.
4. Category: one MSC 2020 top-level name (math) or one CS/ML task discipline (CS).

Format as a JSON array:

[
  {
    "label": "concept name",
    "definition": "...",
    "long_description": "...",
    "prerequisites": ["prereq a", "prereq b"],
    "category": "Category Name"
  }
]

Step C -- Pipe the JSON into ingest:

cat <<'BATCH' | $GR ingest "$GRAPH"
[ ... JSON array ... ]
BATCH

Step D -- Read the ingest output to see how many nodes are pending. Go back to Step A.

3. Finalize

$GR finalize "$GRAPH"
$GR stats "$GRAPH"

Always run finalize before delivering results. finalize enforces that the resulting graph is a DAG (cycles removed + transitive reduction applied). finalize also computes and saves node metrics on the complete DAG (after cycle elimination and before transitive reduction), then writes the reduced DAG. The final saved graph must include _pagerank, _degree_centrality, _betweenness_centrality, _descendant_ratio, _prerequisite_ratio, and _reachability_ratio on every node.

Report the final node count, edge count, top categories, and confirm that node metrics were computed and saved from the complete graph.

Concept decomposition rules

These rules are critical for graph quality:

1. Nodes must be precise concepts, not fields.

   • Yes: "matrix multiplication", "softmax function", "cross-entropy loss", "convolution"
   • No: "linear algebra", "calculus", "deep learning", "statistics"

2. Prerequisites must appear in the definition. Only list concepts that someone must understand to parse the definition. Do not list tangentially related concepts.

3. Use canonical lowercase names: "rectified linear unit", "batch normalization", "bayes theorem", "singular value decomposition".

4. Terminal concepts (do not decompose further -- the bare logical and set-theoretic bedrock): set, element, natural numbers, logical conjunction, logical disjunction, logical negation, logical implication, universal quantifier, existential quantifier, equality.

   Everything above this level must be decomposed. For example:

   • "logarithm" -> "inverse function", "exponential function"
   • "dot product" -> "vector", "multiplication", "summation"
   • "vector" -> "vector space"
   • "vector space" -> "field", "abelian group", "scalar multiplication"
   • "function" -> "relation", "domain", "codomain"
   • "relation" -> "ordered pair", "subset", "cartesian product"
   • "ordered pair" -> "set"
   • "real numbers" -> "complete ordered field", "dedekind cut" or "cauchy sequence"
   • "limit" -> "epsilon-delta definition", "real numbers", "absolute value"
   • "absolute value" -> "real numbers", "function"
   • "addition" -> "binary operation", "natural numbers" (then extends to integers, rationals, reals)
   • "multiplication" -> "binary operation", "natural numbers"
   • "summation" -> "addition", "index set", "sequence"
   • "sequence" -> "function", "natural numbers"

   The goal is a graph that bottoms out at foundational mathematics (set theory, logic, basic algebraic structures) rather than stopping at calculus-level concepts.

5. No self-references: a concept cannot list itself as a prerequisite.

6. Prefer specificity: "convolutional layer" decomposes into "convolution", "activation function", "bias vector" -- not into "neural network" or "deep learning".

7. Result must be acyclic: the final delivered graph must be a DAG. Always run finalize to enforce cycle elimination and transitive reduction.