Knowledge Base Search

Source: https://github.com/aipoch/medical-research-skills

Knowledge Base Search

When to Use

• You need to find specific facts, definitions, or procedures from a local knowledge base and return the exact source location.
• You must provide traceable citations (file path + page/paragraph/section) for audit, compliance, or review.
• You need to verify the original wording of a claim in the source document (quote-level validation).
• You want to compare how multiple local documents discuss the same topic and identify differences.
• You need to assemble supporting snippets for a report, FAQ, or internal knowledge response using only local materials.

Key Features

• Supports multiple retrieval approaches: direct file search, index-based search, and PageIndex-style location mapping.
• Query strategy guidance: keyword splitting, synonym expansion, and optional filters (time range, file type, tags).
• Relevance-oriented result ranking and filtering to keep the most supportive evidence first.
• Outputs verifiable hit snippets with precise citation locations (file + page/paragraph/section when available).
• Enforces local-only boundaries: searches only within authorized directories and does not modify source content.

Dependencies

• glob (>= 10.0.0): file path pattern matching
• grep (>= 3.11): in-file text searching
• Local knowledge base index files (one or more of: filename index, content index, vector index, PageIndex mapping)
• assets/hit_list_template.csv: standardized hit list output template
• Optional reference: references/guide.md (output formats, checklists, inspection points)

Example Usage

The following example demonstrates an end-to-end local search workflow and produces a CSV hit list compatible with assets/hit_list_template.csv.

Inputs

• Knowledge base root: ./kb/
• Query: How do we rotate API keys?
• Filters: file types md,pdf, time range 2024-01-01..2026-12-31

Steps

1. Confirm index and scope

   • Ensure the search scope is limited to authorized paths (e.g., ./kb/).
   • Identify available indices:
     • filename/content index (fast keyword search)
     • vector index (semantic retrieval)
     • PageIndex mapping (page/paragraph location resolution)

2. Build the query

   • Keywords: rotate, API key, key rotation
   • Synonyms/variants: credential rotation, token rotation, regenerate key
   • Filters:
     • file type: *.md, *.pdf
     • time range: 2024-01-01..2026-12-31 (if metadata exists)

3. Execute search (local-only)

   • Path discovery (example):
     • glob("./kb/**/*.md")
     • glob("./kb/**/*.pdf")
   • Content search (example):
     • grep -RIn "API key\|key rotation\|rotate" ./kb/

4. Filter and rank results

   • Keep hits that directly answer the question (procedure, policy, steps, constraints).
   • Rank by:
     • term proximity (e.g., “rotate” near “API key”)
     • section relevance (e.g., “Security”, “Credentials”, “Operations”)
     • coverage (hits that include prerequisites + steps + verification)

5. Output citations and hit list

   • For each hit, output:
     • file_path
     • location (page number for PDFs; heading/paragraph index for Markdown; PageIndex if available)
     • snippet (verbatim excerpt supporting the conclusion)
     • notes (why it is relevant; any assumptions)
   • Save as hit_list.csv using assets/hit_list_template.csv columns.