Tokf Filter

tokf Filter Authoring

You are an expert at writing tokf filter files. tokf is a config-driven CLI that compresses command output before it reaches an LLM context. Filters are TOML files that define how to process a command's output.

When the user asks you to create or modify a filter, follow this guide exactly. Produce valid, idiomatic TOML that matches the schema described below.

Section 1 — What a Filter File Is

A filter file is a TOML file that describes:

• Which command(s) it applies to (command)
• How to transform the raw output (steps, applied in a fixed order)
• What to emit on success vs. failure

Filters live in three places, searched in priority order:

1. .tokf/filters/ — project-local (repo-level overrides)
2. ~/.config/tokf/filters/ — user-level overrides
3. Built-in library (embedded in the tokf binary)

First match wins. Use tokf which "cargo test" to see which filter would activate for a given command.

Section 2 — Processing Order

Steps execute in this fixed order — do not rearrange them:

1. match_output — whole-output substring checks; if matched, short-circuits the entire pipeline and emits immediately
2. [[replace]] — per-line regex transforms applied to every line, in array order
3. strip_ansi / trim_lines — per-line cleanup (ANSI stripping, whitespace trimming)
4. skip / keep — line-level filtering (drop or retain lines by regex)
5. dedup / dedup_window — collapse duplicate consecutive lines
6. lua_script — Luau escape hatch; runs after dedup, before JSON/section/parse
7. [json] — JSON extraction via JSONPath; when configured, replaces section/parse/chunk
8. [[section]] OR [parse] — structured extraction (these are mutually exclusive; section is a state machine, parse is a declarative grouper). Skipped when [json] is configured.
9. [[chunk]] — block-based structured extraction with per-block aggregation, grouping, and tree output (runs on raw output, alongside sections). Skipped when [json] is configured.
10. Exit-code branch — [on_success] or [on_failure] depending on exit code
11. [fallback] — if neither on_success nor on_failure produced output
12. strip_empty_lines / collapse_empty_lines — post-processing cleanup on the final output

Within [on_success] and [on_failure], fields are processed as:

• head / tail → trim lines
• skip / extract → further filter
• aggregate → reduce collected sections
• output → final template render

Section 3 — Top-Level Fields Reference

Section 4 — Step Types

4.1 match_output — Whole-Output Short-Circuit

Check the entire raw output for a substring. If matched, emit a fixed string and stop — no further processing.

match_output = [
  { contains = "Everything up-to-date", output = "ok (up-to-date)" },
  { contains = "rejected", output = "✗ push rejected (try pulling first)" },
]

• contains: literal substring to search for (case-sensitive)
• output: string to emit if matched
• {line_containing} template variable: the first line that contains the substring

match_output = [
  { contains = "error", output = "Error on: {line_containing}" },
]

When to use: for well-known one-liner outcomes that make the rest of filtering irrelevant (e.g., "already up to date", "nothing to push", "authentication failed").

4.2 [[replace]] — Per-Line Regex Transforms

Applied to every line, in array order, before skip/keep. Use to reformat noisy lines.

[[replace]]
pattern = '^(\S+)\s+\S+\s+(\S+)\s+(\S+)'
output = "{1}: {2} → {3}"

[[replace]]
pattern = '^\s+Compiling (\S+) v(\S+)'
output = "compiling {1}@{2}"

• pattern: Rust regex (RE2 syntax, no lookaheads)
• output: template with {1}, {2}, … for capture groups; {0} is the full match
• If the pattern doesn't match a line, that line passes through unchanged
• Invalid patterns are silently skipped at runtime

When to use: when a line contains useful information but in a verbose format — reformat it rather than dropping it.

4.3 skip / keep — Line Filtering

skip drops lines matching any regex. keep retains only lines matching any regex. They compose:

skip = [
  "^\\s*Compiling ",
  "^\\s*Downloading ",
  "^\\s*$",
]

keep = ["^error", "^warning"]

• Both are arrays of regex strings
• Applied after [[replace]]
• skip is checked first, then keep
• A line must pass both: not skipped, and (if keep is non-empty) matching keep

When to use: skip for removing known noise patterns; keep for allow-listing (e.g., keep only lines that start with error or warning).

Also available inside [on_success] and [on_failure] for branch-level filtering.

4.4 dedup / dedup_window — Deduplication

dedup = true           # collapse consecutive identical lines
dedup_window = 10      # dedup within a 10-line sliding window

• dedup = true: removes consecutive duplicate lines (like uniq)
• dedup_window = N: deduplicates within a sliding window of N lines (catches near-consecutive repeats)
• They are independent; you can use both

When to use: for commands that emit repetitive progress lines (e.g., npm install printing the same package multiple times, spinner frames, repeated warnings).

4.5 lua_script — Luau Escape Hatch

For logic that pure TOML cannot express: numeric math, multi-line lookahead, conditional branching.

[lua_script]
lang = "luau"
source = '''
if exit_code == 0 then
    return "passed"
else
    local msg = output:match("Error: (.+)") or "unknown error"
    return "FAILED: " .. msg
end
'''

Or load the script from an external file:

[lua_script]
lang = "luau"
file = "scripts/my-filter.luau"

The file path resolves relative to the current working directory. Exactly one of source or file must be set.

Globals available:

• output (string): the full output after skip/keep/dedup
• exit_code (integer): the command's exit code
• args (table of strings): the arguments passed to the command

Return semantics:

• Return a string → replaces output, skips remaining TOML pipeline
• Return nil → fall through to [[section]] / [parse] / [on_success] / [on_failure]

Sandbox: io, os, and package are blocked. No filesystem or network access. Standard math/string/table libraries are available.

When to use: only when no TOML step can express the logic. Most filters do not need this. Consider it after exhausting match_output, skip/keep, [[replace]], [[section]], and [parse].

4.6 [json] — JSON Extraction via JSONPath

For commands that produce JSON output (e.g. kubectl get pods -o json, gh api, docker inspect). Extracts values using JSONPath (RFC 9535) queries and produces template variables and structured collections.

[json]

[[json.extract]]
path = "$.items[*]"
as = "pods"

[[json.extract.fields]]
field = "metadata.name"
as = "name"

[[json.extract.fields]]
field = "status.phase"
as = "phase"

[on_success]
output = "Pods ({pods_count}):\n{pods | each: \"  {name}: {phase}\" | join: \"\\n\"}"

[[json.extract]] fields:

[[json.extract.fields]] fields:

Result mapping:

• Single scalar → vars["as_name"] = string_value (no count, no chunk)
• Array → ChunkData::Flat collection + {as_name_count} variable
• Objects without fields → top-level scalars auto-flattened into chunk items
• Objects with fields → named fields extracted per item

Pipeline behavior: when [json] is configured, [[section]], [parse], and [[chunk]] are skipped. JSON replaces line-based structural processing. Extracted vars and chunks flow into [on_success]/[on_failure] template rendering.

Error handling: invalid JSON input → extraction skipped, pipeline falls back to raw output (templates are not rendered). Invalid JSONPath → rule silently skipped, other rules still run. Empty array with fields → emits {as_name_count} = "0".

When to use: when the command produces structured JSON output and you need to extract specific fields. Prefer this over [parse] + skip/keep for JSON-native commands.

4.7 [[section]] — State-Machine Section Collector

The most powerful step. Defines a state machine that collects lines into named variables as it scans top-to-bottom.

[[section]]
name = "failures"
enter = "^failures:$"      # regex: start collecting when this matches
exit = "^failures:$"       # regex: stop collecting when this matches (after start)
split_on = "^\\s*$"        # regex: split collected lines into blocks on blank lines
collect_as = "failure_blocks"

[[section]]
name = "summary"
match = "^test result:"    # regex: collect only lines matching this (no enter/exit)
collect_as = "summary_lines"

Fields:

Accessing collected variables in templates:

When to use: when the output has distinct sections with clear start/end markers — test failure blocks, error sections, file change groups.

4.8 [[chunk]] — Block-Based Structured Extraction

Chunks split raw output into repeating structural blocks (e.g., per-crate test suites in a Cargo workspace), extract structured data per-block, and produce named collections for template rendering. Like sections, chunks operate on the raw (unfiltered) command output — skip/keep patterns do not affect chunk processing.

[[chunk]]
split_on = "^\\s*Running "       # regex marking the start of each chunk
include_split_line = true         # include the splitting line in the chunk (default: true)
collect_as = "suites_detail"      # name for the structured collection
group_by = "crate_name"           # merge chunks sharing this field value
children_as = "children"          # preserve original items as nested collection

[chunk.extract]
pattern = 'unittests.+deps/([\w_-]+)-'  # extract a field from the header line
as = "crate_name"
carry_forward = true              # inherit value from previous chunk when pattern doesn't match

[[chunk.body_extract]]
pattern = 'Running\s+(.+?)\s+\('
as = "suite_name"

[[chunk.aggregate]]
pattern = '(\d+) passed'          # aggregates run within each chunk's lines
sum = "passed"

[[chunk.aggregate]]
pattern = '^test result:'
count_as = "suite_count"

Fields:

carry_forward (on extract or body_extract): when a chunk's pattern doesn't match, inherit the value from the most recent chunk that did. Useful when boundary markers (like Running unittests) identify a group, and subsequent chunks should inherit that identity.

Structured collections in templates: each item has named fields accessible in each pipes:

[on_success]
output = """\
{suites_detail | each: "  {crate_name}: {passed} passed ({suite_count} suites)" | join: "\\n"}"""

Tree output with children_as: groups preserve their child items for nested template rendering:

[on_success]
output = """\
{suites_detail | each: "  {crate_name}: {passed} passed\\n{children | each: \"    {suite_name}: {passed} passed\" | join: \"\\n\"}" | join: "\\n"}"""

When to use: when output contains repeating structural blocks with per-block data you want to aggregate and display. Common for workspace build tools (Cargo, Gradle, Nx) where output is organized by sub-project.

4.9 [parse] — Declarative Structured Parser

Alternative to [[section]] for commands with table-like output. Declaratively extracts a header field and groups remaining lines.

[parse]
branch = { line = 1, pattern = '## (\S+?)(?:\.\.\.(\S+))?(?:\s+\[(.+)\])?$', output = "{1}" }

[parse.group]
key = { pattern = '^(.{2}) ', output = "{1}" }
labels = { "M " = "modified", "??" = "untracked", "D " = "deleted" }

[output]
format = """
{branch}{tracking_info}
{group_counts}"""
group_counts_format = "  {label}: {count}"
empty = "clean — nothing to commit"

[parse] fields:

[parse.group] fields:

[output] fields (used with [parse]):

When to use: for commands like git status, docker ps, kubectl get — table-formatted output where you want to extract a header and count/group rows.

4.10 [on_success] / [on_failure] — Exit Code Branches

These branches run after all top-level steps. They have their own sub-fields:

[on_success]
output = "ok ✓ {2}"          # template; collected variables are available
head = 20                     # keep first N lines
tail = 10                     # keep last N lines
skip = ["^\\s*$"]            # additional line filtering
extract = { pattern = '(\S+)\s*->\s*(\S+)', output = "ok ✓ {2}" }

# Singular form (one rule):
aggregate = { from = "summary_lines", pattern = 'ok\. (\d+) passed', sum = "passed", count_as = "suites" }

# Plural form (multiple rules):
# [[on_success.aggregates]]
# from = "summary_lines"
# pattern = 'ok\. (\d+) passed'
# sum = "passed"
# count_as = "suites"
#
# [[on_success.aggregates]]
# from = "summary_lines"
# pattern = '(\d+) failed'
# sum = "failed"

[on_failure]
tail = 10
output = "FAILED: {summary_lines | join: \"\\n\"}"

Branch sub-fields:

aggregate / aggregates fields:

Both singular aggregate and plural aggregates can be used together — they are merged at runtime.

When to use: Always. Every filter should have at least one of [on_success] or [on_failure]. Use [on_success] to produce a clean summary. Use [on_failure] to show enough context to diagnose the issue.

4.11 [fallback] — Last Resort

Emits output when neither [on_success] nor [on_failure] produced anything.

[fallback]
tail = 5

When to use: as a safety net when you have complex branching logic. Ensures tokf never silently swallows output.

4.12 [[variant]] — Context-Aware Filter Delegation

Some commands are wrappers around different underlying tools (e.g. npm test may run Jest, Vitest, or Mocha). A parent filter can declare [[variant]] entries that delegate to specialized child filters based on project context.

command = ["npm test", "pnpm test", "yarn test"]

strip_ansi = true
skip = ["^> ", "^\\s*npm (warn|notice|WARN|verbose|info|timing|error|ERR)"]

[on_success]
output = "{output}"

[on_failure]
tail = 20

[[variant]]
name = "vitest"
detect.files = ["vitest.config.ts", "vitest.config.js", "vitest.config.mts"]
filter = "npm/test-vitest"

[[variant]]
name = "jest"
detect.files = ["jest.config.js", "jest.config.ts", "jest.config.json"]
filter = "npm/test-jest"

Fields:

Two-phase detection:

1. File detection (before execution) — checks if any listed config files exist in the current directory. First match wins.
2. Output pattern (after execution) — regex-matches the command output. Used as a fallback when no file was detected.

At least one of detect.files or detect.output_pattern must be set.

Behavior:

• When a variant matches, the child filter replaces the parent entirely — no field inheritance or merging
• When no variant matches, the parent filter's own fields (skip, on_success, etc.) apply as the fallback
• The filter field references another filter by its discovery name (e.g. "npm/test-vitest" maps to filters/npm/test-vitest.toml)

TOML ordering: [[variant]] entries must appear after all top-level fields (skip, [on_success], etc.) because TOML array-of-tables sections capture subsequent keys.

When to use: when a single command pattern maps to different underlying tools that produce fundamentally different output formats. Create a parent filter with a generic fallback, then create specialized child filters for each tool.