Kimojio JSON Decoder

kimojio-json decoders

kimojio-json tokenizes JSON. It reports borrowed input to a visitor. A decoder maps those events to a typed Rust value.

This skill is for direct, allocation-free decoders. It gives the agent the constraints and the reasons for them. It does not prescribe one state layout for every schema. A previous implementation is evidence, not a specification. If a task gives a type or an interface, treat that interface as a contract unless the task also asks to change it.

Start with the problem

Before writing code, identify:

• the input source and schema;
• the required public interface and input lifetime;
• whether the result is one value, a sequence, or a callback stream;
• the policy for unknown members and optional members;
• maximum sizes and nesting depth;
• error kinds, byte offsets, and completion behavior;
• the expected performance and trust boundary.

Then compare possible designs. For each design, ask:

1. What work or failure mode does it avoid?
2. Does that cost apply to this schema and interface?
3. What is the smallest representation that satisfies the contract?
4. What measurement or test would distinguish the choices?

State the reason for a choice in the module documentation or change summary. Do not copy a detailed technique from an earlier decoder without checking why it was used. A different schema may make another technique better.

Read the tokenizer

Read these files from the repository root before relying on an API detail:

• kimojio-json/src/lib.rs
• kimojio-json/src/tokenizer.rs
• kimojio-json/src/escape_string.rs
• kimojio-json/src/error.rs

Confirm names, lifetimes, return values, and error behavior in the source. Do not invent a pull-token API or assume that a callback has access to the tokenizer.

Prefer the tokenizer's visitor protocol when it fits

The tokenizer already owns the scan loop and calls a Visitor. Prefer this protocol when the decoder can consume events as they arrive. This avoids an extra token iterator and an extra dispatch layer.

This recommendation has a measured reason. In this repository, iterator overhead has dominated tokenization for some workloads. A visitor can provide the same streaming behavior while paying the suspension and output cost only for events that complete an item: return no output for other events and return an output at an item boundary. Use an iterator adapter only when the caller's interface needs one, and compare its cost with the direct visitor design.

The visitor methods return Option<Self::Output>. In the normal case, None means that scanning continues. Some returns control to drive; a later call resumes the same decoder state. This is the mechanism for streaming results. If a decoder never suspends, an uninhabited output such as core::convert::Infallible can make the impossible output path removable. Confirm the exact drive contract in the tokenizer before depending on this.

Other tokenizer facts affect the design:

• A visitor method cannot read the tokenizer offset. Return an error kind from the visitor and let the driver attach the offset.
• Escaped and unescaped strings use different callbacks. Keep escaped text borrowed. If a known member name must be compared, use the tokenizer's allocation-free unescaped comparison operation when available.
• Numbers arrive as validated JSON number text. Convert them only when the schema requires a value.
• The member separator is consumed by the tokenizer. A key callback is followed by the callback for its value.
• Confirm how complete input, trailing input, and sticky tokenizer errors are reported. Do not add a second completion model without a reason.

Choose the output shape from the interface

Streaming sequence. Let the consumer handle an item when the item is complete. This keeps peak memory near one item and avoids an intermediate document. A visitor or callback is often the best fit. An iterator adapter is useful when the caller requires the standard Iterator interface, but it should not add per-token work without a benefit.

Single struct. Keep only the fields needed to construct the value. Check required fields at the object boundary or another unambiguous completion point. Borrow fields from the input when their lifetime permits.

Enum or several document shapes. Use the earliest safe discriminator. If member order is not fixed, retain only the data needed to decide the variant. Do not build a full intermediate tree to postpone a decision.

Callback bridge. A generic bridge can borrow the caller's visitor while the decoder remains a concrete type. This can keep the public API simple and avoid storing a type parameter in decoder state. It is a useful option, not a rule. If profiling shows that a borrowed decoder adds repeated state indirection, test moving small plain decoder state by value across one drive call. One measurement in this repository found about a 7 percent difference on a small document. Treat that result as a clue, not a universal threshold. Keep the consumer borrow if it is touched less often than the decoder state.

Build the smallest safe machine

Use explicit states and a loop when the input can control nesting. Recursive descent turns input depth into call-stack depth and can exhaust the stack. Track only the context needed by the schema:

• a depth counter when only depth matters;
• compact fixed context when the container kind matters;
• a fixed capacity derived from a schema limit when a bound exists.

Unknown values may be skipped by counting nested containers in the decoder. Apply depth and size limits while skipping too. An unknown member must not provide an unbounded path around an input limit.

Emit a value as soon as it is complete. Keep a record only until its fields can be validated. Do not parse, unescape, copy, or store data that the result does not use. Use fixed-size inline storage for data with a real maximum. Reject values over that maximum; do not truncate or grow storage.

Name states after input positions, such as Parameters, AddressOctets, or Skip. A reviewer must be able to relate a state to a position in the input.