Laya Integration

Integrating Laya

Laya answers typed questions about a piece of text. You give it a state (the text) and a dict of questions; it returns a probability for every option of every question in one forward pass. It never generates text, so it cannot answer off-schema, and it is small enough (322M-421M parameters) to run inside the user's own process.

Reach for it when the project needs a decision, not a sentence: which queue, is this spam, how urgent, should the agent stop. Do not reach for it when the task needs reasoning over several steps, arithmetic, extraction of free-form values, or any generated text. For those, keep the LLM and use Laya in front of it as a cheap first pass (see "Cascade" below).

Install and first call

pip install laya        # pulls torch, transformers, safetensors, huggingface_hub

import laya

agent = laya.load("convaiinnovations/laya")          # English checkpoint

result = agent.predict(
    {"subject": "Charged twice", "body": "Refund the duplicate today or we cancel."},
    {
        "department": {
            "type": "choice",
            "instructions": "Which department should handle this request?",
            "criteria": {
                "billing": "invoices, payments, refunds",
                "technical": "bugs, outages, system errors",
                "other": "everything else",
            },
        },
        "urgency": {
            "type": "score",
            "instructions": "How urgent is this request?",
            "criteria": ["not urgent", "soon", "critical deadline or blocking issue"],
        },
        "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
    },
)
answers = result["answers"]

state may be a string, a dict or a list; dicts and lists are serialised to JSON. Load the model once at startup and reuse it. Loading takes 25-35 s; a call takes milliseconds.

The three question types

Every answer also carries action.act_probability from the model's act-or-escalate head, and result["usage"]["input_tokens"] reports the tokens consumed. confidence is one minus the normalised entropy of the distribution, so it is low whenever probability is spread out, even if the top option is right.

All questions in one predict call share a single forward pass, so ask everything you need about a state in one call rather than looping.

Writing questions that work

This matters more than anything else in the integration. Laya is an encoder doing something close to textual entailment, and it rewards questions shaped like that.

• Ask what the text says, not what to do about it. "Where is the bird relative to the gap?" produced clean graded probabilities; "Which way must the bird move?" came out inverted on every checkpoint, because the option word "up" is pulled towards "above" in the state. Ask a perception question, then let your code map the answer to an action.
• Put the state into words, never numbers. Given "Bird altitude: 20. Gap altitude: 60." no checkpoint could tell which was lower. If a decision depends on a comparison, threshold or sum, compute it in code and hand Laya the conclusion ("the bird is far below the gap").
• Describe each option. {"billing": "invoices, payments, refunds"} beats a bare ["billing", ...], because the description is what the model matches the text against.
• Keep option lists short. Each option is truncated to 48 tokens and all options for a question share a budget (192 tokens on the English checkpoint, 256 on the others). Upstream reports accuracy degrading past roughly 20 options. If a question raises ValueError: ... options exceed head_max_len, shorten the descriptions or split the question.
• Keep the state short and front-loaded. The English checkpoint reads 512 tokens in total, the others 1024, and an over-long state is cut from the end. Put what matters first. For emails, laya.email_state(subject, body, sender=...) strips quoted replies and signatures.
• Try two or three phrasings and measure. Small wording changes move results a lot. One concrete case: "blocked by a barrier" separated lanes far better than "blocked by a train".

Ready-made question sets exist for common jobs: laya.triage_questions(), laya.email_questions(), laya.guard_questions(), laya.moderation_questions(), laya.router_questions(). Use them as starting points and read what they contain before relying on them.

Choosing a checkpoint

agent    = laya.load("convaiinnovations/laya")                              # English, 421M
agent_ml = laya.load("convaiinnovations/laya", subfolder="multilingual")    # 100+ languages, 322M, ~1.6x faster
agent_td = laya.load("convaiinnovations/laya", subfolder="typed-decisions") # fine-tuned on four upstream workflows

Use the English checkpoint for English text. It is the best general performer and its probabilities are temperature-calibrated. Use multilingual for anything else. It ships uncalibrated (temperature 1.0), so expect it to report 100% and 0% readily; do not read those as certainty. It also missed explicit cancellation threats that the English checkpoint caught. typed-decisions only helps if your questions resemble the workflows it was tuned on.

laya.Router picks a checkpoint per request from the script and language of the state:

from laya import Router
router = Router(preload=True)                       # all three resident, ~4.6 GB in fp32
res = router.predict(state, questions)              # res["routing"] says which model and why
res = router.predict(state, questions, lang="pl")   # force the language when you know it

Pass lang= whenever the application knows the language. As of laya 0.3.4 the detector only recognises English, French, German, Spanish, Portuguese, Italian and Dutch among Latin-script languages. Polish, Czech, Turkish, Swedish and others are silently sent to the English checkpoint, which then answers confidently and wrongly. Non-Latin scripts route correctly.

Acting on probabilities

The probabilities are the product. Decide per question what happens at each confidence level, and make the threshold reflect the cost of being wrong:

a = answers["department"]
if a["probabilities"][a["choice"]] >= 0.85:
    route(a["choice"])
else:
    send_to_review(a)            # a person, a slower model, or a queue

For yes/no gates compare answers[id]["noul"] against a threshold chosen on real data, not 0.5 by default. For a guardrail, where a miss is expensive, set it low; for an auto-action, where a false alarm is expensive, set it high.

Cascade with an LLM. A good default architecture: Laya handles every request, and only the ones below the threshold go to the LLM or a person. The share that escalates is what you pay LLM latency and cost on, so measure it.

Calibrate on the user's own data before trusting the numbers. Temperatures live on the agent: agent.temperature is [choice, score, noul] and agent.temperature_by_options holds per-option-count overrides (keys like "choice:3-5", "noul:2") that take precedence. To refit: set agent.temperature = [1.0, 1.0, 1.0] and agent.temperature_by_options = {} to read raw probabilities p, fit a scalar T per question type that minimises negative log-likelihood of p ** (1 / T) (renormalised) on a labelled set, then write the fitted values back. A few hundred labelled examples are enough to see whether it helps.