Ak Dev New Evaluator Provider

Adding a New Test Evaluator Provider

This guide walks through adding a new built-in evaluator provider to Agent Kernel's test framework. Use the existing DeepEval implementation (ak-py/src/agentkernel/test/core/evaluator/deepeval.py) as reference.

Before starting, check whether you actually need this skill: if the evaluator only needs to exist for your own project (not addressable by every AK user via a short built-in name), you don't need any of the steps below — just subclass AKEvaluator anywhere importable and point test-config.yaml's evaluator: at its dotted path. That's the "bring your own evaluator" path described in the user-facing ak-test skill and docs/docs/testing/cli-testing.md; examples/cli/custom-evaluator/ is a complete worked example of it. This skill is only for adding a first-party, in-repo provider that ships with AK and gets its own short type name.

Existing Providers

Architecture Overview

• AKEvaluator (ak-py/src/agentkernel/test/core/evaluator/base.py) is the abstract base every evaluator — built-in or bring-your-own — implements. It has exactly two abstract methods:
  • evaluate_by_score(case: AKEvaluationCase) -> AKEvaluationResult — deterministic scoring, no LLM call
  • evaluate_by_llm(case: AKEvaluationCase) -> AKEvaluationResult — LLM-as-judge scoring
• AKEvaluationCase carries the comparison inputs (user_input, actual, expected, threshold, context, criteria); AKEvaluationResult carries the outcome (score, passed, metric, evaluator, reason, cost, attempts, metadata).
• Error contract (every evaluator must honor this, not just DeepEval):
  • Raise AKMissingInput when a field the requested metric needs (e.g. case.expected) wasn't supplied.
  • Raise AKMetricNotSupported from whichever of the two methods your backend structurally cannot implement (e.g. a pure LLM-judge service has no offline scoring mode).
  • Raise AKEvaluationError when a configured backend fails to produce a score (missing credentials, transport error, unparseable judge output). Never raise AssertionError and never silently return a 0.0 to stand in for a failure — 0.0 must only ever mean "scored zero", not "couldn't be scored". Test.compare is the only place that decides pass/fail fatality; evaluators only ever set result.passed.
• Test._resolve_evaluator_class in ak-py/src/agentkernel/test/test.py is the factory. It shares the same pluggable-backend shape as guardrails, sandbox providers, and trace backends (core/util/factory.py's resolve_dotted/require_extra/AKConfigError): an if-per-built-in branch with the SDK import wrapped in require_extra (actionable ImportError naming the pip extra if missing), then a dotted-path bring-your-own fallback for anything else.
• Evaluator instances are cached per-process on Test._evaluator (keyed by the configured value), guarded by Test._evaluator_lock — construction happens once per distinct evaluator: config value, not once per Test.compare call.

Step-by-Step

1. Create the Evaluator Provider File

Create ak-py/src/agentkernel/test/core/evaluator/<provider>.py. Keep the provider's SDK imports inside this file only — test/core/evaluator/__init__.py and base.py stay pure Python with no optional-dependency imports at module level, so importing the AKEvaluator interface never requires your provider's SDK to be installed.

# ak-py/src/agentkernel/test/core/evaluator/<provider>.py
from agentkernel.test.config import AKTestConfig

from .base import AKEvaluationCase, AKEvaluationError, AKEvaluationResult, AKEvaluator, AKMissingInput


class <Provider>AKEvaluator(AKEvaluator):
    def __init__(self, config: AKTestConfig) -> None:
        super().__init__(config)
        # Lazy-init any client/model here only if evaluate_by_score never needs it
        # (mirrors DeepevalAKEvaluator's lazy LiteLLMModel, built only on first evaluate_by_llm call).

    def evaluate_by_score(self, case: AKEvaluationCase) -> AKEvaluationResult:
        if not case.expected:
            raise AKMissingInput("evaluate_by_score requires AKEvaluationCase.expected")
        # Deterministic, offline scoring logic here.
        score = ...  # float
        return AKEvaluationResult(
            metric="<metric_name>",
            evaluator="<provider>",
            score=score,
            passed=score >= case.threshold,
        )

    def evaluate_by_llm(self, case: AKEvaluationCase) -> AKEvaluationResult:
        if not case.expected:
            raise AKMissingInput("evaluate_by_llm requires AKEvaluationCase.expected")
        try:
            score = ...  # call the judge
        except Exception as exc:
            raise AKEvaluationError(f"<provider> llm-based evaluation failed: {exc}") from exc
        return AKEvaluationResult(
            metric="<metric_name>",
            evaluator="<provider>",
            score=score,
            reason=...,  # judge's explanation, if the backend provides one
            passed=score is not None and score >= case.threshold,
        )

If a mode genuinely doesn't apply to your backend (e.g. a provider that is LLM-judge-only), raise AKMetricNotSupported from that method instead of faking a result. Test.compare does not catch it — in fallback mode it propagates out of evaluate_by_score before evaluate_by_llm runs — so document that users of your provider must set the matching mode (e.g. JEV requires mode: llm).

2. Register with the Factory

Add the short name to _BUILTIN_EVALUATORS and a branch in Test._resolve_evaluator_class, both in ak-py/src/agentkernel/test/test.py:

_BUILTIN_EVALUATORS = ["deepeval", "opik", "jev", "<provider>"]          # ADD THIS

class Test:
    ...
    @classmethod
    def _resolve_evaluator_class(cls, configured: str) -> type[AKEvaluator]:
        if configured == "deepeval":
            with require_extra("test", "evaluator: deepeval"):
                from .core.evaluator.deepeval import DeepevalAKEvaluator
            return DeepevalAKEvaluator
        if configured == "opik":
            with require_extra("opik", "evaluator: opik"):
                from .core.evaluator.opik import OpikAKEvaluator
            return OpikAKEvaluator
        if configured == "jev":
            with require_extra("jev", "evaluator: jev"):
                from .core.evaluator.jev import JevAKEvaluator
            return JevAKEvaluator
        if configured == "<provider>":                                        # ADD THIS
            with require_extra("<provider>", "evaluator: <provider>"):
                from .core.evaluator.<provider> import <Provider>AKEvaluator
            return <Provider>AKEvaluator
        if "." not in configured:
            raise AKConfigError(
                f"unknown evaluator '{configured}'; expected one of {_BUILTIN_EVALUATORS} or a dotted path to an AKEvaluator subclass"
            )
        return resolve_dotted(configured, base=AKEvaluator)

A dotted evaluator: value (e.g. myorg.evaluators.CustomEvaluator) resolves via resolve_dotted without any factory edit at all — only add an if branch here for a first-party, in-repo provider you want addressable by a short name.

3. Add Optional Dependencies

Add a new extras group to ak-py/pyproject.toml for the provider's SDK — don't fold it into the existing test extra (that one stays DeepEval's, since every test user already needs it for the framework itself). Follow the pattern of the opik extra, the first provider added on top of the original DeepEval-only test extra:

[project.optional-dependencies]
<provider> = [
    "provider-sdk>=x.y.z",
]