Eval

EvalKit

Overview

EvalKit is a conversational evaluation framework for AI agents that guides you through creating robust evaluations using the Strands Evals SDK. Through natural conversation, you can plan evaluations, generate test data, execute evaluations, and analyze results.

Parameters

• agent_path (required): Path to the agent folder to evaluate (e.g., ./chatbot-agent, /path/to/my-agent)
• evaluation_focus (optional): Specific aspects to evaluate (e.g., "response quality", "tool calling accuracy")
• test_case_count (optional, default: 3): Number of test cases to generate
• output_format (optional, default: "jsonl"): Format for test data output

Constraints for parameter acquisition:

• You MUST ask for the agent_path if not provided because evaluation cannot proceed without knowing which agent to evaluate
• You MUST support multiple input methods including:
  • Direct input: Path provided directly in the conversation
  • Relative paths: Paths relative to current working directory
  • Absolute paths: Full system paths to agent location
• You SHOULD infer evaluation_focus from user's natural language description when not explicitly provided
• You MAY use intelligent defaults for optional parameters based on agent analysis

Steps

1. Setup and Initialization

When a user requests evaluation (any phase), first validate the environment:

Folder Structure: All evaluation artifacts MUST be created in the eval/ folder at the same level as the target agent folder:

<agent-evaluation-project>/       # Example name - can be any name for user's evaluation project
├── <target-agent-folder>/        # Example name - this is the agent you are evaluating
│   └── [agent source code]     # Existing agent code
└── eval/                       # All evaluation files go here (sibling to target-agent-folder)
    ├── eval-plan.md
    ├── test-cases.jsonl
    ├── results/
    ├── run_evaluation.py
    ├── eval-report.md
    └── README.md

Note:

• The eval/ folder is a sibling directory to user's agent folder, not nested inside it
• agent-evaluation-project and target-agent-folder are placeholder names - user may use any names that fit their project

Constraints:

• You MUST check if the agent folder exists
• You MUST verify Python 3.11+ is installed
• You MUST navigate to the evaluation project directory (containing both agent folder and eval/) before any operation
• You MUST create the eval/ folder as a sibling to the agent folder
• You MUST NOT create evaluation folders inside the agent folder
• You MUST create the eval/ directory at the same level as the agent folder if it doesn't exist
• You MUST ensure all evaluation artifacts are within the eval/ folder
• You MUST check for any existing evaluation artifacts in the eval/ folder
• You SHOULD validate that required dependencies are available
• You MUST use relative paths from the evaluation project directory (e.g., "./eval/eval-plan.md") for all file operations

2. Planning Phase

When to Trigger: User requests evaluation planning or mentions creating/designing an evaluation

User Intent Recognition:

• Keywords: "plan", "design", "create evaluation", "evaluate my agent"
• Context: User provides agent path or describes agent to evaluate
• Goal: Understand what the user wants to evaluate and why

Execution Flow:

1. Parse user request: Extract agent path, evaluation focus, and specific requirements from natural language

2. Navigate to evaluation project directory:

   cd <your-evaluation-project>  # Navigate to the directory containing both agent folder and eval/

3. Create evaluation directory structure:

   mkdir -p eval

4. Follow this execution flow:

   1. Parse user evaluation requirements from user input
   2. Analyze agent and user requirements:
      • Parse specific evaluation requirements, scenarios, and constraints from user input
      • Scan codebase for agent architecture and capabilities
      • Check for existing test cases and evaluation files
   3. Design evaluation strategy:
      • Define evaluation areas and metrics (user-request-driven with agent-aware defaults)
      • Identify test data requirements
      • Define file structure
      • Select technology stack

5. Write the complete evaluation plan to eval/eval-plan.md using the template structure (see Appendix A: Evaluation Plan Template), replacing placeholders with concrete details derived from the analysis while preserving section order and headings.

6. Report completion with evaluation plan file path, and suggest next step: "Would you like me to generate test cases based on this plan?"

Evaluation Planning Guidelines

Design Principles

High-Level Design (What & Why):

• Focus on WHAT to evaluate and WHY it matters for the agent
• Define evaluation areas and metrics that are measurable and verifiable
• Ensure requirements can be tested through actual agent execution

Low-Level Implementation (How):

• Select appropriate technology stack and architecture
• Design practical file structure and execution approach
• Choose integration patterns and configuration methods

Metrics Guidelines

Evaluation metrics must be:

1. Measurable: Define what will be measured
2. Verifiable: Can be measured through actual agent execution
3. Implementation-ready: Clear enough to guide technical implementation

Architecture Principles

Key Principles:

• Simple Structure: Use the flat eval/ directory structure
• Real Agent Focus: Always use actual agent execution, never simulation or mock
• Focused Implementation: Avoid over-engineering, focus on core evaluation logic
• Minimal Viable Implementation: Start with essential components, add complexity incrementally
• Framework-First: Leverage existing evaluation frameworks before building custom solutions
• Modular Design: Create reusable components that can be easily tested and maintained

Technology Selection Defaults

Examples of reasonable defaults:

• Evaluation Framework: Strands Evals SDK
• LLM calling service: Built into Strands framework
• LLM provider: Amazon Bedrock
• Data processing: JSON or JSONL
• Agent integration: Direct imports for Python agents

Constraints:

• You MUST prioritize user evaluation requests over detected agent state
• You MUST create eval-plan.md using the template structure
• You MUST analyze agent architecture and capabilities in target-agent-folder
• You MUST define evaluation areas and metrics (user-request-driven with agent-aware defaults)
• You MUST make informed decisions without requiring excessive user clarification
• You MUST save the evaluation plan to eval/eval-plan.md (sibling to agent folder)
• You MUST ensure the eval/ folder is at the same level as the agent folder

3. Test Data Generation Phase

When to Trigger: User requests test case generation or mentions creating test data

User Intent Recognition:

• Keywords: "generate test cases", "create tests", "test data", "test scenarios"
• Context: Evaluation plan exists
• Goal: Create comprehensive test cases

Execution Flow:

1. Parse user request: Extract any specific requirements (e.g., "focus on edge cases", "10 test cases")

2. Navigate to evaluation project directory:

   cd <your-evaluation-project>  # Navigate to the directory containing both agent folder and eval/

3. Load the current evaluation plan (eval/eval-plan.md) to understand evaluation areas and test data requirements.

4. Follow this execution flow:

   1. Parse user context from user input (if provided)
   2. Validate that the evaluation plan contains test data requirements; update the evaluation plan if it does not align with the user's input (if provided); add entry to User Requirements Log in eval-plan.md
   3. Generate proper test cases covering all scenarios and meeting all requirements
   4. Structure test cases in JSONL format
   5. Save test cases to eval/test-cases.jsonl
   6. Update Evaluation Progress section in eval-plan.md with completion status

5. Report completion with test case count, coverage summary, and suggest next step: "Would you like me to run the evaluation with these test cases?"

Data Generation Guidelines

1. Prioritize user-specific data requests: User input takes precedence over the established evaluation plan - always honor specific user requirements and constraints. Update the evaluation plan if needed.

Constraints:

• You MUST load and validate the evaluation plan from eval/eval-plan.md
• You MUST prioritize user-specific data requests over established evaluation plan
• You MUST generate data in JSONL format
• You MUST save test cases to eval/test-cases.jsonl
• You MUST ensure all files remain within the eval/ folder
• You MUST update Evaluation Progress section in eval-plan.md

4. Evaluation Implementation and Execution Phase

When to Trigger: User requests evaluation execution or mentions running tests

User Intent Recognition:

• Keywords: "run evaluation", "execute", "run tests", "evaluate"
• Context: Test cases exist
• Goal: Execute evaluation and generate results

Execution Flow:

1. Parse user request: Extract any specific requirements (e.g., "run on subset", "verbose output")

2. Navigate to evaluation project directory:

   cd <your-evaluation-project>  # Navigate to the directory containing both agent folder and eval/

3. Load the current evaluation plan (eval/eval-plan.md) to understand evaluation requirements and agent architecture.

4. Follow this execution flow:

   1. Parse user context from user input (if provided)
   2. Review evaluation plan to understand requirements; update the evaluation plan if it does not align with the user's input (if provided); add entry to User Requirements Log in eval-plan.md
   3. Implement Strands Evals SDK evaluation pipeline: IMPORTANT: Always navigate to repository root before any operation in the following process to avoid path errors.
      • Create requirements.txt: Detect existing dependencies and consolidate into unified requirements.txt at repository root, adding Strands Evals SDK dependencies
      • Set up environment: Use uv to create virtual environment, activate it, and install requirements.txt
      • Implement run_evaluation.py: Create eval/run_evaluation.py using Strands Evals SDK patterns with Case objects, Experiment class, and appropriate evaluators
      • Create agent integration: Implement agent execution logic within the evaluation framework
      • Execute evaluation: Run the experiment to generate evaluation results
      • Save results: Store evaluation results in eval/results/ directory
      • Create documentation: Create eval/README.md with running instructions for users
   4. Update Evaluation Progress section in eval-plan.md with completion status

5. Report completion with evaluation results summary and suggest next step: "Would you like me to analyze these results and provide recommendations?"

Implementation Guidelines

CRITICAL: Always Create Minimal Working Version: Implement the most basic version that works

Strands Evals SDK Integration

CRITICAL REQUIREMENT - Getting Latest Documentation: Before implementing evaluation code, you MUST retrieve the latest Strands Evals SDK documentation and API usage examples. This is NOT optional. You MUST NOT proceed with implementation without either context7 access or the source code. This ensures you're using the most current patterns and avoiding deprecated APIs.

Step 1: Check Context7 MCP Availability: First, check if context7 MCP server is available by attempting to use it. If you receive an error indicating context7 is not available, proceed to Step 3.

Step 2: Primary Method - Using Context7 (If Available):

1. Use context7 to get library documentation: "Get documentation for strands-agents-evals focusing on Case, Experiment, and Evaluator classes"
2. Review the latest API patterns and examples
3. Implement evaluation code using the current API

Step 3: Fallback Method - REQUIRED If Context7 Is Not Available: If context7 MCP is not installed or doesn't have Strands Evals SDK documentation, you MUST STOP and prompt the user to take one of these actions:

REQUIRED USER ACTION - Choose ONE of the following:

Option 1: Install Context7 MCP Server (Recommended)

Please install the context7 MCP server in your coding assistant to access the latest Strands Evals SDK documentation. Installation steps vary by assistant:

• For your specific coding assistant: Check your assistant's documentation on how to install MCP servers
• Context7 MCP package: @upstash/context7-mcp
• Common installation: Many assistants support adding MCP servers through their settings/configuration

Note: If you're unsure how to install MCP servers in your coding assistant, please consult your assistant's support resources or choose Option 2 below (clone source code).

After installation, you'll be able to query: "Get documentation for strands-agents-evals focusing on Case, Experiment, and Evaluator classes"

Option 2: Clone Strands Evals SDK Source Code

If you cannot install context7 MCP or prefer to work with source code directly:

cd <your-evaluation-project>
git clone https://github.com/strands-agents/evals strands-agents-evals-source

IMPORTANT: You MUST NOT proceed with implementation until the user has completed one of these options. Do NOT attempt to implement evaluation code using only the reference examples in Appendix C, as they may be outdated.

After the user confirms they've completed one of the above options:

If Context7 was installed:

1. Use context7 to get the latest Strands Evals SDK documentation
2. Review the latest API patterns and examples
3. Implement evaluation code using the current API

If source code was cloned:

1. Read the source files to understand the current API: strands-agents-evals-source/src/strands_evals/
2. Check examples in the repository: strands-agents-evals-source/examples/
3. Review API definitions and usage patterns
4. Implement evaluation code based on the actual source code

Core Components:

• Case objects: Individual test cases with input, expected output, and metadata
• Experiment class: Collection of cases with evaluator for running evaluations
• Built-in evaluators: OutputEvaluator, TrajectoryEvaluator, InteractionsEvaluator
• Direct execution: Agent execution with evaluation, no separate trace collection needed

Environment Setup Guidelines

1. Check Existing Requirements: Verify requirements.txt exists in repository root

   # Check if requirements.txt exists
   ls requirements.txt

2. Add Strands Evals SDK Dependencies: Update existing requirements.txt with Strands evaluation dependencies

   # Add Strands Evals SDK and related dependencies
   grep -q "strands-agents-evals" requirements.txt || echo "strands-agents-evals" >> requirements.txt
   # Add other evaluation-specific dependencies as needed based on evaluation plan

3. Installation: Use uv for dependency management

   uv venv
   source .venv/bin/activate
   uv pip install -r requirements.txt

Common Pitfalls to Avoid

• Over-Engineering: Don't add complexity before the basic version works
• Ignoring the Plan: Follow the established evaluation plan structure and requirements
• Separate Trace Collection: Don't implement separate trace collection - Strands Evals SDK handles this automatically

Constraints:

• You MUST implement evaluation in eval/run_evaluation.py using Strands Evals SDK
• You MUST ensure all evaluation code files are within eval/
• You MUST always create minimal working version first
• You MUST execute evaluation and save results to eval/results/
• You MUST create eval/README.md with running instructions
• You MUST keep all evaluation artifacts within the eval/ folder
• You MUST update Evaluation Progress section in eval-plan.md
• You MUST NOT implement separate trace collection - Strands Evals SDK handles this automatically