Arize Phoenix

Arize Phoenix

Phoenix is an open-source AI observability platform built on OpenTelemetry that helps developers understand, debug, and improve AI applications. It provides comprehensive tracing, evaluation, prompt engineering, and experimentation capabilities for LLM-based systems. Phoenix captures detailed execution information from AI applications, measures output quality with evaluators, enables systematic prompt iteration, and supports data-driven experimentation to optimize AI performance.

When to Use This Skill

• Debugging AI application failures by inspecting LLM calls, tool executions, and retrieval operations
• Measuring and improving AI output quality using LLM-based or code-based evaluators
• Iterating on prompts using real production examples and testing variations systematically
• Comparing different versions of AI applications (prompts, models, architectures) using experiments
• Monitoring LLM costs, token usage, latency, and error rates in production
• Building datasets from production traces for evaluation and fine-tuning
• Tracking multi-turn conversations and maintaining context across interactions
• Optimizing RAG systems by analyzing retrieval quality and document relevance
• Evaluating agent performance including tool call accuracy and actionability
• Managing prompt versions and deploying them across different environments

Capabilities

Agents can leverage Phoenix to:

• Trace AI application execution with detailed visibility into LLM calls, tool executions, retrieval operations, embeddings, and prompt templates
• Evaluate output quality using pre-built or custom evaluators with LLM-as-a-judge or code-based evaluation logic
• Annotate traces with human feedback, scores, labels, and quality signals for continuous improvement
• Experiment systematically by comparing different versions of applications using datasets and evaluators
• Monitor performance metrics including latency, token usage, costs, and error rates across projects
• Iterate on prompts using the playground, span replay, and dataset-based testing
• Organize traces into projects and sessions for better management and analysis
• Integrate with 20+ AI frameworks and LLM providers via OpenTelemetry instrumentation

Skills

Tracing

• Capture traces via OpenTelemetry (OTLP) protocol with automatic instrumentation for major frameworks
• View execution flow showing every LLM call, tool execution, retrieval operation, embedding generation, and response generation
• Inspect LLM parameters including temperature, system prompts, function calls, and invocation parameters
• Analyze retrieval operations with document scores, order, and embedding text for RAG systems
• Track token usage with detailed breakdowns by token type (input/output) and model
• Monitor latency at trace, span, and component levels with quantile analysis
• Organize with projects to separate traces by environment, application, team, or use case
• Group with sessions to track multi-turn conversations and maintain context across interactions
• Add metadata to traces with custom attributes, tags, and structured data for filtering and analysis
• Annotate traces with scores, labels, human feedback, and LLM evaluations for quality measurement
• Export and import traces for backup, migration, or analysis in external tools
• Track costs with automatic calculation based on token usage and model pricing

Evaluation

• Run LLM-as-a-judge evaluations using any LLM provider (OpenAI, Anthropic, Gemini, custom endpoints) to assess output quality
• Build custom evaluators with Python or TypeScript using custom prompts, scoring logic, and evaluation criteria
• Use pre-built evaluators for common tasks including faithfulness, relevance, toxicity, summarization, agent evaluation, and RAG quality
• Write code-based evaluators for deterministic checks like exact match, regex patterns, or custom Python/TypeScript logic
• Execute evaluations at scale with automatic concurrency, rate limit handling, error management, and batching via executors
• Map complex inputs using input schemas and mappings to transform nested data structures for evaluators
• View evaluator traces with complete transparency into prompts, model reasoning, scores, and execution metadata
• Run batch evaluations on traces, datasets, or custom data sources with automatic retry and error handling
• Integrate evaluations into workflows by running evals on production traces or test datasets

Datasets & Experiments

• Create datasets from traces, code, CSV files, or manually curated examples with inputs and optional reference outputs
• Build golden datasets with reference outputs (ground truth) for objective evaluation using code-based evaluators
• Version datasets with automatic tracking of inserts, updates, and deletes for reproducibility
• Run experiments by executing task functions against datasets with evaluators to compare different versions
• Compare experiments side-by-side in the UI to see performance differences, score distributions, and individual example results
• Use repetitions to run experiments multiple times for statistical confidence and account for LLM variability
• Organize with splits to separate datasets into train/test/validation splits for proper evaluation workflows
• Export datasets in JSONL or CSV formats for fine-tuning, analysis, or sharing
• View experiment results in the Phoenix UI with task function traces, scores per example, and aggregate performance metrics

Prompt Engineering

• Manage prompts with versioning, storage, and deployment across different environments
• Test prompts interactively in the Prompt Playground with various models, parameters, and tools
• Replay LLM spans from production traces in the playground to debug failures and test improvements
• Test at scale by running prompts against datasets to evaluate performance systematically
• Compare prompt versions side-by-side to see which performs better on your data
• Optimize automatically using automated prompt optimization features
• Sync prompts via SDK to keep prompts in sync across applications and environments programmatically
• Tag prompts for deployment control across development, staging, and production environments
• Track prompt changes with version history, author information, and timestamps

Projects & Organization

• Create projects to organize traces by environment (development, staging, production), application, or team
• Set up sessions to track multi-turn conversations with chatbot-like UI showing conversation history
• View metrics dashboards with pre-defined metrics including latency, errors, token usage, costs, and model performance
• Filter and search traces by metadata, attributes, annotations, or custom tags
• Configure data retention policies to control how long trace and evaluation data is stored

API & Programmatic Access

• Use Python SDK (arize-phoenix-client, arize-phoenix-evals, arize-phoenix-otel) for programmatic access
• Use TypeScript SDK (arizeai-phoenix-client, arizeai-phoenix-evals, arizeai-phoenix-otel) for JavaScript/TypeScript applications
• Access REST API for annotations, datasets, experiments, traces, spans, prompts, projects, and users
• Instrument manually using OpenTelemetry decorators, wrappers, or direct OpenInference SDKs
• Generate API keys for programmatic access with role-based permissions

Authentication & Security

• Configure RBAC with role-based access control for user permissions and project access
• Set up authentication including SSO and user management for self-hosted instances
• Manage API keys for secure programmatic access to Phoenix APIs and SDKs
• Control data privacy with self-hosting options for VPC deployment or local execution

Workflows

Workflow 1: Instrument and Trace an AI Application

1. Choose integration - Select appropriate Phoenix integration for your framework (LangChain, LlamaIndex, OpenAI, etc.)
2. Install package - Install Phoenix client and OpenTelemetry packages for your language (Python or TypeScript)
3. Configure endpoint - Set Phoenix endpoint URL and optionally configure project name and session tracking
4. Instrument application - Add auto-instrumentation or manual instrumentation to capture LLM calls, tool executions, and retrievals
5. View traces - Open Phoenix UI to see execution flow, latency, token usage, and detailed span information
6. Add annotations - Add scores, labels, or human feedback to traces for quality measurement