Arena Reference
Complete technical specifications and reference materials for PromptArena.
Quick Links
Section titled “Quick Links”Complete command-line interface reference with all flags and options.
YAML configuration file structure and all available options.
All available assertion types for validating LLM responses.
Built-in validators for checking response quality and compliance.
Test scenario file structure and specification.
Report generation formats (HTML, JSON, JUnit, Markdown).
Complete duplex streaming configuration for voice testing scenarios.
Picking a tool mode (mock static / mock template / live / mcp / exec / client) and writing each one.
Reference vs. How-To
Section titled “Reference vs. How-To”This is reference documentation - dry, factual, technical specifications.
Looking for task-oriented guides? See:
- Arena How-To Guides - Accomplish specific tasks
- Arena Tutorials - Learn by building
Quick Reference Tables
Section titled “Quick Reference Tables”Command Summary
Section titled “Command Summary”| Command | Purpose |
|---|---|
promptarena run | Execute test scenarios |
promptarena config-inspect | Validate configuration |
promptarena debug | Debug configuration loading |
promptarena prompt-debug | Test prompt rendering |
promptarena render | Generate reports from results |
Common Assertions
Section titled “Common Assertions”| Assertion | Purpose |
|---|---|
content_includes | Response contains specific text |
content_matches | Response matches regex pattern |
tools_called | Specific tools were invoked |
is_valid_json | Response is valid JSON |
json_schema | Response matches JSON schema |
llm_judge | LLM evaluates response quality |
Output Formats
Section titled “Output Formats”| Format | Use Case |
|---|---|
| JSON | Machine processing, APIs |
| HTML | Human-readable reports |
| JUnit | CI/CD integration |
| Markdown | Documentation, sharing |
API Stability
Section titled “API Stability”Arena reference documentation follows semantic versioning:
- Stable: CLI commands, configuration schema
- Beta: Advanced assertions, custom validators
- Experimental: New features marked explicitly
Getting Help
Section titled “Getting Help”- How-To Guides: Task-oriented documentation
- Tutorials: Learning-oriented guides
- Explanations: Conceptual documentation
- Issues: GitHub Issues