Validation Strategy

Understanding PackC’s approach to pack validation and quality assurance.

Overview

PackC implements multi-level validation to catch errors early and ensure pack quality before deployment.

Validation Philosophy

Fail Fast

Catch errors at compile time, not runtime:

# Invalid prompt caught by packc
task_type: ""  # Error: task_type cannot be empty

Better than runtime failure:

// Runtime error (bad!)
conv, err := manager.NewConversation(ctx, pack, config)
// Error: prompt not found

Progressive Validation

Multiple checkpoints throughout compilation:

Parse → Validate Syntax → Validate Structure → Validate References → Final Check

Each stage catches different error types.

Warnings vs. Errors

Errors - Fatal, stop compilation:

Error: Missing required field 'task_type'
Error: Invalid template syntax

Warnings - Non-fatal, allow compilation:

Warning: Missing description for prompt 'support'
Warning: Tool 'search' not defined

Use --strict mode to treat warnings as errors (future).

Validation Levels

Level 1: Syntax Validation

What: Valid YAML/JSON syntax

When: During parsing

Examples:

# Invalid YAML
prompts
  - support.yaml  # Missing colon

Error:

Error parsing config: yaml: line 2: did not find expected key

Level 2: Schema Validation

What: Required fields present, correct types

When: After parsing

Check:

type PromptConfig struct {
  TaskType string `yaml:"task_type"` // Required
  SystemPrompt string `yaml:"system_prompt"` // Required
  UserTemplate string `yaml:"user_template"` // Optional
}

Errors:

Error: Missing required field 'task_type'
Error: Field 'temperature' must be number, got string

Level 3: Template Validation

What: Template syntax valid

When: During compilation

Check:

tmpl, err := template.New("test").Parse(userTemplate)
if err != nil {
  return fmt.Errorf("Invalid template: %w", err)
}

Examples:

# Invalid Go template
user_template: "

Error:

Error: Template parse error: unclosed action

Level 4: Reference Validation

What: Referenced resources exist

When: After pack assembly

Check:

Tools referenced in prompts are defined
Fragments referenced exist
Media files exist on disk

Warnings:

Warning: Tool 'search_kb' referenced but not defined
Warning: Image file not found: images/logo.png

Level 5: Semantic Validation

What: Logical consistency

When: Final validation pass

Check:

No duplicate prompt IDs
No circular fragment references
Reasonable parameter values
Pack size within limits

Warnings:

Warning: temperature=2.0 exceeds recommended max of 1.0
Warning: Pack size 2MB exceeds recommended 1MB

Validation Rules

Required Fields

Pack Level:

{
  "id": "required",
  "version": "required",
  "prompts": "required"
}

Prompt Level:

{
  "id": "required",
  "name": "optional but recommended",
  "description": "optional but recommended",
  "system": "required (system or system_template)",
  "user_template": "optional",
  "parameters": "optional"
}

Type Validation

String fields:

task_type: "support"  # ✅ Valid
task_type: 123        # ❌ Error: must be string

Numeric parameters:

temperature: 0.7      # ✅ Valid
temperature: "0.7"    # ❌ Error: must be number

Boolean fields:

debug: true          # ✅ Valid
debug: "true"        # ❌ Error: must be boolean

Range Validation

Temperature: 0.0 to 2.0

temperature: 0.7     # ✅ Valid
temperature: -1.0    # ⚠️ Warning: below 0
temperature: 3.0     # ⚠️ Warning: above 2.0

Max tokens: 1 to 100,000

max_tokens: 1000     # ✅ Valid
max_tokens: 0        # ❌ Error: must be positive
max_tokens: 200000   # ⚠️ Warning: very large

Pattern Validation

Pack ID: Alphanumeric with hyphens

id: "customer-support"     # ✅ Valid
id: "Customer Support!"    # ❌ Error: invalid characters

Task Type: Alphanumeric with hyphens/underscores

task_type: "support-agent"  # ✅ Valid
task_type: "support agent"  # ❌ Error: no spaces

Template Validation

Go Templates:

# Valid
user_template: "User: "
user_template: ""

# Invalid
user_template: "
user_template: ""  # Unknown function

Variable References:

# Warning if variables not documented
user_template: ""  # ⚠️ Undocumented variable

Validation Implementation

Validation Pipeline

func (p *Pack) Validate() []string {
  var warnings []string

  // 1. Validate pack structure
  warnings = append(warnings, p.validateStructure()...)

  // 2. Validate each prompt
  for _, prompt := range p.Prompts {
    warnings = append(warnings, prompt.Validate()...)
  }

  // 3. Validate references
  warnings = append(warnings, p.validateReferences()...)

  // 4. Validate fragments
  warnings = append(warnings, p.validateFragments()...)

  return warnings
}

Validation Context

Provide context for errors:

type ValidationError struct {
  Field   string  // Which field
  Value   interface{}  // Actual value
  Message string  // Error description
  File    string  // Source file
  Line    int     // Line number
}

Error Message:

Error in prompts/support.yaml:
  Line 8: Missing required field 'task_type'

  7:   spec:
  8:     name: Support Agent
  9:     system_prompt: ...

Validation Strategies

Strict Mode

Treat warnings as errors:

packc compile --strict

Use for:

Production builds
CI/CD pipelines
Release candidates

Lenient Mode (default)

Allow warnings:

packc compile

Use for:

Development
Quick iteration
Testing

Custom Validation

Add custom rules (future):

# .packc-rules.yaml
validation:
  rules:
    - name: require-descriptions
      level: error
      check: prompt.description != ""

    - name: max-system-prompt-length
      level: warning
      check: len(prompt.system) < 1000

Pre-Deployment Validation

Validation Checklist

Before deploying to production:

1. Schema valid

packc validate pack.json

2. Size acceptable

du -h pack.json
# Should be < 1MB

3. JSON valid

jq empty pack.json

4. Required metadata present

jq -e '.compiler_version' pack.json
jq -e '.version' pack.json

5. Load test

# Test with SDK
go run test-pack.go pack.json

Automated Validation

In CI/CD:

- name: Validate packs
  run: |
    packc compile --config arena.yaml --output pack.json --id app
    packc validate pack.json

    # Additional checks
    ./scripts/check-pack-size.sh pack.json
    ./scripts/test-pack-load.sh pack.json

Validation Reports

Summary Report

packc validate pack.json

Output:

Validating pack: pack.json

Pack Information:
  ID: customer-support
  Version: 1.0.0
  Prompts: 3

✓ Schema valid
✓ All prompts valid
✓ All references valid

⚠️ Warnings: 2
  - Missing description for prompt 'support'
  - Tool 'search_kb' referenced but not defined

Summary: Pack is valid with 2 warnings

Detailed Report (future)

packc validate pack.json --detailed

Output:

Validation Report
=================

Pack: customer-support v1.0.0
Compiler: packc-v0.1.0
Validated: 2025-01-16T10:30:00Z

Schema Validation: ✅ PASS
  - Pack ID present
  - Version valid (1.0.0)
  - 3 prompts found

Prompt Validation: ✅ PASS
  - greeting: ✅ Valid
  - support: ⚠️ Missing description
  - escalation: ✅ Valid

Reference Validation: ⚠️ WARNINGS
  - Tool 'search_kb': not defined
  - Tool 'create_ticket': not defined

Template Validation: ✅ PASS
  - All templates syntactically valid
  - 2 variables used: name, message

Fragment Validation: ✅ PASS
  - No fragments used

Size Analysis:
  - Pack size: 45 KB
  - Estimated load time: 5ms
  - Memory footprint: ~100 KB

Overall: ✅ VALID (2 warnings)

Validation Best Practices

1. Validate Early

# During development
packc compile && packc validate

2. Validate Often

# Pre-commit hook
packc validate pack.json || exit 1

3. Automate Validation

.PHONY: validate
validate:
  packc validate packs/*.pack.json

4. Track Warnings

# Log warnings
packc validate pack.json 2>&1 | tee validation.log

5. Fix Warnings

Don’t ignore warnings - they indicate potential issues:

# Bad
task_type: support
# Warning: Missing description

# Good
task_type: support
description: "Handles customer support inquiries"

Validation Evolution

Current (v0.1.0)

Schema validation
Template syntax checking
Basic reference checking
Size warnings

Planned (v0.2.0)

Semantic validation
Custom validation rules
Validation profiles
Detailed reports

Future (v1.0.0)

ML-based validation (detect poor prompts)
Security scanning
Performance predictions
Quality scoring

Summary

PackC’s validation strategy:

Multi-level - Syntax, schema, references, semantics
Progressive - Validation at each pipeline stage
Informative - Clear error messages with context
Flexible - Warnings vs. errors, strict mode
Automated - Easy to integrate in workflows
Evolving - Continuous improvement

This ensures packs are correct, complete, and production-ready before deployment.