Session Recording Architecture

This document explains how PromptKit’s session recording system captures, stores, and replays LLM conversations with full fidelity.

Overview

Session recording provides a complete audit trail of LLM interactions. Unlike simple logging, recordings capture:

• Precise timing: Millisecond-accurate event timestamps
• Complete data: All messages, tool calls, and media
• Reconstructable state: Enough information to replay conversations exactly

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Live Session  │ ──► │   Event Store   │ ──► │   Recording     │
│   (Emitter)     │     │   (JSONL)       │     │   (Replay)      │
└─────────────────┘     └─────────────────┘     └─────────────────┘
        │                       │                       │
        ▼                       ▼                       ▼
   Real-time              Persistent              Synchronized
   events                 storage                 playback

Event-Driven Architecture

Event Emitter

The Emitter captures events as they occur during a conversation:

emitter := events.NewEmitter(eventBus, runID, scenarioID, conversationID)

// Events are emitted automatically during conversation
emitter.ConversationStarted(systemPrompt)
emitter.MessageCreated(role, content, index, parts, toolCalls, toolResult)
emitter.AudioInput(audioData)
emitter.ProviderCallStarted(provider, model)
emitter.ToolCallStarted(toolName, args)
// ...

Binary data handling

The emitter’s MessageCreated() automatically strips binary data (base64 Data and FilePath) from content parts, emitting only metadata (MIMEType, SizeKB, dimensions). This keeps observability events lightweight.

RecordingStage bypasses the emitter and publishes events directly to the EventBus with full binary data intact. Use sdk.WithRecording() to enable this in SDK pipelines, or insert RecordingStages manually in Arena pipelines.

Events flow through an EventBus to registered subscribers:

Emitter ──► EventBus ──┬──► FileEventStore (persistence)
                       ├──► Metrics Collector
                       ├──► Eval Listener (auto-trigger pack evals)
                       └──► Real-time UI

Event Types

Events are categorized by their domain:

Category	Events	Purpose
Conversation	started, ended	Session lifecycle
Message	created, updated	Content exchange
Audio	input, output	Voice data
Provider	call.started, call.completed	LLM API calls
Tool	call.started, call.completed	Function execution
Validation	started, completed	Guardrail checks

Event Structure

Each event contains:

type Event struct {
    Type      EventType       // Event category
    Timestamp time.Time       // When it occurred
    SessionID string          // Session identifier
    Sequence  int64           // Ordering guarantee
    Data      interface{}     // Event-specific payload
}

Eval Integration

The EventBusEvalListener subscribes to message.created events on the EventBus, enabling automatic eval execution on recorded conversations:

Emitter ──► EventBus ──► EventBusEvalListener
                                │
                                ▼
                         SessionAccumulator
                         (builds conversation context)
                                │
                                ▼
                         EvalDispatcher ──► EvalRunner ──► ResultWriter
                                                              │
                                                    ┌─────────┴─────────┐
                                                    ▼                   ▼
                                             MetricContext       Message Metadata
                                             (Prometheus)        (pack_evals)

How it works:

1. RecordingStage (in the pipeline) publishes message.created events as content flows through
2. EventBusEvalListener receives these events and accumulates messages per session in a SessionAccumulator
3. On each assistant message, turn-level evals (every_turn, sample_turns) are dispatched asynchronously
4. On session close, session-level evals (on_session_complete, sample_sessions) run synchronously
5. Results feed into MetricContext (for Prometheus export) and can be attached to message metadata

This means pack evals run automatically against recorded conversations without any additional wiring — the EventBus handles the connection between recording and evaluation. For details on eval types and configuration, see Eval Framework.

Storage Format

FileEventStore

Events are persisted in JSONL (JSON Lines) format:

{"seq":1,"event":{"type":"conversation.started","timestamp":"2024-01-15T10:30:00Z",...}}
{"seq":2,"event":{"type":"message.created","timestamp":"2024-01-15T10:30:00.1Z",...}}
{"seq":3,"event":{"type":"audio.input","timestamp":"2024-01-15T10:30:00.2Z",...}}

Benefits:

• Append-only: Safe for concurrent writes
• Streamable: Process without loading entire file
• Human-readable: Easy debugging

SessionRecording Format

For export/import, recordings use a structured format:

{"type":"metadata","session_id":"...","start_time":"...","duration":"..."}
{"type":"event","offset":"100ms","event_type":"message.created",...}
{"type":"event","offset":"200ms","event_type":"audio.input",...}

The loader auto-detects both formats:

rec, err := recording.Load("session.jsonl")  // Works with either format

Media Timeline

For recordings with audio/video, the MediaTimeline provides synchronized access:

Recording
    │
    ▼
MediaTimeline
    ├── TrackAudioInput ──► User speech segments
    ├── TrackAudioOutput ──► Assistant speech segments
    └── TrackVideo ──► Video frames (if present)

Track Structure

Each track contains time-ordered segments:

type MediaSegment struct {
    Offset    time.Duration  // Start time relative to session
    Duration  time.Duration  // Segment length
    Data      []byte         // Raw media data
    Format    AudioFormat    // Sample rate, channels, encoding
}

Audio Reconstruction

Audio can be exported as standard WAV files:

timeline := rec.ToMediaTimeline(nil)
timeline.ExportAudioToWAV(events.TrackAudioInput, "user.wav")
timeline.ExportAudioToWAV(events.TrackAudioOutput, "assistant.wav")

The export process:

1. Collects all segments for the track
2. Concatenates PCM data in time order
3. Writes RIFF/WAVE header with format info
4. Outputs standard 16-bit PCM WAV

Replay System

ReplayPlayer

The ReplayPlayer provides synchronized access to recordings:

player, _ := recording.NewReplayPlayer(rec)

// Seek to any position
player.Seek(5 * time.Second)

// Query state at current position
state := player.GetState()
// state.CurrentEvents - events at this moment
// state.RecentEvents - events in last 2 seconds
// state.Messages - all messages up to this point
// state.AudioInputActive - is user speaking?
// state.AudioOutputActive - is assistant speaking?

Playback State

At any position, you can access:

Field	Description
Position	Current offset from session start
Timestamp	Absolute timestamp
CurrentEvents	Events within 50ms of position
RecentEvents	Events in last 2 seconds
Messages	Accumulated conversation
AudioInputActive	User audio present
AudioOutputActive	Assistant audio present
ActiveAnnotations	Annotations in scope

Annotation Correlation

Annotations can be attached to recordings for review:

player.SetAnnotations([]*annotations.Annotation{
    // Session-level annotation
    annotations.ForSession().WithScore("quality", 0.92),

    // Time-range annotation
    annotations.InTimeRange(start, end).WithComment("Good response"),

    // Event-targeted annotation
    annotations.ForEvent(eventSeq).WithLabel("category", "greeting"),
})

During playback, active annotations are included in state queries.

Replay Provider

For deterministic replay, the ReplayProvider simulates the original provider:

provider, _ := replay.NewProviderFromRecording(rec)

// Returns the same responses as the original session
response, _ := provider.Complete(ctx, messages, opts)

Use cases:

• Regression testing: Verify behavior against known-good responses
• Debugging: Reproduce exact conversation flows
• Offline testing: No API calls needed

Data Flow Summary

┌─────────────────────────────────────────────────────────────────┐
│                        LIVE SESSION                              │
│                                                                   │
│  User ──► Pipeline ──► Provider ──► Response ──► User            │
│              │                          │                         │
│              ▼                          ▼                         │
│          Emitter ────────────────► EventBus                       │
│                                        │                          │
└────────────────────────────────────────┼──────────────────────────┘
                                         │
                              ┌──────────┼──────────┐
                              ▼          ▼          ▼
┌──────────────────┐  ┌──────────────┐  ┌──────────────────────┐
│     STORAGE      │  │    EVALS     │  │       REPLAY         │
│                  │  │              │  │                      │
│  FileEventStore  │  │ EvalListener │  │ session.jsonl        │
│  ──► .jsonl      │  │ ──► Runner   │  │ ──► SessionRecording │
│                  │  │ ──► Metrics   │  │ ──► ReplayPlayer     │
│                  │  │ ──► Prom     │  │ ──► MediaTimeline     │
└──────────────────┘  └──────────────┘  └──────────────────────┘

Performance Considerations

Recording Overhead

• CPU: Minimal - events are queued and written asynchronously
• Memory: Bounded - segments are written incrementally
• Disk: Proportional to conversation length and audio duration

Audio Data Size

Audio is stored as base64-encoded PCM:

• Input: 16kHz, 16-bit mono = ~32KB/second
• Output: 24kHz, 16-bit mono = ~48KB/second
• Base64 encoding adds ~33% overhead

A 5-minute voice conversation generates approximately:

• Raw audio: ~24MB
• JSONL with metadata: ~32MB

Optimization Tips

1. Disable for CI: Skip recording for quick validation runs
2. Compress archives: JSONL compresses well (70-80% reduction)
3. Retention policies: Auto-delete old recordings
4. Selective recording: Only enable for specific scenarios

Next Steps

• Session Recording How-To - Practical usage guide
• Duplex Architecture - Voice conversation system
• Testing Philosophy - Why test prompts?