Runtime

Core framework powering all PromptKit tools

---

What is the Runtime?

The Runtime is the foundational layer that provides:

• Provider interfaces for implementing custom LLM providers
• Pipeline architecture for streaming request/response processing
• Stage system for extensible, composable processing layers
• Type definitions used across all PromptKit tools
• Tool execution framework for MCP integration
• Validation system for input/output checking

---

Who Should Use Runtime?

The Runtime is for advanced users who want to:

• Build custom providers for new LLM services
• Create custom stages for specialized processing
• Extend validators with domain-specific checks
• Integrate MCP servers with custom tools
• Understand internals for contributions
• Customize pipelines for unique requirements

Note: Most users should use the SDK instead. Use Runtime directly only if you’re extending PromptKit.

---

Quick Start

package main

import (
    "context"
    "github.com/AltairaLabs/PromptKit/runtime/pipeline/stage"
    "github.com/AltairaLabs/PromptKit/runtime/providers/openai"
)

func main() {
    // Create provider
    provider := openai.NewOpenAIProvider(
        "openai", "gpt-4o-mini", "",
        openai.DefaultProviderDefaults(),
        false,
    )

    // Build stage-based pipeline
    pipeline := stage.NewPipelineBuilder().
        Chain(
            stage.NewProviderStage(provider, nil, nil, &stage.ProviderConfig{
                MaxTokens:   1500,
                Temperature: 0.7,
            }),
        ).
        Build()

    // Execute
    input := make(chan stage.StreamElement, 1)
    msg := types.Message{Role: "user"}
    msg.AddTextPart("Hello!")
    input <- stage.NewMessageElement(msg)
    close(input)

    output, _ := pipeline.Execute(context.Background(), input)
    for elem := range output {
        if elem.Text != nil {
            fmt.Print(*elem.Text)
        }
    }
}

Next: Build Your First Pipeline Tutorial

---

Documentation by Type

Tutorials (Learn by Doing)

Step-by-step guides for extending Runtime:

1. First Pipeline - Build your first pipeline
2. Multi-Turn Conversations - State management
3. MCP Integration - Add MCP tools
4. Validation & Guardrails - Content safety
5. Production Deployment - Deploy pipelines
6. Advanced Patterns - Complex pipelines

How-To Guides (Accomplish Specific Tasks)

Focused guides for specific Runtime tasks:

• Configure Pipeline - Pipeline setup
• Setup Providers - Provider configuration
• Handle Errors - Error handling
• Streaming Responses - Real-time output
• Manage State - Conversation persistence
• Integrate MCP - MCP server integration
• Monitor Costs - Cost tracking

Explanation (Understand the Concepts)

Deep dives into Runtime architecture:

• Pipeline Architecture - Stage-based streaming
• Stage Design - Composable stage patterns
• Provider System - LLM provider abstraction
• State Management - Conversation persistence

Reference (Look Up Details)

Complete API documentation:

• Pipeline API - Stage and pipeline interfaces
• Providers API - Provider interfaces
• Types API - Core type definitions
• Tools & MCP API - Tool execution
• Validators API - Validation interfaces
• Storage API - State storage

---

Key Concepts

Stage Interface

Implement custom processing stages:

type Stage interface {
    Name() string
    Type() StageType
    Process(ctx context.Context, input <-chan StreamElement, output chan<- StreamElement) error
}

Pipeline Architecture

Streaming request processing flow:

Input → [Stage 1] → [Stage 2] → [Stage N] → Output

Each stage runs in its own goroutine, enabling concurrent streaming.

Stage Types

Different stage behaviors:

Type	Pattern	Example
Transform	1:1 or 1:N	Validation, enrichment
Accumulate	N:1	VAD buffering, collection
Generate	0:N	LLM streaming, TTS
Sink	N:0	State save, metrics

Tool Execution

MCP tool integration:

type Tool interface {
    Execute(ctx context.Context, params map[string]interface{}) (interface{}, error)
    Schema() *ToolSchema
}

---

Architecture

graph TB
    subgraph "SDK Layer"
        SDK[SDK]
    end

    subgraph "Runtime Layer"
        Pipeline[Pipeline]
        Stages[Stages]
        Providers[Providers]
        Tools[Tool Execution]
        Validators[Validators]
    end

    subgraph "External Services"
        OpenAI[OpenAI]
        Anthropic[Anthropic]
        MCP[MCP Servers]
    end

    SDK --> Pipeline
    Pipeline --> Stages
    Stages --> Providers
    Stages --> Tools
    Stages --> Validators

    Providers --> OpenAI
    Providers --> Anthropic
    Tools --> MCP

    style Runtime Layer fill:#f3e5f5

---

Use Cases

For Framework Contributors

• Add new provider support
• Implement new stage types
• Extend validation system
• Improve core functionality

For Advanced Developers

• Build custom providers for internal APIs
• Create domain-specific stages
• Integrate proprietary LLM services
• Implement custom tool execution

For System Integrators

• Bridge PromptKit with existing systems
• Customize request/response handling
• Add organization-specific features
• Extend for compliance requirements

---

Pipeline Modes

Runtime supports three pipeline configurations:

Text Mode

Standard HTTP-based LLM interactions:

Message → StateStoreLoad → PromptAssembly → Provider → StateStoreSave → Response

VAD Mode (Voice Activity Detection)

For voice applications using text-based LLMs:

Audio → AudioTurn → STT → Provider → TTS → Audio

ASM Mode (Audio Streaming)

For native multimodal LLMs with real-time audio:

Audio/Text → DuplexProvider → Audio/Text

---

Core Types

StreamElement

type StreamElement struct {
    Text      *string
    Audio     *AudioData
    Message   *types.Message
    ToolCall  *types.ToolCall
    Metadata  map[string]interface{}
    Error     error
    Timestamp time.Time
}

Message

type Message struct {
    Role    string
    Content string
    Parts   []ContentPart
}

Provider Response

type Response struct {
    Content   string
    ToolCalls []ToolCall
    Usage     *Usage
    Metadata  map[string]interface{}
}

---

Design Principles

Streaming First

Runtime is designed for streaming from the ground up:

• Channel-based data flow
• Concurrent stage execution
• Backpressure support
• True streaming (not simulated)

Extensibility

Runtime is designed to be extended without modifying core code:

• Plugin-based provider system
• Composable stages
• Flexible validation
• Open for extension, closed for modification

Performance

Optimized for production use:

• Minimal allocations
• Efficient streaming
• Context-aware cancellation
• Connection pooling

Type Safety

Leverage Go’s type system:

• Strong typing throughout
• Interface-based design
• Compile-time checks
• Clear contracts

---

Getting Help

• Quick Start: First Pipeline Tutorial
• Questions: GitHub Discussions
• Issues: Report a Bug
• Contributing: Contribution Guide

---

Related Documentation

• SDK: Built on top of Runtime
• Arena: Uses Runtime for testing
• Architecture: System Design