Welcome to Brokkr

Brokkr is a control plane for Kubernetes that lets you dynamically create and manage applications across clusters. Define what you need, fire it off, and the controller loop takes care of the rest — your applications get created, configured, and reconciled automatically.

Use Cases

On-Demand Application Provisioning

A customer needs a new service spun up? A new tenant needs their own stack? Just create the deployment through Brokkr and it flows through the controller loop to your clusters. No manual kubectl, no waiting on CI pipelines — your infrastructure adapts to your needs in real time.

Dynamic Service Management

As your requirements change, Brokkr lets you define, reconfigure, and scale the services running across your clusters. Generators can programmatically create deployment objects, templates let you stamp out standardized configurations, and the agent reconciliation loop keeps everything in the desired state.

Multi-Cluster Orchestration

Manage applications across multiple Kubernetes clusters from a single control plane. Target specific clusters with labels, push updates to all of them at once, and let each agent independently reconcile its own state. Brokkr handles the coordination so you can focus on what to deploy, not where and how.

Explore Brokkr

• Getting Started — Install, configure, and get Brokkr running
• How-To Guides — Practical guides for common tasks
• Explanation — Architecture, concepts, and design decisions
• Reference — API reference and technical details

What Makes Brokkr Different?

While tools like FluxCD and ArgoCD excel at GitOps-based state management, Brokkr takes a different approach — it’s built for dynamic, on-demand application lifecycle management rather than static manifest synchronization.

Programmatic Resource Creation

Brokkr’s generators and templates let external systems programmatically create Kubernetes resources through an API. CI/CD pipelines, customer provisioning systems, or internal tools can fire off deployments without touching git repos or manifest files.

Controller Loop Reconciliation

Every agent runs its own reconciliation loop, continuously pulling its target state from the broker and applying it to its cluster. Resources drift? The agent corrects it. New deployment object pushed? The agent picks it up on the next poll.

Built for Dynamic Workloads

Where GitOps tools work best with a known, static set of manifests, Brokkr is designed for environments where the set of applications changes frequently — multi-tenant platforms, on-demand infrastructure, and systems where what needs to run is determined at runtime, not at commit time.

Getting Started with Brokkr

Welcome to Brokkr! This section will guide you through the process of installing and setting up Brokkr for your environment.

Prerequisites

Before you begin, ensure you have:

• Kubernetes cluster access
• kubectl installed and configured
• Rust toolchain (for building from source)
• Docker (for container deployments)

Quick Navigation

1. Installation - Install Brokkr on your system
2. Quick Start - Get up and running quickly
3. Configuration - Configure Brokkr for your environment

What’s Next?

After completing the getting started guide, you can:

• Follow our tutorials for hands-on learning
• Check out the how-to guides for specific tasks
• Dive into the reference documentation for detailed information

Installing Brokkr

This guide will help you install Brokkr using Helm, the recommended installation method.

Prerequisites

Before installing Brokkr, ensure you have:

• Kubernetes cluster (v1.20 or later)
• kubectl CLI configured to access your cluster
• Helm 3.8 or later installed (installation guide)

Verifying Prerequisites

# Check Kubernetes version
kubectl version --short

# Check Helm version
helm version --short

# Verify cluster access
kubectl cluster-info

Quick Start

Get Brokkr up and running in under 10 minutes with a broker and agent in your Kubernetes cluster.

1. Install the Broker

Install the broker with bundled PostgreSQL for development:

# Install broker with bundled PostgreSQL
helm install brokkr-broker oci://ghcr.io/colliery-io/charts/brokkr-broker \
  --set postgresql.enabled=true \
  --wait

# Verify broker is running
kubectl get pods -l app.kubernetes.io/name=brokkr-broker

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
brokkr-broker-xxxxxxxxxx-xxxxx   1/1     Running   0          2m

2. Get Broker URL

# Port forward to access the broker locally
kubectl port-forward svc/brokkr-broker 3000:3000 &

# The broker is now accessible at http://localhost:3000

3. Create an Agent and Get PAK

Create an agent registration and retrieve its Prefixed API Key (PAK):

# Create a new agent
curl -X POST http://localhost:3000/api/v1/agents \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-agent",
    "cluster_name": "development"
  }'

The response will include the agent’s PAK:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "my-agent",
  "cluster_name": "development",
  "status": "ACTIVE",
  "pak": "brokkr_BRxxxxxxxx_yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}

Save the pak value - you’ll need it to install the agent.

4. Install the Agent

Install the agent using the PAK from step 3:

# Install agent (replace <PAK> with the actual PAK from step 3)
helm install brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --set broker.url=http://brokkr-broker:3000 \
  --set broker.pak="<PAK>" \
  --wait

# Verify agent is running
kubectl get pods -l app.kubernetes.io/name=brokkr-agent

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
brokkr-agent-xxxxxxxxxxx-xxxxx   1/1     Running   0          1m

5. Verify Installation

Check that both components are healthy:

# Check broker health
kubectl exec deploy/brokkr-broker -- wget -qO- http://localhost:3000/healthz

# Check agent health
kubectl exec deploy/brokkr-agent -- wget -qO- http://localhost:8080/healthz

# View agent registration in broker logs
kubectl logs deploy/brokkr-broker | grep -i agent

You should see “OK” from both health checks and agent registration messages in the broker logs.

Detailed Installation

Broker Installation

The broker is the central management service that coordinates deployments across your Kubernetes clusters.

Development Setup (Bundled PostgreSQL)

For development and testing, use the bundled PostgreSQL:

helm install brokkr-broker oci://ghcr.io/colliery-io/charts/brokkr-broker \
  --set postgresql.enabled=true \
  --set postgresql.auth.password=brokkr \
  --wait

Using Provided Values Files

Brokkr includes pre-configured values files for different environments:

# Development (bundled PostgreSQL, minimal resources)
helm install brokkr-broker oci://ghcr.io/colliery-io/charts/brokkr-broker \
  -f https://raw.githubusercontent.com/colliery-io/brokkr/main/charts/brokkr-broker/values/development.yaml

# Staging (external PostgreSQL, moderate resources)
helm install brokkr-broker oci://ghcr.io/colliery-io/charts/brokkr-broker \
  -f https://raw.githubusercontent.com/colliery-io/brokkr/main/charts/brokkr-broker/values/staging.yaml

# Production (external PostgreSQL, production-grade resources)
helm install brokkr-broker oci://ghcr.io/colliery-io/charts/brokkr-broker \
  -f https://raw.githubusercontent.com/colliery-io/brokkr/main/charts/brokkr-broker/values/production.yaml

You can also download these files and customize them:

# Download development values
curl -O https://raw.githubusercontent.com/colliery-io/brokkr/main/charts/brokkr-broker/values/development.yaml

# Edit as needed
vi development.yaml

# Install with custom values
helm install brokkr-broker oci://ghcr.io/colliery-io/charts/brokkr-broker \
  -f development.yaml

View all available values files:

• Development
• Staging
• Production

Agent Installation

The agent runs in each Kubernetes cluster you want to manage and communicates with the broker.

Basic Agent Installation

# Create agent via broker API (see Quick Start step 3)
# Then install with the returned PAK:

helm install brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --set broker.url=http://brokkr-broker:3000 \
  --set broker.pak="<PAK_FROM_BROKER>" \
  --wait

Using Provided Values Files

Brokkr includes pre-configured values files for agents:

# Development (minimal resources, cluster-wide RBAC)
helm install brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --set broker.url=http://brokkr-broker:3000 \
  --set broker.pak="<PAK>" \
  -f https://raw.githubusercontent.com/colliery-io/brokkr/main/charts/brokkr-agent/values/development.yaml

# Staging (moderate resources)
helm install brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --set broker.url=http://brokkr-broker:3000 \
  --set broker.pak="<PAK>" \
  -f https://raw.githubusercontent.com/colliery-io/brokkr/main/charts/brokkr-agent/values/staging.yaml

# Production (production-grade resources)
helm install brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --set broker.url=http://brokkr-broker:3000 \
  --set broker.pak="<PAK>" \
  -f https://raw.githubusercontent.com/colliery-io/brokkr/main/charts/brokkr-agent/values/production.yaml

View all available agent values files:

• Development
• Staging
• Production

Chart Versions

Brokkr Helm charts are published to GitHub Container Registry (GHCR).

Installing Specific Versions

# Install a specific release version
helm install brokkr-broker oci://ghcr.io/colliery-io/charts/brokkr-broker \
  --version 1.0.0 \
  --set postgresql.enabled=true

# List available versions
# Visit: https://github.com/orgs/colliery-io/packages/container/package/charts%2Fbrokkr-broker

Development Builds

Development builds are available for testing:

# Development builds use semver pre-release format with timestamps
# Example: 0.0.0-develop.20251021150606

# Find the latest development build at:
# https://github.com/orgs/colliery-io/packages/container/package/charts%2Fbrokkr-broker

# Install development build (replace timestamp with actual version)
helm install brokkr-broker oci://ghcr.io/colliery-io/charts/brokkr-broker \
  --version 0.0.0-develop.20251021150606 \
  --set postgresql.enabled=true

Upgrading Brokkr

Upgrade your Brokkr installation to a newer version:

# Upgrade broker
helm upgrade brokkr-broker oci://ghcr.io/colliery-io/charts/brokkr-broker \
  --version 1.1.0 \
  --reuse-values

# Upgrade agent
helm upgrade brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --version 1.1.0 \
  --reuse-values

Uninstalling Brokkr

Remove Brokkr from your cluster:

# Uninstall agent
helm uninstall brokkr-agent

# Uninstall broker (this will also remove bundled PostgreSQL if enabled)
helm uninstall brokkr-broker

# Note: PersistentVolumes may remain - delete manually if needed
kubectl get pv
kubectl delete pv <pv-name>

Verifying the Installation

Health Checks

# Broker health endpoint
kubectl exec deploy/brokkr-broker -- wget -qO- http://localhost:3000/healthz

# Agent health endpoint
kubectl exec deploy/brokkr-agent -- wget -qO- http://localhost:8080/healthz

Both should return “OK”.

Connectivity Tests

# Check agent registration in broker
kubectl logs deploy/brokkr-broker | grep "agent registered"

# Check agent connection to broker
kubectl logs deploy/brokkr-agent | grep "connected to broker"

# List registered agents via API
kubectl port-forward svc/brokkr-broker 3000:3000 &
curl http://localhost:3000/api/v1/agents

Test Deployment

Create a test namespace to verify end-to-end functionality:

# Port forward to broker
kubectl port-forward svc/brokkr-broker 3000:3000 &

# Create a stack
STACK_ID=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Content-Type: application/json" \
  -d '{"name": "test-stack", "description": "Test stack"}' \
  | jq -r '.id')

# Deploy a test namespace
curl -X POST http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: brokkr-test",
    "is_deletion_marker": false
  }'

# Verify namespace was created
kubectl get namespace brokkr-test

# Clean up
curl -X POST http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: brokkr-test",
    "is_deletion_marker": true
  }'

kubectl get namespace brokkr-test  # Should show Terminating/NotFound

Configuration Reference

Broker Values

Key configuration options for the broker chart:

Agent Values

Key configuration options for the agent chart:

For complete configuration options, see the chart values files:

• Broker Chart Values
• Agent Chart Values

Next Steps

• Follow our Quick Start Guide to deploy your first application
• Learn about Basic Concepts in Brokkr
• Explore Configuration Guide

Troubleshooting

Common Issues

Broker pod not starting:

# Check pod status
kubectl describe pod -l app.kubernetes.io/name=brokkr-broker

# Check logs
kubectl logs -l app.kubernetes.io/name=brokkr-broker

Agent not connecting to broker:

# Verify broker URL is accessible from agent
kubectl exec deploy/brokkr-agent -- wget -qO- http://brokkr-broker:3000/healthz

# Check agent logs for connection errors
kubectl logs -l app.kubernetes.io/name=brokkr-agent

Database connection errors:

# Check PostgreSQL is running
kubectl get pods -l app.kubernetes.io/name=postgresql

# Check database credentials
kubectl get secret brokkr-broker-postgresql -o yaml

PAK authentication failures:

• Verify the PAK is correct and not expired
• Check that the agent name matches the registration
• Ensure the broker URL is accessible

Getting Help

• Check our GitHub Issues
• Check our GitHub Issues for known issues and solutions

Building from Source

For contributors or advanced users who want to build Brokkr from source:

Prerequisites

• Rust toolchain (1.8+)
• PostgreSQL database (v12+)
• Kubernetes cluster
• Docker (for building images)

Build Instructions

# Clone the repository
git clone https://github.com/colliery-io/brokkr.git
cd brokkr

# Build using Cargo
cargo build --release

# The binaries will be available in target/release/
# - brokkr-broker: The central management service
# - brokkr-agent: The Kubernetes cluster agent

Running Locally

# Set up database
export BROKKR__DATABASE__URL="postgres://brokkr:brokkr@localhost:5432/brokkr"

# Run broker
./target/release/brokkr-broker serve

# Run agent (in another terminal)
export BROKKR__AGENT__PAK="<your-pak>"
export BROKKR__AGENT__BROKER_URL="http://localhost:3000"
./target/release/brokkr-agent start

Development Environment

For active development:

# Install Angreal (development task runner)
pip install angreal

# Start the development environment
angreal local up

# Rebuild specific services
angreal local rebuild broker
angreal local rebuild agent

For more details on contributing, see the project README.

Quick Start Guide

This guide walks through deploying your first application using Brokkr. By the end, you will understand the core deployment workflow: creating stacks, associating them with agents, deploying Kubernetes resources, and cleaning up when you’re done.

Prerequisites

Before starting, ensure you have completed the Installation Guide with both the broker and at least one agent running. You will need the admin PAK (Prefixed API Key) from the broker setup and kubectl configured to access your target cluster. The examples below assume the broker is accessible at http://localhost:3000 via port-forward.

Understanding the Deployment Model

Brokkr organizes deployments around three interconnected concepts. A stack represents a logical grouping of Kubernetes resources that belong together—an application, a service, or any collection of related resources. An agent runs in a Kubernetes cluster and handles the actual application of resources. Targeting connects stacks to agents, telling Brokkr which agents should receive which stacks.

The deployment flow works as follows: when you create a deployment object in a stack, Brokkr stores it in the broker’s database. Agents that target that stack poll the broker on a regular interval and retrieve pending deployment objects. Each agent then applies the resources to its cluster using Kubernetes server-side apply, ensuring resources are created or updated to match the desired state.

Step 1: Create a Stack

Stacks serve as containers for related Kubernetes resources. Every deployment in Brokkr belongs to a stack, so you’ll start by creating one for your quick-start application.

# Set your admin PAK for convenience
export ADMIN_PAK="brokkr_BR..."  # Replace with your actual PAK

# Create a stack
curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -d '{
    "name": "quick-start-app",
    "description": "My first Brokkr deployment"
  }' | jq .

The response includes a stack ID that you’ll use in subsequent commands:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "quick-start-app",
  "description": "My first Brokkr deployment",
  "created_at": "2024-01-15T10:30:00Z"
}

# Save the stack ID for later use
export STACK_ID="550e8400-e29b-41d4-a716-446655440000"  # Use your actual ID

Step 2: Target the Stack to Your Agent

Agents only receive deployment objects from stacks they’re targeting. This targeting relationship must be created explicitly, giving you control over which agents manage which resources.

First, find your agent’s ID by listing registered agents:

# List agents to find the ID
curl -s http://localhost:3000/api/v1/agents \
  -H "Authorization: Bearer $ADMIN_PAK" | jq '.[].id'

Then create the targeting relationship that connects your agent to the stack:

# Save the agent ID
export AGENT_ID="your-agent-id-here"

# Create the targeting relationship
curl -s -X POST http://localhost:3000/api/v1/agents/$AGENT_ID/targets \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -d "{
    \"agent_id\": \"$AGENT_ID\",
    \"stack_id\": \"$STACK_ID\"
  }" | jq .

With the targeting in place, the agent will poll for deployment objects from this stack on its next polling cycle.

Step 3: Deploy the Application

Now you’ll create a deployment object containing Kubernetes resources. This example deploys a simple application with a namespace, a ConfigMap for configuration, and a Deployment that reads from that ConfigMap.

Create a YAML file with the resources:

cat > quick-start.yaml << 'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: quick-start
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: quick-start
data:
  message: "Hello from Brokkr!"
  environment: development
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: quick-start-app
  namespace: quick-start
spec:
  replicas: 1
  selector:
    matchLabels:
      app: quick-start-app
  template:
    metadata:
      labels:
        app: quick-start-app
    spec:
      containers:
      - name: app
        image: busybox:1.36
        command: ["sh", "-c", "while true; do echo $MESSAGE; sleep 30; done"]
        env:
        - name: MESSAGE
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: message
EOF

Submit this YAML to Brokkr as a deployment object. The jq command properly encodes the YAML content for the JSON request:

curl -s -X POST "http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -d "$(jq -n --arg yaml "$(cat quick-start.yaml)" '{yaml_content: $yaml, is_deletion_marker: false}')" | jq .

The broker stores this deployment object and marks it pending. On the agent’s next polling cycle (default: every 30 seconds), it retrieves the object and applies the resources to the cluster.

Step 4: Verify the Deployment

After waiting a few seconds for the agent to poll and apply the resources, verify they were created correctly in your cluster:

# Check the namespace was created
kubectl get namespace quick-start

# Verify the ConfigMap contains your data
kubectl get configmap app-config -n quick-start -o yaml

# Check the Deployment and its pods
kubectl get deployment quick-start-app -n quick-start
kubectl get pods -n quick-start

# View the application logs to confirm it's working
kubectl logs -n quick-start -l app=quick-start-app

You should see the pod running and logging “Hello from Brokkr!” every 30 seconds.

You can also check the deployment status through the Brokkr API to see how the broker tracked this deployment:

# View deployment objects in the stack
curl -s "http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects" \
  -H "Authorization: Bearer $ADMIN_PAK" | jq '.[] | {id, sequence_id, status, created_at}'

Step 5: Update the Application

Updating resources in Brokkr works by submitting a new deployment object with the complete desired state. Brokkr uses full-state deployments rather than partial updates—each deployment object represents all the resources that should exist.

Create an updated version of the application with more replicas and a changed message:

cat > quick-start-updated.yaml << 'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: quick-start
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: quick-start
data:
  message: "Updated: Brokkr is working!"
  environment: production
  debug: "false"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: quick-start-app
  namespace: quick-start
spec:
  replicas: 2
  selector:
    matchLabels:
      app: quick-start-app
  template:
    metadata:
      labels:
        app: quick-start-app
    spec:
      containers:
      - name: app
        image: busybox:1.36
        command: ["sh", "-c", "while true; do echo $MESSAGE; sleep 30; done"]
        env:
        - name: MESSAGE
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: message
EOF

Deploy the updated resources through Brokkr:

curl -s -X POST "http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -d "$(jq -n --arg yaml "$(cat quick-start-updated.yaml)" '{yaml_content: $yaml, is_deletion_marker: false}')" | jq .

After the agent polls and applies the update, verify the changes took effect:

# Check that replicas scaled to 2
kubectl get pods -n quick-start

# Verify the ConfigMap was updated
kubectl get configmap app-config -n quick-start -o jsonpath='{.data.message}'

Step 6: Clean Up

To delete resources through Brokkr, submit a deployment object with is_deletion_marker: true. This signals the agent to remove the specified resources rather than apply them.

curl -s -X POST "http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -d "$(jq -n --arg yaml "$(cat quick-start-updated.yaml)" '{yaml_content: $yaml, is_deletion_marker: true}')" | jq .

The YAML content is required even for deletions so the agent knows exactly which resources to remove. After the agent processes this deletion marker, verify the resources are gone:

# The namespace should be terminating or gone
kubectl get namespace quick-start

Finally, clean up the Brokkr resources themselves if you no longer need them:

# Delete the targeting relationship
curl -s -X DELETE "http://localhost:3000/api/v1/agents/$AGENT_ID/targets/$STACK_ID" \
  -H "Authorization: Bearer $ADMIN_PAK"

# Delete the stack
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/$STACK_ID" \
  -H "Authorization: Bearer $ADMIN_PAK"

Next Steps

You have now completed the core Brokkr workflow: creating stacks, targeting them to agents, deploying resources, updating them, and cleaning up. This pattern—declarative state stored in the broker, pulled and applied by agents—scales to managing deployments across many clusters.

From here, explore more advanced capabilities:

• Read about Core Concepts to understand Brokkr’s design philosophy
• Learn about the Data Model to understand how entities relate
• Explore the Configuration Guide for production deployment options
• Check the Work Orders reference for container builds and other transient operations

Configuration Guide

Brokkr uses a layered configuration system that allows settings to be defined through default values, configuration files, and environment variables. This guide provides a comprehensive reference for all configuration options and explains how to configure both the broker and agent components.

Configuration Sources

Configuration values are loaded from multiple sources, with later sources taking precedence over earlier ones:

1. Default values embedded in the application from default.toml
2. Configuration file (optional) specified at startup
3. Environment variables prefixed with BROKKR__

This layering enables a flexible deployment model where defaults work out of the box, configuration files provide environment-specific settings, and environment variables allow runtime overrides without modifying files.

Environment Variable Naming

All environment variables use the BROKKR__ prefix with double underscores (__) as separators for nested configuration. The naming convention converts configuration paths to uppercase with underscores.

For example, the configuration path broker.webhook_delivery_interval_seconds becomes the environment variable BROKKR__BROKER__WEBHOOK_DELIVERY_INTERVAL_SECONDS.

Database Configuration

The database configuration controls the connection to PostgreSQL.

The schema setting enables multi-tenant deployments where each tenant’s data is isolated in a separate PostgreSQL schema. When configured, all queries automatically set search_path to the specified schema.

# Standard single-tenant configuration
BROKKR__DATABASE__URL=postgres://user:password@db.example.com:5432/brokkr

# Multi-tenant configuration with schema isolation
BROKKR__DATABASE__URL=postgres://user:password@db.example.com:5432/brokkr
BROKKR__DATABASE__SCHEMA=tenant_acme

Logging Configuration

Logging settings control the verbosity and format of application logs.

The log level is hot-reloadable—changes take effect without restarting the application. Use json format in production environments for easier log aggregation and parsing.

# Production logging configuration
BROKKR__LOG__LEVEL=info
BROKKR__LOG__FORMAT=json

Broker Configuration

The broker configuration controls the central management service’s behavior.

Core Settings

Webhook Settings

Webhooks deliver event notifications to external systems. These settings control the delivery worker’s behavior.

The encryption key protects webhook URLs and authentication headers at rest. If not configured, the broker generates a random key at startup and logs a warning. This means encrypted data will become unreadable if the broker restarts. For production deployments, always configure an explicit encryption key.

# Production webhook configuration
BROKKR__BROKER__WEBHOOK_ENCRYPTION_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
BROKKR__BROKER__WEBHOOK_DELIVERY_INTERVAL_SECONDS=5
BROKKR__BROKER__WEBHOOK_DELIVERY_BATCH_SIZE=100
BROKKR__BROKER__WEBHOOK_CLEANUP_RETENTION_DAYS=30

Diagnostic Settings

Diagnostics are temporary operations that agents execute for debugging purposes.

Audit Log Settings

Audit logs record all significant actions for security and compliance.

# Extended audit retention for compliance
BROKKR__BROKER__AUDIT_LOG_RETENTION_DAYS=365

Auth Cache Settings

The broker caches PAK authentication results in memory to reduce database queries per request. Each authenticated request would otherwise require 2-3 database lookups (admin, agent, generator tables). The cache is automatically invalidated when PAKs are rotated or entities are deleted.

# Increase cache TTL for high-throughput deployments
BROKKR__BROKER__AUTH_CACHE_TTL_SECONDS=120

# Disable auth caching entirely
BROKKR__BROKER__AUTH_CACHE_TTL_SECONDS=0

CORS Configuration

Cross-Origin Resource Sharing (CORS) settings control which origins can access the broker API.

CORS settings are hot-reloadable. In production, restrict allowed_origins to specific domains rather than using *.

Agent Configuration

The agent configuration controls the Kubernetes cluster agent’s behavior.

Required Settings

Polling Settings

Health and Monitoring

Kubernetes Settings

# Complete agent configuration
BROKKR__AGENT__BROKER_URL=https://broker.example.com:3000
BROKKR__AGENT__PAK=brokkr_BRabc123_xyzSecretTokenHere
BROKKR__AGENT__AGENT_NAME=production-east
BROKKR__AGENT__CLUSTER_NAME=prod-us-east-1
BROKKR__AGENT__POLLING_INTERVAL=10
BROKKR__AGENT__HEALTH_PORT=8080
BROKKR__AGENT__DEPLOYMENT_HEALTH_ENABLED=true
BROKKR__AGENT__DEPLOYMENT_HEALTH_INTERVAL=60

Telemetry Configuration

Telemetry settings control OpenTelemetry trace export for distributed tracing.

Base Settings

Component Overrides

The broker and agent can have independent telemetry configurations that override the base settings.

# Enable telemetry with different sampling for broker and agent
BROKKR__TELEMETRY__ENABLED=true
BROKKR__TELEMETRY__OTLP_ENDPOINT=http://otel-collector:4317
BROKKR__TELEMETRY__SERVICE_NAME=brokkr
BROKKR__TELEMETRY__SAMPLING_RATE=0.1
BROKKR__TELEMETRY__BROKER__SERVICE_NAME=brokkr-broker
BROKKR__TELEMETRY__BROKER__SAMPLING_RATE=0.5
BROKKR__TELEMETRY__AGENT__SERVICE_NAME=brokkr-agent
BROKKR__TELEMETRY__AGENT__SAMPLING_RATE=0.1

PAK Configuration

Prefixed API Key (PAK) settings control token generation characteristics.

These settings are typically left at their defaults. Changing them affects only newly generated PAKs—existing PAKs remain valid.

Configuration File Format

Configuration files use TOML format. All settings can be specified in a configuration file as an alternative to environment variables.

[database]
url = "postgres://user:password@localhost:5432/brokkr"
schema = "tenant_acme"

[log]
level = "info"
format = "json"

[broker]
webhook_encryption_key = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
webhook_delivery_interval_seconds = 5
webhook_delivery_batch_size = 50
webhook_cleanup_retention_days = 7
diagnostic_cleanup_interval_seconds = 900
diagnostic_max_age_hours = 1
audit_log_retention_days = 90
auth_cache_ttl_seconds = 60

[cors]
allowed_origins = ["https://admin.example.com"]
allowed_methods = ["GET", "POST", "PUT", "DELETE", "OPTIONS"]
allowed_headers = ["Content-Type", "Authorization"]
max_age_seconds = 3600

[agent]
broker_url = "https://broker.example.com:3000"
pak = "brokkr_BRabc123_xyzSecretTokenHere"
agent_name = "production-east"
cluster_name = "prod-us-east-1"
polling_interval = 10
max_retries = 60
health_port = 8080
deployment_health_enabled = true
deployment_health_interval = 60

[telemetry]
enabled = true
otlp_endpoint = "http://otel-collector:4317"
service_name = "brokkr"
sampling_rate = 0.1

[telemetry.broker]
service_name = "brokkr-broker"
sampling_rate = 0.5

[telemetry.agent]
service_name = "brokkr-agent"
sampling_rate = 0.1

[pak]
prefix = "brokkr"
short_token_prefix = "BR"
short_token_length = 8
long_token_length = 24

Hot-Reload Configuration

The broker supports dynamic configuration reloading for certain settings without requiring a restart.

Hot-Reloadable Settings

These settings can be changed at runtime:

• log.level - Log verbosity
• cors.allowed_origins - CORS origins
• cors.max_age_seconds - CORS preflight cache
• broker.diagnostic_cleanup_interval_seconds - Diagnostic cleanup interval
• broker.diagnostic_max_age_hours - Diagnostic retention
• broker.webhook_delivery_interval_seconds - Webhook delivery interval
• broker.webhook_delivery_batch_size - Webhook batch size
• broker.webhook_cleanup_retention_days - Webhook retention

Static Settings (Require Restart)

These settings require an application restart to change:

• database.url - Database connection
• database.schema - Database schema
• broker.webhook_encryption_key - Encryption key
• broker.pak_hash - Admin PAK hash
• broker.auth_cache_ttl_seconds - Auth cache TTL
• telemetry.* - All telemetry settings
• pak.* - All PAK generation settings

Triggering a Reload

Reload configuration via the admin API:

curl -X POST https://broker.example.com/api/v1/admin/config/reload \
  -H "Authorization: Bearer $ADMIN_PAK"

When a configuration file is specified via BROKKR_CONFIG_FILE, the broker automatically watches it for filesystem changes with a 5-second debounce period.

Troubleshooting

Common Configuration Issues

Database connection failures typically indicate incorrect credentials or network issues. Verify the database URL is correct, the database server is running, and network connectivity exists between the broker and database.

# Test database connectivity
psql "postgres://user:password@localhost:5432/brokkr" -c "SELECT 1"

Agent authentication failures usually result from an invalid PAK. Verify the PAK was copied correctly without extra whitespace and that the agent record hasn’t been deleted from the broker.

Kubernetes access issues in agents may indicate missing or invalid credentials. When running outside a cluster, ensure BROKKR__AGENT__KUBECONFIG_PATH points to a valid kubeconfig file. When running inside a cluster, verify the service account has appropriate RBAC permissions.

Debugging Configuration

Enable trace-level logging to see configuration loading details:

BROKKR__LOG__LEVEL=trace brokkr-broker

The broker logs configuration values at startup (with sensitive values redacted), making it easy to verify which settings were applied.

Getting Help

If you encounter configuration issues:

1. Check the logs for detailed error messages
2. Verify all required configuration values are set
3. Test connectivity to external dependencies (database, Kubernetes API)
4. Consult the GitHub Issues for known issues

Tutorials

Step-by-step tutorials for learning Brokkr. Each tutorial walks you through a complete workflow from start to finish, building practical skills along the way.

These tutorials assume you have a working Brokkr development environment. If you haven’t set one up yet, follow the Installation Guide first.

Tutorial: Deploy Your First Application

In this tutorial, you’ll deploy an nginx web server to a Kubernetes cluster through Brokkr. You’ll learn the core workflow: creating a stack, adding Kubernetes manifests as deployment objects, and watching an agent apply them to a cluster.

What you’ll learn:

• How stacks organize Kubernetes resources
• How deployment objects carry YAML manifests
• How agents poll for and apply resources
• How to verify deployments succeeded

Prerequisites:

• A running Brokkr development environment (see Installation Guide — angreal local up starts the full stack)
• The admin PAK (Pre-Authentication Key) printed during first startup (check broker logs if you missed it)
• curl and jq installed

Step 1: Verify the Environment

First, confirm the broker is running and healthy:

curl -s http://localhost:3000/healthz

You should see:

OK

Check that at least one agent is registered:

curl -s http://localhost:3000/api/v1/agents \
  -H "Authorization: <your-admin-pak>" | jq '.[].name'

You should see the default agent name (typically "DEFAULT"). Note the agent’s id field — you’ll need it later.

Tip: Throughout this tutorial, replace <your-admin-pak> with the actual admin PAK from your broker startup logs. It looks like brokkr_BR3rVsDa_GK3QN7CDUzYc6iKgMkJ98M2WSimM5t6U8.

Step 2: Create a Stack

A stack is a named container that groups related Kubernetes resources. Think of it as a logical application — everything needed to run your service lives inside one stack.

Stacks are always owned by a generator (the entity that manages deployments, typically a CI/CD pipeline). As an admin, you can create a stack on behalf of any generator. For this tutorial, we’ll use a nil generator ID to indicate the stack is admin-managed:

STACK_ID=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "tutorial-nginx",
    "description": "Tutorial: a simple nginx deployment",
    "generator_id": "00000000-0000-0000-0000-000000000000"
  }' | jq -r '.id')

echo "Stack ID: $STACK_ID"

The response contains the new stack with its ID. The generator_id field ties the stack to its owning generator — we’ll explore generators in a later tutorial.

Step 3: Target the Agent to the Stack

Agents don’t automatically receive every stack’s resources. You need to explicitly target an agent to a stack, telling it “you are responsible for deploying this stack’s resources.”

First, get the agent ID:

AGENT_ID=$(curl -s http://localhost:3000/api/v1/agents \
  -H "Authorization: <your-admin-pak>" | jq -r '.[0].id')

echo "Agent ID: $AGENT_ID"

Now target the agent to your stack:

curl -s -X POST "http://localhost:3000/api/v1/agents/${AGENT_ID}/targets" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"stack_id\": \"${STACK_ID}\"}" | jq .

The agent will now receive deployment objects from this stack on its next poll cycle.

Step 4: Create a Deployment Object

A deployment object contains the actual Kubernetes YAML that the agent will apply to its cluster. You can include multiple Kubernetes resources in a single deployment object using multi-document YAML (separated by ---).

Create a deployment object with an nginx namespace, deployment, and service:

curl -s -X POST "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: tutorial-nginx\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: nginx\n  namespace: tutorial-nginx\n  labels:\n    app: nginx\nspec:\n  replicas: 2\n  selector:\n    matchLabels:\n      app: nginx\n  template:\n    metadata:\n      labels:\n        app: nginx\n    spec:\n      containers:\n      - name: nginx\n        image: nginx:1.27\n        ports:\n        - containerPort: 80\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: nginx\n  namespace: tutorial-nginx\nspec:\n  selector:\n    app: nginx\n  ports:\n  - port: 80\n    targetPort: 80"
  }' | jq .

The response includes a sequence_id — an auto-incrementing number that orders deployment objects within a stack. The agent uses this to know which version is latest.

Step 5: Watch the Agent Apply Resources

The agent polls the broker at its configured interval (default: 10 seconds). Within a few seconds, you should see the resources appear in your Kubernetes cluster.

Check the agent events to confirm the deployment was applied:

curl -s "http://localhost:3000/api/v1/agents/${AGENT_ID}/events" \
  -H "Authorization: <your-admin-pak>" | jq '.[] | {event_type, status, message, created_at}'

You should see a SUCCESS event:

{
  "event_type": "DEPLOYMENT",
  "status": "SUCCESS",
  "message": "Successfully applied deployment object",
  "created_at": "2025-01-15T10:01:30Z"
}

If you have kubectl configured to talk to your development cluster (k3s), verify the resources directly:

kubectl get all -n tutorial-nginx

Expected output:

NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-7c5ddbdf54-abc12   1/1     Running   0          30s
pod/nginx-7c5ddbdf54-def34   1/1     Running   0          30s

NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/nginx   ClusterIP   10.43.120.50   <none>        80/TCP    30s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   2/2     2            2           30s

Step 6: Update the Deployment

To update a deployment, create a new deployment object in the same stack. The agent detects the new sequence_id and applies the updated manifests, reconciling the cluster state.

Scale nginx to 3 replicas:

curl -s -X POST "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: tutorial-nginx\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: nginx\n  namespace: tutorial-nginx\n  labels:\n    app: nginx\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: nginx\n  template:\n    metadata:\n      labels:\n        app: nginx\n    spec:\n      containers:\n      - name: nginx\n        image: nginx:1.27\n        ports:\n        - containerPort: 80\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: nginx\n  namespace: tutorial-nginx\nspec:\n  selector:\n    app: nginx\n  ports:\n  - port: 80\n    targetPort: 80"
  }' | jq .

After the next poll cycle, verify the update:

kubectl get deployment nginx -n tutorial-nginx

You should see 3/3 in the READY column.

Step 7: Clean Up

To remove the deployed resources, create a deletion marker — a special deployment object with is_deletion_marker: true. This tells the agent to delete all resources previously applied for this stack from the cluster (not just the resources listed in the YAML content):

curl -s -X POST "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: tutorial-nginx",
    "is_deletion_marker": true
  }' | jq .

The agent will remove the Kubernetes resources on its next poll. Verify:

kubectl get namespace tutorial-nginx

After a few seconds, the namespace and all its contents will be gone.

Optionally, remove the agent target and delete the stack:

# Remove the target
curl -s -X DELETE "http://localhost:3000/api/v1/agents/${AGENT_ID}/targets/${STACK_ID}" \
  -H "Authorization: <your-admin-pak>"

# Delete the stack (soft delete — marks as deleted but preserves the record)
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${STACK_ID}" \
  -H "Authorization: <your-admin-pak>"

Note: Deletion in Brokkr is a “soft delete” — the record is marked with a deleted_at timestamp but not removed from the database. See Soft Deletion for details.

What You’ve Learned

• Stacks group related Kubernetes resources under a single name
• Deployment objects carry the YAML manifests inside a stack
• Agent targets connect agents to stacks, controlling which clusters receive which resources
• Sequence IDs let the agent know when a newer version is available
• Deletion markers trigger resource cleanup on the cluster
• Agents use a pull-based model — they poll the broker, so clusters behind firewalls work without inbound connections

Next Steps

• Multi-Cluster Targeting — direct deployments to specific clusters using labels
• CI/CD with Generators — automate deployment pushes from a CI pipeline
• Managing Stacks — deeper guide on stack lifecycle management
• Configuration Guide — tune polling intervals, database settings, and more

Tutorial: Multi-Cluster Targeting

In this tutorial, you’ll deploy different configurations to different clusters using Brokkr’s label and annotation targeting system. You’ll register two agents representing two environments (staging and production), then direct deployments to each one selectively.

What you’ll learn:

• How labels categorize agents and stacks
• How annotations attach key-value metadata
• How targeting rules match deployments to specific agents
• How one stack can reach multiple clusters while another targets only one

Prerequisites:

• A running Brokkr development environment (angreal local up)
• Your admin PAK
• Completed the Deploy Your First Application tutorial

Step 1: Create Two Agents

In a real deployment, each agent runs in a different Kubernetes cluster. For this tutorial, we’ll create two agent records in the broker to simulate a multi-cluster setup.

# Create a staging agent
brokkr-broker create agent --name staging-agent --cluster_name staging-cluster

Note the PAK printed for the staging agent. Then create the production agent:

# Create a production agent
brokkr-broker create agent --name prod-agent --cluster_name prod-cluster

If you’re using the API instead of the CLI:

STAGING=$(curl -s -X POST http://localhost:3000/api/v1/agents \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "staging-agent", "cluster_name": "staging-cluster"}' | jq -r '.agent.id')

PROD=$(curl -s -X POST http://localhost:3000/api/v1/agents \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "prod-agent", "cluster_name": "prod-cluster"}' | jq -r '.agent.id')

echo "Staging agent: $STAGING"
echo "Production agent: $PROD"

Step 2: Label the Agents

Labels are simple tags that categorize agents (e.g., env:staging, tier:web). Annotations are key-value metadata pairs (e.g., region=us-east-1). Both are used for organizing and filtering agents. Labels are typically used for broad categories, while annotations carry specific configuration values.

Add environment labels to each agent:

# Label the staging agent
curl -s -X POST "http://localhost:3000/api/v1/agents/${STAGING}/labels" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '"env:staging"' | jq .

# Label the production agent
curl -s -X POST "http://localhost:3000/api/v1/agents/${PROD}/labels" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '"env:production"' | jq .

Add a region annotation to the production agent:

curl -s -X POST "http://localhost:3000/api/v1/agents/${PROD}/annotations" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"key": "region", "value": "us-east-1"}' | jq .

Verify the labels are set:

curl -s "http://localhost:3000/api/v1/agents/${STAGING}/labels" \
  -H "Authorization: <your-admin-pak>" | jq '.[].label'

curl -s "http://localhost:3000/api/v1/agents/${PROD}/labels" \
  -H "Authorization: <your-admin-pak>" | jq '.[].label'

Step 3: Create Stacks with Matching Labels

Now create two stacks — one for staging, one for production — and label them to match the agents.

Important: Labels on agents and stacks are metadata for organization and filtering. The actual connection between an agent and a stack is created by agent targets (Step 4). Labels help you categorize resources; targets tell the agent “deploy this stack’s resources.”

# Create staging stack
STAGING_STACK=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "myapp-staging", "description": "My app - staging environment", "generator_id": "00000000-0000-0000-0000-000000000000"}' \
  | jq -r '.id')

# Add label to staging stack
curl -s -X POST "http://localhost:3000/api/v1/stacks/${STAGING_STACK}/labels" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '"env:staging"' | jq .

# Create production stack
PROD_STACK=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "myapp-production", "description": "My app - production environment", "generator_id": "00000000-0000-0000-0000-000000000000"}' \
  | jq -r '.id')

# Add label to production stack
curl -s -X POST "http://localhost:3000/api/v1/stacks/${PROD_STACK}/labels" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '"env:production"' | jq .

Step 4: Target Agents to Stacks

Connect each agent to its corresponding stack:

# Staging agent targets staging stack
curl -s -X POST "http://localhost:3000/api/v1/agents/${STAGING}/targets" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"stack_id\": \"${STAGING_STACK}\"}" | jq .

# Production agent targets production stack
curl -s -X POST "http://localhost:3000/api/v1/agents/${PROD}/targets" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"stack_id\": \"${PROD_STACK}\"}" | jq .

Step 5: Deploy to Staging Only

Push a deployment to the staging stack. Only the staging agent will receive it:

curl -s -X POST "http://localhost:3000/api/v1/stacks/${STAGING_STACK}/deployment-objects" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: myapp-staging\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: myapp\n  namespace: myapp-staging\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: myapp\n  template:\n    metadata:\n      labels:\n        app: myapp\n    spec:\n      containers:\n      - name: myapp\n        image: nginx:1.27\n        env:\n        - name: ENVIRONMENT\n          value: staging"
  }' | jq .

Check agent events — only the staging agent should have a deployment event:

# Staging agent should have an event
curl -s "http://localhost:3000/api/v1/agents/${STAGING}/events" \
  -H "Authorization: <your-admin-pak>" | jq 'length'

# Production agent should have no new events
curl -s "http://localhost:3000/api/v1/agents/${PROD}/events" \
  -H "Authorization: <your-admin-pak>" | jq 'length'

Step 6: Deploy to Production

Now push to production with a different replica count:

curl -s -X POST "http://localhost:3000/api/v1/stacks/${PROD_STACK}/deployment-objects" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: myapp-production\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: myapp\n  namespace: myapp-production\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: myapp\n  template:\n    metadata:\n      labels:\n        app: myapp\n    spec:\n      containers:\n      - name: myapp\n        image: nginx:1.27\n        env:\n        - name: ENVIRONMENT\n          value: production"
  }' | jq .

The production agent receives 3 replicas while staging has 1 — each environment gets exactly what it needs.

Step 7: Create a Shared Stack

What if you have infrastructure (like monitoring) that should go to both environments? Create a stack that both agents target:

# Create shared stack
SHARED_STACK=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "monitoring-shared", "description": "Monitoring stack for all clusters", "generator_id": "00000000-0000-0000-0000-000000000000"}' \
  | jq -r '.id')

# Both agents target the shared stack
curl -s -X POST "http://localhost:3000/api/v1/agents/${STAGING}/targets" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"stack_id\": \"${SHARED_STACK}\"}" | jq .

curl -s -X POST "http://localhost:3000/api/v1/agents/${PROD}/targets" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"stack_id\": \"${SHARED_STACK}\"}" | jq .

Now any deployment object pushed to monitoring-shared will be applied by both agents.

Step 8: Verify Targeting

Review each agent’s full target list:

echo "=== Staging Agent Targets ==="
curl -s "http://localhost:3000/api/v1/agents/${STAGING}/targets" \
  -H "Authorization: <your-admin-pak>" | jq '.[].stack_id'

echo "=== Production Agent Targets ==="
curl -s "http://localhost:3000/api/v1/agents/${PROD}/targets" \
  -H "Authorization: <your-admin-pak>" | jq '.[].stack_id'

Staging should show two stacks (its own + shared), and production should also show two (its own + shared).

Clean Up

Remove the test resources:

# Delete stacks (soft delete)
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${STAGING_STACK}" \
  -H "Authorization: <your-admin-pak>"
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${PROD_STACK}" \
  -H "Authorization: <your-admin-pak>"
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${SHARED_STACK}" \
  -H "Authorization: <your-admin-pak>"

# Delete agents (soft delete)
curl -s -X DELETE "http://localhost:3000/api/v1/agents/${STAGING}" \
  -H "Authorization: <your-admin-pak>"
curl -s -X DELETE "http://localhost:3000/api/v1/agents/${PROD}" \
  -H "Authorization: <your-admin-pak>"

What You’ve Learned

• Labels categorize agents and stacks with simple tags like env:staging
• Annotations attach key-value metadata like region=us-east-1
• Agent targets create explicit bindings between agents and stacks
• An agent only receives deployment objects from stacks it targets
• Multiple agents can target the same stack for shared infrastructure
• One agent can target multiple stacks for layered deployments

Next Steps

• CI/CD with Generators — automate deployments from pipelines
• Standardized Deployments with Templates — use templates to reduce YAML duplication across environments
• Core Concepts — deeper understanding of the targeting model

Tutorial: CI/CD with Generators

In this tutorial, you’ll set up a generator — Brokkr’s mechanism for CI/CD integration — and use it to push deployments from a simulated pipeline. Generators are non-admin identities with scoped permissions, designed for automation.

What you’ll learn:

• What generators are and why they exist
• How to create a generator and manage its PAK
• How generators create and manage stacks
• How to push deployment objects from a CI/CD pipeline
• Access control differences between admin and generator roles

Prerequisites:

• A running Brokkr development environment (angreal local up)
• Your admin PAK
• Completed the Deploy Your First Application tutorial

Step 1: Create a Generator

Generators represent automated systems (CI/CD pipelines, GitOps controllers, deployment scripts) that push resources to Brokkr. They have their own PAK and can only manage resources they own.

Create a generator using the CLI:

brokkr-broker create generator --name "github-actions" --description "GitHub Actions deployment pipeline"

This prints the generator’s ID and PAK:

Generator created successfully:
ID: f8e7d6c5-b4a3-...
Name: github-actions
Initial PAK: brokkr_BRx9y2Kq_A1B2C3D4E5F6G7H8I9J0K1L2

Save this PAK immediately — it’s only shown once. You’ll store it as a CI secret.

Alternatively, use the API:

GENERATOR_RESPONSE=$(curl -s -X POST http://localhost:3000/api/v1/generators \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "github-actions", "description": "GitHub Actions deployment pipeline"}')

GENERATOR_ID=$(echo "$GENERATOR_RESPONSE" | jq -r '.generator.id')
GENERATOR_PAK=$(echo "$GENERATOR_RESPONSE" | jq -r '.pak')

echo "Generator ID: $GENERATOR_ID"
echo "Generator PAK: $GENERATOR_PAK"

Step 2: Create a Stack as the Generator

Switch to using the generator’s PAK. The generator can create stacks, and those stacks become “owned” by the generator.

GENERATOR_PAK="brokkr_BRx9y2Kq_A1B2C3D4E5F6G7H8I9J0K1L2"  # your actual PAK

STACK_ID=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: ${GENERATOR_PAK}" \
  -H "Content-Type: application/json" \
  -d '{"name": "myapp-v2", "description": "My application deployed via CI/CD"}' \
  | jq -r '.id')

echo "Stack ID: $STACK_ID"

The stack’s generator_id field is now set to your generator’s ID. This ownership means:

• The generator can update and delete this stack
• The generator can push deployment objects to this stack
• Other generators cannot modify this stack
• Admins can still manage any stack

Step 3: See Generator Permissions in Action

Generators have scoped access — they can only manage resources they own. Let’s see this in practice:

# Generator can list its own stacks
curl -s http://localhost:3000/api/v1/stacks \
  -H "Authorization: ${GENERATOR_PAK}" | jq '.[].name'

# Generator CANNOT list agents (admin-only)
curl -s http://localhost:3000/api/v1/agents \
  -H "Authorization: ${GENERATOR_PAK}"
# Returns: 403 Forbidden

The key rule: generators can create, update, and delete their own stacks and push deployment objects to them, but they cannot manage agents, targets, or other generators’ resources. See the Security Model for the complete access control matrix.

Step 4: Target an Agent (So Deployments Reach a Cluster)

Before pushing deployment objects, an agent must be targeted to the stack. Otherwise the deployment exists in the broker but no agent will apply it. As an admin, target the default agent:

AGENT_ID=$(curl -s http://localhost:3000/api/v1/agents \
  -H "Authorization: <your-admin-pak>" | jq -r '.[0].id')

curl -s -X POST "http://localhost:3000/api/v1/agents/${AGENT_ID}/targets" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"stack_id\": \"${STACK_ID}\"}" | jq .

Note: Generators cannot manage agents or targets — that requires admin access. In production, an admin sets up the targeting once and the generator just pushes deployments.

Step 5: Push a Deployment (Simulating CI/CD)

Now simulate what a CI/CD pipeline would do — push a deployment object after building an image:

# This is what your CI/CD pipeline would run after building
curl -s -X POST "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects" \
  -H "Authorization: ${GENERATOR_PAK}" \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: myapp\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: myapp\n  namespace: myapp\n  labels:\n    app: myapp\nspec:\n  replicas: 2\n  selector:\n    matchLabels:\n      app: myapp\n  template:\n    metadata:\n      labels:\n        app: myapp\n    spec:\n      containers:\n      - name: myapp\n        image: myregistry.example.com/myapp:v1.2.3\n        ports:\n        - containerPort: 8080\n        env:\n        - name: VERSION\n          value: v1.2.3"
  }' | jq '{id, sequence_id, yaml_checksum}'

Each push creates a new deployment object with an incrementing sequence_id. The agent sees the new sequence and applies the latest version.

Step 6: Simulate a Deployment Update

Push a new version (as a CI/CD pipeline would on the next merge):

curl -s -X POST "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects" \
  -H "Authorization: ${GENERATOR_PAK}" \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: myapp\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: myapp\n  namespace: myapp\n  labels:\n    app: myapp\nspec:\n  replicas: 2\n  selector:\n    matchLabels:\n      app: myapp\n  template:\n    metadata:\n      labels:\n        app: myapp\n    spec:\n      containers:\n      - name: myapp\n        image: myregistry.example.com/myapp:v1.3.0\n        ports:\n        - containerPort: 8080\n        env:\n        - name: VERSION\n          value: v1.3.0"
  }' | jq '{id, sequence_id, yaml_checksum}'

Notice the sequence_id incremented. The agent will apply this new version.

Step 7: A Real GitHub Actions Workflow

Here’s how you’d integrate Brokkr into a real GitHub Actions pipeline:

# .github/workflows/deploy.yml
name: Deploy to Brokkr

on:
  push:
    branches: [main]

env:
  BROKKR_URL: https://brokkr.example.com
  STACK_ID: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build and push image
        run: |
          docker build -t myregistry.example.com/myapp:${{ github.sha }} .
          docker push myregistry.example.com/myapp:${{ github.sha }}

      - name: Generate deployment YAML
        run: |
          cat > deployment.yaml << 'YAML'
          apiVersion: v1
          kind: Namespace
          metadata:
            name: myapp
          ---
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: myapp
            namespace: myapp
          spec:
            replicas: 2
            selector:
              matchLabels:
                app: myapp
            template:
              metadata:
                labels:
                  app: myapp
              spec:
                containers:
                - name: myapp
                  image: myregistry.example.com/myapp:${{ github.sha }}
                  ports:
                  - containerPort: 8080
          YAML

      - name: Push to Brokkr
        run: |
          YAML_CONTENT=$(cat deployment.yaml | jq -Rs .)
          curl -sf -X POST "${BROKKR_URL}/api/v1/stacks/${STACK_ID}/deployment-objects" \
            -H "Authorization: ${{ secrets.BROKKR_GENERATOR_PAK }}" \
            -H "Content-Type: application/json" \
            -d "{\"yaml_content\": ${YAML_CONTENT}}"

Store the generator PAK as BROKKR_GENERATOR_PAK in your repository’s GitHub Actions secrets.

Step 8: Rotate the Generator PAK

PAKs should be rotated periodically. You can rotate via CLI or API:

# Via CLI (requires admin access to the broker host)
brokkr-broker rotate generator --uuid <generator-uuid>

# Via API
curl -s -X POST "http://localhost:3000/api/v1/generators/${GENERATOR_ID}/rotate-pak" \
  -H "Authorization: ${GENERATOR_PAK}" | jq .

The response contains the new PAK. Update your CI secrets immediately — the old PAK stops working.

Clean Up

# Delete the stack (as generator)
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${STACK_ID}" \
  -H "Authorization: ${GENERATOR_PAK}"

# Delete the generator (requires admin)
curl -s -X DELETE "http://localhost:3000/api/v1/generators/${GENERATOR_ID}" \
  -H "Authorization: <your-admin-pak>"

What You’ve Learned

• Generators are scoped identities for CI/CD pipeline integration
• Each generator gets its own PAK for authentication
• Generators own the stacks they create — other generators can’t modify them
• Pushing deployment objects is as simple as a curl POST with YAML content
• Sequence IDs ensure agents always apply the latest version
• Generator PAKs should be stored as CI secrets and rotated periodically

Next Steps

• Standardized Deployments with Templates — reduce YAML duplication with templates
• Working with Generators — detailed generator management guide
• Generators Reference — complete API reference
• Security Model — understand the full authorization model

Tutorial: Standardized Deployments with Templates

In this tutorial, you’ll create a reusable deployment template with parameterized values and JSON Schema validation, then instantiate it across multiple stacks. Templates eliminate YAML duplication and enforce consistency.

What you’ll learn:

• How to create a template with Tera syntax
• How to define a parameter schema using JSON Schema
• How to instantiate a template into a stack
• How template versioning works
• How template targeting restricts which stacks can use a template

Prerequisites:

• A running Brokkr development environment (angreal local up)
• Your admin PAK
• Completed the Deploy Your First Application tutorial

Step 1: Understand the Template Concept

A Brokkr template has two parts:

1. Template content — Kubernetes YAML with Tera placeholders (e.g., {{ replicas }}, {{ image_tag }})
2. Parameters schema — a JSON Schema that defines which parameters exist, their types, defaults, and constraints

When you instantiate a template into a stack, you provide parameter values. Brokkr validates them against the schema, renders the Tera template, and creates a deployment object with the resulting YAML.

Step 2: Create a Template

Create a template for a standard web service deployment:

curl -s -X POST http://localhost:3000/api/v1/templates \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "web-service",
    "description": "Standard web service with configurable replicas, image, and resource limits",
    "template_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: {{ namespace }}\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: {{ service_name }}\n  namespace: {{ namespace }}\n  labels:\n    app: {{ service_name }}\n    managed-by: brokkr\nspec:\n  replicas: {{ replicas }}\n  selector:\n    matchLabels:\n      app: {{ service_name }}\n  template:\n    metadata:\n      labels:\n        app: {{ service_name }}\n    spec:\n      containers:\n      - name: {{ service_name }}\n        image: {{ image_repository }}:{{ image_tag }}\n        ports:\n        - containerPort: {{ container_port }}\n        resources:\n          requests:\n            cpu: {{ cpu_request }}\n            memory: {{ memory_request }}\n          limits:\n            cpu: {{ cpu_limit }}\n            memory: {{ memory_limit }}\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: {{ service_name }}\n  namespace: {{ namespace }}\nspec:\n  selector:\n    app: {{ service_name }}\n  ports:\n  - port: 80\n    targetPort: {{ container_port }}",
    "parameters_schema": "{\"type\": \"object\", \"required\": [\"service_name\", \"namespace\", \"image_repository\", \"image_tag\"], \"properties\": {\"service_name\": {\"type\": \"string\", \"description\": \"Name of the service\", \"minLength\": 1, \"maxLength\": 63}, \"namespace\": {\"type\": \"string\", \"description\": \"Kubernetes namespace\", \"minLength\": 1}, \"image_repository\": {\"type\": \"string\", \"description\": \"Container image repository\"}, \"image_tag\": {\"type\": \"string\", \"description\": \"Container image tag\", \"default\": \"latest\"}, \"replicas\": {\"type\": \"integer\", \"description\": \"Number of replicas\", \"default\": 2, \"minimum\": 1, \"maximum\": 20}, \"container_port\": {\"type\": \"integer\", \"description\": \"Container port\", \"default\": 8080}, \"cpu_request\": {\"type\": \"string\", \"description\": \"CPU request\", \"default\": \"100m\"}, \"memory_request\": {\"type\": \"string\", \"description\": \"Memory request\", \"default\": \"128Mi\"}, \"cpu_limit\": {\"type\": \"string\", \"description\": \"CPU limit\", \"default\": \"500m\"}, \"memory_limit\": {\"type\": \"string\", \"description\": \"Memory limit\", \"default\": \"256Mi\"}}}"
  }' | jq '{id, name, version, checksum}'

The response shows the template was created with version: 1:

{
  "id": "t1234567-...",
  "name": "web-service",
  "version": 1,
  "checksum": "abc123..."
}

Save the template ID:

TEMPLATE_ID="t1234567-..."  # use the actual ID from the response

Step 3: Understand the Parameters Schema

The JSON Schema you provided defines 10 parameters: four required (service_name, namespace, image_repository, image_tag) and six optional with defaults (replicas defaults to 2, container_port to 8080, etc.). The schema enforces constraints — for example, replicas must be between 1 and 20, and service_name must be 1-63 characters.

The schema ensures that callers provide the required values and that constraints are enforced at instantiation time, before any YAML is rendered. See the Templates Reference for the full JSON Schema syntax guide.

Step 4: Create a Stack and Instantiate the Template

Create a stack, then instantiate the template into it:

# Create the stack
STACK_ID=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "frontend-app", "description": "Frontend web application", "generator_id": "00000000-0000-0000-0000-000000000000"}' \
  | jq -r '.id')

# Instantiate the template
curl -s -X POST "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects/from-template" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{
    \"template_id\": \"${TEMPLATE_ID}\",
    \"parameters\": {
      \"service_name\": \"frontend\",
      \"namespace\": \"frontend-app\",
      \"image_repository\": \"myregistry.example.com/frontend\",
      \"image_tag\": \"v2.1.0\",
      \"replicas\": 3,
      \"container_port\": 3000,
      \"memory_limit\": \"512Mi\"
    }
  }" | jq '.[0] | {id, sequence_id, yaml_checksum}'

Brokkr validated the parameters, rendered the Tera template, and created a deployment object. The resulting YAML has all placeholders replaced with actual values.

Step 5: Verify the Rendered Output

Fetch the deployment object to see the rendered YAML:

DO_ID=$(curl -s "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects" \
  -H "Authorization: <your-admin-pak>" | jq -r '.[0].id')

curl -s "http://localhost:3000/api/v1/deployment-objects/${DO_ID}" \
  -H "Authorization: <your-admin-pak>" | jq -r '.yaml_content'

You’ll see fully-rendered Kubernetes YAML with all template variables replaced:

apiVersion: v1
kind: Namespace
metadata:
  name: frontend-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: frontend-app
  labels:
    app: frontend
    managed-by: brokkr
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: myregistry.example.com/frontend:v2.1.0
        ports:
        - containerPort: 3000
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: frontend-app
spec:
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 3000

Step 6: Re-use the Template for Another Service

The same template works for a different service by changing the parameters:

# Create a backend stack
BACKEND_STACK=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "backend-api", "description": "Backend API service", "generator_id": "00000000-0000-0000-0000-000000000000"}' \
  | jq -r '.id')

# Instantiate with different parameters
curl -s -X POST "http://localhost:3000/api/v1/stacks/${BACKEND_STACK}/deployment-objects/from-template" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{
    \"template_id\": \"${TEMPLATE_ID}\",
    \"parameters\": {
      \"service_name\": \"api\",
      \"namespace\": \"backend-api\",
      \"image_repository\": \"myregistry.example.com/api\",
      \"image_tag\": \"v3.0.1\",
      \"replicas\": 5,
      \"container_port\": 8080,
      \"cpu_limit\": \"1000m\",
      \"memory_limit\": \"1Gi\"
    }
  }" | jq '.[0] | {id, sequence_id}'

One template, multiple services, each with appropriate configuration.

Step 7: Update the Template (Versioning)

Templates are versioned. Updating a template creates a new version while preserving the old one:

curl -s -X PUT "http://localhost:3000/api/v1/templates/${TEMPLATE_ID}" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Standard web service v2 - adds liveness probe",
    "template_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: {{ namespace }}\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: {{ service_name }}\n  namespace: {{ namespace }}\n  labels:\n    app: {{ service_name }}\n    managed-by: brokkr\nspec:\n  replicas: {{ replicas }}\n  selector:\n    matchLabels:\n      app: {{ service_name }}\n  template:\n    metadata:\n      labels:\n        app: {{ service_name }}\n    spec:\n      containers:\n      - name: {{ service_name }}\n        image: {{ image_repository }}:{{ image_tag }}\n        ports:\n        - containerPort: {{ container_port }}\n        livenessProbe:\n          httpGet:\n            path: /healthz\n            port: {{ container_port }}\n          initialDelaySeconds: 10\n          periodSeconds: 30\n        resources:\n          requests:\n            cpu: {{ cpu_request }}\n            memory: {{ memory_request }}\n          limits:\n            cpu: {{ cpu_limit }}\n            memory: {{ memory_limit }}\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: {{ service_name }}\n  namespace: {{ namespace }}\nspec:\n  selector:\n    app: {{ service_name }}\n  ports:\n  - port: 80\n    targetPort: {{ container_port }}",
    "parameters_schema": "{\"type\": \"object\", \"required\": [\"service_name\", \"namespace\", \"image_repository\", \"image_tag\"], \"properties\": {\"service_name\": {\"type\": \"string\", \"minLength\": 1, \"maxLength\": 63}, \"namespace\": {\"type\": \"string\", \"minLength\": 1}, \"image_repository\": {\"type\": \"string\"}, \"image_tag\": {\"type\": \"string\", \"default\": \"latest\"}, \"replicas\": {\"type\": \"integer\", \"default\": 2, \"minimum\": 1, \"maximum\": 20}, \"container_port\": {\"type\": \"integer\", \"default\": 8080}, \"cpu_request\": {\"type\": \"string\", \"default\": \"100m\"}, \"memory_request\": {\"type\": \"string\", \"default\": \"128Mi\"}, \"cpu_limit\": {\"type\": \"string\", \"default\": \"500m\"}, \"memory_limit\": {\"type\": \"string\", \"default\": \"256Mi\"}}}"
  }' | jq '{id, name, version}'

The response shows version: 2. Existing deployment objects rendered from version 1 are unaffected. New instantiations will use the latest version.

Step 8: Schema Validation in Action

Try instantiating with invalid parameters to see validation:

# Missing required field (service_name)
curl -s -X POST "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects/from-template" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{
    \"template_id\": \"${TEMPLATE_ID}\",
    \"parameters\": {
      \"namespace\": \"test\",
      \"image_repository\": \"nginx\",
      \"image_tag\": \"latest\"
    }
  }" | jq .

# Replicas out of range (max is 20)
curl -s -X POST "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects/from-template" \
  -H "Authorization: <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{
    \"template_id\": \"${TEMPLATE_ID}\",
    \"parameters\": {
      \"service_name\": \"test\",
      \"namespace\": \"test\",
      \"image_repository\": \"nginx\",
      \"image_tag\": \"latest\",
      \"replicas\": 100
    }
  }" | jq .

Both requests return validation errors, preventing invalid YAML from reaching your clusters.

Clean Up

curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${STACK_ID}" \
  -H "Authorization: <your-admin-pak>"
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${BACKEND_STACK}" \
  -H "Authorization: <your-admin-pak>"
curl -s -X DELETE "http://localhost:3000/api/v1/templates/${TEMPLATE_ID}" \
  -H "Authorization: <your-admin-pak>"

What You’ve Learned

• Templates combine Tera-syntax YAML with JSON Schema parameter validation
• Instantiation validates parameters, renders the template, and creates a deployment object
• Versioning preserves old template versions while allowing updates
• JSON Schema enforces types, required fields, ranges, and string constraints
• Templates reduce duplication — one template serves many stacks with different parameters

For the complete Tera template syntax (conditionals, loops, filters) and JSON Schema reference, see the Templates Reference.

Next Steps

• Using Stack Templates — detailed how-to guide for template workflows
• Templates Reference — complete API reference for templates
• Core Concepts — how templates fit into the Brokkr architecture

How-To Guides

Practical guides for accomplishing specific tasks with Brokkr. These are goal-oriented — each guide walks you through a particular operation or workflow.

Container Builds with Shipwright

Brokkr integrates with Shipwright Build to provide native container image building capabilities. This guide covers installation, configuration, and usage of Shipwright builds through Brokkr’s work order system.

Overview

Shipwright Build is a CNCF Sandbox project that provides a framework for building container images on Kubernetes. Brokkr uses Shipwright as the execution engine for build work orders, allowing you to:

• Build container images from Git repositories
• Push images to container registries
• Leverage production-ready build strategies (buildah, kaniko)
• Manage builds through Brokkr’s work order API

Prerequisites

Kubernetes Version

Shipwright integration requires Kubernetes 1.29 or later due to dependencies on newer API features.

# Verify your Kubernetes version
kubectl version --short

Cluster Requirements

• Sufficient resources for build pods (recommended: 4GB memory, 2 CPU cores available)
• Network access to container registries you’ll push to
• Network access to Git repositories you’ll build from

Installation Options

Option 1: Bundled Installation (Recommended)

The brokkr-agent Helm chart includes Shipwright Build and Tekton Pipelines as vendored dependencies. This is enabled by default.

# Install agent with bundled Shipwright (default)
helm install brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --set broker.url=http://brokkr-broker:3000 \
  --set broker.pak="<YOUR_PAK>" \
  --wait

This installs:

• Tekton Pipelines (v0.37.2) - Task execution engine
• Shipwright Build (v0.10.0) - Build orchestration
• buildah ClusterBuildStrategy - Default build strategy

Option 2: Bring Your Own Shipwright

If you already have Shipwright and Tekton installed, or need specific versions:

# Disable bundled Shipwright
helm install brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --set broker.url=http://brokkr-broker:3000 \
  --set broker.pak="<YOUR_PAK>" \
  --set shipwright.enabled=false \
  --wait

Manual Shipwright Installation

If installing Shipwright manually, ensure you have compatible versions:

# Install Tekton Pipelines (v0.59.0 or later recommended)
kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.59.0/release.yaml

# Wait for Tekton to be ready
kubectl wait --for=condition=Ready pods -l app=tekton-pipelines-controller -n tekton-pipelines --timeout=300s

# Install Shipwright Build (v0.13.0 or later recommended)
kubectl apply -f https://github.com/shipwright-io/build/releases/download/v0.13.0/release.yaml

# Wait for Shipwright to be ready
kubectl wait --for=condition=Ready pods -l app=shipwright-build-controller -n shipwright-build --timeout=300s

# Install sample build strategies
kubectl apply -f https://github.com/shipwright-io/build/releases/download/v0.13.0/sample-strategies.yaml

Verifying Installation

Check Components

# Verify Tekton Pipelines
kubectl get pods -n tekton-pipelines
# Expected: tekton-pipelines-controller and tekton-pipelines-webhook Running

# Verify Shipwright Build
kubectl get pods -n shipwright-build
# Expected: shipwright-build-controller Running

# Verify ClusterBuildStrategies
kubectl get clusterbuildstrategies
# Expected: buildah (and others if sample strategies installed)

Test Build Capability

Create a simple test build to verify the installation:

# test-build.yaml
apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: test-build
spec:
  source:
    type: Git
    git:
      url: https://github.com/shipwright-io/sample-go
  strategy:
    name: buildah
    kind: ClusterBuildStrategy
  output:
    image: ttl.sh/brokkr-test-$(date +%s):1h
---
apiVersion: shipwright.io/v1beta1
kind: BuildRun
metadata:
  generateName: test-build-run-
spec:
  build:
    name: test-build

# Apply the test build
kubectl apply -f test-build.yaml

# Watch the build progress
kubectl get buildruns -w

# Check build logs
kubectl logs -l buildrun.shipwright.io/name=test-build-run-xxxxx -c step-build

Configuration

Build Strategies

The bundled installation includes a buildah ClusterBuildStrategy. You can install additional strategies:

# Install all sample strategies (buildah, kaniko, buildpacks, etc.)
helm upgrade brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --reuse-values \
  --set shipwright.installSampleStrategies=true

Disable Sample Strategies

If you only want Shipwright without sample strategies:

helm install brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --set broker.url=http://brokkr-broker:3000 \
  --set broker.pak="<YOUR_PAK>" \
  --set shipwright.enabled=true \
  --set shipwright.installSampleStrategies=false

RBAC Configuration

The brokkr-agent automatically includes RBAC rules for Shipwright when enabled:

# Automatically included when shipwright.enabled=true
- apiGroups: ["shipwright.io"]
  resources: ["builds", "buildruns"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["shipwright.io"]
  resources: ["buildstrategies", "clusterbuildstrategies"]
  verbs: ["get", "list", "watch"]

Registry Authentication

To push images to private registries, create a Kubernetes secret:

# Create registry credentials secret
kubectl create secret docker-registry registry-creds \
  --docker-server=ghcr.io \
  --docker-username=<username> \
  --docker-password=<token> \
  --docker-email=<email>

Reference this secret in your Build spec:

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: my-build
spec:
  source:
    type: Git
    git:
      url: https://github.com/org/repo
  strategy:
    name: buildah
    kind: ClusterBuildStrategy
  output:
    image: ghcr.io/org/my-image:latest
    pushSecret: registry-creds  # Reference your secret here

Git Authentication

For private Git repositories:

# Create Git credentials secret (HTTPS)
kubectl create secret generic git-creds \
  --from-literal=username=<username> \
  --from-literal=password=<token>

# Or for SSH
kubectl create secret generic git-ssh-creds \
  --from-file=ssh-privatekey=/path/to/id_rsa

Reference in your Build:

spec:
  source:
    type: Git
    git:
      url: https://github.com/org/private-repo
      cloneSecret: git-creds

Troubleshooting

Build Pods Not Starting

# Check for pending pods
kubectl get pods -l build.shipwright.io/name

# Check events
kubectl get events --sort-by='.lastTimestamp'

# Verify ServiceAccount has required permissions
kubectl auth can-i create pods --as=system:serviceaccount:default:default

Shipwright Controller Not Ready

# Check controller logs
kubectl logs -n shipwright-build -l app=shipwright-build-controller

# Check for CRD installation
kubectl get crd builds.shipwright.io buildruns.shipwright.io

Tekton Pipeline Failures

# Check Tekton controller logs
kubectl logs -n tekton-pipelines -l app=tekton-pipelines-controller

# Check TaskRun status
kubectl get taskruns
kubectl describe taskrun <taskrun-name>

Next Steps

• Learn about Work Orders for managing builds through Brokkr
• Configure Monitoring for build metrics

Configuring Webhooks

Brokkr’s webhook system enables external systems to receive real-time notifications when events occur. This guide covers creating webhook subscriptions, configuring delivery options, and integrating with external services.

Overview

Webhooks provide HTTP callbacks for events such as:

• Deployment applied or failed
• Work order completed or failed
• Agent registered or deregistered
• Stack created or deleted

Brokkr supports two delivery modes:

• Broker delivery (default): The broker sends webhooks directly
• Agent delivery: An agent in the target cluster delivers webhooks, enabling access to in-cluster services

Prerequisites

• Admin PAK for creating webhook subscriptions
• Target endpoint accessible from the broker or agent (depending on delivery mode)
• HTTPS recommended for production endpoints

Creating a Webhook Subscription

Basic Webhook (Broker Delivery)

Create a webhook subscription using the API:

curl -X POST "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Deployment Notifications",
    "url": "https://my-service.example.com/webhooks/brokkr",
    "event_types": ["deployment.applied", "deployment.failed"],
    "auth_header": "Bearer my-webhook-secret"
  }'

Response:

{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "name": "Deployment Notifications",
  "has_url": true,
  "has_auth_header": true,
  "event_types": ["deployment.applied", "deployment.failed"],
  "enabled": true,
  "max_retries": 5,
  "timeout_seconds": 30,
  "created_at": "2025-01-02T10:00:00Z"
}

Webhook with Agent Delivery

For in-cluster targets that the broker cannot reach, configure agent delivery using target_labels:

curl -X POST "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "In-Cluster Alerts",
    "url": "http://alertmanager.monitoring.svc.cluster.local:9093/api/v2/alerts",
    "event_types": ["deployment.failed", "workorder.failed"],
    "target_labels": ["env:production"]
  }'

When target_labels is set:

1. Deliveries are queued for agents matching ALL specified labels
2. The matching agent fetches pending deliveries during its polling loop
3. The agent delivers the webhook from inside the cluster
4. The agent reports success/failure back to the broker

Wildcard Event Types

Subscribe to multiple events using wildcards:

curl -X POST "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "All Deployment Events",
    "url": "https://webhook.example.com/deployments",
    "event_types": ["deployment.*"]
  }'

Supported wildcards:

• deployment.* - All deployment events
• workorder.* - All work order events
• agent.* - All agent events
• stack.* - All stack events
• * - All events

Configuring Delivery Options

Retry Settings

Configure retry behavior for failed deliveries:

curl -X POST "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Critical Alerts",
    "url": "https://pagerduty.example.com/webhook",
    "event_types": ["deployment.failed"],
    "max_retries": 10,
    "timeout_seconds": 60
  }'

Retry behavior:

• Failed deliveries use exponential backoff: 2, 4, 8, 16… seconds
• After max_retries failures, deliveries are marked as “dead”
• Delivery timeouts count as failures

Filters

Filter events by specific agents or stacks:

curl -X POST "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production Stack Alerts",
    "url": "https://slack.example.com/webhook",
    "event_types": ["deployment.*"],
    "filters": {
      "labels": {"env": "production"}
    }
  }'

Managing Webhooks

List All Webhooks

curl "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK"

Get Webhook Details

curl "http://broker:3000/api/v1/webhooks/{webhook_id}" \
  -H "Authorization: Bearer $ADMIN_PAK"

Update a Webhook

curl -X PUT "http://broker:3000/api/v1/webhooks/{webhook_id}" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": false
  }'

Delete a Webhook

curl -X DELETE "http://broker:3000/api/v1/webhooks/{webhook_id}" \
  -H "Authorization: Bearer $ADMIN_PAK"

Test a Webhook

Send a test event to verify connectivity:

curl -X POST "http://broker:3000/api/v1/webhooks/{webhook_id}/test" \
  -H "Authorization: Bearer $ADMIN_PAK"

Viewing Delivery Status

List Deliveries for a Subscription

curl "http://broker:3000/api/v1/webhooks/{webhook_id}/deliveries" \
  -H "Authorization: Bearer $ADMIN_PAK"

Filter by Status

# Show only failed deliveries
curl "http://broker:3000/api/v1/webhooks/{webhook_id}/deliveries?status=failed" \
  -H "Authorization: Bearer $ADMIN_PAK"

# Show only dead (max retries exceeded)
curl "http://broker:3000/api/v1/webhooks/{webhook_id}/deliveries?status=dead" \
  -H "Authorization: Bearer $ADMIN_PAK"

Delivery statuses:

• pending - Waiting to be delivered
• acquired - Claimed by broker or agent, delivery in progress
• success - Successfully delivered
• failed - Delivery failed, will retry
• dead - Max retries exceeded

Webhook Payload Format

All webhook deliveries include these headers:

Content-Type: application/json
X-Brokkr-Event-Type: deployment.applied
X-Brokkr-Delivery-Id: a1b2c3d4-e5f6-7890-abcd-ef1234567890
Authorization: <your-configured-auth-header>

Payload structure:

{
  "id": "event-uuid",
  "event_type": "deployment.applied",
  "timestamp": "2025-01-02T10:00:00Z",
  "data": {
    "deployment_object_id": "...",
    "agent_id": "...",
    "status": "SUCCESS"
  }
}

Common Patterns

Slack Integration

curl -X POST "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Slack Deployment Alerts",
    "url": "https://hooks.slack.com/services/T00/B00/XXX",
    "event_types": ["deployment.applied", "deployment.failed"]
  }'

PagerDuty Integration

curl -X POST "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "PagerDuty Critical Alerts",
    "url": "https://events.pagerduty.com/v2/enqueue",
    "event_types": ["deployment.failed", "workorder.failed"],
    "auth_header": "Token token=your-pagerduty-token",
    "max_retries": 10
  }'

In-Cluster Alertmanager

curl -X POST "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Alertmanager Notifications",
    "url": "http://alertmanager.monitoring.svc.cluster.local:9093/api/v2/alerts",
    "event_types": ["deployment.failed"],
    "target_labels": ["role:monitoring"]
  }'

Troubleshooting

Webhooks Not Being Delivered

1. Check if the subscription is enabled:

   curl "http://broker:3000/api/v1/webhooks/{id}" \
     -H "Authorization: Bearer $ADMIN_PAK"
   

2. Check delivery status for failures:

   curl "http://broker:3000/api/v1/webhooks/{id}/deliveries?status=failed" \
     -H "Authorization: Bearer $ADMIN_PAK"
   

3. Verify endpoint is reachable from broker/agent

Agent-Delivered Webhooks Failing

1. Verify agent has matching labels:

   curl "http://broker:3000/api/v1/agents/{agent_id}" \
     -H "Authorization: Bearer $ADMIN_PAK"
   

2. Check agent logs for delivery errors:

   kubectl logs -l app=brokkr-agent -c agent
   

3. Ensure the agent is ACTIVE and polling

Deliveries Stuck in “Acquired” State

Deliveries have a 60-second TTL. If they remain acquired longer, they’ll be released back to pending. This can happen if:

• The delivering agent/broker crashed mid-delivery
• Network issues prevented result reporting

The system automatically recovers these deliveries.

Related Documentation

• Webhooks Reference - Complete API reference
• Event Types - List of all event types
• Architecture - How webhooks fit into Brokkr

Managing Stacks

Stacks are the fundamental organizational unit in Brokkr, representing collections of related Kubernetes resources that belong together. This guide covers creating stacks, configuring them with labels and annotations for targeting, managing their lifecycle, and understanding how they connect to agents and deployment objects.

Understanding Stacks

A stack serves as a container for deployment objects—the versioned snapshots of Kubernetes resources you want to deploy. When you create a stack, you establish a logical boundary for a set of resources, whether that’s an application, a service, or any collection of related Kubernetes manifests. Agents are then assigned to stacks through targeting relationships, enabling you to control which clusters receive which resources.

Each stack maintains a history of deployment objects, providing an immutable audit trail of every configuration change. This history enables rollback capabilities and satisfies compliance requirements for tracking what was deployed and when.

Creating Stacks

Basic Stack Creation

To create a stack, send a POST request to the stacks endpoint with a name and optional description:

curl -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "payment-service",
    "description": "Payment processing microservice and dependencies"
  }'

The response includes the stack’s UUID which you’ll use for all subsequent operations:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "payment-service",
  "description": "Payment processing microservice and dependencies",
  "created_at": "2025-01-02T10:00:00Z",
  "updated_at": "2025-01-02T10:00:00Z",
  "generator_id": "00000000-0000-0000-0000-000000000000"
}

Stack Naming Conventions

Stack names must be non-empty strings up to 255 characters. While Brokkr doesn’t enforce naming conventions, consistent patterns make your infrastructure easier to navigate. Consider including:

• The application or service name
• The environment (if not using labels)
• A version or variant indicator for parallel deployments

Examples: frontend-app, database-cluster, monitoring-stack, api-gateway-v2

Configuring Labels and Annotations

Labels and annotations provide metadata for stacks that enables dynamic targeting and integration with external systems. Labels are simple string values used for categorization and selection. Annotations are key-value pairs for richer metadata.

Adding Labels

Labels enable pattern-based targeting where agents with matching labels automatically receive stacks. Add a label with a POST request:

curl -X POST http://localhost:3000/api/v1/stacks/$STACK_ID/labels \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '"production"'

Labels must be non-empty strings up to 64 characters with no whitespace. Common labeling patterns include:

Listing Labels

View all labels for a stack:

curl http://localhost:3000/api/v1/stacks/$STACK_ID/labels \
  -H "Authorization: Bearer $ADMIN_PAK"

Removing Labels

Remove a label by its value:

curl -X DELETE http://localhost:3000/api/v1/stacks/$STACK_ID/labels/production \
  -H "Authorization: Bearer $ADMIN_PAK"

Adding Annotations

Annotations carry key-value metadata for integration with external systems or configuration hints:

curl -X POST http://localhost:3000/api/v1/stacks/$STACK_ID/annotations \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "stack_id": "550e8400-e29b-41d4-a716-446655440000",
    "key": "cost-center",
    "value": "engineering-team-a"
  }'

Both keys and values must be non-empty strings up to 64 characters with no whitespace. Annotations are useful for:

Listing Annotations

View all annotations for a stack:

curl http://localhost:3000/api/v1/stacks/$STACK_ID/annotations \
  -H "Authorization: Bearer $ADMIN_PAK"

Removing Annotations

Remove an annotation by its key:

curl -X DELETE http://localhost:3000/api/v1/stacks/$STACK_ID/annotations/cost-center \
  -H "Authorization: Bearer $ADMIN_PAK"

Targeting Stacks to Agents

Targeting establishes the relationship between stacks and the agents that should manage them. Without targeting, an agent won’t receive deployment objects from a stack regardless of other configuration.

Direct Assignment

Create a targeting relationship by associating an agent with a specific stack:

curl -X POST http://localhost:3000/api/v1/agents/$AGENT_ID/targets \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d "{
    \"agent_id\": \"$AGENT_ID\",
    \"stack_id\": \"$STACK_ID\"
  }"

Direct assignment is appropriate when you have explicit control over which agents manage which stacks and the relationship is stable.

Label-Based Targeting

When both agents and stacks carry matching labels, you can configure agents to automatically target all stacks with those labels. This enables patterns like “all production agents receive all production stacks” without maintaining explicit per-pair associations.

To set up label-based targeting:

1. Add labels to your stacks that represent their characteristics
2. Add corresponding labels to agents that should manage those stacks
3. Configure the targeting policy (see agent configuration)

The agent polls for stacks matching its label configuration and creates targeting relationships automatically.

Multi-Cluster Deployments

A single stack can be targeted by multiple agents, enabling multi-cluster deployment scenarios. Each agent independently polls for the stack’s deployment objects and applies them to its cluster. This is useful for:

• High availability across regions
• Disaster recovery setups
• Consistent infrastructure across environments

Working with Deployment Objects

Once a stack exists and is targeted to agents, you populate it with deployment objects containing Kubernetes resources.

Creating Deployment Objects

Submit Kubernetes YAML as a deployment object:

curl -X POST "http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg yaml "$(cat resources.yaml)" '{yaml_content: $yaml, is_deletion_marker: false}')"

Each deployment object receives a sequence ID that guarantees ordering. Agents process deployment objects in sequence order, ensuring resources are applied in the intended order.

Listing Deployment Objects

View all deployment objects in a stack:

curl "http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects" \
  -H "Authorization: Bearer $ADMIN_PAK"

Using Templates

For standardized deployments, instantiate a template into a deployment object:

curl -X POST "http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects/from-template" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "template_id": "template-uuid-here",
    "parameters": {
      "replicas": 3,
      "image_tag": "v1.2.3"
    }
  }'

The template’s parameters are validated against its JSON Schema before rendering. If the template has labels, they must match the stack’s labels for instantiation to succeed.

Stack Lifecycle

Updating Stacks

Modify a stack’s name or description:

curl -X PUT http://localhost:3000/api/v1/stacks/$STACK_ID \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "name": "payment-service-v2",
    "description": "Updated payment service with new features"
  }'

Deleting Stacks

Brokkr uses soft deletion for stacks. When you delete a stack, it’s marked with a deleted_at timestamp rather than being removed from the database:

curl -X DELETE http://localhost:3000/api/v1/stacks/$STACK_ID \
  -H "Authorization: Bearer $ADMIN_PAK"

Soft deletion marks the stack with a deleted_at timestamp:

• The stack stops appearing in list queries
• The underlying data remains intact for audit purposes and potential recovery

Note that soft-deleting a stack does not automatically cascade to its deployment objects or create deletion markers. To ensure agents remove resources from clusters, create a deletion marker deployment object before deleting the stack.

Understanding Deletion Markers

To clean up cluster resources, create a deployment object with is_deletion_marker: true for the stack before deleting it. Agents receiving this marker understand they should delete the stack’s resources rather than apply them. This ensures resources are cleaned up from clusters.

Generator Integration

Generators are external systems like CI/CD pipelines that create stacks programmatically. When a generator creates a stack, that stack’s generator_id links it to the creating generator, establishing ownership and access control.

Generators can only access stacks they created. This scoping ensures pipelines can’t accidentally modify resources belonging to other systems. See the Generators Guide for details on integrating CI/CD systems.

Best Practices

Organize by responsibility: Group resources into stacks based on what changes together. A stack should contain resources that are deployed, updated, and scaled as a unit.

Use labels for targeting: Rather than creating explicit targeting relationships for each stack-agent pair, use labels to establish patterns. This reduces configuration overhead as your infrastructure grows.

Keep stacks focused: While you can put any resources in a stack, keeping stacks focused on specific applications or services makes management clearer and reduces blast radius for changes.

Document with annotations: Use annotations to record metadata that helps teams understand the stack’s purpose, ownership, and relationship to business systems.

Plan for deletion: Remember that deleting a stack triggers resource deletion on targeted clusters. Ensure you understand which clusters are affected before deleting.

Related Documentation

• Quick Start Guide - First deployment walkthrough
• Core Concepts - Understanding Brokkr’s architecture
• Generators - CI/CD integration
• Templates - Standardized deployments

Working with Generators

Generators are identity principals that enable external systems to interact with Brokkr. They provide a way for CI/CD pipelines, automation tools, and other services to authenticate and manage resources within defined boundaries. This guide covers creating generators, integrating them with CI/CD systems, and managing their lifecycle.

Understanding Generators

A generator represents an external system that creates and manages Brokkr resources. Each generator receives a Pre-Authentication Key (PAK) that grants it permission to create stacks, templates, and deployment objects. Resources created by a generator are scoped to that generator, providing natural isolation between different automation pipelines or teams.

Generators differ from the admin PAK in important ways. The admin PAK has full access to all resources and administrative functions. Generator PAKs can only access resources they created and cannot perform administrative operations like creating other generators or managing agents.

Prerequisites

• Admin PAK for creating and managing generators
• Access to the Brokkr broker API
• CI/CD system or automation tool to configure

Creating a Generator

Step 1: Create the Generator

Create a new generator using the admin PAK:

curl -X POST "http://broker:3000/api/v1/generators" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "github-actions-prod",
    "description": "Production deployment pipeline"
  }'

The response includes the generator details and its PAK:

{
  "generator": {
    "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "name": "github-actions-prod",
    "description": "Production deployment pipeline",
    "created_at": "2025-01-02T10:00:00Z",
    "updated_at": "2025-01-02T10:00:00Z"
  },
  "pak": "brk_gen_abc123...xyz789"
}

Step 2: Store the PAK Securely

The PAK is only returned once at creation time. Store it immediately in your secret management system:

• GitHub Actions: Add as a repository or organization secret
• GitLab CI: Add as a protected variable
• Jenkins: Store in credentials manager
• Vault/AWS Secrets Manager: Store with appropriate access policies

If you lose the PAK, you’ll need to rotate it (see PAK Rotation below).

CI/CD Integration

GitHub Actions Example

Configure your workflow to deploy through Brokkr:

name: Deploy to Production
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Create Stack
        env:
          BROKKR_PAK: ${{ secrets.BROKKR_GENERATOR_PAK }}
          BROKKR_URL: ${{ vars.BROKKR_URL }}
        run: |
          curl -X POST "$BROKKR_URL/api/v1/stacks" \
            -H "Authorization: Bearer $BROKKR_PAK" \
            -H "Content-Type: application/json" \
            -d '{
              "name": "my-app-${{ github.sha }}",
              "description": "Deployed from commit ${{ github.sha }}"
            }'

      - name: Add Deployment Objects
        env:
          BROKKR_PAK: ${{ secrets.BROKKR_GENERATOR_PAK }}
          BROKKR_URL: ${{ vars.BROKKR_URL }}
        run: |
          STACK_ID=$(cat stack-response.json | jq -r '.id')
          curl -X POST "$BROKKR_URL/api/v1/stacks/$STACK_ID/deployment-objects" \
            -H "Authorization: Bearer $BROKKR_PAK" \
            -H "Content-Type: application/json" \
            -d @deployment.json

GitLab CI Example

deploy:
  stage: deploy
  script:
    - |
      curl -X POST "$BROKKR_URL/api/v1/stacks" \
        -H "Authorization: Bearer $BROKKR_GENERATOR_PAK" \
        -H "Content-Type: application/json" \
        -d "{
          \"name\": \"my-app-$CI_COMMIT_SHA\",
          \"description\": \"Pipeline $CI_PIPELINE_ID\"
        }"
  only:
    - main

Using Templates

Generators can create and use stack templates for consistent deployments:

# Create a template (using generator PAK)
curl -X POST "http://broker:3000/api/v1/templates" \
  -H "Authorization: Bearer $GENERATOR_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "web-service",
    "description": "Standard web service deployment",
    "template_yaml": "...",
    "schema_json": "..."
  }'

# Create stack from template
curl -X POST "http://broker:3000/api/v1/stacks" \
  -H "Authorization: Bearer $GENERATOR_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-web-service",
    "template_name": "web-service",
    "parameters": {
      "replicas": 3,
      "image": "myapp:v1.2.3"
    }
  }'

Managing Generators

List Generators

View all generators (admin only):

curl "http://broker:3000/api/v1/generators" \
  -H "Authorization: Bearer $ADMIN_PAK"

Get Generator Details

A generator can view its own details:

curl "http://broker:3000/api/v1/generators/$GENERATOR_ID" \
  -H "Authorization: Bearer $GENERATOR_PAK"

Update Generator

Update the generator’s metadata:

curl -X PUT "http://broker:3000/api/v1/generators/$GENERATOR_ID" \
  -H "Authorization: Bearer $GENERATOR_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Updated description"
  }'

Delete Generator

Soft-delete a generator (admin or generator itself):

curl -X DELETE "http://broker:3000/api/v1/generators/$GENERATOR_ID" \
  -H "Authorization: Bearer $ADMIN_PAK"

Deleting a generator cascades the soft-delete to all stacks owned by the generator and their deployment objects. This is handled by a database trigger, so the cascade is atomic.

PAK Rotation

Rotate the generator’s PAK for security best practices or if the current PAK is compromised:

curl -X POST "http://broker:3000/api/v1/generators/$GENERATOR_ID/rotate-pak" \
  -H "Authorization: Bearer $GENERATOR_PAK"

The response contains the new PAK:

{
  "generator": {
    "id": "a1b2c3d4-...",
    "name": "github-actions-prod",
    ...
  },
  "pak": "brk_gen_new123...newxyz"
}

After rotation:

1. The old PAK is immediately invalidated
2. Update all CI/CD systems with the new PAK
3. Verify deployments work with the new credentials

Consider rotating PAKs:

• On a regular schedule (quarterly, annually)
• When team members with access leave
• If the PAK may have been exposed
• After security incidents

Access Control

Generators operate under a scoped permission model:

Best Practices

One Generator Per Pipeline

Create separate generators for each deployment pipeline or team. This provides:

• Clear ownership of resources
• Independent PAK rotation
• Easier auditing and troubleshooting
• Isolation between environments

Naming Conventions

Use descriptive names that identify the purpose and scope:

• github-actions-prod - Production pipeline in GitHub Actions
• gitlab-ci-staging - Staging pipeline in GitLab CI
• jenkins-nightly-builds - Nightly build automation
• team-platform-prod - Platform team’s production deployments

Secret Management

Never store PAKs in:

• Source code repositories
• Unencrypted configuration files
• Logs or console output

Always use:

• CI/CD secret management (GitHub Secrets, GitLab Variables)
• Secret management systems (Vault, AWS Secrets Manager)
• Encrypted environment variables

Troubleshooting

Authentication Failures

If API calls fail with 401 or 403:

1. Verify the PAK is correct and not expired
2. Check if the PAK was rotated
3. Ensure you’re using the generator PAK, not the admin PAK (for generator-scoped operations)

Cannot See Resources

If a generator cannot see expected stacks or templates:

1. Verify the resources were created with this generator’s PAK
2. Resources created by other generators are not visible
3. Use admin PAK to view all resources across generators

PAK Lost

If you’ve lost a generator’s PAK:

1. Use the admin PAK to rotate: POST /api/v1/generators/{id}/rotate-pak
2. Store the new PAK securely
3. Update all systems using the old PAK

Related Documentation

• Generators API Reference - Complete API documentation
• Stack Templates - Using templates with generators
• Authentication - Understanding Brokkr authentication

Monitoring Deployment Health

Brokkr agents continuously monitor the health of deployed Kubernetes resources and report status back to the broker. This provides centralized visibility into deployment health across all clusters without requiring direct cluster access. This guide covers configuring health monitoring, interpreting health status, and troubleshooting common issues.

How Health Monitoring Works

When an agent applies deployment objects to a Kubernetes cluster, it tracks those resources and periodically checks their health. The agent examines pod status, container states, and Kubernetes conditions to determine overall health. This information is reported to the broker, where it can be viewed through the API or UI.

Health monitoring runs as a background process on each agent. On each check interval, the agent queries the Kubernetes API for pods associated with each deployment object, analyzes their status, and sends a consolidated health report to the broker.

Health Status Values

The health monitoring system reports one of four status values:

Detected Conditions

The agent detects and reports these problematic conditions:

Container Issues:

• ImagePullBackOff - Unable to pull container image
• ErrImagePull - Error pulling container image
• CrashLoopBackOff - Container repeatedly crashing
• CreateContainerConfigError - Invalid container configuration
• InvalidImageName - Malformed image reference
• RunContainerError - Error starting container
• ContainerCannotRun - Container failed to run

Resource Issues:

• OOMKilled - Container killed due to memory limits
• Error - Container exited with error

Pod Issues:

• PodFailed - Pod entered failed phase

Configuring Health Monitoring

Enabling Health Monitoring

Health monitoring is enabled by default. Configure it through environment variables:

# Helm values for agent
agent:
  config:
    deploymentHealthEnabled: true
    deploymentHealthInterval: 60

Or set environment variables directly:

BROKKR__AGENT__DEPLOYMENT_HEALTH_ENABLED=true
BROKKR__AGENT__DEPLOYMENT_HEALTH_INTERVAL=60

Adjusting Check Interval

The check interval determines how frequently the agent evaluates deployment health. The default is 60 seconds, which balances responsiveness with API load.

For environments where rapid detection is critical:

agent:
  config:
    deploymentHealthInterval: 30  # Check every 30 seconds

For large clusters with many deployments, increase the interval to reduce API load:

agent:
  config:
    deploymentHealthInterval: 120  # Check every 2 minutes

Disabling Health Monitoring

To disable health monitoring entirely:

agent:
  config:
    deploymentHealthEnabled: false

Note that disabling health monitoring means the broker will not have visibility into deployment status.

Viewing Health Status

Via API

Query health status for a specific deployment object:

curl "http://broker:3000/api/v1/deployment-objects/{id}/health" \
  -H "Authorization: Bearer $ADMIN_PAK"

Response:

{
  "deployment_object_id": "a1b2c3d4-...",
  "overall_status": "healthy",
  "health_records": [
    {
      "agent_id": "e5f6g7h8-...",
      "status": "healthy",
      "summary": {
        "pods_ready": 3,
        "pods_total": 3,
        "conditions": []
      },
      "checked_at": "2025-01-02T10:00:00Z"
    }
  ]
}

Understanding the Summary

The health summary provides details about pod status:

{
  "pods_ready": 2,
  "pods_total": 3,
  "conditions": ["ImagePullBackOff"],
  "resources": [
    {
      "kind": "Pod",
      "name": "my-app-abc123",
      "namespace": "production",
      "ready": false,
      "message": "Back-off pulling image \"myapp:invalid\""
    }
  ]
}

Common Scenarios

ImagePullBackOff

When the agent reports ImagePullBackOff:

1. Verify the image name and tag are correct
2. Check that the image exists in the registry
3. Verify the cluster has network access to the registry
4. Check image pull secrets are configured correctly

# Check pod events for details
kubectl describe pod <pod-name> -n <namespace>

# Check image pull secrets
kubectl get secrets -n <namespace>

CrashLoopBackOff

When containers repeatedly crash:

1. Check container logs for error messages:

   kubectl logs <pod-name> -n <namespace> --previous
   

2. Verify the application configuration is correct

3. Check resource limits aren’t too restrictive

4. Ensure required environment variables and secrets are present

OOMKilled

When containers are killed for memory:

1. Increase memory limits:

   resources:
     limits:
       memory: "512Mi"  # Increase as needed
   

2. Investigate application memory usage

3. Consider memory profiling to identify leaks

Unknown Status

When status shows as unknown:

1. Verify pods exist for the deployment object
2. Check the agent has RBAC permissions to list pods
3. Check agent logs for API errors:

   kubectl logs -l app=brokkr-agent -c agent
   

Multi-Agent Deployments

When a deployment object is targeted to multiple agents, each agent reports its own health status. The broker stores health per agent, reflecting that the same deployment may have different health on different clusters.

The health endpoint always returns records from all reporting agents in the health_records array, along with an overall_status that reflects the aggregate state:

curl "http://broker:3000/api/v1/deployment-objects/{id}/health" \
  -H "Authorization: Bearer $ADMIN_PAK"

Webhook Integration

Configure webhooks to receive notifications when deployment health changes:

curl -X POST "http://broker:3000/api/v1/webhooks" \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Health Alerts",
    "url": "https://alerts.example.com/webhook",
    "event_types": ["deployment.failed"]
  }'

The deployment.failed event fires when a deployment transitions to failing status.

Troubleshooting

Health Not Updating

If health status isn’t updating:

1. Check the agent is running and connected:

   kubectl get pods -l app=brokkr-agent
   

2. Verify health monitoring is enabled:

   kubectl get configmap brokkr-agent-config -o yaml
   

3. Check agent logs for health check errors:

   kubectl logs -l app=brokkr-agent -c agent | grep -i health
   

Incorrect Health Status

If reported health doesn’t match actual pod status:

1. Verify pods have the correct deployment object ID label
2. Check the health check interval - status may be stale
3. Confirm the agent has permission to list pods across namespaces

High API Load

If health monitoring causes excessive Kubernetes API load:

1. Increase the check interval
2. Consider reducing the number of deployment objects per agent
3. Monitor agent metrics for API call rates

Related Documentation

• Configuration Reference - Agent configuration options
• Architecture - How agents monitor health
• Webhooks - Alert on health changes

Understanding Reconciliation

Brokkr agents continuously reconcile the desired state defined in deployment objects with the actual state in Kubernetes clusters. This guide explains how the reconciliation loop works, how sequence IDs ensure correct ordering, how the system handles pre-existing resources, and what to do when things go wrong.

The Reconciliation Model

Brokkr uses a pull-based reconciliation model where agents periodically fetch their target state from the broker and ensure their cluster matches. This differs from push-based systems where a central controller actively pushes changes—instead, each agent owns the reconciliation responsibility for its cluster.

The core loop runs at a configurable polling interval (default: 10 seconds). On each cycle, the agent fetches deployment objects from the broker, validates the resources, applies them using Kubernetes server-side apply, and reports the result back to the broker as an event. This cycle continues indefinitely, ensuring clusters stay aligned with the desired state even if drift occurs.

Polling and Target State

When an agent polls for its target state, it receives deployment objects from all stacks it’s targeting. The broker returns objects ordered by sequence ID, ensuring the agent processes them in the order they were created.

Agent -> Broker: GET /api/v1/agents/{id}/target-state
Broker -> Agent: [DeploymentObject, DeploymentObject, ...]

Each deployment object contains the complete YAML content to be applied and a checksum calculated from that content. The checksum serves as a version identifier—resources in the cluster are annotated with the checksum so the agent can identify which version they belong to.

Sequence IDs and Ordering

Every deployment object receives a monotonically increasing sequence ID when created. This provides a global ordering that’s essential for correct reconciliation:

• The agent processes objects in sequence order
• Later objects supersede earlier ones for the same stack
• The most recent (highest sequence ID) deployment object represents current desired state

When multiple deployment objects exist for a stack, the agent only needs to apply the most recent one. Earlier objects are historical records useful for audit purposes but don’t affect the current cluster state.

The Apply Process

For each deployment object, the reconciliation proceeds through several stages:

1. Priority Resource Application: Namespaces and CustomResourceDefinitions are applied first. These resources must exist before namespaced resources can be validated or created.

2. Validation: Remaining resources are validated using a Kubernetes dry-run apply. This catches schema errors, missing fields, and other issues before attempting the real apply.

3. Server-Side Apply: Resources are applied using Kubernetes server-side apply with force enabled. This approach allows Brokkr to take ownership of fields it manages while preserving fields managed by other controllers.

4. Annotation Injection: Each applied resource receives annotations identifying its stack and the deployment object checksum:

   • k8s.brokkr.io/stack: Links the resource to its stack (label)
   • k8s.brokkr.io/deployment-checksum: Identifies which deployment object version created it (annotation)

5. Pruning: After applying desired resources, the agent queries the cluster for all resources with the stack annotation and deletes any that don’t match the current checksum. This removes resources that were part of previous deployments but aren’t in the current desired state.

Handling Pre-Existing Resources

When Brokkr encounters resources that already exist in the cluster, several scenarios arise:

Resources without Brokkr annotations: Server-side apply succeeds, and Brokkr adds its annotations. The resource becomes managed by Brokkr going forward. Any fields not specified in the deployment object remain unchanged.

Resources with matching annotations: The apply is idempotent—the resource is updated to match the desired state. If the content hasn’t changed, this is effectively a no-op.

Resources with different checksum: The resource was created by a previous deployment object. It gets updated with the new content and checksum. If the resource is no longer in the desired state, it gets pruned during the cleanup phase.

Resources with owner references: During pruning, Brokkr skips resources that have owner references. These resources are managed by Kubernetes controllers and will be garbage collected when their owner is deleted.

Rollback on Failure

If reconciliation fails partway through applying resources, the agent attempts to roll back changes to maintain cluster consistency:

Namespace rollback: If the agent created new namespaces as part of this reconciliation and a later step fails, those namespaces are deleted. This prevents orphaned namespaces from accumulating after failed deployments.

No resource rollback: Individual resources that were successfully applied are not rolled back. This means partial applies can occur. The next reconciliation cycle will attempt the full apply again.

Error reporting: Failed reconciliation generates a failure event that’s sent to the broker. The event includes the error message, enabling visibility into what went wrong through the broker’s API or webhooks.

Deletion Markers

When a stack is deleted, Brokkr creates a special deployment object with is_deletion_marker: true. When the agent receives this:

1. The agent identifies all cluster resources belonging to that stack
2. Each resource is deleted from the cluster
3. A success event is reported to the broker

Deletion markers ensure resources are cleaned up even if the agent was offline when the stack was deleted. The agent will process the deletion marker on its next poll and remove the resources.

Configuration Options

Polling Interval

Control how frequently the agent checks for changes:

agent:
  polling_interval: 10  # seconds

Shorter intervals mean faster propagation of changes but higher API load on both the broker and Kubernetes API server. For production deployments, 10-60 seconds is typically appropriate.

Retry Behavior

The agent implements exponential backoff for transient Kubernetes API errors:

• Initial retry interval: 1 second
• Maximum retry interval: 60 seconds
• Maximum elapsed time: 5 minutes
• Backoff multiplier: 2.0

Retryable errors include:

• HTTP 429 (Too Many Requests)
• HTTP 500 (Internal Server Error)
• HTTP 503 (Service Unavailable)
• HTTP 504 (Gateway Timeout)

Non-retryable errors fail immediately and are reported to the broker.

Troubleshooting Reconciliation Issues

Resources Not Being Applied

If resources aren’t appearing in your cluster:

1. Check agent status: Verify the agent is running and in ACTIVE status. Inactive agents skip deployment object requests.

2. Check targeting: Confirm the stack is targeted to the agent via GET /api/v1/agents/{id}/targets.

3. Check agent logs: Look for validation errors or API failures in the agent container logs.

4. Check events: Query the broker for events related to the deployment object to see if failures were reported.

Resources Not Being Deleted

If resources persist after stack deletion:

1. Verify deletion marker: Check that a deletion marker deployment object was created for the stack.

2. Check labels: Verify the resources have the k8s.brokkr.io/stack label. Resources without this label aren’t managed by Brokkr.

3. Check owner references: Resources with owner references are skipped during pruning. They’ll be cleaned up when their owner is deleted.

Validation Failures

If deployments fail validation:

1. Check YAML syntax: Ensure the YAML in your deployment object is valid.

2. Check API versions: Verify the apiVersion and kind are correct for your Kubernetes version.

3. Check namespaces: If referencing a namespace, ensure it’s either included in the deployment object or already exists in the cluster.

4. Check CRDs: If using custom resources, ensure the CRD is either included in the deployment object or already installed.

Drift Detection

Brokkr doesn’t continuously monitor for drift—it only reconciles during polling cycles. If resources are modified outside of Brokkr:

• The next deployment object apply will restore the Brokkr-managed state
• Fields not managed by Brokkr (not in the deployment object) will be preserved
• To force a reconcile, submit the same deployment object content again (it gets a new sequence ID)

Related Documentation

• Core Concepts - Understanding the pull-based model
• Deployment Health - Monitoring applied resources
• Quick Start Guide - First deployment walkthrough
• Managing Stacks - Stack lifecycle and deletion

Using Stack Templates

Stack templates allow you to define reusable Kubernetes manifests with parameterized values. Templates use Tera for templating and JSON Schema for parameter validation.

Concepts

What Templates Provide

• Reusability: Define common patterns once, instantiate many times
• Validation: Parameters are validated against JSON Schema before rendering
• Safety: Template syntax is validated at creation time
• Versioning: Updates create new versions, preserving history
• Access Control: System templates (admin) vs generator-owned templates

Template Matching

Templates can be constrained to specific stacks using labels and annotations:

• No labels/annotations: Template can be used with any stack
• With labels: ALL template labels must exist on the target stack
• With annotations: ALL template annotation key-value pairs must exist on the target stack

Creating a Template

Basic Template Structure

A template consists of:

1. Name: Identifier for the template
2. Template Content: Tera-templated YAML
3. Parameters Schema: JSON Schema defining valid parameters

Example: Nginx Deployment Template

curl -X POST http://localhost:3000/api/v1/templates \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nginx-deployment",
    "description": "Simple nginx deployment with configurable replicas and image",
    "template_content": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: {{ name }}\n  namespace: {{ namespace | default(value=\"default\") }}\nspec:\n  replicas: {{ replicas | default(value=1) }}\n  selector:\n    matchLabels:\n      app: {{ name }}\n  template:\n    metadata:\n      labels:\n        app: {{ name }}\n    spec:\n      containers:\n      - name: nginx\n        image: nginx:{{ version | default(value=\"latest\") }}\n        ports:\n        - containerPort: 80",
    "parameters_schema": "{\"type\": \"object\", \"required\": [\"name\"], \"properties\": {\"name\": {\"type\": \"string\", \"minLength\": 1, \"description\": \"Deployment name\"}, \"namespace\": {\"type\": \"string\", \"description\": \"Target namespace\"}, \"replicas\": {\"type\": \"integer\", \"minimum\": 1, \"maximum\": 10, \"description\": \"Number of replicas\"}, \"version\": {\"type\": \"string\", \"description\": \"Nginx image tag\"}}}"
  }'

Tera Templating

Variable Substitution

Use {{ variable }} syntax for simple substitution:

metadata:
  name: {{ name }}
  namespace: {{ namespace }}

Default Values

Use the default filter for optional parameters:

spec:
  replicas: {{ replicas | default(value=1) }}

Conditionals

Use {% if %} blocks for conditional content:

{% if enable_hpa %}
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ name }}-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ name }}
  minReplicas: {{ min_replicas | default(value=1) }}
  maxReplicas: {{ max_replicas | default(value=10) }}
{% endif %}

Loops

Use {% for %} to iterate over arrays:

spec:
  containers:
  {% for container in containers %}
  - name: {{ container.name }}
    image: {{ container.image }}
    ports:
    {% for port in container.ports %}
    - containerPort: {{ port }}
    {% endfor %}
  {% endfor %}

Filters

Tera provides many built-in filters:

metadata:
  name: {{ name | lower }}           # lowercase
  labels:
    version: "{{ version | upper }}" # uppercase
    slug: {{ name | slugify }}        # URL-safe slug

See the Tera documentation for all available filters.

JSON Schema Validation

Basic Schema

Define required and optional parameters:

{
  "type": "object",
  "required": ["name", "image"],
  "properties": {
    "name": {
      "type": "string",
      "minLength": 1,
      "description": "Resource name"
    },
    "image": {
      "type": "string",
      "pattern": "^[a-z0-9./-]+:[a-zA-Z0-9.-]+$"
    },
    "replicas": {
      "type": "integer",
      "minimum": 1,
      "maximum": 100,
      "default": 1
    }
  }
}

Validation Constraints

Common JSON Schema constraints:

Nested Objects

{
  "type": "object",
  "properties": {
    "resources": {
      "type": "object",
      "properties": {
        "cpu": {"type": "string", "pattern": "^[0-9]+m?$"},
        "memory": {"type": "string", "pattern": "^[0-9]+[GMK]i$"}
      }
    }
  }
}

Instantiating Templates

Once a template is created, instantiate it to create deployment objects:

curl -X POST http://localhost:3000/api/v1/stacks/$STACK_ID/deployment-objects/from-template \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "template_id": "'"$TEMPLATE_ID"'",
    "parameters": {
      "name": "my-nginx",
      "namespace": "production",
      "replicas": 3,
      "version": "1.25"
    }
  }'

The broker will:

1. Validate template labels match the stack
2. Validate parameters against the JSON Schema
3. Render the template with Tera
4. Create a deployment object in the stack

Template Labels and Annotations

Restricting Template Usage

Add labels to restrict which stacks can use a template:

# Add label to template
curl -X POST http://localhost:3000/api/v1/templates/$TEMPLATE_ID/labels \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '"env=production"'

# Add annotation to template
curl -X POST http://localhost:3000/api/v1/templates/$TEMPLATE_ID/annotations \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{"key": "tier", "value": "1"}'

Matching Rules

When instantiation fails due to label mismatch, you’ll receive a 422 response with details:

{
  "error": "Template labels do not match stack",
  "missing_labels": ["tier=1"],
  "missing_annotations": []
}

Template Versioning

Templates are immutable. Updates create new versions:

# Update template (creates version 2)
curl -X PUT http://localhost:3000/api/v1/templates/$TEMPLATE_ID \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Updated nginx template with HPA support",
    "template_content": "...",
    "parameters_schema": "..."
  }'

Each version has a unique ID. Deployment objects reference the specific template version used.

Generator-Owned Templates

Generators can create and manage their own templates:

# Generator creates template
curl -X POST http://localhost:3000/api/v1/templates \
  -H "Authorization: Bearer $GENERATOR_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-app-template",
    "template_content": "...",
    "parameters_schema": "..."
  }'

Generators can only:

• View system templates (no generator_id) and their own templates
• Modify/delete only their own templates
• Instantiate templates into stacks they own

Complete Example: PostgreSQL Database

1. Create the Template

curl -X POST http://localhost:3000/api/v1/templates \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "postgresql-database",
    "description": "PostgreSQL StatefulSet with PVC",
    "template_content": "apiVersion: v1\nkind: Secret\nmetadata:\n  name: {{ name }}-credentials\n  namespace: {{ namespace }}\nstringData:\n  POSTGRES_USER: {{ username | default(value=\"postgres\") }}\n  POSTGRES_PASSWORD: {{ password }}\n  POSTGRES_DB: {{ database }}\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: {{ name }}\n  namespace: {{ namespace }}\nspec:\n  ports:\n  - port: 5432\n  clusterIP: None\n  selector:\n    app: {{ name }}\n---\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n  name: {{ name }}\n  namespace: {{ namespace }}\nspec:\n  serviceName: {{ name }}\n  replicas: {{ replicas | default(value=1) }}\n  selector:\n    matchLabels:\n      app: {{ name }}\n  template:\n    metadata:\n      labels:\n        app: {{ name }}\n    spec:\n      containers:\n      - name: postgres\n        image: postgres:{{ version | default(value=\"15\") }}\n        ports:\n        - containerPort: 5432\n        envFrom:\n        - secretRef:\n            name: {{ name }}-credentials\n        volumeMounts:\n        - name: data\n          mountPath: /var/lib/postgresql/data\n  volumeClaimTemplates:\n  - metadata:\n      name: data\n    spec:\n      accessModes: [\"ReadWriteOnce\"]\n      resources:\n        requests:\n          storage: {{ storage_size }}",
    "parameters_schema": "{\"type\": \"object\", \"required\": [\"name\", \"namespace\", \"database\", \"password\", \"storage_size\"], \"properties\": {\"name\": {\"type\": \"string\", \"minLength\": 1}, \"namespace\": {\"type\": \"string\", \"minLength\": 1}, \"database\": {\"type\": \"string\", \"minLength\": 1}, \"username\": {\"type\": \"string\"}, \"password\": {\"type\": \"string\", \"minLength\": 8}, \"version\": {\"type\": \"string\"}, \"replicas\": {\"type\": \"integer\", \"minimum\": 1}, \"storage_size\": {\"type\": \"string\", \"pattern\": \"^[0-9]+[GMK]i$\"}}}"
  }'

2. Add Production Label

curl -X POST http://localhost:3000/api/v1/templates/$TEMPLATE_ID/labels \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '"database=postgresql"'

3. Instantiate for Production

curl -X POST http://localhost:3000/api/v1/stacks/$PROD_STACK_ID/deployment-objects/from-template \
  -H "Authorization: Bearer $ADMIN_PAK" \
  -H "Content-Type: application/json" \
  -d '{
    "template_id": "'"$TEMPLATE_ID"'",
    "parameters": {
      "name": "orders-db",
      "namespace": "production",
      "database": "orders",
      "password": "secure-password-here",
      "version": "15",
      "replicas": 3,
      "storage_size": "100Gi"
    }
  }'

Troubleshooting

Invalid Tera Syntax

Template creation fails with syntax errors:

{
  "error": "Invalid Tera syntax: ..."
}

Check for:

• Unclosed {{ }} or {% %} blocks
• Missing {% endif %} or {% endfor %}
• Invalid filter names

Invalid JSON Schema

{
  "error": "Invalid JSON Schema: ..."
}

Validate your schema at jsonschemavalidator.net.

Parameter Validation Failed

{
  "error": "Invalid parameters",
  "validation_errors": [
    "/replicas: 0 is less than the minimum of 1"
  ]
}

Check that parameters match the schema constraints.

Template Rendering Failed

{
  "error": "Template rendering failed: Variable `name` not found"
}

Ensure all required template variables are provided in parameters, or use | default(value=...) for optional ones.

How-To: Running On-Demand Diagnostics

This guide shows how to collect pod statuses, Kubernetes events, and container logs from a remote cluster when a deployment is misbehaving. Brokkr’s diagnostic system lets you request this data through the broker API without direct kubectl access to the target cluster.

When to Use Diagnostics

Use on-demand diagnostics when:

• A deployment shows degraded or failing health status
• You need to see pod conditions, restart counts, or OOMKill events
• You want container logs from a remote cluster you can’t directly access
• You’re troubleshooting why a deployment object failed to apply

Prerequisites

• Admin PAK for the broker
• The deployment_object_id of the resource you want to diagnose
• The agent_id of the agent running in the target cluster
• The agent must be connected and sending heartbeats

Step 1: Identify the Deployment Object

If you know the stack, list its deployment objects:

curl -s "http://localhost:3000/api/v1/stacks/${STACK_ID}/deployment-objects" \
  -H "Authorization: <admin-pak>" | jq '.[] | {id, sequence_id, created_at}'

Check the health status to confirm something is wrong:

curl -s "http://localhost:3000/api/v1/deployment-objects/${DO_ID}/health" \
  -H "Authorization: <admin-pak>" | jq .

Step 2: Find the Target Agent

List agents that target this stack:

curl -s "http://localhost:3000/api/v1/agents" \
  -H "Authorization: <admin-pak>" | jq '.[] | {id, name, cluster_name, last_heartbeat}'

Verify the agent has a recent heartbeat (within the last few minutes).

Step 3: Request Diagnostics

Create a diagnostic request:

curl -s -X POST "http://localhost:3000/api/v1/deployment-objects/${DO_ID}/diagnostics" \
  -H "Authorization: <admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{
    \"agent_id\": \"${AGENT_ID}\",
    \"requested_by\": \"oncall-engineer\",
    \"retention_minutes\": 120
  }" | jq .

Save the diagnostic request ID from the response:

DIAG_ID="..."

The retention_minutes field controls how long the request stays active before expiring. Default is 60 minutes, maximum is 1440 (24 hours).

Step 4: Wait for Results

The agent picks up the diagnostic request on its next poll cycle. Poll the diagnostic status:

curl -s "http://localhost:3000/api/v1/diagnostics/${DIAG_ID}" \
  -H "Authorization: <admin-pak>" | jq '.request.status'

Status progression: pending → claimed → completed

Step 5: Read the Results

Once the status is completed, the full results are available:

# Pod statuses
curl -s "http://localhost:3000/api/v1/diagnostics/${DIAG_ID}" \
  -H "Authorization: <admin-pak>" | jq -r '.result.pod_statuses' | jq .

# Kubernetes events
curl -s "http://localhost:3000/api/v1/diagnostics/${DIAG_ID}" \
  -H "Authorization: <admin-pak>" | jq -r '.result.events' | jq .

# Container logs
curl -s "http://localhost:3000/api/v1/diagnostics/${DIAG_ID}" \
  -H "Authorization: <admin-pak>" | jq -r '.result.log_tails' | jq .

Reading Pod Statuses

Look for:

• Phase: Pending or Failed indicates problems
• Conditions: Check Ready=False with the reason
• Containers: Look for restart_count > 0, state=waiting with reasons like CrashLoopBackOff, or state=terminated with reason OOMKilled

Reading Events

Filter for warnings:

curl -s "http://localhost:3000/api/v1/diagnostics/${DIAG_ID}" \
  -H "Authorization: <admin-pak>" \
  | jq -r '.result.events' \
  | jq '.[] | select(.event_type == "Warning")'

Common warning events: FailedScheduling, Unhealthy, BackOff, FailedMount.

Reading Logs

Log tails are keyed by pod-name/container-name:

curl -s "http://localhost:3000/api/v1/diagnostics/${DIAG_ID}" \
  -H "Authorization: <admin-pak>" \
  | jq -r '.result.log_tails' \
  | jq 'to_entries[] | "\(.key):\n\(.value)\n---"' -r

Each container’s last 100 log lines are included.

Troubleshooting

Diagnostic stays in pending state:

• Check the agent’s heartbeat — it may be disconnected
• Verify the agent is targeting the stack that contains the deployment object
• Check the agent logs for errors

Diagnostic moves to expired:

• The retention period elapsed before the agent could claim it
• Increase retention_minutes and try again
• Check if the agent is running and polling

Diagnostic moves to failed:

• The agent encountered an error collecting data
• Check the agent logs for Kubernetes API errors
• Verify the agent has RBAC permissions to read pods, events, and logs

Cleanup

Diagnostics are automatically cleaned up by the broker’s background task based on broker.diagnostic_cleanup_interval_seconds (default: 15 minutes) and broker.diagnostic_max_age_hours (default: 1 hour).

Related Documentation

• Diagnostics Reference — complete API and data model reference
• Monitoring Deployment Health — continuous health monitoring
• Health Endpoints — health check configuration

How-To: Managing PAKs (Key Rotation)

Pre-Authentication Keys (PAKs) are the authentication credentials for all Brokkr entities — admins, agents, and generators. This guide covers creating, rotating, and managing PAKs.

Overview

PAKs look like brokkr_BR3rVsDa_GK3QN7CDUzYc6iKgMkJ98M2WSimM5t6U8. Brokkr stores only the hash — once a PAK is displayed at creation, it cannot be recovered, only rotated. See the Environment Variables Reference for PAK configuration details.

Rotating the Admin PAK

Via CLI (recommended)

Run on the broker host:

brokkr-broker rotate admin

The new PAK is printed to stdout. The old PAK immediately stops working.

Via API

Not available — admin PAK rotation requires CLI access to prevent an attacker with a compromised admin PAK from locking out the real admin.

When to Rotate

• After personnel changes (someone with admin access leaves)
• If the PAK may have been exposed in logs, version control, or screenshots
• As part of a regular rotation schedule (e.g., quarterly)

Rotating Agent PAKs

Via CLI

brokkr-broker rotate agent --uuid <agent-uuid>

Via API

An agent can rotate its own PAK, or an admin can rotate any agent’s PAK:

# As admin
curl -s -X POST "http://localhost:3000/api/v1/agents/${AGENT_ID}/rotate-pak" \
  -H "Authorization: <admin-pak>" | jq .

# As the agent itself
curl -s -X POST "http://localhost:3000/api/v1/agents/${AGENT_ID}/rotate-pak" \
  -H "Authorization: <agent-pak>" | jq .

Response:

{
  "agent": { "id": "...", "name": "prod-1", ... },
  "pak": "brokkr_BRnewKey_NewLongTokenValue1234567890"
}

Updating the Agent After Rotation

After rotating, update the agent’s configuration with the new PAK:

Helm deployment:

helm upgrade brokkr-agent oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --set broker.pak="brokkr_BRnewKey_NewLongTokenValue1234567890"

Environment variable:

BROKKR__AGENT__PAK=brokkr_BRnewKey_NewLongTokenValue1234567890

Kubernetes secret:

kubectl create secret generic brokkr-agent-pak \
  --from-literal=pak="brokkr_BRnewKey_NewLongTokenValue1234567890" \
  --dry-run=client -o yaml | kubectl apply -f -

Warning: The agent will fail to authenticate with the old PAK immediately after rotation. Ensure you update the agent configuration before the next poll cycle, or the agent will lose connectivity until updated.

Rotating Generator PAKs

Via CLI

brokkr-broker rotate generator --uuid <generator-uuid>

Via API

# As admin
curl -s -X POST "http://localhost:3000/api/v1/generators/${GEN_ID}/rotate-pak" \
  -H "Authorization: <admin-pak>" | jq .

# As the generator itself
curl -s -X POST "http://localhost:3000/api/v1/generators/${GEN_ID}/rotate-pak" \
  -H "Authorization: <generator-pak>" | jq .

Updating CI/CD After Rotation

Update the stored secret in your CI/CD system:

GitHub Actions:

1. Go to Settings → Secrets and variables → Actions
2. Update the BROKKR_GENERATOR_PAK secret with the new value

GitLab CI:

1. Go to Settings → CI/CD → Variables
2. Update the BROKKR_GENERATOR_PAK variable

Cache Considerations After CLI Rotation

API-based rotation automatically invalidates the auth cache. CLI-based rotation operates directly on the database, so the old PAK may still work for up to 60 seconds (the default broker.auth_cache_ttl_seconds). To force immediate invalidation after a CLI rotation:

curl -s -X POST "http://localhost:3000/api/v1/admin/config/reload" \
  -H "Authorization: <admin-pak>"

Verifying Rotation via Audit Logs

All PAK operations are recorded. Query PAK-related audit events:

curl -s "http://localhost:3000/api/v1/admin/audit-logs?action=pak.*" \
  -H "Authorization: <admin-pak>" | jq .

See the Audit Logs Reference for the full list of audit event types.

Related Documentation

• Security Model — authentication and authorization architecture
• Audit Logs Reference — audit event format and querying
• Configuration Guide — PAK and auth cache settings

How-To: Setting Up Multi-Tenant Isolation

This guide walks through configuring Brokkr for multi-tenant operation using PostgreSQL schema isolation. Each tenant gets a fully isolated dataset while sharing a single database server.

Goal

Set up two tenants (acme and globex) on a shared PostgreSQL instance, each with their own broker instance and complete data isolation.

Prerequisites

• A PostgreSQL server accessible to both broker instances
• The brokkr user with permission to create schemas
• Helm (for Kubernetes deployment) or direct access to run broker binaries

Step 1: Prepare the Database

Create the shared database if it doesn’t exist:

CREATE DATABASE brokkr;
CREATE USER brokkr WITH PASSWORD 'your-secure-password';
GRANT ALL PRIVILEGES ON DATABASE brokkr TO brokkr;

-- Grant schema creation permission
GRANT CREATE ON DATABASE brokkr TO brokkr;

You don’t need to create the schemas manually — Brokkr creates them on first startup.

Step 2: Deploy Tenant Broker Instances

Option A: Helm (Kubernetes)

Deploy each tenant as a separate Helm release:

# Tenant: Acme
helm install brokkr-acme oci://ghcr.io/colliery-io/charts/brokkr-broker \
  --namespace brokkr-acme --create-namespace \
  --set postgresql.enabled=false \
  --set postgresql.external.host=postgres.example.com \
  --set postgresql.external.port=5432 \
  --set postgresql.external.database=brokkr \
  --set postgresql.external.username=brokkr \
  --set postgresql.external.password=your-secure-password \
  --set postgresql.external.schema=tenant_acme

# Tenant: Globex
helm install brokkr-globex oci://ghcr.io/colliery-io/charts/brokkr-broker \
  --namespace brokkr-globex --create-namespace \
  --set postgresql.enabled=false \
  --set postgresql.external.host=postgres.example.com \
  --set postgresql.external.port=5432 \
  --set postgresql.external.database=brokkr \
  --set postgresql.external.username=brokkr \
  --set postgresql.external.password=your-secure-password \
  --set postgresql.external.schema=tenant_globex

Option B: Environment Variables (Direct)

Run each broker with different schema settings:

# Terminal 1: Acme broker (port 3000)
BROKKR__DATABASE__URL=postgres://brokkr:password@postgres.example.com:5432/brokkr \
BROKKR__DATABASE__SCHEMA=tenant_acme \
BROKKR__LOG__LEVEL=info \
  brokkr-broker serve

# Terminal 2: Globex broker (port 3001 - change bind port in config)
BROKKR__DATABASE__URL=postgres://brokkr:password@postgres.example.com:5432/brokkr \
BROKKR__DATABASE__SCHEMA=tenant_globex \
BROKKR__LOG__LEVEL=info \
  brokkr-broker serve

Option C: Configuration Files

Create a config file per tenant:

# /etc/brokkr/acme.toml
[database]
url = "postgres://brokkr:password@postgres.example.com:5432/brokkr"
schema = "tenant_acme"

[log]
level = "info"
format = "json"

# /etc/brokkr/globex.toml
[database]
url = "postgres://brokkr:password@postgres.example.com:5432/brokkr"
schema = "tenant_globex"

[log]
level = "info"
format = "json"

BROKKR_CONFIG_FILE=/etc/brokkr/acme.toml brokkr-broker serve
BROKKR_CONFIG_FILE=/etc/brokkr/globex.toml brokkr-broker serve

Step 3: First Startup

On first startup, each broker instance:

1. Creates the schema (CREATE SCHEMA IF NOT EXISTS tenant_acme)
2. Runs all database migrations within the schema
3. Creates the admin role and generates an admin PAK
4. Logs the admin PAK to stdout

Capture the admin PAK for each tenant — it’s only shown once:

# Look for this line in the logs
# INFO  Admin PAK: brokkr_BR...

Step 4: Register Agents Per Tenant

Each tenant’s agents connect to their tenant’s broker instance:

# Create agent for Acme tenant
curl -s -X POST http://acme-broker:3000/api/v1/agents \
  -H "Authorization: <acme-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "acme-prod", "cluster_name": "us-east-1"}'

# Create agent for Globex tenant
curl -s -X POST http://globex-broker:3000/api/v1/agents \
  -H "Authorization: <globex-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{"name": "globex-prod", "cluster_name": "eu-west-1"}'

Step 5: Deploy Tenant Agents

Point each agent at the correct tenant broker:

# Acme agent
helm install brokkr-agent-acme oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --namespace brokkr-acme \
  --set broker.url=http://brokkr-acme-brokkr-broker:3000 \
  --set broker.pak="<acme-agent-pak>" \
  --set broker.agentName=acme-prod \
  --set broker.clusterName=us-east-1

# Globex agent
helm install brokkr-agent-globex oci://ghcr.io/colliery-io/charts/brokkr-agent \
  --namespace brokkr-globex \
  --set broker.url=http://brokkr-globex-brokkr-broker:3000 \
  --set broker.pak="<globex-agent-pak>" \
  --set broker.agentName=globex-prod \
  --set broker.clusterName=eu-west-1

Step 6: Verify Isolation

Confirm that each tenant only sees its own data:

# Acme sees only Acme agents
curl -s http://acme-broker:3000/api/v1/agents \
  -H "Authorization: <acme-admin-pak>" | jq '.[].name'
# Output: "acme-prod"

# Globex sees only Globex agents
curl -s http://globex-broker:3000/api/v1/agents \
  -H "Authorization: <globex-admin-pak>" | jq '.[].name'
# Output: "globex-prod"

Acme’s admin PAK does not work against Globex’s broker, and vice versa.

Connection Pool Sizing

Each broker instance uses a connection pool (default: 50 connections). With multiple tenants on one database, the total connections across all broker instances must stay under PostgreSQL’s max_connections (default: 100). Increase it or reduce per-tenant pool sizes for many tenants. See Multi-Tenancy Reference for detailed capacity planning.

Schema Naming

Use a consistent pattern like tenant_{name} (e.g., tenant_acme). Schema names allow only alphanumeric characters and underscores, max 63 characters. See Multi-Tenancy Reference for full constraints.

Related Documentation

• Multi-Tenancy Reference — data isolation details and limitations
• Configuration Guide — database configuration
• Installation Guide — deployment options

How-To: Querying Audit Logs

Brokkr maintains an immutable audit trail of all significant operations. This guide shows how to query audit logs to investigate events, track changes, and monitor security.

Audit Log API

All audit log queries go through the admin API:

GET /api/v1/admin/audit-logs

Auth: Admin only.

Basic Query

List the most recent audit events:

curl -s "http://localhost:3000/api/v1/admin/audit-logs" \
  -H "Authorization: <admin-pak>" | jq .

Default: returns the 100 most recent entries, ordered by timestamp (newest first). The response is a JSON object with a logs array, plus total, count, limit, and offset fields for pagination.

Filtering

By Actor Type

See all actions performed by agents:

curl -s "http://localhost:3000/api/v1/admin/audit-logs?actor_type=agent" \
  -H "Authorization: <admin-pak>" | jq '.logs[] | {action, resource_type, timestamp}'

Valid actor types: admin, agent, generator, system

By Action

Find all agent creation events:

curl -s "http://localhost:3000/api/v1/admin/audit-logs?action=agent.created" \
  -H "Authorization: <admin-pak>" | jq .

Actions support wildcard prefix matching. To see all agent-related events:

curl -s "http://localhost:3000/api/v1/admin/audit-logs?action=agent.*" \
  -H "Authorization: <admin-pak>" | jq .

By Resource

Track all changes to a specific agent:

curl -s "http://localhost:3000/api/v1/admin/audit-logs?resource_type=agent&resource_id=${AGENT_ID}" \
  -H "Authorization: <admin-pak>" | jq .

By Time Range

Query events within a specific window:

curl -s "http://localhost:3000/api/v1/admin/audit-logs?from=2025-01-15T00:00:00Z&to=2025-01-16T00:00:00Z" \
  -H "Authorization: <admin-pak>" | jq .

By Actor ID

See everything a specific generator has done:

curl -s "http://localhost:3000/api/v1/admin/audit-logs?actor_type=generator&actor_id=${GEN_ID}" \
  -H "Authorization: <admin-pak>" | jq .

Pagination

Use limit and offset for large result sets:

# First page
curl -s "http://localhost:3000/api/v1/admin/audit-logs?limit=50&offset=0" \
  -H "Authorization: <admin-pak>" | jq .

# Second page
curl -s "http://localhost:3000/api/v1/admin/audit-logs?limit=50&offset=50" \
  -H "Authorization: <admin-pak>" | jq .

Maximum limit is 1000.

Common Investigation Patterns

Who Changed This Agent?

curl -s "http://localhost:3000/api/v1/admin/audit-logs?resource_type=agent&resource_id=${AGENT_ID}" \
  -H "Authorization: <admin-pak>" \
  | jq '.logs[] | {actor_type, actor_id, action, timestamp, details}'

Failed Authentication Attempts

curl -s "http://localhost:3000/api/v1/admin/audit-logs?action=auth.failed" \
  -H "Authorization: <admin-pak>" \
  | jq '.logs[] | {timestamp, ip_address, user_agent, details}'

Recent PAK Rotations

curl -s "http://localhost:3000/api/v1/admin/audit-logs?action=pak.rotated" \
  -H "Authorization: <admin-pak>" \
  | jq '.logs[] | {actor_type, resource_type, resource_id, timestamp}'

All Admin Actions Today

TODAY=$(date -u +%Y-%m-%dT00:00:00Z)
curl -s "http://localhost:3000/api/v1/admin/audit-logs?actor_type=admin&from=${TODAY}" \
  -H "Authorization: <admin-pak>" | jq .

Webhook Configuration Changes

curl -s "http://localhost:3000/api/v1/admin/audit-logs?action=webhook.*" \
  -H "Authorization: <admin-pak>" \
  | jq '.logs[] | {action, resource_id, timestamp, details}'

Stack Deletion History

curl -s "http://localhost:3000/api/v1/admin/audit-logs?action=stack.deleted" \
  -H "Authorization: <admin-pak>" \
  | jq '.logs[] | {actor_type, actor_id, resource_id, timestamp}'

Audit Event Types

Actions follow the pattern resource.verb (e.g., agent.created, pak.rotated, auth.failed). You can use wildcard queries like action=agent.* to match all events for a resource type.

For the complete list of all audit event types, see the Audit Logs Reference.

Retention

Audit logs are retained for 90 days by default (broker.audit_log_retention_days). The broker runs a daily cleanup task to purge older entries.

To change retention:

BROKKR__BROKER__AUDIT_LOG_RETENTION_DAYS=365

Related Documentation

• Audit Logs Reference — schema and data model details
• Security Model — authentication and authorization
• Managing PAKs — PAK rotation and security

Explanation

In-depth discussion of Brokkr’s architecture, design decisions, and internal workings. These documents help you understand why Brokkr works the way it does.

Core Concepts

What is Brokkr?

Brokkr is an environment-aware control plane for dynamically distributing Kubernetes objects. Think of it as a smart traffic controller for your Kubernetes resources—it knows not just what to deploy, but where and when to deploy it based on your environment’s specific needs and policies.

graph LR
    subgraph "Control Plane"
        UA[User/Admin] -->|Creates/Updates| BR[Broker]
    end

    subgraph "Agents"
        AG[Agent]
    end

    subgraph "Kubernetes Clusters"
        KC[K8s Cluster]
    end

    AG -- Fetches Target State --> BR
    AG -- Reports Status --> BR
    AG -- Applies --> KC

Note: This diagram shows a single agent and cluster for clarity. In real deployments, Brokkr supports multiple agents and clusters, each following the same pattern.

Key Components

The Broker: The Source of Truth

The Broker serves as the central source of truth for Brokkr. It records the desired state of your applications and environments while providing APIs for users and agents to interact with this state. Importantly, the broker does not directly control clusters or push deployments. Instead, it maintains the authoritative record of what should exist and lets agents pull that information on their own schedule.

When users create or update resources, the broker records these changes and makes them available via its REST API. It handles authentication and authorization for all requests, ensuring that agents and generators can only access resources they’re permitted to see. As agents report back their activities, the broker records these events to maintain a complete audit trail of what has happened across your infrastructure.

The Agent: The Executor

Agents are the workhorses that make Brokkr’s desired state a reality in your Kubernetes clusters. Each agent runs within a specific environment, typically a single Kubernetes cluster, and takes full responsibility for that environment’s alignment with the broker’s desired state.

On a regular polling interval, agents contact the broker to fetch their target state—the deployment objects they should apply. They then validate these resources locally, checking YAML syntax and ensuring the resources make sense for their environment. After validation, agents apply the resources to their local Kubernetes cluster and report the results back to the broker.

This pull-based model has important advantages. Agents in restricted networks or behind firewalls can still receive deployments by initiating outbound connections to the broker. The model also provides natural resilience; if an agent goes offline temporarily, it simply catches up on missed changes when it reconnects.

Internal Data Architecture

Brokkr’s data model tracks what should be deployed, where, and by whom, while maintaining a clear audit trail of what has actually occurred. Understanding these entities helps you work effectively with the system.

Stacks

A Stack is a collection of related Kubernetes objects managed as a unit. Stacks provide the organizational boundary for grouping resources that belong together—perhaps all the components of a microservice, or all the infrastructure for a particular application. Beyond this grouping, Brokkr imposes no particular structure or semantics on stacks.

Deployment Objects

A Deployment Object is a versioned snapshot of all Kubernetes resources in a Stack at a particular point in time. Each time you update a Stack, Brokkr creates a new Deployment Object capturing that desired state. These objects are immutable once created, providing a complete historical record of changes. This immutability means you can always see exactly what was deployed at any point in the past.

Agents

An Agent represents a Brokkr process running in a specific environment. Agents have unique identities, authentication credentials, and metadata describing their capabilities and characteristics. The broker tracks which agents are registered, their current status, and their assignment to various stacks.

Agent Targets

An Agent Target connects an Agent to a Stack, defining which agents are responsible for managing which stacks. This mapping layer allows Brokkr to distribute workloads across multiple clusters and environments. A single stack might be targeted by multiple agents (for multi-cluster deployments), and a single agent might be responsible for multiple stacks.

Agent Events

Agent Events record the outcome of each attempt to apply a Deployment Object. When an agent applies resources and reports back to the broker, that report becomes an event in the system’s history. Events capture both successes and failures, providing an audit trail that’s essential for troubleshooting and compliance requirements.

Targeting Mechanisms

Brokkr provides flexible mechanisms for associating agents with stacks, allowing you to model a variety of deployment scenarios.

Direct Assignment offers the simplest approach: explicitly associate an agent with a stack by their IDs. This works well when you have a clear one-to-one mapping between agents and the stacks they should manage.

Label-Based Targeting enables dynamic, scalable associations. Both agents and stacks can carry labels, and you can configure stacks to target all agents with matching labels. This supports patterns like “all production agents should receive all production stacks” without maintaining explicit associations for each pair.

Annotation-Based Targeting extends the label concept with key-value pairs that can encode more complex matching rules. Annotations are useful when targeting logic requires more nuance than simple label presence—for example, targeting agents in a specific region or with particular capabilities.

How These Pieces Fit Together

The data entities connect to form a complete deployment workflow. Users create Stacks to group their Kubernetes resources. Each Stack accumulates Deployment Objects as its contents change over time. Agents register with the broker and are assigned responsibility for one or more Stacks via Agent Targets.

When an Agent polls the broker, it receives the latest Deployment Objects for its assigned Stacks. The Agent validates and applies these resources to its Kubernetes cluster, then reports the outcome as Agent Events. This cycle repeats continuously, keeping all clusters aligned with the desired state recorded in the broker.

erDiagram
    STACK ||--o{ DEPLOYMENT_OBJECT : has
    AGENT ||--o{ AGENT_TARGET : assigned_to
    STACK ||--o{ AGENT_TARGET : targeted_by
    DEPLOYMENT_OBJECT ||--o{ AGENT_EVENT : triggers
    AGENT ||--o{ AGENT_EVENT : reports

This architecture provides a clear, auditable, and scalable foundation for managing Kubernetes resources across many environments.

The Deployment Journey

The deployment process follows a pull-based model where agents take responsibility for fetching, validating, and applying their assigned target state. The broker maintains the source of truth and records events but never pushes deployments or performs environment-specific validation.

The journey begins when a user creates or updates a stack, which results in a new deployment object being created in the broker. Each agent then polls the broker on its regular interval, receiving the latest deployment objects for its assigned stacks. The agent validates these locally—checking YAML syntax, resource constraints, and any environment-specific rules—before applying the resources to its Kubernetes cluster.

After applying resources, the agent reports the outcome back to the broker as an event. Whether the application succeeded or failed, this information becomes part of the permanent audit trail. Over time, the broker accumulates a complete history of every deployment attempt across all your environments.

sequenceDiagram
    participant User
    participant Broker
    participant Agent
    participant Cluster

    User->>Broker: Create/Update Stack (creates Deployment Object)
    loop Every polling interval
        Agent->>Broker: Fetch Target State (Deployment Objects)
        Broker-->>Agent: Return Deployment Objects
        Agent->>Agent: Validate & Apply Resources
        Agent->>Cluster: Apply Resources
        Cluster-->>Agent: Result
        Agent->>Broker: Report Event (Success/Failure)
    end

Security Model

Brokkr uses API key authentication and role-based authorization for all API access. Every request must include a valid PAK (Prefixed API Key) in the Authorization header.

Authentication

The system supports three types of PAKs, each granting different levels of access. Admin PAKs provide full administrative access to all API endpoints and resources. Agent PAKs grant access only to endpoints and data relevant to a specific agent, such as fetching target state and reporting events. Generator PAKs allow external systems to create resources within their designated scope.

When a request arrives, the API middleware extracts the PAK from the Authorization header and verifies it against stored hashes. If the PAK matches a known admin, agent, or generator, the request proceeds with that identity and role attached. Invalid or missing PAKs result in authentication failures.

Authorization

Beyond authentication, Brokkr enforces role-based access control at every endpoint. Certain operations require admin privileges: creating agents, listing all resources, managing system configuration. Agent endpoints ensure that each agent can only access its own target state and report its own events. Generator endpoints similarly restrict access to each generator’s own resources.

The system also enforces row-based access control within endpoints. After authenticating a request, the API verifies that the requesting entity has permission to access each specific resource. An agent fetching deployment objects receives only those for stacks it’s assigned to. A generator creating a stack can only access stacks it created. This fine-grained control ensures that even authenticated entities can only see and modify what they’re supposed to.

sequenceDiagram
    participant Client
    participant API
    participant DB

    Client->>API: Request (with PAK)
    API->>API: Authenticate PAK
    API->>API: Determine role/identity
    API->>DB: Query resource (with access check)
    alt Access allowed
        DB-->>API: Resource data
        API-->>Client: Success/Resource
    else Access denied
        API-->>Client: Forbidden/Unauthorized
    end

Key Management

PAKs are generated using secure random generation and stored as hashes in the database. The actual PAK value is shown only once at creation or rotation time, so it must be captured and stored securely at that moment. Both agents and generators can rotate their own PAKs, and administrators can rotate any PAK in the system.

Next Steps

With an understanding of Brokkr’s core concepts, you can explore further:

• Follow the Quick Start Guide to deploy your first application
• Study the Technical Architecture for implementation details
• Explore the Data Model to understand entity relationships
• Read the Security Model for comprehensive authentication and authorization details

Technical Architecture

Brokkr is a deployment orchestration platform designed to manage Kubernetes resources across multiple clusters from a central control plane. This document provides a comprehensive technical overview of Brokkr’s architecture, explaining how its components are implemented, how they interact, and the design decisions that shape its behavior.

System Overview

C4Context
    title Brokkr System Context (C4 Level 1)

    Person(engineer, "Platform Engineer", "Manages deployments and configuration")
    System_Ext(cicd, "CI/CD Pipeline", "Automated deployment source (Generator)")

    System(brokkr, "Brokkr", "Deployment orchestration platform for multi-cluster Kubernetes")

    System_Ext(k8s, "Kubernetes Clusters", "Target clusters where resources are deployed")
    SystemDb(pg, "PostgreSQL", "Persistent state storage")
    System_Ext(webhooks, "External Systems", "Receive event notifications via webhooks")

    Rel(engineer, brokkr, "Manages stacks and deployments", "HTTPS REST API")
    Rel(cicd, brokkr, "Creates stacks and deployment objects", "HTTPS REST API")
    Rel(brokkr, pg, "Stores state", "SQL")
    Rel(brokkr, k8s, "Deploys resources via agents", "HTTPS")
    Rel(brokkr, webhooks, "Sends event notifications", "HTTPS")

At its core, Brokkr follows a hub-and-spoke architecture where a central broker service coordinates deployments across multiple agent instances. The broker maintains the desired state of all deployments in a PostgreSQL database, while agents running in target Kubernetes clusters continuously poll the broker to fetch their assigned work and reconcile the actual cluster state to match the desired state.

C4Container
    title Brokkr Container Diagram (C4 Level 2)

    Person(engineer, "Platform Engineer", "Manages deployments")
    System_Ext(cicd, "CI/CD Pipeline", "Generator identity")

    Container_Boundary(brokkr, "Brokkr Platform") {
        Container(broker, "Broker", "Rust, Axum", "REST API, background tasks, webhook dispatch")
        Container(agent, "Agent", "Rust, kube-rs", "Polls broker, reconciles cluster state")
        ContainerDb(db, "PostgreSQL", "PostgreSQL 15+", "Stacks, deployment objects, events, audit logs")
    }

    System_Ext(k8s, "Kubernetes API", "Target cluster")
    System_Ext(webhooks, "Webhook Endpoints", "External notification receivers")

    Rel(engineer, broker, "Manages", "HTTPS REST API")
    Rel(cicd, broker, "Deploys", "HTTPS REST API")
    Rel(agent, broker, "Polls for state, reports events", "HTTPS REST")
    Rel(broker, db, "Reads/writes state", "SQL :5432")
    Rel(agent, k8s, "Applies/deletes resources", "HTTPS :6443")
    Rel(broker, webhooks, "Delivers events", "HTTPS")

This pull-based model was chosen deliberately over a push-based approach for several reasons. First, it simplifies network topology since agents only need outbound connectivity to the broker rather than requiring the broker to reach into potentially firewalled cluster networks. Second, it provides natural resilience since agents can continue operating with their cached state during temporary broker unavailability. Third, it allows agents to control their own reconciliation pace based on their cluster’s capabilities and load.

Broker Service Architecture

The broker is implemented as an asynchronous Rust service built on the Axum web framework with the Tokio runtime providing the async execution environment. When the broker starts, it initializes several interconnected subsystems that work together to provide the complete platform functionality.

Initialization Sequence

The broker follows a carefully ordered startup sequence to ensure all dependencies are properly initialized before accepting requests. First, the configuration is loaded from environment variables and configuration files, establishing database connection parameters, authentication settings, and operational parameters like polling intervals.

Next, a PostgreSQL connection pool is established using the r2d2 connection manager with Diesel as the ORM layer. The pool is configured with a default of 50 connections to accommodate background tasks, middleware, and concurrent request handling. If multi-tenant mode is enabled, the broker sets up the appropriate PostgreSQL schema to isolate tenant data.

After the database connection is established, Diesel migrations run automatically to ensure the schema is up to date. The broker then checks the app_initialization table to determine if this is a first-time startup. On first run, it creates the admin role and an associated admin generator, writing the initial admin PAK (Prefixed API Key) to a temporary file for retrieval.

With the database ready, the broker initializes its runtime subsystems in sequence: the Data Access Layer (DAL), the encryption subsystem for webhook secrets, the event emission system for webhook dispatch, the audit logger for compliance tracking, and finally the five background task workers. If a configuration file is specified, it also starts a filesystem watcher for hot-reload capability.

API Layer

The API layer exposes a RESTful interface under the /api/v1 prefix, with all endpoints documented via OpenAPI annotations that generate Swagger documentation. Every request to an API endpoint passes through authentication middleware that extracts and validates the PAK from the Authorization header.

The authentication middleware performs verification against three possible identity types in order: first checking the admin role table, then the agents table, and finally the generators table. This lookup is optimized with partial database indexes on the pak_hash column that exclude soft-deleted records, enabling O(1) lookup performance rather than full table scans.

Upon successful authentication, the middleware injects an AuthPayload structure into the request context containing the authenticated identity’s type and ID. Individual endpoint handlers then perform authorization checks against this payload to determine if the caller has permission for the requested operation.

C4Component
    title Broker Request Processing (C4 Component)

    Person(client, "API Client", "Admin, Agent, or Generator")

    Component(auth, "Auth Middleware", "Axum Middleware", "Extracts PAK, verifies against DB, injects AuthPayload")
    Component(handler, "Route Handler", "Axum Handler", "Authorization check and business logic")
    Component(dal, "Data Access Layer", "Diesel ORM", "Type-safe database operations via accessor types")
    ComponentDb(pool, "Connection Pool", "r2d2", "Manages PostgreSQL connections (default: 50)")
    ComponentDb(db, "PostgreSQL", "Database", "Persistent storage")

    Rel(client, auth, "HTTP Request", "Authorization: Bearer {PAK}")
    Rel(auth, handler, "Authenticated request", "AuthPayload injected")
    Rel(handler, dal, "Data operations", "Type-safe queries")
    Rel(dal, pool, "Get connection", "")
    Rel(pool, db, "SQL", "TCP :5432")

The API is organized into resource-specific modules: agents, generators, stacks, deployment objects, templates, work orders, webhooks, diagnostics, health status, and admin operations. Each module defines its routes, request/response structures, and authorization requirements.

Data Access Layer

The DAL provides a clean abstraction over database operations, exposing specialized accessor types for each entity in the system. Rather than having a monolithic data access object, the DAL struct provides factory methods that return purpose-built accessor types: dal.agents() returns an AgentsDAL, dal.stacks() returns a StacksDAL, and so on for all twenty-two entity types in the system.

Each accessor type implements the standard CRUD operations appropriate for its entity, along with any specialized queries needed. For example, the AgentsDAL provides not only basic create, get, update, and delete operations, but also get_by_pak_hash for authentication lookups and filtered listing methods that support complex queries combining label and annotation filters.

The DAL uses Diesel’s query builder for type-safe SQL generation, ensuring that queries are checked at compile time rather than failing at runtime with SQL syntax errors. All operations use the connection pool, automatically returning connections when operations complete.

Soft deletion is implemented throughout the system using a deleted_at timestamp column. When an entity is “deleted,” this timestamp is set rather than removing the row, preserving the audit trail and enabling potential recovery. Most queries automatically filter out soft-deleted records.

Event Emission

The event emission system provides a database-centric approach to webhook dispatch. Rather than using an in-memory pub/sub bus, events are directly matched against webhook subscriptions and inserted into the webhook_deliveries table for processing by the webhook delivery worker.

When an event occurs (such as a deployment being applied or a work order completing), the emit_event() function queries the database for all webhook subscriptions whose event type pattern matches the event. Pattern matching supports exact matches like workorder.completed, wildcard suffixes like deployment.* that match any deployment-related event, and a catch-all * pattern that matches everything. For each matching subscription, a webhook delivery record is created in PENDING status for the webhook delivery worker to process.

Background Tasks

The broker runs five concurrent background tasks that handle various maintenance and processing duties:

Diagnostic Cleanup Task runs on a configurable interval (default: every 15 minutes) to expire pending diagnostic requests that have exceeded their timeout and delete completed, expired, or failed diagnostic records older than the configured maximum age (default: 1 hour). This prevents the database from accumulating stale diagnostic data.

Work Order Maintenance Task runs frequently (default: every 10 seconds) to handle work order lifecycle transitions. It moves work orders from RETRY_PENDING status back to PENDING when their backoff period has elapsed, allowing them to be claimed again. It also reclaims work orders that were claimed but never completed within the timeout period, returning them to PENDING for another agent to attempt.

Webhook Delivery Worker runs on a short interval (default: every 5 seconds) to process pending webhook deliveries. It fetches a batch of pending deliveries (default: 50), retrieves and decrypts the webhook URL and authentication header for each subscription, performs an HTTP POST with a 30-second timeout, and records the result. Failed deliveries are scheduled for retry with exponential backoff until they exceed the maximum retry count, at which point they are marked as dead.

Webhook Cleanup Task runs less frequently (default: hourly) to delete completed and dead webhook deliveries older than the retention period (default: 7 days), preventing unbounded growth of delivery history.

Audit Log Cleanup Task runs daily to delete audit log entries older than the configured retention period (default: 90 days), balancing compliance requirements against storage costs.

Audit Logger

The audit logger provides asynchronous, batched recording of security-relevant events for compliance and forensic purposes. Like the event bus, it is implemented as a singleton with an internal mpsc channel and background writer task.

Rather than writing each audit entry immediately to the database, which would create significant write amplification, the logger buffers entries and writes them in batches. The writer task accumulates entries up to a configurable batch size (default: 100) or until a flush interval elapses (default: 1 second), whichever comes first. This approach dramatically reduces database write pressure while maintaining a nearly real-time audit trail.

Audit entries capture the actor type (admin, agent, generator, or system), the actor’s ID, the action performed, the resource type affected, the resource’s ID, and additional contextual details. This information enables security teams to reconstruct the sequence of events leading to any system state.

Configuration Hot-Reload

When a configuration file is specified via the BROKKR_CONFIG_FILE environment variable, the broker can automatically detect and apply configuration changes without requiring a pod restart. The config watcher uses the notify crate to monitor the configuration file’s parent directory for filesystem events, detecting modifications through OS-level file watching.

When a change is detected, the watcher applies a configurable debounce period (default: 5 seconds, configurable via BROKKR_CONFIG_WATCHER_DEBOUNCE_SECONDS) to coalesce rapid successive changes into a single reload operation. The watcher can be disabled via BROKKR_CONFIG_WATCHER_ENABLED=false. Only certain configuration values support hot-reload: log level, diagnostic cleanup intervals, webhook delivery settings, and CORS configuration. Settings that affect initialization, like database connection parameters or TLS configuration, require a pod restart.

The admin API also exposes a manual reload endpoint at POST /api/v1/admin/config/reload that triggers an immediate configuration reload, useful when changes need to take effect immediately.

Agent Service Architecture

The agent is a Kubernetes-native component that runs inside target clusters, responsible for applying deployment objects to the cluster and reporting status back to the broker. It is designed to be resilient, continuing to operate with cached state during broker unavailability.

Startup and Initialization

When the agent starts, it first loads its configuration, which must include the broker URL, the agent’s name and cluster name, and its PAK for authentication. It then enters a readiness loop, repeatedly checking the broker’s health endpoint until it receives a successful response or exceeds its retry limit.

Once the broker is reachable, the agent verifies its PAK by calling the authentication endpoint. A successful response confirms the agent is properly registered and authorized. The agent then fetches its full details from the broker, including its assigned labels and annotations that determine which deployments it will receive.

With authentication confirmed, the agent initializes its Kubernetes client using in-cluster configuration (when running as a pod) or a specified kubeconfig path (for development). It validates connectivity by fetching the cluster’s API server version.

Finally, the agent starts an HTTP server on port 8080 for health checks and metrics, then enters its main control loop.

Main Control Loop

The agent’s control loop is implemented using Tokio’s select! macro to concurrently await multiple timer-based tasks. This design allows multiple activities to proceed in parallel while ensuring orderly shutdown when a termination signal is received.

C4Component
    title Agent Internal Architecture (C4 Component)

    Container_Boundary(agent, "Brokkr Agent") {
        Component(loop, "Control Loop", "Tokio select!", "Concurrent timer-based task dispatcher")
        Component(heartbeat, "Heartbeat", "Timer Task", "Sends periodic heartbeat to broker")
        Component(deployer, "Deployment Checker", "Timer Task", "Polls for deployment objects and reconciles")
        Component(workorder, "Work Order Processor", "Timer Task", "Claims and executes work orders")
        Component(healthcheck, "Health Monitor", "Timer Task", "Evaluates deployment health status")
        Component(diagnostics, "Diagnostics Handler", "Timer Task", "Processes on-demand diagnostic requests")
        Component(reconciler, "Reconciliation Engine", "kube-rs", "Server-side apply with ordering and ownership")
    }

    System_Ext(broker, "Broker API", "Central control plane")
    System_Ext(k8s, "Kubernetes API", "Target cluster")

    Rel(loop, heartbeat, "Triggers")
    Rel(loop, deployer, "Triggers")
    Rel(loop, workorder, "Triggers")
    Rel(loop, healthcheck, "Triggers")
    Rel(loop, diagnostics, "Triggers")
    Rel(deployer, reconciler, "Delegates apply/delete")
    Rel(heartbeat, broker, "POST heartbeat", "HTTPS")
    Rel(deployer, broker, "GET target-state", "HTTPS")
    Rel(reconciler, k8s, "Apply/Delete resources", "HTTPS :6443")
    Rel(healthcheck, k8s, "Query pod status", "HTTPS :6443")

Heartbeat Timer fires at a configurable interval (derived from the polling interval) to send a heartbeat to the broker, maintaining the agent’s “alive” status. The response includes updated agent details, allowing the broker to push configuration changes like label updates.

Deployment Check Timer fires at the configured polling interval to fetch the agent’s target state from the broker. The agent compares this desired state against what it has previously applied and performs reconciliation to converge them.

Work Order Timer polls for pending work orders assigned to the agent. Work orders represent transient operations like container image builds that don’t fit the declarative deployment model.

Health Check Timer (when enabled) periodically evaluates the health of deployments the agent has applied, checking pod status, container states, and conditions to produce health summaries reported back to the broker.

Diagnostics Timer polls for on-demand diagnostic requests, which allow administrators to collect debugging information from specific deployments.

Reconciliation Engine

The reconciliation engine is the heart of the agent, responsible for applying Kubernetes resources to achieve the desired state while handling the complexities of resource dependencies, conflicts, and failures.

When the agent receives deployment objects from the broker, it first parses the YAML content into Kubernetes DynamicObject instances. The objects are then sorted to apply cluster-scoped resources (Namespaces and CustomResourceDefinitions) before namespace-scoped resources, ensuring dependencies exist before resources that need them.

Before applying each resource, the agent injects standard metadata labels that link the resource back to Brokkr: the stack ID, a checksum of the YAML content for change detection, the deployment object ID, and the agent’s ID as the owner. This metadata enables the agent to identify which resources it manages and detect changes.

Application uses Kubernetes server-side apply with field ownership, which handles merge conflicts more gracefully than client-side apply. Before applying in earnest, resources are validated with a dry-run request to catch schema errors or policy violations early.

When deployment objects are removed (indicated by deletion markers), the agent deletes the corresponding resources from the cluster. However, it only deletes resources it owns, verified by checking the owner ID label. This prevents accidental deletion of resources created by other systems.

The reconciliation engine implements exponential backoff retry logic for transient failures like rate limiting (HTTP 429) or temporary server errors (5xx). Non-retryable errors like authorization failures (403) or resource not found (404) fail immediately to avoid wasting retry attempts.

Work Order Processing

Work orders handle operations that don’t fit the declarative resource model, such as building container images. The agent polls for pending work orders, claims one for exclusive processing, executes it, and reports the result.

For build work orders, the agent parses the YAML to extract Shipwright Build resources, applies them to the cluster, creates a BuildRun to trigger execution, and monitors the BuildRun status until completion. Build progress is checked every 5 seconds with a 15-minute overall timeout. On success, the agent reports the resulting image digest; on failure, it reports the error reason.

The work order system includes retry logic: when a work order fails with a retryable error, it is scheduled for retry with exponential backoff. Non-retryable errors are marked as permanently failed.

Health Monitoring

When deployment health monitoring is enabled, the agent periodically evaluates the health of resources it has applied. For each tracked deployment object, it queries for pods matching the deployment object ID label and analyzes their status.

The health checker examines pod phase, conditions, and container states to produce a health assessment. It specifically looks for problematic conditions like ImagePullBackOff, CrashLoopBackOff, OOMKilled, and various container creation errors. Based on its analysis, it assigns one of four health statuses: healthy (all pods running and ready), degraded (issues detected but not failed), failing (pod in Failed phase), or unknown (unable to determine status).

Health summaries include the count of ready versus total pods, a list of detected issues, and detailed resource information. This data is reported to the broker, which stores it for display in management interfaces and can trigger webhook notifications based on health changes.

Component Interaction Patterns

Understanding how the broker and agents interact is essential for operating and troubleshooting Brokkr deployments.

Deployment Flow

The deployment lifecycle begins when an administrator or CI/CD system (acting as a generator) creates a stack and adds deployment objects to it. The stack serves as a logical grouping with labels that determine which agents will receive its deployments.

When a deployment object is created, the broker’s matching engine evaluates all registered agents to find those whose labels satisfy the stack’s targeting requirements. For each matching agent, it creates an agent target record linking the agent to the stack.

sequenceDiagram
    participant Admin as Administrator
    participant Broker as Broker
    participant DB as PostgreSQL
    participant Agent as Agent
    participant K8s as Kubernetes

    Admin->>Broker: Create Stack with Labels
    Broker->>DB: Store Stack
    Admin->>Broker: Create Deployment Object
    Broker->>DB: Store Deployment Object
    Broker->>Broker: Find Matching Agents
    Broker->>DB: Create Agent Targets

    loop Every Polling Interval
        Agent->>Broker: Request Target State
        Broker->>DB: Query Agent's Deployments
        Broker-->>Agent: Deployment Objects
        Agent->>Agent: Compare Desired vs Actual
        Agent->>K8s: Apply Resources
        K8s-->>Agent: Result
        Agent->>Broker: Report Event
        Broker->>DB: Store Event
        Broker->>Broker: Dispatch Webhooks
    end

On the agent side, the deployment check timer fires and requests the agent’s target state from the broker. The broker returns all deployment objects from stacks the agent is targeting, along with sequence IDs that enable incremental synchronization. The agent compares this desired state against what it has applied, performs the necessary create, update, or delete operations in Kubernetes, and reports events back to the broker.

Event and Webhook Flow

Events flow from agents through the broker to external systems via webhooks. When an agent reports an event (success, failure, health change), the broker stores it in the database and emits it through the event emission system.

The event emitter queries the database for matching webhook subscriptions. For each match, it creates a webhook delivery record. The webhook delivery worker picks up these records and attempts HTTP delivery to the configured endpoints.

This decoupled architecture ensures that webhook delivery proceeds asynchronously with its own retry logic, separate from the initial event recording.

Performance Characteristics

The broker is designed to handle substantial load with modest resources. The API layer uses Axum’s async request handling to efficiently multiplex many concurrent connections onto a small number of OS threads. Connection pooling minimizes database connection overhead.

API Performance: Under typical conditions, API requests complete in under 50 milliseconds at the 95th percentile, with the broker capable of handling over 1,000 requests per second on a single instance.

Database Performance: Query latency is typically under 20 milliseconds, with indexed lookups for authentication completing in single-digit milliseconds. The connection pool defaults to 50 connections.

Agent Performance: Resource application completes in under 100 milliseconds per resource, with the reconciliation loop typically completing in under 1 second. Event reporting has sub-50ms latency to the broker.

For larger deployments, the broker can be horizontally scaled behind a load balancer since it maintains no in-memory state beyond caches—all persistent state lives in PostgreSQL.

Scaling Patterns

Horizontal Broker Scaling

The broker’s stateless design enables straightforward horizontal scaling. Multiple broker instances can run behind a load balancer, all connecting to the same PostgreSQL database. Each instance runs its own background tasks, but these are designed to be safe for concurrent execution through database-level coordination (e.g., work order claiming uses atomic updates to prevent double-claiming).

C4Container
    title Horizontal Scaling Pattern (C4 Container)

    System_Ext(lb, "Load Balancer", "Distributes API traffic")

    Container_Boundary(brokers, "Broker Instances") {
        Container(b1, "Broker 1", "Rust/Axum", "Stateless API + background tasks")
        Container(b2, "Broker 2", "Rust/Axum", "Stateless API + background tasks")
        Container(b3, "Broker 3", "Rust/Axum", "Stateless API + background tasks")
    }

    ContainerDb(db, "PostgreSQL Primary", "PostgreSQL", "All persistent state")
    ContainerDb(dbr, "PostgreSQL Replica", "PostgreSQL", "Read replicas (optional)")

    Rel(lb, b1, "Routes requests", "HTTPS")
    Rel(lb, b2, "Routes requests", "HTTPS")
    Rel(lb, b3, "Routes requests", "HTTPS")
    Rel(b1, db, "Read/Write", "SQL :5432")
    Rel(b2, db, "Read/Write", "SQL :5432")
    Rel(b3, db, "Read/Write", "SQL :5432")
    Rel(db, dbr, "Replication", "Streaming")

Agent Deployment

Each Kubernetes cluster runs one agent instance. The agent handles all deployments for that cluster, with no need for multiple agents per cluster. Agents operate independently with no inter-agent communication, simplifying the operational model.

For very large clusters or high deployment volumes, the agent’s polling interval can be tuned to balance responsiveness against API load.

Resource Requirements

These are conservative defaults suitable for small to medium deployments. Production deployments handling thousands of deployment objects or hundreds of agents should increase these limits based on observed resource utilization.

Multi-Tenancy

The broker supports multi-tenant deployments through PostgreSQL schema isolation. When configured with a schema name, the broker creates a dedicated schema and sets the connection’s search_path to use it for all queries. This provides data isolation between tenants sharing a PostgreSQL instance without requiring separate databases.

Schema names are validated to prevent SQL injection, accepting only alphanumeric characters and underscores, and requiring the name to start with a letter.

Component Implementation Details

This document provides detailed technical implementation information about each component in the Brokkr system. Understanding these implementation details helps operators debug issues, extend functionality, and optimize performance.

Broker Components

The broker is implemented in Rust using the Axum web framework with Tokio as the async runtime. It uses Diesel ORM with r2d2 connection pooling for PostgreSQL database access.

API Module

The API module implements the broker’s RESTful interface using Axum’s router and middleware patterns. Routes are organized hierarchically with authentication applied uniformly through middleware.

Route Organization

Routes are defined in submodules and merged into a unified router with authentication middleware applied:

#![allow(unused)]
fn main() {
pub fn routes(dal: DAL, cors_config: &Cors, reloadable_config: Option<ReloadableConfig>) -> Router<DAL> {
    let cors = build_cors_layer(cors_config);
    let api_routes = Router::new()
        .merge(agent_events::routes())
        .merge(agents::routes())
        .merge(stacks::routes())
        .merge(webhooks::routes())
        .merge(work_orders::routes())
        // Additional route modules...
        .layer(from_fn_with_state(dal.clone(), middleware::auth_middleware))
        .layer(cors);

    Router::new().nest("/api/v1", api_routes)
}
}

The authentication middleware intercepts every request, extracts the PAK from the Authorization header, verifies it against the database, and attaches an AuthPayload to the request extensions. Handlers can then access the authenticated identity to make authorization decisions.

CORS Configuration

CORS is configured dynamically based on settings, supporting three modes: allow all origins when "*" is specified, restrict to specific origins otherwise, with configurable methods, headers, and preflight cache duration.

#![allow(unused)]
fn main() {
pub struct Cors {
    pub allowed_origins: Vec<String>,
    pub allowed_methods: Vec<String>,
    pub allowed_headers: Vec<String>,
    pub max_age_seconds: u64,
}
}

Debugging

Enable debug logging with environment variables:

• RUST_LOG=debug - General debug output
• RUST_LOG=brokkr_broker=trace - Detailed broker tracing
• RUST_LOG=tower_http=debug - HTTP layer debugging

DAL (Data Access Layer) Module

The Data Access Layer provides structured access to the PostgreSQL database using Diesel ORM with r2d2 connection pooling. Each entity type has a dedicated accessor class that encapsulates all database operations.

Implementation Architecture

The DAL uses Diesel’s compile-time query checking and type-safe schema definitions. Connection management uses r2d2’s pooling with automatic connection recycling:

#![allow(unused)]
fn main() {
pub struct DAL {
    pool: Pool<ConnectionManager<PgConnection>>,
    schema: Option<String>,
}

impl DAL {
    pub fn agents(&self) -> AgentsDAL {
        AgentsDAL::new(self.pool.clone(), self.schema.clone())
    }

    pub fn stacks(&self) -> StacksDAL {
        StacksDAL::new(self.pool.clone(), self.schema.clone())
    }
    // Additional accessors...
}
}

Each accessor obtains a connection from the pool, optionally sets the PostgreSQL search path for multi-tenant schema isolation, executes the query, and returns the connection to the pool.

Multi-Tenant Schema Support

The DAL supports PostgreSQL schema isolation for multi-tenant deployments. When a schema is configured, every connection sets search_path before executing queries:

#![allow(unused)]
fn main() {
if let Some(schema) = &self.schema {
    diesel::sql_query(format!("SET search_path TO {}", schema))
        .execute(&mut conn)?;
}
}

Schema names are validated to prevent SQL injection before use.

Error Handling

Database errors are wrapped in a unified DalError type:

#![allow(unused)]
fn main() {
pub enum DalError {
    ConnectionPool(r2d2::Error),
    Query(diesel::result::Error),
    NotFound,
}
}

This provides consistent error handling across all database operations while preserving the underlying error details for debugging.

CLI Module

The CLI module handles command-line argument parsing, configuration loading, and service initialization. It supports running database migrations, starting the broker server, and administrative operations.

Configuration is loaded from environment variables using the BROKKR__ prefix with double underscore separators for nesting:

BROKKR__DATABASE__URL=postgres://user:pass@localhost/brokkr
BROKKR__DATABASE__SCHEMA=tenant_a
BROKKR__LOG__LEVEL=info
BROKKR__BROKER__WEBHOOK_DELIVERY_INTERVAL_SECONDS=5

Background Tasks Module

The broker runs several background tasks for maintenance operations:

Diagnostic Cleanup runs every 15 minutes (configurable) to remove diagnostic results older than 1 hour (configurable).

Work Order Maintenance runs every 10 seconds to process retry scheduling and detect stale claims.

Webhook Delivery runs every 5 seconds (configurable) to process pending webhook deliveries in batches of 50 (configurable).

Webhook Cleanup runs hourly to remove delivery records older than 7 days (configurable).

Audit Log Cleanup runs daily to remove audit entries older than 90 days (configurable).

Utils Module

The utils module provides shared functionality:

Event Emission provides database-centric webhook dispatch by matching events against subscriptions and inserting delivery records directly.

Audit Logger provides non-blocking audit logging with batched database writes (100 entries or 1 second flush).

Encryption implements AES-256-GCM encryption for webhook secrets with versioned format for algorithm upgrades.

PAK Controller generates and verifies Prefixed API Keys using SHA-256 hashing with indexed lookups.

Agent Components

The agent is implemented in Rust using Tokio for async operations and kube-rs for Kubernetes API interaction. It runs a continuous control loop that polls the broker and reconciles cluster state.

Broker Communication Module

The broker module handles all communication with the Brokkr Broker service using REST API calls with PAK authentication. The agent polls the broker at configurable intervals rather than maintaining persistent connections.

Communication Pattern

All broker communication uses HTTP requests with Bearer token authentication:

#![allow(unused)]
fn main() {
pub async fn fetch_deployment_objects(
    config: &Settings,
    client: &Client,
    agent: &Agent,
) -> Result<Vec<DeploymentObject>, Error> {
    let url = format!(
        "{}/api/v1/agents/{}/target-state",
        config.agent.broker_url, agent.id
    );

    let response = client
        .get(&url)
        .header("Authorization", format!("Bearer {}", config.agent.pak))
        .send()
        .await?;

    response.json().await
}
}

Key Endpoints

The agent communicates with these broker endpoints:

Retry Logic

Failed broker requests use exponential backoff with configurable parameters:

#![allow(unused)]
fn main() {
pub struct Agent {
    pub max_retries: u32,
    pub event_message_retry_delay: u64,
    // ...
}
}

Kubernetes Module

The Kubernetes module manages all interactions with the Kubernetes API using the kube-rs client library. It implements server-side apply for resource management with intelligent ordering and ownership tracking.

Server-Side Apply

Resources are applied using Kubernetes server-side apply, which provides declarative management with conflict detection:

#![allow(unused)]
fn main() {
pub async fn apply_k8s_objects(
    k8s_objects: &[DynamicObject],
    k8s_client: Client,
    patch_params: PatchParams,
) -> Result<(), Error> {
    for obj in k8s_objects {
        let api = get_api_for_object(&k8s_client, obj)?;
        let patch = Patch::Apply(obj);
        api.patch(&obj.name_any(), &patch_params, &patch).await?;
    }
    Ok(())
}
}

Resource Ordering

Resources are applied in priority order to respect dependencies:

1. Namespaces are applied first as other resources may depend on them
2. CustomResourceDefinitions are applied second as custom resources require their definitions
3. All other resources are applied after dependencies exist

This ordering prevents failures from missing dependencies during initial deployment.

Ownership Tracking

The agent tracks resource ownership using a combination of labels and annotations:

#![allow(unused)]
fn main() {
// Labels (used for selection and filtering)
pub static STACK_LABEL: &str = "k8s.brokkr.io/stack";
pub static DEPLOYMENT_OBJECT_ID_LABEL: &str = "brokkr.io/deployment-object-id";

// Annotations (used for metadata)
pub static CHECKSUM_ANNOTATION: &str = "k8s.brokkr.io/deployment-checksum";
pub static LAST_CONFIG_ANNOTATION: &str = "k8s.brokkr.io/last-config-applied";
pub static BROKKR_AGENT_OWNER_ANNOTATION: &str = "brokkr.io/owner-id";
}

Before deleting resources, the agent verifies ownership by checking the owner annotation to prevent removing resources managed by other systems. The checksum annotation enables detection of configuration drift.

Reconciliation

Full reconciliation applies the desired state and prunes resources that no longer belong:

#![allow(unused)]
fn main() {
pub async fn reconcile_target_state(
    objects: &[DynamicObject],
    client: Client,
    stack_id: &str,
    checksum: &str,
) -> Result<(), Error> {
    // Apply priority objects first
    apply_priority_objects(&objects, &client).await?;

    // Validate remaining objects
    validate_objects(&objects)?;

    // Apply all resources
    apply_all_objects(&objects, &client).await?;

    // Prune old resources with mismatched checksums
    prune_old_resources(&client, stack_id, checksum).await?;

    Ok(())
}
}

Error Handling

Kubernetes operations use retry logic for transient failures:

#![allow(unused)]
fn main() {
// Retryable HTTP status codes
const RETRYABLE_CODES: [u16; 4] = [429, 500, 503, 504];

// Retryable error reasons
const RETRYABLE_REASONS: [&str; 3] = [
    "ServiceUnavailable",
    "InternalError",
    "Timeout",
];
}

Exponential backoff prevents overwhelming a recovering API server.

Health Module

The health module exposes HTTP endpoints for Kubernetes probes and Prometheus metrics:

The health server runs on a configurable port (default: 8080) separately from the main control loop.

CLI Module

The agent CLI handles configuration loading and service initialization. Configuration uses the same environment variable pattern as the broker:

BROKKR__AGENT__BROKER_URL=https://broker.example.com:3000
BROKKR__AGENT__PAK=brokkr_BR...
BROKKR__AGENT__AGENT_NAME=production-cluster
BROKKR__AGENT__CLUSTER_NAME=prod-us-east-1
BROKKR__AGENT__POLLING_INTERVAL=10
BROKKR__AGENT__HEALTH_PORT=8080

Configuration Reference

Broker Configuration

#![allow(unused)]
fn main() {
pub struct Settings {
    pub database: Database,
    pub log: Log,
    pub broker: Broker,
    pub cors: Cors,
    pub telemetry: Telemetry,
}

pub struct Database {
    pub url: String,
    pub schema: Option<String>,
}

pub struct Broker {
    pub diagnostic_cleanup_interval_seconds: Option<u64>,  // default: 900
    pub diagnostic_max_age_hours: Option<i64>,              // default: 1
    pub webhook_encryption_key: Option<String>,
    pub webhook_delivery_interval_seconds: Option<u64>,     // default: 5
    pub webhook_delivery_batch_size: Option<i64>,           // default: 50
    pub webhook_cleanup_retention_days: Option<i64>,        // default: 7
    pub audit_log_retention_days: Option<i64>,              // default: 90
}

pub struct Cors {
    pub allowed_origins: Vec<String>,
    pub allowed_methods: Vec<String>,
    pub allowed_headers: Vec<String>,
    pub max_age_seconds: u64,
}
}

Agent Configuration

#![allow(unused)]
fn main() {
pub struct Settings {
    pub agent: Agent,
    pub log: Log,
    pub telemetry: Telemetry,
}

pub struct Agent {
    pub broker_url: String,
    pub pak: String,
    pub agent_name: String,
    pub cluster_name: String,
    pub polling_interval: u64,                    // default: 10
    pub kubeconfig_path: Option<String>,
    pub max_retries: u32,
    pub max_event_message_retries: usize,
    pub event_message_retry_delay: u64,
    pub health_port: Option<u16>,                 // default: 8080
    pub deployment_health_enabled: Option<bool>,  // default: true
    pub deployment_health_interval: Option<u64>,  // default: 60
}
}

Hot-Reload Configuration

The broker supports dynamic configuration reloading for certain settings:

Hot-reloadable (apply without restart):

• Log level
• CORS settings (origins, methods, headers, max-age)
• Webhook delivery interval and batch size
• Diagnostic cleanup settings

Static (require restart):

• Database URL and schema
• Webhook encryption key
• PAK configuration
• Telemetry settings

Trigger a manual reload via the admin API:

curl -X POST https://broker/api/v1/admin/config/reload \
  -H "Authorization: Bearer <admin-pak>"

When a configuration file is specified, the broker automatically watches it for filesystem changes with a 5-second debounce.

Component Lifecycle

Broker Startup Sequence

1. Configuration Loading - Parse environment variables and configuration files
2. Database Connection - Establish r2d2 connection pool to PostgreSQL
3. Migration Check - Verify database schema is current
4. Encryption Initialization - Load or generate webhook encryption key
5. Event Bus Initialization - Start event dispatcher with mpsc channel
6. Audit Logger Initialization - Start background writer with batching
7. Background Tasks - Spawn diagnostic cleanup, work order, webhook, and audit tasks
8. API Server - Bind to configured port and start accepting requests

Agent Startup Sequence

1. Configuration Loading - Parse environment variables
2. PAK Verification - Authenticate with broker and retrieve agent identity
3. Kubernetes Client - Initialize kube-rs client with in-cluster or kubeconfig credentials
4. Health Server - Start HTTP server for probes and metrics
5. Control Loop - Enter main loop with polling, health checks, and work order processing

Graceful Shutdown

Both components handle SIGTERM and SIGINT for graceful shutdown:

1. Stop accepting new requests
2. Complete in-flight operations
3. Flush pending data (audit logs, events)
4. Close database connections
5. Exit cleanly

Performance Considerations

Broker Optimization

Connection Pooling - r2d2 maintains a pool of database connections, avoiding connection establishment overhead for each request.

Batched Writes - Audit logs and webhook deliveries are batched to reduce database round trips.

Indexed Lookups - PAK verification uses indexed columns for O(1) authentication performance.

Async I/O - Tokio provides non-blocking I/O for high concurrency without thread-per-request overhead.

Agent Optimization

Incremental Fetching - Agents track processed sequence IDs to fetch only new deployment objects.

Parallel Apply - Independent resources can be applied concurrently within priority groups.

Connection Reuse - HTTP client maintains connection pools to the broker and Kubernetes API.

Efficient Diffing - Checksum-based change detection avoids unnecessary applies for unchanged resources.

Data Model Design

This document explains the design decisions and architectural philosophy behind Brokkr’s data model.

Entity Relationship Overview

classDiagram
    class stacks
    class agents
    class deployment_objects
    class agent_events
    class agent_targets
    class stack_labels
    class stack_annotations
    class agent_labels
    class agent_annotations
    class generators
    class stack_templates
    class work_orders

    generators "1" -- "0..*" stacks : owns
    stacks "1" -- "0..*" deployment_objects : contains
    stacks "1" -- "0..*" agent_targets : targeted by
    agents "1" -- "0..*" agent_events : reports
    agents "1" -- "0..*" agent_targets : targets
    deployment_objects "1" -- "0..*" agent_events : triggers
    stacks "1" -- "0..*" stack_labels : has
    stacks "1" -- "0..*" stack_annotations : has
    agents "1" -- "0..*" agent_labels : has
    agents "1" -- "0..*" agent_annotations : has
    generators "1" -- "0..*" stack_templates : owns
    stacks "1" -- "0..*" work_orders : targets

Design Philosophy

Immutability of Deployment Objects

Deployment objects are immutable after creation (except for soft deletion). This design decision ensures:

• Audit Trail: Every deployment can be traced back to its exact configuration
• Rollback Capability: Previous configurations are always available
• Consistency: No accidental modifications to deployed resources

Soft Deletion Strategy

All primary entities support soft deletion via deleted_at timestamps. This approach provides:

• Recovery: Accidentally deleted items can be restored
• Referential Integrity: Related data remains intact
• Historical Analysis: Past configurations and relationships are preserved
• Compliance: Audit requirements are met without data loss

Cascading Operations

The system implements intelligent cascading for both soft and hard deletes:

Soft Delete Cascades

• Generator → Stacks and Deployment Objects
• Stack → Deployment Objects (with deletion marker)
• Agent → Agent Events

Hard Delete Cascades

• Stack → Agent Targets, Agent Events, Deployment Objects
• Agent → Agent Targets, Agent Events
• Generator → (handled by foreign key constraints)

Key Architectural Decisions

Why Generators?

Generators represent external systems that create stacks and deployment objects. This abstraction:

• Provides authentication boundaries for automated systems
• Tracks which system created which resources
• Enables rate limiting and access control per generator
• Maintains audit trail of automated deployments

Why Agent Targets?

The many-to-many relationship between agents and stacks enables:

• Flexible deployment topologies
• Multi-cluster deployments
• Gradual rollouts
• Environment-specific targeting

Labels vs Annotations

Labels (single values):

• Used for selection and filtering
• Simple categorization
• Fast queries

Annotations (key-value pairs):

• Rich metadata
• Configuration hints
• Integration with external systems

Trigger Behavior

Stack Deletion Flow

sequenceDiagram
    participant User
    participant DB

    Note over User,DB: Soft Delete
    User->>DB: UPDATE stacks SET deleted_at = NOW()
    DB->>DB: Soft delete all deployment objects
    DB->>DB: Insert deletion marker object

    Note over User,DB: Hard Delete
    User->>DB: DELETE FROM stacks
    DB->>DB: Delete agent_targets
    DB->>DB: Delete agent_events
    DB->>DB: Delete deployment_objects

Deployment Object Protection

Deployment objects cannot be modified except for:

• Setting deleted_at (soft delete)
• Updating deletion markers

This is enforced by database triggers to ensure immutability.

Performance Considerations

Indexing Strategy

Key indexes are created on:

• Foreign keys for join performance
• deleted_at for filtering active records
• Unique constraints for data integrity
• Frequently queried fields (name, status)

Sequence IDs

Deployment objects use BIGSERIAL sequence_id for:

• Guaranteed ordering
• Efficient pagination
• Conflict-free concurrent inserts

Migration Strategy

The data model is managed through versioned SQL migrations in crates/brokkr-models/migrations/. Each migration:

• Is idempotent
• Includes both up and down scripts
• Is tested in CI/CD pipeline

For detailed field definitions and constraints, refer to the API documentation or the source code in crates/brokkr-models/.

Network Flows

Understanding the network architecture of a distributed system is essential for proper deployment, security hardening, and troubleshooting. This document provides a comprehensive analysis of the network traffic patterns between Brokkr components, including detailed port and protocol specifications, firewall requirements, and Kubernetes NetworkPolicy configurations.

Network Topology

Brokkr implements a hub-and-spoke network topology where the broker acts as the central coordination point. All agents initiate outbound connections to the broker—there are no inbound connections required to agents. This pull-based model simplifies firewall configuration and enables agents to operate behind NAT without special accommodations.

C4Context
    title Brokkr Network Topology (C4 System Context)

    Person(admin, "Platform Engineer", "Manages deployments and configuration")
    System_Ext(generator, "Generator / CI Pipeline", "Automated deployment source")
    System_Ext(webhook, "Webhook Endpoints", "External notification receivers")
    System_Ext(otlp, "OTLP Collector", "Distributed tracing (optional)")

    Enterprise_Boundary(broker_cluster, "Broker Cluster") {
        System_Ext(ingress, "Ingress Controller", "TLS termination")
        System(broker, "Broker Service", "Central API on :3000")
        SystemDb(db, "PostgreSQL", "Persistent state on :5432")
    }

    Enterprise_Boundary(target_cluster, "Target Cluster(s)") {
        System(agent, "Brokkr Agent", "Cluster-local operator")
        System_Ext(k8sapi, "Kubernetes API", "Cluster API on :6443")
    }

    Rel(admin, ingress, "Manages", "HTTPS :443")
    Rel(generator, ingress, "Deploys", "HTTPS :443")
    Rel(ingress, broker, "Forwards", "HTTP :3000")
    Rel(broker, db, "Queries", "TCP :5432")
    Rel(broker, webhook, "Delivers events", "HTTPS :443")
    Rel(broker, otlp, "Traces", "gRPC :4317")
    Rel(agent, broker, "Polls & reports", "HTTPS :3000")
    Rel(agent, k8sapi, "Applies resources", "HTTPS :6443")
    Rel(agent, otlp, "Traces", "gRPC :4317")

The diagram above illustrates the three primary network zones in a typical Brokkr deployment. External traffic from administrators and generators enters through an ingress controller, which terminates TLS and forwards requests to the broker service. The broker maintains persistent connectivity to its PostgreSQL database and sends outbound webhook deliveries to configured external endpoints. Meanwhile, agents in target clusters poll the broker for deployment instructions and interact with their local Kubernetes API servers to apply resources.

Connection Specifications

Complete Connection Matrix

The following table enumerates every network connection in the Brokkr system, including the source and destination components, ports, protocols, and whether each connection is required for basic operation.

Port Assignments

Brokkr uses a small number of well-defined ports. The broker service listens on port 3000 for all API traffic, including agent communication, administrator operations, and generator requests. This single-port design simplifies ingress configuration and firewall rules. Agents expose a health and metrics server on port 8080, which serves the /healthz, /readyz, /health, and /metrics endpoints used by Kubernetes liveness probes and Prometheus scraping.

The PostgreSQL database uses the standard port 5432. When deploying the bundled PostgreSQL instance via the Helm chart, this connection remains internal to the broker cluster. External PostgreSQL deployments may use different ports, which can be configured via the postgresql.external.port value.

OpenTelemetry tracing, when enabled, uses gRPC on port 4317 to communicate with OTLP collectors. This optional integration provides distributed tracing capabilities for debugging and performance analysis.

Broker Network Requirements

Inbound Traffic

The broker service accepts all inbound traffic on a single port, simplifying both service exposure and network policy configuration. The default configuration exposes port 3000, though this is rarely accessed directly in production. Instead, an ingress controller typically terminates TLS and forwards traffic to the broker.

The broker service supports three exposure methods through its Helm chart:

ClusterIP is the default service type, restricting access to within the Kubernetes cluster. This configuration is appropriate when agents run in the same cluster as the broker or when an ingress controller handles external access.

LoadBalancer creates a cloud provider load balancer that exposes the service directly to external traffic. While simpler to configure than ingress, this approach requires managing TLS termination separately and may incur additional cloud provider costs.

Ingress (recommended for production) delegates external access and TLS termination to a Kubernetes ingress controller. This approach integrates with cert-manager for automatic certificate management and provides flexible routing options.

Outbound Traffic

The broker initiates three types of outbound connections. Database connectivity to PostgreSQL is essential—the broker cannot operate without it. The Helm chart supports both bundled PostgreSQL (deployed as a subchart) and external PostgreSQL instances. For bundled deployments, the connection uses internal cluster DNS (brokkr-broker-postgresql:5432). External databases are configured via the postgresql.external values or by providing a complete connection URL through postgresql.existingSecret.

Webhook delivery represents the second outbound connection type. When webhooks are configured, the broker dispatches event notifications to external HTTP/HTTPS endpoints. The webhook delivery worker processes deliveries in batches, with the batch size and interval configurable via broker.webhookDeliveryBatchSize (default: 50) and broker.webhookDeliveryIntervalSeconds (default: 5). Failed deliveries are retried with exponential backoff.

OpenTelemetry tracing, when enabled, establishes gRPC connections to an OTLP collector. The collector endpoint is configured via telemetry.otlpEndpoint, and the sampling rate via telemetry.samplingRate. The Helm chart optionally deploys an OTel collector sidecar for environments where the main collector is not directly accessible.

Database Connectivity

The broker uses Diesel ORM with an r2d2 connection pool for PostgreSQL connectivity. Connection strings follow the standard PostgreSQL URI format:

postgres://user:password@host:5432/database

For production deployments, TLS should be enabled by appending ?sslmode=require or stronger modes to the connection string. The broker supports multi-tenant deployments through PostgreSQL schema isolation—the postgresql.external.schema value specifies which schema to use for data storage.

Agent Network Requirements

Outbound-Only Architecture

Agents operate with an outbound-only network model. They initiate all connections and require no inbound ports for their primary function. This design enables agents to operate behind restrictive firewalls and NAT gateways without special configuration—a critical feature for edge deployments and air-gapped environments.

The agent’s network requirements are minimal: connectivity to the broker API and the local Kubernetes API server. When metrics scraping is enabled, the agent also accepts inbound connections from Prometheus on port 8080.

Kubernetes API Access

Agents communicate with their local Kubernetes API server to apply and manage resources. When deployed via the Helm chart, agents use in-cluster configuration automatically—the Kubernetes client discovers the API server address from the cluster’s DNS and service account credentials.

The Helm chart creates RBAC resources that grant agents permission to manage resources across the cluster (when rbac.clusterWide: true) or within specific namespaces. The agent requires broad resource access for deployment management but can be restricted from sensitive resources like Secrets through the rbac.secretAccess configuration.

Broker Connectivity

Agents poll the broker at a configurable interval (default: 10 seconds, set via agent.pollingInterval). Each polling cycle fetches pending deployment objects and reports events for completed operations. The agent also sends deployment health status updates at a separate interval (default: 60 seconds, set via agent.deploymentHealth.intervalSeconds).

The broker URL is configured via the broker.url value in the agent’s Helm chart. For deployments where the agent and broker share a cluster, an internal URL like http://brokkr-broker:3000 provides optimal performance. For multi-cluster deployments, agents use the broker’s external URL with TLS: https://broker.example.com.

Authentication uses Prefixed API Keys (PAKs), which agents include in the Authorization header of every request. The PAK is generated when an agent is registered and should be provided via broker.pak in the Helm values or through a Kubernetes Secret.

TLS Configuration

Broker TLS Options

The broker supports three TLS configuration approaches, each suited to different deployment scenarios.

Ingress TLS Termination is the recommended approach for most production deployments. TLS terminates at the ingress controller, and internal traffic between the ingress and broker uses plain HTTP. This approach centralizes certificate management and integrates smoothly with cert-manager:

ingress:
  enabled: true
  className: 'nginx'
  tls:
    - secretName: brokkr-tls
      hosts:
        - broker.example.com

Direct TLS on Broker enables TLS termination at the broker itself, useful for deployments without an ingress controller or when end-to-end encryption is required. Enable via tls.enabled: true and provide certificates either through tls.existingSecret or inline via tls.cert and tls.key.

Cert-Manager Integration automates certificate provisioning when combined with ingress TLS. The Helm chart can configure cert-manager annotations to automatically request and renew certificates:

tls:
  enabled: true
  certManager:
    enabled: true
    issuer: 'letsencrypt-prod'
    issuerKind: 'ClusterIssuer'

Agent-to-Broker TLS

Agents should always communicate with the broker over HTTPS in production. The agent validates the broker’s TLS certificate using the system’s trusted certificate authorities. For deployments with self-signed or private CA certificates, the CA must be added to the agent’s trust store or mounted as a volume.

Kubernetes NetworkPolicy Configuration

NetworkPolicies provide defense-in-depth by restricting pod-to-pod and pod-to-external communication at the network layer. Both the broker and agent Helm charts include optional NetworkPolicy resources that implement least-privilege network access.

Broker NetworkPolicy

The broker’s NetworkPolicy allows inbound connections from configured sources and outbound connections to the database and webhook destinations. When networkPolicy.enabled: true, the generated policy includes:

Ingress rules permit connections on port 3000 from pods matching the selectors specified in networkPolicy.allowIngressFrom. If no selectors are specified, the policy allows connections from any pod in the same namespace. Metrics scraping can be separately controlled via networkPolicy.allowMetricsFrom.

Egress rules permit DNS resolution (UDP/TCP port 53), database connectivity (port 5432 to PostgreSQL pods or external IPs), and optionally webhook delivery (HTTPS port 443 to external IPs, excluding private ranges). The networkPolicy.allowWebhookEgress value controls whether webhook egress is permitted.

networkPolicy:
  enabled: true
  allowIngressFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
      podSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
  allowMetricsFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
  allowWebhookEgress: true

Agent NetworkPolicy

The agent’s NetworkPolicy restricts traffic to essential destinations only. When enabled, the policy permits:

Egress to DNS (UDP/TCP port 53) for name resolution.

Egress to the Kubernetes API server (ports 443 and 6443). The networkPolicy.kubernetesApiCidr value controls which IP ranges can receive this traffic—in production, restrict this to the actual API server IP for maximum security.

Egress to the broker on the configured port (default 3000). The destination can be specified as a pod selector for same-cluster deployments or as an IP block for external brokers.

Ingress for metrics (port 8080) when metrics.enabled: true and networkPolicy.allowMetricsFrom specifies allowed scrapers.

networkPolicy:
  enabled: true
  kubernetesApiCidr: "10.0.0.1/32"  # API server IP
  brokerEndpoint:
    podSelector:
      matchLabels:
        app.kubernetes.io/name: brokkr-broker
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: brokkr
  allowMetricsFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring

Firewall Configuration

Minimum Required Ports

Organizations deploying Brokkr must configure firewalls to permit the following traffic:

For the broker host:

For the agent host:

Cloud Provider Security Groups

Cloud deployments require security group configuration that permits the traffic flows described above. The following examples demonstrate typical configurations for major cloud providers.

AWS Security Groups:

For the broker, create an inbound rule permitting TCP port 3000 from the VPC CIDR (or specific agent security groups). Create outbound rules for PostgreSQL (port 5432 to the database security group) and webhook delivery (port 443 to 0.0.0.0/0 or specific webhook destinations).

GCP Firewall Rules:

Create an ingress rule with a target tag for broker instances, permitting TCP port 3000 from authorized sources. Create egress rules permitting port 5432 to the Cloud SQL instance and port 443 for webhooks.

Azure Network Security Groups:

Configure inbound rules for port 3000 from the virtual network address space. Configure outbound rules for database connectivity and webhook delivery similar to the AWS and GCP examples.

Troubleshooting Network Issues

Common Problems

Agent cannot reach broker: This typically manifests as repeated connection timeouts or DNS resolution failures in agent logs. Begin by verifying the broker URL is correct and resolvable from the agent pod. Use nslookup or dig to test DNS resolution. Check that no NetworkPolicy or firewall rule blocks egress on the broker port. For TLS connections, verify the agent trusts the broker’s certificate.

Broker cannot reach database: Database connectivity failures prevent the broker from starting. Verify the database host is resolvable and the credentials are correct. Check security group rules permit traffic on port 5432. For TLS-enabled database connections, verify the sslmode parameter is correctly configured.

Webhooks not being delivered: Check the broker logs for delivery errors, which indicate whether the issue is connection-related or response-related. Verify egress rules permit HTTPS traffic to external IPs. If NetworkPolicy is enabled, confirm allowWebhookEgress: true is set. Test webhook endpoint accessibility using curl from a pod in the broker namespace.

Metrics not being scraped: If Prometheus cannot reach the metrics endpoints, verify the ServiceMonitor is correctly configured and the Prometheus operator’s selector matches. Check that NetworkPolicy allows ingress from the monitoring namespace on the metrics port (3000 for broker, 8080 for agent).

Diagnostic Commands

The following commands help diagnose network connectivity issues:

# Test broker connectivity from agent pod
kubectl exec -it deploy/brokkr-agent -- wget -qO- http://brokkr-broker:3000/healthz

# Test database connectivity from broker pod
kubectl exec -it deploy/brokkr-broker -- nc -zv postgresql 5432

# List NetworkPolicies in the namespace
kubectl get networkpolicy -n brokkr

# Examine NetworkPolicy details
kubectl describe networkpolicy brokkr-broker -n brokkr

# Check agent logs for connection errors
kubectl logs deploy/brokkr-agent | grep -i "connection\|error\|timeout"

# Verify DNS resolution from within a pod
kubectl exec -it deploy/brokkr-agent -- nslookup brokkr-broker

Data Flows

This document traces the journey of data through the Brokkr system, from initial deployment creation through resource application in target clusters and event propagation to external systems. Understanding these flows is essential for debugging issues, optimizing performance, and building integrations with Brokkr.

Deployment Lifecycle

The deployment lifecycle encompasses the complete journey of a deployment object from creation through application on target clusters. This flow demonstrates Brokkr’s immutable, append-only data model and its approach to eventual consistency.

Creating a Deployment

Deployments begin their lifecycle when an administrator or generator creates a stack and submits deployment objects to it. The broker processes these submissions through several stages, ultimately targeting them to appropriate agents.

sequenceDiagram
    participant Client as Admin/Generator
    participant Broker as Broker API
    participant DB as PostgreSQL
    participant Match as Targeting Logic

    Client->>Broker: POST /api/v1/stacks
    Broker->>DB: INSERT stack
    DB-->>Broker: Stack created
    Broker-->>Client: Stack response (with ID)

    Client->>Broker: POST /api/v1/stacks/{id}/labels
    Broker->>DB: INSERT stack_labels
    Note over Broker: Labels used for agent targeting

    Client->>Broker: POST /api/v1/stacks/{id}/deployment-objects
    Broker->>DB: INSERT deployment_object
    Broker->>Match: Find matching agents
    Match->>DB: Query agents by labels
    DB-->>Match: Matching agents
    Match->>DB: INSERT agent_targets
    Broker-->>Client: Deployment object response

    Note over DB: Deployment objects are immutable after creation

The broker assigns each deployment object a sequence ID upon creation, establishing a strict ordering that agents use to process updates in the correct sequence. This sequence ID is monotonically increasing within each stack, ensuring that newer deployment objects always have higher sequence IDs than older ones. The combination of stack ID and sequence ID provides a reliable mechanism for agents to track which objects they have already processed.

When a deployment object is created, the broker does not immediately push it to agents. Instead, the targeting logic creates entries in the agent_targets table that associate the stack with eligible agents. Agents discover these targets during their next polling cycle and fetch the relevant deployment objects.

Agent Reconciliation

Agents continuously poll the broker and reconcile their cluster state to match the desired state defined by deployment objects. The reconciliation loop runs at a configurable interval, defaulting to 10 seconds.

sequenceDiagram
    participant Agent as Agent
    participant Broker as Broker API
    participant DB as PostgreSQL
    participant K8s as Kubernetes API
    participant Cluster as Cluster State

    loop Every polling interval (default: 10s)
        Agent->>Broker: GET /api/v1/agents/{id}/target-state
        Broker->>DB: Query targeted objects for agent
        DB-->>Broker: Deployment objects list
        Broker-->>Agent: Deployment objects (with sequence IDs)

        Agent->>Agent: Calculate diff (desired vs actual)

        alt New objects to apply
            Agent->>K8s: Apply resource (create/update)
            K8s->>Cluster: Resource created/updated
            K8s-->>Agent: Success/Failure
            Agent->>Broker: POST /api/v1/agents/{id}/events
            Broker->>DB: INSERT agent_event
        end

        alt Objects to delete (deletion markers)
            Agent->>K8s: Delete resource
            K8s->>Cluster: Resource deleted
            K8s-->>Agent: Success/Failure
            Agent->>Broker: POST /api/v1/agents/{id}/events
            Broker->>DB: INSERT agent_event
        end

        Agent->>Agent: Update local state cache
    end

The agent’s GET /api/v1/agents/{id}/target-state endpoint returns deployment objects the agent is responsible for, filtered to exclude objects already deployed (based on agent events). This optimization reduces payload size and processing time for agents managing large numbers of deployments.

During reconciliation, the agent uses Kubernetes server-side apply to create or update resources. This approach preserves fields managed by other controllers while allowing the agent to manage its own fields. The agent orders resource application to respect dependencies: Namespaces and CustomResourceDefinitions are applied before resources that depend on them.

After each successful operation, the agent reports an event to the broker. These events serve multiple purposes: they update the broker’s view of deployment state, they trigger webhook notifications to external systems, and they provide an audit trail of all operations.

Deployment Object States

Deployment objects follow an implicit lifecycle tracked through their presence, associated agent events, and deletion markers. The state model uses soft deletion to maintain a complete audit trail while supporting reliable cleanup.

stateDiagram-v2
    [*] --> Created: POST deployment-object
    Created --> Targeted: Agent matching
    Targeted --> Applied: Agent applies
    Applied --> Updated: New version created
    Updated --> Applied: Agent applies update
    Applied --> MarkedForDeletion: Deletion marker created
    MarkedForDeletion --> Deleted: Agent deletes
    Deleted --> [*]: Soft delete (retained)

Created indicates a deployment object exists in the database but has not yet been targeted to any agents. This state typically transitions quickly to Targeted as the broker processes agent matching.

Targeted means one or more agents are responsible for this deployment object. The agent_targets table records these associations, linking stacks to agents based on label matching.

Applied indicates the agent has successfully applied the resource to its cluster and reported an event confirming the operation. The agent event records the deployment object ID, timestamp, and outcome.

Updated represents a transitional state where a new deployment object with a higher sequence ID has been created for the same logical resource. The agent detects this during reconciliation and applies the update.

MarkedForDeletion occurs when a deletion marker deployment object is created. Deletion markers are deployment objects with a special flag indicating the agent should delete the referenced resource rather than apply it.

Deleted indicates the agent has removed the resource from the cluster. Both the original deployment object and the deletion marker remain in the database with deleted_at timestamps for audit purposes.

Deletion Flow

Deleting resources uses a marker pattern that ensures reliable cleanup even when agents are temporarily unavailable. Rather than immediately removing data, the broker creates a deletion marker that agents process during their normal reconciliation cycle.

sequenceDiagram
    participant Client as Admin
    participant Broker as Broker API
    participant DB as PostgreSQL
    participant Agent as Agent
    participant K8s as Kubernetes

    Client->>Broker: POST /api/v1/stacks/{id}/deployment-objects
    Note over Client,Broker: is_deletion_marker: true

    Broker->>DB: INSERT deployment_object (deletion marker)
    Broker-->>Client: Deletion marker created

    Note over Agent: Next polling interval
    Agent->>Broker: GET /api/v1/agents/{id}/target-state
    Broker-->>Agent: Includes deletion marker

    Agent->>Agent: Detect deletion marker
    Agent->>K8s: DELETE resource
    K8s-->>Agent: Deleted

    Agent->>Broker: POST /api/v1/agents/{id}/events
    Note over Broker: Event type: DELETED

    Note over DB: Both original and marker<br/>remain for audit trail

This approach has several advantages over immediate deletion. First, it provides reliable cleanup even when agents are offline—when they reconnect, they process accumulated deletion markers. Second, it maintains a complete history of what was deployed and when it was removed. Third, it allows for rollback by creating new deployment objects that restore deleted resources.

Event Flow

Events form the nervous system of Brokkr, propagating state changes from agents through the broker to external systems. The event system handles agent reports, webhook notifications, and audit logging through an asynchronous architecture designed for high throughput and reliability.

Event Architecture

The broker uses a database-centric approach to event emission. Rather than an in-memory pub/sub bus, events are directly matched against webhook subscriptions and inserted into the delivery queue. Audit logging operates independently through its own asynchronous channel.

flowchart LR
    subgraph Agent
        Apply[Apply Resource]
        Report[Event Reporter]
    end

    subgraph Broker
        API[API Handler]
        DB[(Database)]
        Emit[Event Emitter]
        Webhook[Webhook Worker]
        Audit[Audit Logger]
    end

    subgraph External
        Endpoints[Webhook Endpoints]
        Logs[Audit Logs]
    end

    Apply --> Report
    Report -->|POST /agents/{id}/events| API
    API --> DB
    API --> Emit
    Emit -->|Match subscriptions & insert deliveries| DB
    Webhook -->|Poll pending deliveries| DB
    Webhook --> Endpoints
    API --> Audit
    Audit --> Logs

When an event occurs, the emit_event() function queries the database for webhook subscriptions whose event type pattern matches the event. For each matching subscription, a delivery record is created in PENDING status. The webhook delivery worker then processes these records independently, ensuring webhook delivery doesn’t block API responses.

Audit logging uses a separate asynchronous channel with a 10,000-entry buffer. A background writer task batches entries (up to 100 per batch or every 1 second) for efficient database writes.

Agent Event Reporting

Agents report events to the broker after completing each operation. The POST /api/v1/agents/{id}/events endpoint accepts event data and persists it to the agent_events table.

The agent includes comprehensive metadata with each event: the deployment object ID, resource GVK (Group/Version/Kind), namespace and name, operation result, and any error messages. This data enables precise tracking of deployment state and troubleshooting of failures.

Events are processed synchronously in the API handler—the database insert must succeed before the endpoint returns. However, downstream processing (webhook delivery, audit logging) happens asynchronously through the event bus.

Webhook Delivery

Webhook subscriptions enable external systems to receive notifications when events occur in Brokkr. The delivery system prioritizes reliability through persistent queuing and automatic retries, with two delivery modes for different network topologies.

Broker Delivery (Default)

When no target_labels are specified on a subscription, the broker delivers webhooks directly. This is suitable for external endpoints accessible from the broker’s network.

sequenceDiagram
    participant EventBus as Event Bus
    participant DB as PostgreSQL
    participant Worker as Webhook Worker
    participant Endpoint as External Endpoint

    EventBus->>DB: Find matching subscriptions
    DB-->>EventBus: Subscription list
    EventBus->>DB: INSERT webhook_delivery (per subscription)

    loop Every 5 seconds
        Worker->>DB: SELECT pending deliveries (batch of 50)
        DB-->>Worker: Delivery batch

        par For each delivery
            Worker->>DB: Get subscription details
            Worker->>Worker: Decrypt URL and auth header
            Worker->>Endpoint: POST event payload
            alt Success (2xx)
                Endpoint-->>Worker: 200 OK
                Worker->>DB: Mark success
            else Failure
                Endpoint-->>Worker: Error
                Worker->>DB: Schedule retry (exponential backoff)
            end
        end
    end

The webhook worker runs as a background task, polling for pending deliveries every 5 seconds (configurable via broker.webhookDeliveryIntervalSeconds). Each polling cycle processes up to 50 deliveries (configurable via broker.webhookDeliveryBatchSize), enabling high throughput while controlling resource usage.

Agent Delivery

When target_labels are specified on a subscription, agents matching those labels deliver the webhooks. This enables webhooks to reach in-cluster endpoints (e.g., http://service.namespace.svc.cluster.local) that the broker cannot access due to network separation.

sequenceDiagram
    participant EventBus as Event Bus
    participant DB as PostgreSQL
    participant Agent as Agent (matching labels)
    participant Endpoint as In-Cluster Endpoint

    EventBus->>DB: Find matching subscriptions
    DB-->>EventBus: Subscription list
    EventBus->>DB: INSERT webhook_delivery with target_labels

    loop Every 10 seconds (agent heartbeat)
        Agent->>DB: Fetch pending deliveries matching my labels
        DB-->>Agent: Delivery batch

        par For each delivery
            Agent->>Agent: Decrypt URL and auth header
            Agent->>Endpoint: POST event payload
            alt Success (2xx)
                Endpoint-->>Agent: 200 OK
                Agent->>DB: Report success
            else Failure
                Endpoint-->>Agent: Error
                Agent->>DB: Report failure (schedules retry)
            end
        end
    end

Agent delivery requires the agent to have ALL labels specified in target_labels. For example, a subscription with target_labels: ["env:prod", "region:us"] will only be delivered by agents with both labels. This allows precise control over which agents handle which webhooks.

Encryption and Security

Delivery URLs and authentication headers are stored encrypted in the database using AES-256-GCM. The worker decrypts these values just before making the HTTP request, minimizing the time sensitive data exists in memory.

Retry Behavior

Failed deliveries are retried with exponential backoff. The first retry occurs after 2 seconds, the second after 4 seconds, then 8, 16, and so on. After exhausting the maximum retry count (configurable), deliveries are marked as “dead” and no longer retried. A cleanup task removes old delivery records after 7 days (configurable via broker.webhookCleanupRetentionDays).

Event Types

Brokkr emits events for various system activities, enabling external systems to react to state changes.

Webhook subscriptions can filter by event type using exact matches or wildcards (e.g., deployment.* matches all deployment events). This filtering reduces unnecessary network traffic and processing on the receiving end. See the Webhooks Reference for complete details on event payloads and subscription configuration.

Authentication Flows

All actors in Brokkr authenticate using Prefixed API Keys (PAKs) sent via the Authorization: Bearer header. The authentication middleware checks the PAK against three tables in order—admin roles, agents, and generators—to determine the identity type.

PAK Authentication

Prefixed API Keys (PAKs) provide secure, stateless authentication for agents. The PAK contains both an identifier and a secret component, enabling the broker to authenticate requests without storing plaintext secrets.

sequenceDiagram
    participant Agent as Agent
    participant Broker as Broker API
    participant Auth as Auth Middleware
    participant DB as PostgreSQL

    Note over Agent: Agent startup
    Agent->>Agent: Load PAK from config

    Agent->>Broker: GET /api/v1/agents/{id}/target-state
    Note over Agent,Broker: Authorization: Bearer {PAK}

    Broker->>Auth: Validate PAK
    Auth->>Auth: Parse PAK structure
    Auth->>Auth: Extract short token (identifier)
    Auth->>DB: Lookup agent by short token
    DB-->>Auth: Agent record with hash
    Auth->>Auth: Hash long token from request
    Auth->>Auth: Compare with stored hash

    alt Hashes match
        Auth-->>Broker: Agent identity
        Broker->>Broker: Continue with request
        Broker-->>Agent: Response
    else Invalid/Revoked
        Auth-->>Broker: Authentication failed
        Broker-->>Agent: 401 Unauthorized
    end

PAK structure follows a defined format: brokkr_BR{short_token}_{long_token}. The short token serves as an identifier that can be safely logged and displayed. The long token is the secret component—it is hashed with SHA-256 before storage, and the plaintext is never persisted.

When an agent authenticates, the middleware extracts the short token to locate the agent record, then hashes the provided long token and compares it with the stored hash. This constant-time comparison prevents timing attacks that could reveal information about valid tokens.

PAKs can be rotated through the POST /api/v1/agents/{id}/rotate-pak endpoint, which generates a new PAK and invalidates the previous one. The new PAK is returned only once—it cannot be retrieved later.

Admin Authentication

Administrators authenticate using PAKs stored in the admin_role table. Admin PAKs grant access to sensitive management operations that regular agents and generators cannot perform.

sequenceDiagram
    participant Admin as Admin Client
    participant Broker as Broker API
    participant Auth as Auth Middleware
    participant DB as PostgreSQL

    Admin->>Broker: POST /api/v1/admin/config/reload
    Note over Admin,Broker: Authorization: Bearer {PAK}

    Broker->>Auth: Validate PAK
    Auth->>Auth: Parse PAK structure
    Auth->>Auth: Extract short token
    Auth->>DB: Lookup admin by pak_hash
    DB-->>Auth: Admin record with hash
    Auth->>Auth: Hash long token and compare

    alt Valid PAK
        Auth-->>Broker: Admin identity (admin flag set)
        Broker->>Broker: Execute admin operation
        Broker-->>Admin: Response
    else Invalid PAK
        Auth-->>Broker: Authentication failed
        Broker-->>Admin: 401 Unauthorized
    end

Admin PAKs enable access to sensitive operations including configuration reload, audit log queries, agent management, and system health endpoints. The PAK is verified using the same mechanism as agent authentication—SHA-256 hashing with constant-time comparison.

Generator Authentication

Generators, typically CI/CD systems, authenticate using PAKs just like agents and admins. These keys enable automated deployment workflows while maintaining security boundaries.

sequenceDiagram
    participant Generator as Generator/CI
    participant Broker as Broker API
    participant Auth as Auth Middleware
    participant DB as PostgreSQL

    Generator->>Broker: POST /api/v1/stacks
    Note over Generator,Broker: Authorization: Bearer {PAK}

    Broker->>Auth: Validate PAK
    Auth->>Auth: Parse PAK structure
    Auth->>Auth: Extract short token
    Auth->>DB: Lookup generator by pak_hash
    DB-->>Auth: Generator record with hash
    Auth->>Auth: Hash long token and compare

    alt Valid PAK
        Auth-->>Broker: Generator identity
        Broker->>DB: Create stack (with generator_id)
        Broker-->>Generator: Stack created
    else Invalid PAK
        Auth-->>Broker: Authentication failed
        Broker-->>Generator: 401 Unauthorized
    end

Generators can create and manage stacks and deployment objects, but they cannot access admin endpoints or manage other generators. Resources created by a generator are associated with its identity, enabling audit tracking and future access control enhancements.

Work Order Flow

Work orders enable the broker to dispatch tasks to agents for execution. Unlike deployment objects which represent desired state, work orders represent one-time operations like container image builds or diagnostic commands.

sequenceDiagram
    participant Client as Admin/API
    participant Broker as Broker API
    participant DB as PostgreSQL
    participant Agent as Agent
    participant K8s as Kubernetes
    participant Build as Build System

    Client->>Broker: POST /api/v1/work-orders
    Broker->>DB: INSERT work_order (status: PENDING)
    Broker-->>Client: Work order created

    Note over Agent: Next polling interval
    Agent->>Broker: GET /api/v1/agents/{id}/work-orders/pending
    Broker->>DB: Query matching work orders
    DB-->>Broker: Work orders
    Broker-->>Agent: Work order details

    Agent->>Broker: POST /api/v1/work-orders/{id}/claim
    Broker->>DB: Update status: CLAIMED
    Broker-->>Agent: Claim confirmed

    Agent->>K8s: Create Build resource
    K8s->>Build: Execute build

    loop Monitor progress
        Agent->>K8s: Check build status
        K8s-->>Agent: Build status
    end

    Build-->>K8s: Build complete
    K8s-->>Agent: Success + artifacts
    Agent->>Broker: POST /api/v1/work-orders/{id}/complete
    Broker->>DB: Move to work_order_log
    Broker->>DB: DELETE work_order
    Broker-->>Agent: Acknowledged

Work orders support sophisticated targeting through three mechanisms: hard targets (specific agent IDs), label matching (agents with matching labels), and annotation matching (agents with matching annotations). An agent is eligible to claim a work order if it matches any of these criteria.

The claiming mechanism prevents multiple agents from processing the same work order. When an agent claims a work order, the broker atomically updates its status to CLAIMED and records the claiming agent’s ID. If the claim succeeds, the agent proceeds with execution; if another agent already claimed it, the claim fails.

Work Order States

Work orders transition through a defined set of states, with automatic retry handling for transient failures.

Successful completion moves the work order to the work_order_log table and deletes the original record. This design keeps the active work order table small while maintaining a complete history of completed operations.

Failed work orders may be retried depending on configuration. When a retryable failure occurs, the work order enters RETRY_PENDING status with a scheduled retry time based on exponential backoff. A background task runs every 10 seconds, resetting RETRY_PENDING work orders to PENDING once their retry time has elapsed.

Stale claims are automatically detected and reset. If an agent claims a work order but fails to complete it within the configured timeout (due to crash or network partition), the broker resets the work order to PENDING, allowing another agent to claim it.

Data Retention

Brokkr maintains extensive data for auditing, debugging, and compliance purposes. Different data types have different retention characteristics based on their importance and storage requirements.

Immutability Pattern

Deployment objects use an append-only pattern that preserves complete history. Objects are never modified after creation—updates create new objects with higher sequence IDs, and deletions create deletion markers rather than removing data. This approach enables precise audit trails and potential rollback to any previous state.

The deleted_at timestamp implements soft deletion across most entity types. Queries filter by deleted_at IS NULL by default, hiding deleted records from normal operations while preserving them for auditing. Special “include deleted” query variants provide access to the full history when needed.

Retention Policies

Background tasks run at regular intervals to enforce retention policies. The webhook cleanup task runs hourly, removing deliveries older than the configured retention period. The audit log cleanup task runs daily, removing entries beyond the retention window. Diagnostic results have a short retention period (1 hour by default) as they contain point-in-time debugging information.

Sequence ID Tracking

Agents track the highest sequence ID they have processed for each stack, enabling efficient incremental fetching. When an agent reconnects after downtime, it requests only objects with sequence IDs higher than its last processed value. This mechanism ensures reliable delivery of all updates while minimizing network traffic and processing time.

The GET /api/v1/agents/{id}/target-state endpoint leverages sequence tracking to return only unprocessed objects, reducing response size for agents managing large deployments.

Security Model

Security in Brokkr follows a defense-in-depth approach, implementing multiple layers of protection from network boundaries through application-level access controls. This document describes the trust boundaries, authentication mechanisms, authorization model, and operational security practices that protect Brokkr deployments.

Trust Boundaries

Brokkr defines four distinct security zones, each with different trust levels and access controls. Understanding these boundaries is essential for secure deployment architecture and incident response.

C4Context
    title Brokkr Security Trust Boundaries (C4 System Context)

    Person(admin, "Admin Users", "Untrusted zone — must authenticate")
    System_Ext(generator, "Generators / CI", "Untrusted zone — must authenticate")
    System_Ext(internet, "Internet / External", "Untrusted zone")

    Enterprise_Boundary(dmz, "DMZ / Edge") {
        System_Ext(ingress, "Ingress Controller", "TLS termination and routing")
    }

    Enterprise_Boundary(trusted, "Trusted Zone") {
        System(broker, "Broker Service", "Authenticated API with authorization")
        SystemDb(db, "PostgreSQL", "Encrypted secrets, hashed credentials")
    }

    Enterprise_Boundary(semi_trusted, "Semi-Trusted Zone (per cluster)") {
        System(agent, "Brokkr Agent", "Scoped access — own resources only")
        System_Ext(k8s, "Kubernetes API", "Target cluster API server")
    }

    Rel(admin, ingress, "API requests", "HTTPS")
    Rel(generator, ingress, "Deployment requests", "HTTPS")
    Rel(internet, ingress, "External traffic", "HTTPS")
    Rel(ingress, broker, "Authenticated traffic", "HTTP :3000")
    Rel(broker, db, "Read/Write", "TCP :5432")
    Rel(agent, broker, "Poll & report", "HTTPS :3000")
    Rel(agent, k8s, "Manage resources", "HTTPS :6443")

The Untrusted Zone encompasses all external entities: internet traffic, administrator clients, and CI/CD generators. Nothing in this zone receives implicit trust—every request must authenticate before accessing protected resources.

The DMZ provides the transition layer where external traffic enters the system. The ingress controller terminates TLS connections, validating certificates and establishing encrypted channels. This layer handles the initial security negotiation before traffic reaches application components.

The Trusted Zone contains the broker service and its PostgreSQL database. Components in this zone communicate over internal networks with mutual trust. The database accepts connections only from the broker, and the broker applies authentication and authorization before exposing data to external zones.

The Semi-Trusted Zone exists in each target cluster where agents operate. Agents receive scoped trust: they can access resources targeted specifically to them but cannot see resources belonging to other agents. This isolation prevents a compromised agent from affecting deployments on other clusters.

Security Principles

Four principles guide Brokkr’s security architecture:

Zero Trust by Default requires all external requests to authenticate. The broker’s API middleware rejects any request without valid credentials before route handlers execute. There are no anonymous endpoints except health checks.

Least Privilege restricts each identity to the minimum permissions necessary. Agents can only access deployment objects targeted to them through the agent_targets association. Generators can only manage stacks they created. This scoping limits the blast radius of credential compromise.

Defense in Depth implements multiple overlapping security controls. Even if an attacker bypasses network security, they face application-level authentication. Even with valid credentials, authorization limits accessible resources. Even with resource access, audit logging records all actions.

Immutable Audit Trail records every significant action in the system. Audit logs support only create and read operations—no updates or deletions are possible. This immutability ensures forensic evidence remains intact regardless of what an attacker does after gaining access.

Authentication Mechanisms

Brokkr implements three authentication mechanisms, each designed for different actor types and usage patterns.

Prefixed API Keys (PAKs)

PAKs serve as the primary authentication mechanism for agents and can also authenticate administrators and generators. The PAK design balances security with operational simplicity, enabling stateless authentication without storing plaintext secrets.

PAK Structure

Every PAK follows a structured format that embeds both an identifier and a secret component:

brokkr_BR{short_token}_{long_token}
       ^  ^            ^
       |  |            |
       |  |            +-- Long token (secret, used for verification)
       |  +--------------- Short token (identifier, safe to log)
       +------------------ Prefix (identifies key type)

The prefix brokkr_BR identifies this as a Brokkr PAK, distinguishing it from other credentials. The short token serves as an identifier that can appear in logs and error messages without compromising security. The long token is the secret component—it proves the holder possesses the original PAK.

A typical PAK looks like brokkr_BRabc123_xyzSecretTokenHere.... In this example, abc123 is the short token (identifier) and xyzSecretTokenHere... is the long token (secret).

Generation Process

PAK generation occurs when creating agents, generators, or admin credentials. The process uses cryptographically secure randomness from the operating system’s entropy source:

1. The system generates a random short token of configurable length. This token uses URL-safe characters and serves as the lookup key in the database.

2. A separate random long token is generated with sufficient entropy for cryptographic security. This token never leaves the generation response.

3. The long token is hashed using SHA-256, producing a fixed-size digest that represents the secret without revealing it.

4. The database stores only the hash. The original long token exists only in the complete PAK string returned to the caller.

5. The complete PAK is returned exactly once. If the caller loses it, a new PAK must be generated—the original cannot be recovered.

This design means the broker never stores information sufficient to reconstruct a PAK. A database breach reveals only hashes, which cannot authenticate without the original long tokens.

Verification Process

When a request arrives with a PAK, the authentication middleware executes a verification sequence:

sequenceDiagram
    participant Client
    participant Middleware as Auth Middleware
    participant DB as PostgreSQL

    Client->>Middleware: Request with PAK header
    Middleware->>Middleware: Parse PAK structure
    Middleware->>Middleware: Extract short token
    Middleware->>DB: Lookup by pak_hash index
    DB-->>Middleware: Record with stored hash

    Middleware->>Middleware: Hash long token from request
    Middleware->>Middleware: Constant-time hash comparison

    alt Hashes match
        Middleware-->>Client: Authenticated (continue to handler)
    else Hashes don't match
        Middleware-->>Client: 401 Unauthorized
    end

The middleware first parses the PAK to extract its components. Using the short token as an identifier, it performs an indexed database lookup to find the associated record. This lookup uses the pak_hash column index, ensuring O(1) performance regardless of how many credentials exist.

The middleware then hashes the long token from the incoming request using the same SHA-256 algorithm used during generation. Finally, it compares this computed hash with the stored hash. The comparison uses constant-time algorithms to prevent timing attacks that could reveal information about valid hashes.

If verification succeeds, the middleware populates an AuthPayload structure identifying the authenticated entity (agent, generator, or admin) and attaches it to the request for downstream handlers. If verification fails, the request is rejected with a 401 status before reaching any route handler.

PAK Security Properties

Admin Authentication

Administrative users authenticate using PAKs stored in the admin_role table. Admin PAKs grant access to sensitive management operations that regular agents and generators cannot perform.

Admin authentication follows the same verification process as agent authentication, but the resulting AuthPayload sets the admin flag to true. Route handlers check this flag to authorize access to admin-only endpoints.

# Example admin API call
curl -X POST https://broker.example.com/api/v1/admin/config/reload \
  -H "Authorization: Bearer brokkr_BR..."

Admin credentials should be treated with extreme care. A compromised admin PAK grants access to all system data, configuration changes, and audit logs. Organizations should implement additional controls around admin credential storage and usage, such as hardware security modules or secrets management systems.

Generator Authentication

Generators authenticate using PAKs stored in the generators table. Generator credentials enable CI/CD systems to create and manage deployments programmatically.

Generator permissions are scoped to resources they create. When a generator creates a stack, the broker records the generator’s ID with that stack. Future operations on the stack verify the requesting generator matches the owner. This scoping prevents one generator from modifying another’s deployments.

# Example generator API call
curl -X POST https://broker.example.com/api/v1/stacks \
  -H "Authorization: Bearer brokkr_BR..." \
  -H "Content-Type: application/json" \
  -d '{"name": "my-stack"}'

Generators cannot access admin endpoints regardless of their PAK. The authorization layer checks identity type before granting access to protected routes.

Authorization Model

Brokkr implements implicit role-based access control (RBAC) where roles are determined by authentication type rather than explicit role assignments.

Role Definitions

Endpoint Authorization

The following table summarizes which roles can access each API endpoint category:

Resource-Level Access Control

Beyond endpoint-level authorization, Brokkr enforces resource-level access control through database queries.

Agent Scope limits agents to resources explicitly targeted to them. When an agent requests deployment objects, the query joins through the agent_targets table:

SELECT do.* FROM deployment_objects do
JOIN agent_targets at ON at.stack_id = do.stack_id
WHERE at.agent_id = :requesting_agent_id
  AND at.deleted_at IS NULL
  AND do.deleted_at IS NULL;

This query structure ensures agents can never see deployment objects from stacks not targeted to them, regardless of what parameters they provide in API requests.

Generator Scope restricts generators to stacks they created:

SELECT * FROM stacks
WHERE generator_id = :requesting_generator_id
  AND deleted_at IS NULL;

Generators cannot list, read, or modify stacks created by other generators or through admin operations.

Credential Management

Storage

Brokkr stores credentials using appropriate protection levels based on sensitivity:

Webhook Secret Encryption

Webhook URLs and authentication headers may contain sensitive information like API keys or tokens. Brokkr encrypts these values at rest using AES-256-GCM, a modern authenticated encryption algorithm.

The encryption format includes version information for future algorithm upgrades:

version (1 byte) || nonce (12 bytes) || ciphertext || tag (16 bytes)

The current version byte (0x01) indicates AES-256-GCM encryption. The 12-byte nonce ensures each encryption produces unique ciphertext even for identical plaintexts. The 16-byte authentication tag detects any tampering with the encrypted data.

The encryption key is configured via the BROKKR__BROKER__WEBHOOK_ENCRYPTION_KEY environment variable as a 64-character hexadecimal string (representing 32 bytes). If no key is configured, the broker generates a random key at startup and logs a warning. Production deployments should always configure an explicit key to ensure encrypted data survives broker restarts.

PAK Rotation

PAK rotation replaces an entity’s authentication credential without disrupting its identity or permissions. The POST /api/v1/agents/{id}/rotate-pak endpoint (and similar endpoints for generators) generates a new PAK and invalidates the previous one atomically.

# Rotate an agent's PAK
curl -X POST https://broker/api/v1/agents/{id}/rotate-pak \
  -H "Authorization: Bearer <admin-pak>"
# Returns: new PAK (store immediately; cannot be retrieved again)

# Update agent configuration with new PAK
kubectl set env deployment/brokkr-agent BROKKR__AGENT__PAK=<new-pak>

After rotation, the old PAK becomes invalid immediately. Any requests using the old PAK receive 401 Unauthorized responses. The agent must be updated with the new PAK before its next broker communication.

Credential Revocation

Revoking access involves soft-deleting the associated entity. When an agent is deleted, its record remains in the database with a deleted_at timestamp, but authentication queries filter out deleted records:

# Revoke agent access
curl -X DELETE https://broker/api/v1/agents/{id} \
  -H "Authorization: Bearer <admin-pak>"

After revocation, the agent’s PAK becomes invalid immediately. Existing deployments remain in the target cluster (Brokkr doesn’t forcibly remove resources), but the agent can no longer fetch new deployments or report events.

Audit Logging

The audit logging system records significant actions for security monitoring, compliance, and forensic analysis. The system prioritizes completeness and immutability while minimizing performance impact on normal operations.

Architecture

The audit logger uses an asynchronous design to avoid blocking API request handlers:

API Handler → mpsc channel (10,000 buffer) → Background Writer → PostgreSQL

When an action occurs, the handler sends an audit entry to a bounded channel. A background task collects entries from this channel and writes them to PostgreSQL in batches. This design ensures audit logging never blocks request processing, even under high load.

The background writer batches entries for efficiency, flushing when the batch reaches 100 entries or after 1 second, whichever comes first. This batching reduces database round trips while ensuring entries are persisted within a predictable time window.

Audit Log Contents

Each audit log entry captures comprehensive context about the action:

Recorded Actions

The audit system records actions across several categories:

Authentication Events track credential usage and failures:

• auth.success - Successful authentication
• auth.failed - Failed authentication attempt
• pak.created - New PAK generated
• pak.rotated - PAK rotated
• pak.deleted - PAK revoked

Resource Lifecycle tracks creation, modification, and deletion:

• agent.created, agent.updated, agent.deleted
• stack.created, stack.updated, stack.deleted
• generator.created, generator.updated, generator.deleted
• webhook.created, webhook.updated, webhook.deleted

Operational Events record system activities:

• workorder.created, workorder.claimed, workorder.completed, workorder.failed
• config.reloaded - Configuration hot-reload triggered
• webhook.delivery_failed - Webhook delivery failure

Query Capabilities

The audit log API supports filtering to find relevant entries:

# Recent authentication failures
curl "https://broker/api/v1/admin/audit-logs?action=auth.failed&limit=100" \
  -H "Authorization: Bearer <admin-pak>"

# Actions by specific agent
curl "https://broker/api/v1/admin/audit-logs?actor_type=agent&actor_id=<uuid>" \
  -H "Authorization: Bearer <admin-pak>"

# All admin actions in a time range
curl "https://broker/api/v1/admin/audit-logs?actor_type=admin&from=2024-01-01T00:00:00Z" \
  -H "Authorization: Bearer <admin-pak>"

Filters support actor type and ID, action (with wildcard prefix matching), resource type and ID, and time ranges. Pagination limits responses to manageable sizes, with a maximum of 1000 entries per request.

Retention and Cleanup

Audit logs are retained for a configurable period (default: 90 days). A background task runs daily to remove entries older than the retention period. This cleanup prevents unbounded database growth while maintaining sufficient history for security investigations.

The retention period is configurable via broker settings, allowing organizations to meet their specific compliance requirements.

Kubernetes Security

Pod Security

Both broker and agent deployments configure security contexts that follow Kubernetes security best practices:

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 10001
  runAsGroup: 10001
  fsGroup: 10001

containerSecurityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false  # Configurable
  capabilities:
    drop:
      - ALL

Non-root execution prevents containers from running as the root user, limiting the impact of container escape vulnerabilities. The UID 10001 is arbitrary but consistent across components.

Privilege escalation prevention blocks processes from gaining additional capabilities after container startup, closing a common attack vector.

Capability dropping removes all Linux capabilities by default. Most applications don’t need capabilities, and removing them reduces attack surface.

AppArmor support is available when enabled via apparmor.enabled: true. AppArmor provides mandatory access control at the kernel level, further restricting what container processes can do.

Agent RBAC

The agent requires Kubernetes permissions to manage resources in target clusters. The Helm chart creates RBAC resources that implement least-privilege access:

Resource management access to core resources (pods, services, configmaps, deployments, and others) enables the agent to apply, update, and delete resources as part of deployment management. The agent requires create, get, list, watch, update, patch, and delete permissions on the resources it manages.

Shipwright access (when enabled) grants create, update, and delete permissions on Build and BuildRun resources for container image builds.

Optional secret access is disabled by default. When enabled via rbac.secretAccess.enabled, the agent can list and watch secrets. The rbac.secretAccess.readContents flag controls whether the agent can read actual secret values.

The RBAC configuration supports both namespace-scoped (Role/RoleBinding) and cluster-wide (ClusterRole/ClusterRoleBinding) permissions, configured via rbac.clusterWide.

Security Best Practices

Production Deployment Checklist

Before deploying Brokkr to production, verify these security configurations:

• TLS Everywhere: Enable TLS for all external connections via ingress or direct TLS
• Strong Secrets: Use cryptographically secure random values for all PAKs and encryption keys
• External Database: Use managed PostgreSQL with encryption at rest
• Secret Management: Store credentials in Kubernetes Secrets or external vault
• NetworkPolicy: Enable and configure network policies to restrict traffic
• RBAC: Use minimal required permissions for service accounts
• Pod Security: Enable pod security standards (restricted profile)
• Audit Logging: Enable and monitor audit logs
• Resource Limits: Set CPU/memory limits to prevent resource exhaustion
• Image Scanning: Scan container images for vulnerabilities before deployment

Monitoring for Security Events

Monitor these metrics and events for security-relevant activity:

Incident Response

Suspected Agent Compromise

If you suspect an agent’s credentials have been compromised:

1. Revoke immediately: Delete or disable the agent via the admin API
2. Review audit logs: Search for unusual actions by the agent’s actor_id
3. Inspect cluster: Review resources the agent may have created or modified
4. Rotate secrets: Generate new PAK if re-enabling the agent
5. Investigate: Determine how the compromise occurred

Suspected Broker Compromise

If you suspect the broker itself has been compromised:

1. Isolate: Remove external network access to the broker
2. Preserve evidence: Capture logs, database state, and container images
3. Rotate all credentials: Generate new PAKs for all agents, generators, and admins
4. Review webhooks: Check for unauthorized webhook subscriptions
5. Audit database: Look for unauthorized modifications to stacks or agents
6. Rebuild: Consider deploying fresh broker instances rather than cleaning compromised ones

Compliance Considerations

Data Protection

Brokkr implements several controls relevant to data protection regulations:

Access logging records all API access with actor identification, supporting accountability requirements.

Encryption at rest protects sensitive webhook data using AES-256-GCM.

Encryption in transit via TLS protects all external communications.

Data minimization ensures PAK secrets are never stored, only their hashes.

Retention controls automatically remove old audit logs after the configured period.

Regulatory Mapping

Container Image Publishing Strategy

This document explains Brokkr’s approach to building and publishing container images, including the reasoning behind our tagging strategy, multi-architecture support, and distribution decisions.

Publishing to GitHub Container Registry

Brokkr uses GitHub Container Registry (GHCR) as its primary container image registry for several reasons:

• Integrated authentication: Leverages GitHub’s existing access control and tokens
• Co-located with source: Images live alongside the code repository
• Cost effective: Free for public open-source projects
• Multi-architecture support: Full support for AMD64 and ARM64 platforms
• OCI compliance: Standards-compliant container registry

Public vs Private Distribution

Brokkr container images are published publicly despite being licensed under Elastic License 2.0. This decision balances openness with commercial protection:

Why Public?

• Easy evaluation: Users can try Brokkr without requesting access
• Community adoption: Lower barrier to entry encourages experimentation
• Source already available: The code is public on GitHub, so binaries being public is consistent
• Modern expectations: Developers expect to docker pull open-source-adjacent projects

License Protection Remains

Making images publicly accessible does not grant additional rights beyond the Elastic License 2.0:

• Users cannot offer Brokkr as a managed service
• Commercial restrictions still apply
• The license terms must be honored regardless of distribution method
• Source-available ≠ open-source

This approach follows the model used by Elasticsearch, Kibana, and other ELv2 projects.

Image Tagging Strategy

Brokkr uses multiple tagging strategies to support different use cases and deployment patterns.

Semantic Versioning for Releases

When a release is tagged (e.g., v1.2.3), multiple version tags are created:

• Full version (v1.2.3): Exact release identifier
• Minor version (v1.2): Latest patch within minor version
• Major version (v1): Latest minor within major version
• Latest (latest): Most recent stable release

Rationale: This allows users to choose their update cadence:

• Pin to v1.2.3 for no automatic updates
• Use v1.2 to get patch updates automatically
• Use v1 to track the major version
• Use latest for the bleeding edge (not recommended for production)

Commit SHA Tags

Every commit that triggers a build gets a SHA-based tag (e.g., sha-abc1234).

Rationale:

• Enables exact reproducibility
• Supports bisecting issues to specific commits
• Provides audit trail for security and compliance
• Immutable and unique across the repository’s lifetime

Branch Tags

Active branches get updated tags matching the branch name (e.g., main, develop).

Rationale:

• Development teams can track branch heads
• Continuous integration environments can use consistent tag names
• Useful for testing changes before they’re released

Pull Request Tags (Optional)

Pull requests can optionally generate tags (e.g., pr-123).

Rationale:

• Test changes in isolation before merging
• Share pre-merge builds with reviewers or QA
• Verify changes work in containerized environments

Tag Immutability

Not all tags are created equal. Understanding mutability is critical for production deployments:

Immutable Tags

These tags never change once created:

• Semantic versions: v1.2.3, v1.2, v1
• SHA tags: sha-abc1234

Mutable Tags

These tags are updated with new pushes:

• Branch names: main, develop, feature-xyz
• Latest: latest
• PR tags: pr-123 (if the PR is updated)

Production Deployment Recommendation

For production deployments, always use digest references instead of tags:

# Best - digest reference (immutable)
image: ghcr.io/colliery-io/brokkr-broker@sha256:9fc91fae...

# Good - semantic version (immutable)
image: ghcr.io/colliery-io/brokkr-broker:v1.2.3

# Acceptable - minor version (gets patches)
image: ghcr.io/colliery-io/brokkr-broker:v1.2

# Avoid - mutable tags
image: ghcr.io/colliery-io/brokkr-broker:latest

Using digests ensures that a deployment always references the exact image that was tested and approved, preventing unexpected changes from tag updates.

Multi-Architecture Support

All Brokkr images are built for both AMD64 and ARM64 architectures.

Why Multi-Architecture?

• Apple Silicon support: Developers on M1/M2/M3 Macs run ARM64 natively
• AWS Graviton: ARM64 instances offer better price/performance
• Edge computing: ARM64 is common in edge and IoT deployments
• Future-proofing: ARM64 adoption is accelerating across cloud providers

Implementation

Brokkr uses Docker Buildx to create multi-architecture manifest lists. When you pull an image, Docker automatically selects the correct architecture:

# Same command works on AMD64 and ARM64
docker pull ghcr.io/colliery-io/brokkr-broker:v1.0.0

The manifest list contains references to both architectures, and Docker pulls the appropriate one based on the host platform.

Local Development Considerations

Local builds with --load can only target a single architecture due to Docker limitations. The build tools automatically detect your platform and build for it:

• Apple Silicon (M1/M2/M3): Builds linux/arm64
• Intel/AMD systems: Builds linux/amd64

For multi-architecture builds, use --push to publish directly to the registry without loading locally.

Security Considerations

Image Content Security

Before any image is published:

• No embedded secrets: Credentials must never be baked into images
• Build argument hygiene: Ensure build args don’t leak sensitive data
• Minimal base images: Use slim Debian images to reduce attack surface
• Dependency scanning: Automated scanning for known vulnerabilities (planned)

Authentication and Authorization

• GitHub Actions: Uses built-in GITHUB_TOKEN with automatic permissions
• Manual publishing: Requires Personal Access Token with write:packages scope
• Token security: Tokens stored as GitHub secrets, never committed to source

Public Registry Security

Public images mean:

• Anyone can pull and inspect the images
• Image layers and content are visible
• Security through obscurity does not apply

Therefore, all security must be built into the application itself, not rely on image privacy.

Automated vs Manual Publishing

Automated Publishing (Preferred)

GitHub Actions workflows handle publishing for:

• Release tags → semantic version tags
• Main branch pushes → main tag and SHA tags
• Develop branch pushes → develop tag and SHA tags

Benefits:

• Consistent build environment
• Multi-architecture builds guaranteed
• Security scanning integrated
• Audit trail in GitHub Actions logs

Manual Publishing

Manual publishing is supported for:

• Testing the build process
• Emergency releases
• Local development verification

Manual builds use the angreal build multi-arch command with authentication to GHCR.

Future Enhancements

Planned improvements to the publishing strategy:

• Image signing: Cosign signatures for supply chain security
• SBOM generation: Software Bill of Materials for dependency tracking
• Vulnerability scanning: Automated Trivy or Grype integration
• Helm chart publishing: OCI-based Helm chart distribution via GHCR
• Image attestations: Build provenance and SLSA compliance

Related Documentation

• Container Images Reference - Repository URLs and tag formats

Work Orders & Build System

This document explains the design rationale behind Brokkr’s work order system and its integration with Shipwright for container builds. Work orders extend Brokkr beyond Kubernetes manifest distribution into task orchestration across clusters.

Why Work Orders Exist

Brokkr’s core workflow — pushing YAML deployment objects to agents — solves the “distribute manifests” problem well. But real-world operations need more:

• Build container images on specific clusters (near source registries, with GPU access, etc.)
• Run one-off tasks like database migrations, cleanup scripts, or certificate rotations
• Execute maintenance operations that need to happen on specific clusters

Work orders address these needs without changing the pull-based deployment model. They’re a parallel system: deployment objects are continuous desired state; work orders are discrete, one-time tasks.

Design Decisions

Pull-Based, Not Push-Based

Like deployment objects, work orders use a pull model. The broker doesn’t push commands to agents. Instead:

1. Admin creates a work order in the broker
2. Broker determines eligible agents via targeting rules
3. Agents poll for pending work orders they’re authorized to claim
4. One agent claims and executes the work order
5. Agent reports completion (success or failure)

This preserves Brokkr’s key property: clusters behind firewalls work without inbound connections.

Single-Claim Semantics

A work order is claimed by exactly one agent. This prevents duplicate execution — you don’t want two clusters running the same database migration. The claiming process is atomic: the work order transitions from PENDING to CLAIMED in a single database transaction.

Retry with Exponential Backoff

When a work order fails, the system supports automatic retries:

PENDING → CLAIMED → (failure) → RETRY_PENDING → (backoff expires) → PENDING → CLAIMED

The backoff follows the formula: 2^retry_count × backoff_seconds

After max_retries failures, the work order moves to the log with success: false.

Why Retryability Is Caller-Declared

The retryable flag on completion is set by the agent, not the broker. This is intentional: only the agent knows whether the failure was transient (network timeout, resource contention) or permanent (invalid YAML, missing permissions). Non-retryable failures skip the retry loop entirely.

Stale Claim Detection

If an agent claims a work order but crashes before completing it, the work order would be stuck in CLAIMED forever. The broker’s maintenance task detects stale claims:

• Each work order has a claim_timeout_seconds (default: 3600 = 1 hour)
• If a claimed work order exceeds its timeout, it’s released back to PENDING
• The maintenance task runs every 10 seconds

State Machine

                    ┌──────────┐
                    │ PENDING  │◄─────────────────────┐
                    └────┬─────┘                      │
                         │ claim()                    │ backoff expires
                         ▼                            │
                    ┌──────────┐              ┌───────┴──────┐
                    │ CLAIMED  │              │RETRY_PENDING │
                    └────┬─────┘              └──────────────┘
                         │                            ▲
              ┌──────────┴──────────┐                 │
              ▼                     ▼                  │
     ┌────────────────┐    ┌────────────────┐         │
     │ complete_success│    │complete_failure │────────┘
     └────────┬───────┘    └────────┬───────┘  (retryable + retries left)
              │                     │
              ▼                     ▼
     ┌────────────────┐    ┌────────────────┐
     │  WORK_ORDER_LOG│    │  WORK_ORDER_LOG│
     │  success=true  │    │  success=false │
     └────────────────┘    └────────────────┘

Targeting

Work orders support three targeting mechanisms: hard targets (explicit agent UUIDs), label matching, and annotation matching. At least one must be specified — the API rejects work orders with no targeting.

All three mechanisms use OR logic: an agent is eligible if it matches any of the specified targets, labels, or annotations. When multiple types are combined, they’re also OR’d together — agent UUID-1 OR any agent with label builder:true can claim the work order.

Design note: This differs from template matching, which uses AND logic (a template’s labels must all be present on the stack). The rationale: work orders need to reach at least one capable agent (OR is permissive), while template matching needs to ensure full compatibility with a stack (AND is restrictive). See Template Matching & Rendering for comparison.

See the Work Orders Reference for the complete targeting API with request body examples.

Shipwright Build Integration

The primary built-in work order type is build, which integrates with Shipwright for container image builds.

The agent claims a build work order, applies the Shipwright Build resource, creates a BuildRun, watches it until completion, and reports back (including the image digest on success). See Container Builds with Shipwright for the complete operational guide.

Why Shipwright?

Shipwright provides a Kubernetes-native build abstraction:

• No privileged containers — uses unprivileged build strategies (Buildah, Kaniko, etc.)
• Cluster-native — builds run as Kubernetes resources, leverage cluster scheduling
• Strategy flexibility — swap between Buildah, Kaniko, ko, S2I without changing build definitions
• Build caching — strategies can cache layers for faster rebuilds

Builds have a 15-minute timeout — if the BuildRun doesn’t complete in time, it’s reported as failed. These timeouts are compile-time constants in the agent, not configurable at runtime.

Work Order Log

Completed work orders (success or failure) are moved from the active work_orders table to the work_order_log table. This is an immutable audit trail:

The log records: original ID, work type, timing, claiming agent, success/failure, retry count, and result message.

Custom Work Orders

Beyond builds, work orders support arbitrary YAML:

{
  "work_type": "custom",
  "yaml_content": "apiVersion: batch/v1\nkind: Job\nmetadata:\n  name: db-migration\nspec:\n  template:\n    spec:\n      containers:\n      - name: migrate\n        image: myapp/migrate:v1\n      restartPolicy: Never"
}

Custom work orders apply the YAML to the cluster and monitor completion. This enables arbitrary Kubernetes jobs, CronJobs, or any other resource to be orchestrated through Brokkr.

Related Documentation

• Work Orders Reference — API endpoints and data model
• Container Builds with Shipwright — setup and usage guide
• Data Flows — work order lifecycle in context
• Architecture — system-level view

Template Matching & Rendering

This document explains the design of Brokkr’s template system — how templates are matched to stacks and rendered into deployment objects via the Tera engine with JSON Schema validation.

The Problem Templates Solve

Without templates, every deployment requires hand-crafted YAML. In a multi-environment setup (staging, production, 10 regional clusters), you end up with nearly-identical YAML files that differ only in replica counts, image tags, resource limits, and environment variables. This leads to:

• Duplication drift — copies fall out of sync
• Manual errors — wrong value in wrong environment
• No validation — any YAML is accepted, mistakes caught only at apply time

Templates solve this with parameterized YAML, schema validation, and matching rules that prevent production templates from being instantiated into staging stacks.

Architecture

The template system has three components:

┌─────────────────────────────────────────────┐
│ Template                                     │
│  ┌─────────────────┐  ┌──────────────────┐  │
│  │ Tera Content     │  │ JSON Schema      │  │
│  │ (YAML with       │  │ (parameter       │  │
│  │  placeholders)   │  │  validation)     │  │
│  └─────────────────┘  └──────────────────┘  │
│  ┌─────────────────────────────────────────┐ │
│  │ Labels + Annotations (matching rules)   │ │
│  └─────────────────────────────────────────┘ │
└──────────────────────┬──────────────────────┘
                       │ instantiate(parameters)
                       ▼
              ┌────────────────┐
              │ Validate params │ ← JSON Schema check
              │ Match stack     │ ← Label/annotation check
              │ Render Tera     │ ← Variable substitution
              └────────┬───────┘
                       ▼
              ┌────────────────┐
              │ Deployment     │
              │ Object         │ (rendered YAML)
              └────────────────┘

Rendering Pipeline

Step 1: Parameter Validation

Before touching the template, Brokkr validates the provided parameters against the JSON Schema. This catches issues early:

// Schema requires service_name (string, 1-63 chars) and replicas (integer, 1-20)
// Caller provides: {"service_name": "", "replicas": 100}
// Result: Validation fails — service_name too short, replicas exceeds maximum

If validation fails, the request is rejected with a detailed error message explaining which constraints were violated. No YAML is rendered.

Step 2: Stack Matching

Templates can have labels and annotations that restrict which stacks they can be instantiated into. This is a safety mechanism — it prevents, for example, a production-hardened template from being used in a development stack where the configuration doesn’t make sense.

The matching is strict AND logic: the stack must have every label and annotation the template requires. A template with no labels/annotations matches any stack (universal). Extra labels on the stack are ignored — it only matters that the required ones are present.

For the complete matching rules table with examples, see the Templates Reference.

Step 3: Tera Rendering

With validation and matching passed, Brokkr renders the Tera template:

1. Creates a Tera context from the JSON parameters (flat key-value mapping)
2. Adds the parameter values to the context
3. Renders the template content through Tera
4. The resulting string is the final Kubernetes YAML

Tera supports rich template features:

• Variables: {{ service_name }}
• Conditionals: {% if enable_monitoring %}...{% endif %}
• Loops: {% for port in ports %}...{% endfor %}
• Filters: {{ name | upper }}, {{ x | default(value="fallback") }}
• Math: {{ replicas * 2 }}

Step 4: Deployment Object Creation

The rendered YAML is stored as a new deployment object in the target stack, along with provenance metadata:

• rendered_deployment_objects.template_id — which template was used
• rendered_deployment_objects.template_version — which version
• rendered_deployment_objects.template_parameters — the exact parameters provided

This provenance enables re-rendering with different parameters or auditing what parameters produced a given deployment.

Versioning Design

Why Version Templates?

Templates evolve: you add a liveness probe, change resource defaults, introduce a new parameter. Without versioning, updating a template could silently change the meaning of existing deployments.

Brokkr’s versioning ensures:

• Existing deployments are immutable — a deployment object rendered from template v1 stays as-is even after v2 is created
• New instantiations use latest — when you instantiate a template, you always get the newest version
• Provenance is preserved — you can trace any deployment back to the exact template version and parameters

Version Lifecycle

Version 1 ────── Version 2 ────── Version 3 (latest)
    │                 │
    │                 ├── Deployment Object A (rendered from v2)
    │                 └── Deployment Object B (rendered from v2)
    │
    ├── Deployment Object C (rendered from v1, still lives in cluster)
    └── Deployment Object D (rendered from v1)

Updating a template via PUT /api/v1/templates/{id} creates a new version. The version number auto-increments. Old versions remain in the database.

System Templates vs. Generator Templates

Templates have two ownership modes:

System Templates (generator_id = NULL)

• Created by admins
• Visible to all generators and admins
• Represent organization-wide standards (e.g., “standard web service”, “batch job”)
• Cannot be modified by generators

Generator Templates (generator_id = UUID)

• Created by a specific generator
• Visible only to the owning generator and admins
• Represent pipeline-specific templates (e.g., templates tailored for a particular CI/CD system)
• Can be modified by the owning generator

This separation allows centralized governance (admin-managed standards) while still allowing individual teams (generators) to create specialized templates.

Why Tera?

Brokkr chose Tera over alternatives:

Tera was chosen because:

• Native Rust integration (no FFI or subprocess)
• Familiar Jinja2-like syntax widely known by DevOps engineers
• Rich filter and function library
• Compile-time syntax validation via add_raw_template

Why JSON Schema?

JSON Schema was chosen for parameter validation because:

• Industry standard — widely understood, extensive tooling
• Declarative — schema defines constraints, engine enforces them
• Rich constraints — types, ranges, patterns, required fields, enums, string lengths
• Self-documenting — the description field in each property serves as parameter documentation
• Client-side validation — CI/CD systems can validate parameters before hitting the API

Related Documentation

• Templates Reference — API endpoints and data model
• Using Stack Templates — how-to guide
• Tutorial: Standardized Deployments — step-by-step tutorial
• Data Model — template entity relationships

Technical Reference

This section provides comprehensive technical documentation for Brokkr, including API references, code documentation, and implementation details.

API Documentation

The Brokkr API provides a comprehensive set of endpoints for managing deployments, agents, and system configuration. You can explore the complete API documentation in two ways:

Interactive API Documentation

Our interactive Swagger UI provides a complete reference of all available endpoints, including:

• Detailed request/response schemas
• Authentication requirements
• Example requests and responses
• Interactive testing interface

View Interactive API Documentation

API Endpoints Overview

The Brokkr API is organized into the following main sections:

Health Check

• GET /healthz - Liveness check
• GET /readyz - Readiness check
• GET /api/v1/health - Detailed health diagnostics

Agent Management

• POST /api/v1/agents - Create a new agent
• GET /api/v1/agents - List all agents
• GET /api/v1/agents/{agent_id} - Get agent details
• PUT /api/v1/agents/{agent_id} - Update an agent
• DELETE /api/v1/agents/{agent_id} - Delete an agent

Stack Management

• POST /api/v1/stacks - Create a new stack
• GET /api/v1/stacks - List all stacks
• GET /api/v1/stacks/{stack_id} - Get stack details
• PUT /api/v1/stacks/{stack_id} - Update a stack
• DELETE /api/v1/stacks/{stack_id} - Delete a stack

Deployment Object Management

• POST /api/v1/stacks/{stack_id}/deployment-objects - Create a deployment object
• GET /api/v1/stacks/{stack_id}/deployment-objects - List deployment objects

Event Management

• POST /api/v1/events - Report a deployment event
• GET /api/v1/events - List events

For detailed information about each endpoint, including request/response formats and examples, please refer to the Interactive API Documentation.

Rust API Documentation

The Brokkr codebase is written in Rust and provides a rich set of APIs for both the Broker and Agent components. You can explore the complete Rust API documentation here:

View Rust API Documentation

The Rust documentation includes:

• Detailed module and function documentation
• Type definitions and trait implementations
• Code examples and usage patterns
• Implementation details for core components

Key Components

Broker

• API Server implementation
• Database layer
• Event system
• Authentication and authorization

Agent

• Kubernetes client
• Broker communication
• State management
• Deployment orchestration

For detailed information about the Rust implementation, including module structure and function documentation, please refer to the Rust API Documentation.

API Reference

Brokkr provides a comprehensive REST API for managing deployments, agents, stacks, templates, and work orders across your Kubernetes clusters.

Interactive API Documentation

The Brokkr broker includes an interactive Swagger UI that provides complete API documentation with:

• All available endpoints with request/response schemas
• Authentication requirements for each endpoint
• Try-it-out functionality for testing endpoints
• Example requests and responses

Access Swagger UI at: http://<broker-url>/swagger-ui

OpenAPI spec available at: http://<broker-url>/docs/openapi.json

API Overview

All API endpoints are prefixed with /api/v1/ and require authentication via PAK (Pre-Authenticated Key) in the Authorization header.

Authentication

# All requests require a PAK in the Authorization header
curl -H "Authorization: Bearer <your-pak>" http://localhost:3000/api/v1/...

There are three types of PAKs:

• Admin PAK: Full access to all endpoints
• Agent PAK: Access to agent-specific endpoints (target state, events, heartbeat)
• Generator PAK: Access to create deployment objects for assigned stacks

Core Resources

Stacks

Stacks are collections of Kubernetes resources managed as a unit.

Agents

Agents run in Kubernetes clusters and apply deployment objects.

Templates

Reusable stack templates with Tera templating and JSON Schema validation.

Work Orders

Transient operations like container builds routed to agents.

Generators

External systems that create deployment objects.

Other Endpoints

Webhooks

Admin

Health Monitoring

Health Endpoints

The broker exposes health endpoints (not under /api/v1/):

See Health Endpoints for details.

Error Handling

All API errors return JSON with an error field:

{
  "error": "Description of what went wrong"
}

Common HTTP status codes:

• 400 - Bad request (invalid input)
• 401 - Unauthorized (missing or invalid PAK)
• 403 - Forbidden (valid PAK but insufficient permissions)
• 404 - Not found
• 422 - Unprocessable entity (validation failed)
• 500 - Internal server error

Rate Limiting

The API does not currently implement rate limiting. For production deployments, consider placing a reverse proxy in front of the broker.

CLI Reference

Brokkr provides two command-line binaries: brokkr-broker for the central management server and brokkr-agent for the cluster-side agent. Both accept configuration via files, environment variables, and command-line flags.

brokkr-broker

The broker binary runs the central API server and provides administrative commands for managing agents, generators, and keys.

Commands

brokkr-broker serve

Starts the broker HTTP server on 0.0.0.0:3000.

brokkr-broker serve

Endpoints exposed:

brokkr-broker create agent

Creates a new agent record and generates its initial PAK.

brokkr-broker create agent --name <name> --cluster-name <cluster>

Flags:

Output:

Agent created successfully:
ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890
Name: production-us-east
Cluster: us-east-1-prod
Initial PAK: brokkr_BRx9y2Kq_A1B2C3D4E5F6G7H8I9J0K1L2

Important: The PAK is only displayed once. Store it securely.

brokkr-broker create generator

Creates a new generator for CI/CD integration.

brokkr-broker create generator --name <name> [--description <desc>]

Flags:

Output:

Generator created successfully:
ID: f8e7d6c5-b4a3-2109-8765-432109876543
Name: github-actions
Initial PAK: brokkr_BRy8z3Lp_M1N2O3P4Q5R6S7T8U9V0W1X2

brokkr-broker rotate admin

Generates a new admin PAK, replacing the existing one.

brokkr-broker rotate admin

The new PAK is printed to stdout. The old PAK immediately stops working.

brokkr-broker rotate agent

Rotates an agent’s PAK.

brokkr-broker rotate agent --uuid <uuid>

Flags:

brokkr-broker rotate generator

Rotates a generator’s PAK.

brokkr-broker rotate generator --uuid <uuid>

Flags:

brokkr-agent

The agent binary runs in each target Kubernetes cluster and polls the broker for deployment objects to apply.

Commands

brokkr-agent start

Starts the agent process.

brokkr-agent start

Health endpoints exposed on agent.health_port (default: 8080):

Configuration

Both binaries read configuration from the same layered system:

1. Embedded defaults (default.toml compiled into the binary)
2. Configuration file (optional, path passed at startup or via BROKKR_CONFIG_FILE)
3. Environment variables (prefix: BROKKR__, separator: __)

See the Configuration Guide for all available settings and the Environment Variables Reference for the complete variable listing.

Exit Codes

Examples

# Start broker with custom config file
BROKKR_CONFIG_FILE=/etc/brokkr/broker.toml brokkr-broker serve

# Start broker with environment overrides
BROKKR__DATABASE__URL=postgres://user:pass@db:5432/brokkr \
BROKKR__LOG__LEVEL=info \
BROKKR__LOG__FORMAT=json \
  brokkr-broker serve

# Create an agent and capture its PAK
brokkr-broker create agent --name prod-1 --cluster_name us-east-1 2>&1 | grep "Initial PAK"

# Start agent with environment config
BROKKR__AGENT__BROKER_URL=https://broker.example.com \
BROKKR__AGENT__PAK=brokkr_BRx9y2Kq_A1B2C3D4E5F6G7H8I9J0K1L2 \
BROKKR__AGENT__AGENT_NAME=prod-1 \
BROKKR__AGENT__CLUSTER_NAME=us-east-1 \
  brokkr-agent start

Environment Variables Reference

Complete listing of all environment variables supported by Brokkr. All variables use the BROKKR__ prefix with double underscores (__) as nested separators.

Configuration precedence (highest wins): Environment variables > Config file > Embedded defaults.

Database

Logging

The log level is hot-reloadable — changes take effect without restarting.

Broker

Agent

PAK (Pre-Authentication Key) Generation

Generated PAK format: {prefix}_{short_token_prefix}{short_token}_{long_token}

Example: brokkr_BR3rVsDa_GK3QN7CDUzYc6iKgMkJ98M2WSimM5t6U8

CORS

Note: Array-type CORS settings accept comma-separated strings when set via environment variables (e.g., BROKKR__CORS__ALLOWED_ORIGINS=http://a.com,http://b.com).

Telemetry (OpenTelemetry)

Base Settings

Broker-Specific Overrides

These override the base telemetry settings for the broker component only. If unset, the base value is used.

Agent-Specific Overrides

Configuration File and Hot-Reload

These environment variables control the configuration system itself and are not part of the BROKKR__ namespace:

Hot-Reloadable Settings

The following settings can be changed at runtime without restarting the broker (via config file change or admin API):

• log.level
• broker.diagnostic_cleanup_interval_seconds
• broker.diagnostic_max_age_hours
• broker.webhook_delivery_interval_seconds
• broker.webhook_delivery_batch_size
• broker.webhook_cleanup_retention_days
• cors.allowed_origins
• cors.max_age_seconds

Related Documentation

• Configuration Guide — configuration system overview
• CLI Reference — command-line usage
• Multi-Tenancy — schema-based isolation
• Monitoring & Observability — telemetry and metrics setup

Templates Reference

Stack templates provide reusable, parameterized Kubernetes manifests with JSON Schema validation. This reference covers the data model, API endpoints, Tera template syntax, and matching rules.

Data Model

StackTemplate

Constraints:

• Unique combination of (generator_id, name, version)
• version must be >= 1
• name, template_content, and parameters_schema cannot be empty
• checksum is auto-computed on creation

Template Types

RenderedDeploymentObject

When a template is instantiated, Brokkr records the provenance:

API Endpoints

List Templates

GET /api/v1/templates

Auth: Admin sees all templates. Generator sees system templates + own templates.

Response: 200 OK — StackTemplate[]

Create Template

POST /api/v1/templates

Auth: Admin only (creates system templates). Generators can also create templates (owned by the generator).

Request body:

{
  "name": "web-service",
  "description": "Standard web service template",
  "template_content": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: {{ service_name }}\nspec:\n  replicas: {{ replicas }}",
  "parameters_schema": "{\"type\": \"object\", \"required\": [\"service_name\"], \"properties\": {\"service_name\": {\"type\": \"string\"}, \"replicas\": {\"type\": \"integer\", \"default\": 2}}}"
}

Validation:

• Template content is validated for Tera syntax errors
• Parameters schema is validated as a valid JSON Schema
• Name must be 1-255 characters

Response: 201 Created — StackTemplate

Get Template

GET /api/v1/templates/{id}

Auth: Admin or owning generator.

Response: 200 OK — StackTemplate

Update Template (New Version)

PUT /api/v1/templates/{id}

Auth: Admin or owning generator.

Updating a template creates a new version. The previous version remains available. The version number auto-increments.

Request body:

{
  "description": "Standard web service template v2",
  "template_content": "...",
  "parameters_schema": "..."
}

Note: The name field is not accepted on update — it is preserved from the existing template.

Response: 200 OK — StackTemplate (with incremented version)

Delete Template

DELETE /api/v1/templates/{id}

Auth: Admin or owning generator.

Response: 204 No Content

Template Labels

GET    /api/v1/templates/{id}/labels
POST   /api/v1/templates/{id}/labels          Body: "label-string"
DELETE /api/v1/templates/{id}/labels/{label}

Auth: Admin or owning generator.

Labels control which stacks a template can be instantiated into. See Matching Rules.

Template Annotations

GET    /api/v1/templates/{id}/annotations
POST   /api/v1/templates/{id}/annotations     Body: {"key": "k", "value": "v"}
DELETE /api/v1/templates/{id}/annotations/{key}

Auth: Admin or owning generator.

Instantiate Template

POST /api/v1/stacks/{stack_id}/deployment-objects/from-template

Auth: Admin or owning generator (for the stack).

Request body:

{
  "template_id": "uuid-of-template",
  "parameters": {
    "service_name": "frontend",
    "replicas": 3
  }
}

Process:

1. Fetches the latest version of the template
2. Validates parameters against the JSON Schema
3. Checks template-to-stack matching rules (labels/annotations)
4. Renders the Tera template with the provided parameters
5. Creates a deployment object with the rendered YAML
6. Records the rendered deployment object provenance

Response: 200 OK — DeploymentObject[]

Tera Template Syntax

Templates use the Tera engine. Key features:

Variable Substitution

name: {{ service_name }}
replicas: {{ replicas }}
image: {{ repository }}:{{ tag }}

Conditionals

{% if enable_hpa %}
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ service_name }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ service_name }}
  minReplicas: {{ min_replicas }}
  maxReplicas: {{ max_replicas }}
{% endif %}

Loops

env:
{% for key, value in env_vars %}
- name: {{ key }}
  value: "{{ value }}"
{% endfor %}

Filters

See the Tera documentation for the complete filter and function reference.

JSON Schema for Parameters

The parameters_schema field accepts a standard JSON Schema document. Commonly used features:

Type Constraints

{
  "type": "object",
  "properties": {
    "replicas": { "type": "integer", "minimum": 1, "maximum": 100 },
    "name": { "type": "string", "minLength": 1, "maxLength": 63 },
    "debug": { "type": "boolean" },
    "cpu": { "type": "string", "pattern": "^[0-9]+m$" }
  }
}

Required Fields

{
  "type": "object",
  "required": ["name", "image"],
  "properties": {
    "name": { "type": "string" },
    "image": { "type": "string" }
  }
}

Defaults

{
  "properties": {
    "replicas": { "type": "integer", "default": 2 },
    "port": { "type": "integer", "default": 8080 }
  }
}

Enum Values

{
  "properties": {
    "environment": {
      "type": "string",
      "enum": ["development", "staging", "production"]
    }
  }
}

Matching Rules

Templates with labels or annotations are restricted to stacks with matching metadata. This prevents production-only templates from being instantiated into staging stacks.

Rules:

1. Template with no labels and no annotations → matches any stack (universal)
2. Template with labels → stack must have all of the template’s labels
3. Template with annotations → stack must have all of the template’s annotations (key-value match)
4. Template with both → stack must satisfy both label AND annotation requirements

Example:

Template with labels ["env:production", "tier:frontend"]:

• Stack with ["env:production", "tier:frontend", "region:us"] → matches (has all required)
• Stack with ["env:production"] → no match (missing tier:frontend)
• Stack with ["env:staging", "tier:frontend"] → no match (wrong env)

Versioning Behavior

• Creating a template starts at version 1
• Updating via PUT auto-increments the version
• Instantiation always uses the latest version of the template
• Old versions remain in the database for provenance
• Deployment objects rendered from old versions are not affected by template updates
• The rendered_deployment_objects table records which version was used

Related Documentation

• Using Stack Templates — how-to guide for template workflows
• Tutorial: Standardized Deployments — step-by-step tutorial
• API Reference — complete API documentation
• Data Model — entity relationships

Work Orders

Work orders are transient operations that Brokkr routes to agents for execution. Unlike deployment objects which represent persistent desired state, work orders are one-time operations such as container builds, tests, or backups.

Concepts

Work Order vs Deployment Object

Work Order Lifecycle

PENDING -> CLAIMED -> (success) -> work_order_log
                  \-> (failure) -> RETRY_PENDING -> PENDING (retry)
                                \-> work_order_log (max retries)

1. PENDING: Work order created, waiting for an agent to claim
2. CLAIMED: Agent has claimed the work order and is executing
3. RETRY_PENDING: Execution failed, waiting for retry backoff
4. Completed: Moved to work_order_log (success or max retries exceeded)

Targeting

Work orders are routed to agents using the same targeting mechanisms as stacks:

• Direct agent IDs: Route to specific agents by UUID
• Labels: Route to agents with matching labels (OR logic)
• Annotations: Route to agents with matching annotations (OR logic)