Technical Architecture

Brokkr is a deployment orchestration platform designed to manage Kubernetes resources across multiple clusters from a central control plane. This document provides a comprehensive technical overview of Brokkr’s architecture, explaining how its components are implemented, how they interact, and the design decisions that shape its behavior.

System Overview

C4Context
    title Brokkr System Context (C4 Level 1)

    Person(engineer, "Platform Engineer", "Manages deployments and configuration")
    System_Ext(cicd, "CI/CD Pipeline", "Automated deployment source (Generator)")

    System(brokkr, "Brokkr", "Deployment orchestration platform for multi-cluster Kubernetes")

    System_Ext(k8s, "Kubernetes Clusters", "Target clusters where resources are deployed")
    SystemDb(pg, "PostgreSQL", "Persistent state storage")
    System_Ext(webhooks, "External Systems", "Receive event notifications via webhooks")

    Rel(engineer, brokkr, "Manages stacks and deployments", "brokkr CLI / SDKs / REST API")
    Rel(cicd, brokkr, "Creates stacks and deployment objects", "brokkr CLI / SDKs / REST API")
    Rel(brokkr, pg, "Stores state", "SQL")
    Rel(brokkr, k8s, "Deploys resources via agents", "HTTPS")
    Rel(brokkr, webhooks, "Sends event notifications", "HTTPS")

At its core, Brokkr follows a hub-and-spoke architecture where a central broker service coordinates deployments across multiple agent instances. The broker maintains the desired state of all deployments in a PostgreSQL database, while agents running in target Kubernetes clusters continuously poll the broker to fetch their assigned work and reconcile the actual cluster state to match the desired state.

C4Container
    title Brokkr Container Diagram (C4 Level 2)

    Person(engineer, "Platform Engineer", "Manages deployments")
    System_Ext(cicd, "CI/CD Pipeline", "Generator identity")

    Container_Boundary(brokkr, "Control Plane") {
        Container(broker, "Broker", "Rust, Axum", "REST API, background tasks, webhook dispatch")
        Container(console, "Operator Console", "Rust, Leptos/WASM (brokkr-web)", "Read-only browser view; compiled into the broker binary and served at its root URL")
        ContainerDb(db, "PostgreSQL", "PostgreSQL 15+", "Stacks, deployment objects, events, audit logs")
    }

    Container_Boundary(cluster, "Target Kubernetes Cluster (one per cluster)") {
        Container(agent, "Agent", "Rust, kube-rs", "Polls broker, reconciles cluster state")
        System_Ext(k8s, "Kubernetes API", "Target cluster")
    }

    System_Ext(webhooks, "Webhook Endpoints", "External notification receivers")

    Rel(engineer, broker, "Manages", "brokkr CLI / SDKs / REST API")
    Rel(engineer, console, "Watches", "Browser, broker root URL")
    Rel(console, broker, "Reads fleet, deployments, telemetry", "Same-origin REST /api/v1")
    Rel(cicd, broker, "Deploys", "brokkr CLI / SDKs / REST API")
    Rel(agent, broker, "Polls for state, reports events", "HTTPS REST + internal WebSocket")
    Rel(broker, db, "Reads/writes state", "SQL :5432")
    Rel(agent, k8s, "Applies/deletes resources", "HTTPS :6443")
    Rel(broker, webhooks, "Delivers events", "HTTPS")

The Operator Console is drawn as a separate container because operators experience it as one, but it deploys as part of the broker: the brokkr-web crate compiles to WebAssembly, the bundle is embedded into the broker binary at build time (the published image is built this way), and the broker serves it as a fallback behind the API on the same port. It is an ordinary read-only client of the v1 REST API with no privileged channel of its own, and the page the broker serves carries an ephemeral read-only credential so it needs no configuration — which makes network reach its authentication boundary. See Using the Operator Console and Read-Only Console Authentication. The unsupported examples/ui-slim React demo is a separate consumer sample, neither built nor shipped with the broker.

The broker and database form the control plane and typically run together; each agent runs inside the cluster it manages, reaching the broker over outbound connections only. Alongside REST polling, the agent maintains an internal WebSocket channel to the broker for low-latency pushes and telemetry streaming — REST remains the load-bearing path (see Internal Broker↔Agent WS Channel).

This pull-based model was chosen deliberately: agents need only outbound connectivity (no inbound path into firewalled clusters), they keep operating from cached state during temporary broker unavailability, and they control their own reconciliation pace based on cluster capacity and load.

Broker Service Architecture

The broker is implemented as an asynchronous Rust service built on the Axum web framework with the Tokio runtime providing the async execution environment. When the broker starts, it initializes several interconnected subsystems that work together to provide the complete platform functionality.

Initialization Sequence

The broker follows an ordered startup sequence so every dependency is ready before it accepts requests. Configuration is loaded first from environment variables and files, establishing database connection parameters, authentication settings, and operational parameters like polling intervals.

A PostgreSQL connection pool is then established via the r2d2 connection manager with Diesel as the ORM, defaulting to 50 connections to cover background tasks, middleware, and concurrent requests. In multi-tenant mode, the broker sets up the appropriate PostgreSQL schema to isolate tenant data.

Diesel migrations then run automatically to bring the schema up to date, and the broker checks the app_initialization table for a first-time startup. On first run it creates the admin role and an associated admin generator, writing the initial admin PAK (Prefixed API Key) to a temporary file for retrieval.

The broker also provisions a system generator (named __system__), which serves as the default application scope for fleet-level and system stacks that reach every agent without per-agent opt-in. Every agent is automatically registered with the system generator when it is created, so it can always receive system-level deployments; this registration is excluded from the public generator listing. The admin generator is a separate entity tied to administrative operations, and agents are not automatically registered with it. The system generator is the foundation of Brokkr’s application-level tenancy model — see Generator Registration and Application Scopes.

With the database ready, the broker initializes its runtime subsystems in sequence: the Data Access Layer (DAL), the encryption subsystem for webhook secrets, the ephemeral read-only UI PAK that authenticates the embedded Operator Console (minted in memory, never persisted), the event emission system for webhook dispatch, the audit logger for compliance tracking, and finally a set of background maintenance tasks. If a configuration file is specified, it also starts a filesystem watcher for hot-reload capability.

API Layer

The API layer exposes a RESTful interface under the /api/v1 prefix, with all endpoints documented via OpenAPI annotations that generate Swagger documentation. Every request to an API endpoint passes through authentication middleware that extracts and validates the PAK from the Authorization header.

The authentication middleware resolves one of four identity classes. It first compares the presented PAK’s hash, in constant time, against the ephemeral read-only UI PAK the broker mints in memory at startup for the embedded operator console — a match yields a read-only admin identity without touching the database. It then consults a short-lived auth cache of recent verifications (TTL configurable, default 60 seconds), and on a cache miss checks the remaining identity types in order: the admin role table, then the agents table, and finally the generators table. The database lookups are optimized with partial indexes on the pak_hash column that exclude soft-deleted records, enabling O(1) lookup performance rather than full table scans.

Upon successful authentication, the middleware injects an AuthPayload structure into the request context containing the authenticated identity’s type and ID. Individual endpoint handlers then perform authorization checks against this payload to determine if the caller has permission for the requested operation.

C4Component
    title Broker Request Processing (C4 Component)

    Person(client, "API Client", "Admin, Agent, or Generator")

    Component(auth, "Auth Middleware", "Axum Middleware", "Extracts PAK, verifies against DB, injects AuthPayload")
    Component(handler, "Route Handler", "Axum Handler", "Authorization check and business logic")
    Component(dal, "Data Access Layer", "Diesel ORM", "Type-safe database operations via accessor types")
    ComponentDb(pool, "Connection Pool", "r2d2", "Manages PostgreSQL connections (default: 50)")
    ComponentDb(db, "PostgreSQL", "Database", "Persistent storage")

    Rel(client, auth, "HTTP Request", "Authorization: Bearer {PAK}")
    Rel(auth, handler, "Authenticated request", "AuthPayload injected")
    Rel(handler, dal, "Data operations", "Type-safe queries")
    Rel(dal, pool, "Get connection", "")
    Rel(pool, db, "SQL", "TCP :5432")

The API is organized into resource-specific modules: agents, generators, stacks, deployment objects, templates, work orders, webhooks, diagnostics, health status, and admin operations. Each module defines its routes, request/response structures, and authorization requirements. The generators and agents modules also expose endpoints for managing generator registrations — registering an agent with or deregistering it from an application scope, and listing the scopes an agent or generator is associated with — which underpin the application-level isolation model (see the API reference).

Data Access Layer

The DAL provides a clean abstraction over database operations, exposing specialized accessor types for each entity in the system. Rather than having a monolithic data access object, the DAL struct provides factory methods that return purpose-built accessor types: dal.agents() returns an AgentsDAL, dal.stacks() returns a StacksDAL, and so on for the system’s two dozen entity types.

Each accessor type implements the standard CRUD operations appropriate for its entity, along with any specialized queries needed. For example, the AgentsDAL provides not only basic create, get, update, and delete operations, but also get_by_pak_hash for authentication lookups and filtered listing methods that support complex queries combining label and annotation filters.

The DAL uses Diesel’s query builder for type-safe SQL generation, ensuring that queries are checked at compile time rather than failing at runtime with SQL syntax errors. All operations use the connection pool, automatically returning connections when operations complete.

Soft deletion is implemented throughout the system using a deleted_at timestamp column. When an entity is “deleted,” this timestamp is set rather than removing the row, preserving the audit trail and enabling potential recovery. Most queries automatically filter out soft-deleted records.

Event Emission

The event emission system provides a database-centric approach to webhook dispatch. Rather than using an in-memory pub/sub bus, events are directly matched against webhook subscriptions and inserted into the webhook_deliveries table for processing by the webhook delivery worker.

When an event occurs (such as a deployment being applied or a work order completing), the emit_event() function queries the database for all webhook subscriptions whose event type pattern matches the event. Pattern matching supports exact matches like workorder.completed, wildcard suffixes like deployment.* that match any deployment-related event, and a catch-all * pattern that matches everything. For each matching subscription, a webhook delivery record is created in PENDING status for the webhook delivery worker to process.

Background Tasks

The broker runs a set of concurrent background maintenance tasks:

Diagnostic Cleanup Task (default: every 15 minutes) expires timed-out diagnostic requests and deletes completed, expired, or failed records older than the configured maximum age (default: 1 hour).

Work Order Maintenance Task (default: every 10 seconds) drives work order lifecycle transitions: it returns RETRY_PENDING orders to PENDING once their backoff elapses, and reclaims orders that were claimed but never completed within the timeout.

Webhook Delivery Worker (default: every 5 seconds) processes a batch of pending deliveries (default: 50) — decrypting each subscription’s URL and auth header, issuing an HTTP POST with a 30-second timeout, and recording the result. Failures retry with exponential backoff until the maximum count, then are marked dead.

Webhook Cleanup Task (default: hourly) deletes completed and dead deliveries older than the retention period (default: 7 days).

Audit Log Cleanup Task (default: daily) deletes audit entries older than the configured retention period (default: 90 days).

Additional workers — agent-metrics-refresh, agent-events-cleanup, the fleet-sweep, and WebSocket-eviction tasks — run on their own intervals to keep agent telemetry, event history, fleet liveness, and stale WS connections in order.

Audit Logger

The audit logger provides asynchronous, batched recording of security-relevant events for compliance and forensic purposes. It is implemented as a singleton with an internal mpsc channel and background writer task (unlike event emission, which is database-centric with no in-memory channel).

Rather than writing each audit entry immediately to the database, which would create significant write amplification, the logger buffers entries and writes them in batches. The writer task accumulates entries up to a configurable batch size (default: 100) or until a flush interval elapses (default: 1 second), whichever comes first. This approach dramatically reduces database write pressure while maintaining a nearly real-time audit trail.

Audit entries capture the actor type (admin, agent, generator, or system), the actor’s ID, the action performed, the resource type affected, the resource’s ID, and additional contextual details. This information enables security teams to reconstruct the sequence of events leading to any system state.

Configuration Hot-Reload

When the BROKKR_CONFIG_FILE environment variable names a configuration file, the broker watches that file and applies some changes without a pod restart. The broker Helm chart mounts such a file by default (configReload.enabled). The config watcher uses the notify crate to monitor the configuration file’s parent directory for filesystem events, detecting modifications through OS-level file watching.

When a change is detected, the watcher applies a configurable debounce period (default: 5 seconds, configurable via BROKKR_CONFIG_WATCHER_DEBOUNCE_SECONDS) to coalesce rapid successive changes into a single reload operation. The watcher can be disabled via BROKKR_CONFIG_WATCHER_ENABLED=false. Only some values reload: the log level, the diagnostic cleanup interval and maximum age, the webhook delivery interval and batch size, and the webhook cleanup retention. The background tasks read these values again on each pass. The log level does not reload when RUST_LOG was set at startup. All other settings need a pod restart, including the database settings and CORS: the broker builds the CORS layer one time, at startup.

The admin API also exposes a manual reload endpoint at POST /api/v1/admin/config/reload that triggers an immediate configuration reload, useful when changes need to take effect immediately.

Agent Service Architecture

The agent is a Kubernetes-native component that runs inside target clusters, responsible for applying deployment objects to the cluster and reporting status back to the broker. It is designed to be resilient, continuing to operate with cached state during broker unavailability.

Startup and Initialization

When the agent starts, it first loads its configuration, which must include the broker URL, the agent’s name and cluster name, and its PAK for authentication. It then enters a readiness loop, repeatedly checking the broker’s health endpoint until it receives a successful response or exceeds its retry limit.

Once the broker is reachable, the agent verifies its PAK by calling the authentication endpoint. A successful response confirms the agent is properly registered and authorized. The agent then fetches its full details from the broker, including its assigned labels and annotations that determine which deployments it will receive.

The agent also registers with its configured generator scope(s), opting into application-level deployments beyond the system/fleet scope it already receives by default. The scope is resolved by precedence: the --generator-ids CLI flag takes priority over the BROKKR__AGENT__GENERATOR_IDS environment variable (equivalently agent.generator_ids in a config file), which in turn takes priority over the deprecated bare BROKKR_GENERATOR_IDS variable (still honored, but it logs a deprecation warning). When no generator scope is configured, the agent operates in system/fleet scope only. This opt-in is the agent’s consent boundary: it can only be targeted by stacks from application scopes it has explicitly joined. See Generator Registration and Application Scopes for the concept, the agent registration how-to for operational steps, and environment variables for the exact configuration keys.

With authentication confirmed, the agent initializes its Kubernetes client using in-cluster configuration (when running as a pod) or a specified kubeconfig path (for development). It validates connectivity by fetching the cluster’s API server version.

Finally, the agent starts an HTTP server on port 8080 for health checks and metrics, then enters its main control loop.

Main Control Loop

The agent’s control loop is implemented using Tokio’s select! macro to concurrently await multiple timer-based tasks. This design allows multiple activities to proceed in parallel while ensuring orderly shutdown when a termination signal is received.

C4Component
    title Agent Internal Architecture (C4 Component)

    Container_Boundary(agent, "Brokkr Agent") {
        Component(loop, "Control Loop", "Tokio select!", "Concurrent timer-based task dispatcher")
        Component(heartbeat, "Heartbeat", "Timer Task", "Sends periodic heartbeat to broker")
        Component(deployer, "Deployment Checker", "Timer Task", "Polls for deployment objects and reconciles")
        Component(workorder, "Work Order Processor", "Timer Task", "Claims and executes work orders")
        Component(healthcheck, "Health Monitor", "Timer Task", "Evaluates deployment health status")
        Component(diagnostics, "Diagnostics Handler", "Timer Task", "Processes on-demand diagnostic requests")
        Component(reconciler, "Reconciliation Engine", "kube-rs", "Server-side apply with ordering and ownership")
    }

    System_Ext(broker, "Broker API", "Central control plane")
    System_Ext(k8s, "Kubernetes API", "Target cluster")

    Rel(loop, heartbeat, "Triggers")
    Rel(loop, deployer, "Triggers")
    Rel(loop, workorder, "Triggers")
    Rel(loop, healthcheck, "Triggers")
    Rel(loop, diagnostics, "Triggers")
    Rel(deployer, reconciler, "Delegates apply/delete")
    Rel(heartbeat, broker, "POST heartbeat", "HTTPS")
    Rel(deployer, broker, "GET target-state", "HTTPS")
    Rel(reconciler, k8s, "Apply/Delete resources", "HTTPS :6443")
    Rel(healthcheck, k8s, "Query pod status", "HTTPS :6443")

Heartbeat Timer fires at a configurable interval (derived from the polling interval) to send a heartbeat to the broker, maintaining the agent’s “alive” status. The response includes updated agent details, allowing the broker to push configuration changes like label updates.

Deployment Check Timer fires at the configured polling interval to fetch the agent’s target state from the broker. The agent compares this desired state against what it has previously applied and performs reconciliation to converge them.

Work Order Timer polls for pending work orders assigned to the agent. Work orders represent transient operations like container image builds that don’t fit the declarative deployment model.

Health Check Timer (when enabled) periodically evaluates the health of deployments the agent has applied, checking pod status, container states, and conditions to produce health summaries reported back to the broker.

Diagnostics Timer polls for on-demand diagnostic requests, which allow administrators to collect debugging information from specific deployments.

Reconciliation Engine

The reconciliation engine is the heart of the agent, responsible for applying Kubernetes resources to achieve the desired state while handling the complexities of resource dependencies, conflicts, and failures.

When the agent receives deployment objects from the broker, it first parses the YAML content into Kubernetes DynamicObject instances. The objects are then sorted to apply cluster-scoped resources (Namespaces and CustomResourceDefinitions) before namespace-scoped resources, ensuring dependencies exist before resources that need them.

Before applying each resource, the agent injects standard metadata labels that link the resource back to Brokkr: the stack ID, a checksum of the YAML content for change detection, the deployment object ID, and the agent’s ID as the owner. This metadata enables the agent to identify which resources it manages and detect changes.

Application uses Kubernetes server-side apply with field ownership, which handles merge conflicts more gracefully than client-side apply. Before applying in earnest, resources are validated with a dry-run request to catch schema errors or policy violations early.

When deployment objects are removed (indicated by deletion markers), the agent deletes the corresponding resources from the cluster. However, it only deletes resources it owns, verified by checking the owner ID label. This prevents accidental deletion of resources created by other systems.

The reconciliation engine implements exponential backoff retry logic for transient failures like rate limiting (HTTP 429) or temporary server errors (5xx). Non-retryable errors like authorization failures (403) or resource not found (404) fail immediately to avoid wasting retry attempts.

Work Order Processing

Work orders handle operations that don’t fit the declarative resource model, such as building container images. The agent polls for pending work orders, claims one for exclusive processing, executes it, and reports the result.

For build work orders, the agent parses the YAML to extract Shipwright Build resources, applies them to the cluster, creates a BuildRun to trigger execution, and monitors the BuildRun status until completion. Build progress is checked every 5 seconds with a 15-minute overall timeout. On success, the agent reports the resulting image digest; on failure, it reports the error reason.

The work order system includes retry logic: when a work order fails with a retryable error, it is scheduled for retry with exponential backoff. Non-retryable errors are marked as permanently failed.

Health Monitoring

When deployment health monitoring is enabled, the agent periodically evaluates the health of resources it has applied. For each tracked deployment object, it queries for pods matching the deployment object ID label and analyzes their status.

The health checker examines pod phase, conditions, and container states to produce a health assessment. It specifically looks for problematic conditions like ImagePullBackOff, CrashLoopBackOff, OOMKilled, and various container creation errors. Based on its analysis, it assigns one of four health statuses: healthy (all pods running and ready), degraded (issues detected but not failed), failing (pod in Failed phase), or unknown (unable to determine status).

Health reports are stored per agent and deployment object and surfaced through the health API; health-status changes do not emit webhook events (webhooks fire on apply outcomes such as deployment.failed).

Component Interaction Patterns

Understanding how the broker and agents interact is essential for operating and troubleshooting Brokkr deployments.

Deployment Flow

The deployment lifecycle begins when an administrator or CI/CD system (acting as a generator) creates a stack and adds deployment objects to it. The stack serves as a logical grouping with labels and annotations that determine which agents will receive its deployments.

Creating a deployment object does not trigger any matching or write any agent associations. Instead, the agent-to-stack association is resolved dynamically at read time, on every poll. Rows in the agent_targets table exist only when an operator explicitly creates them via POST /api/v1/agents/{id}/targets. When an agent requests its target state, the broker computes the union of three sets: stacks explicitly targeted to the agent, stacks sharing any label with the agent (OR semantics), and stacks sharing any annotation key/value pair with the agent (also OR). Deployment objects are then served from that union.

Creating an explicit target is gated by generator registration: before an operator can add an agent_targets row via POST /api/v1/agents/{id}/targets, the agent must be registered with the stack’s owning generator — otherwise the request is rejected. Registration also constrains the read-time association above: the label and annotation legs of that union contribute only stacks whose owning generator the agent is registered with, so matching narrows within consented generators instead of reaching across them. Explicit targets need no read-time filter, having been checked at creation. This is how an agent’s opt-in consent boundary is enforced — see Generator Registration and Application Scopes.

sequenceDiagram
    participant Admin as Administrator
    participant Broker as Broker
    participant DB as PostgreSQL
    participant Agent as Agent
    participant K8s as Kubernetes

    Admin->>Broker: Create Stack with Labels
    Broker->>DB: Store Stack
    Admin->>Broker: Create Deployment Object
    Broker->>DB: Store Deployment Object
    opt Explicit targeting only
        Admin->>Broker: POST /agents/{id}/targets
        Broker->>DB: Create Agent Target
    end

    loop Every Polling Interval
        Agent->>Broker: Request Target State
        Broker->>DB: Resolve associated stacks (explicit targets ∪ label matches ∪ annotation matches)
        Broker->>DB: Query Deployment Objects for those stacks
        Broker-->>Agent: Deployment Objects
        Agent->>Agent: Compare Desired vs Actual
        Agent->>K8s: Apply Resources
        K8s-->>Agent: Result
        Agent->>Broker: Report Event
        Broker->>DB: Store Event
        Broker->>Broker: Dispatch Webhooks
    end

On the agent side, the deployment check timer requests the target state; the broker returns the resolved stacks’ deployment objects with sequence IDs that establish ordering. The agent compares this desired state against what it has applied, performs the necessary create, update, or delete operations in Kubernetes, and reports events back to the broker.

Event and Webhook Flow

Events flow from agents through the broker to external systems via webhooks. When an agent reports an event (success, failure, health change), the broker stores it in the database and emits it through the event emission system.

The event emitter queries the database for matching webhook subscriptions. For each match, it creates a webhook delivery record. The webhook delivery worker picks up these records and attempts HTTP delivery to the configured endpoints.

This decoupled architecture ensures that webhook delivery proceeds asynchronously with its own retry logic, separate from the initial event recording.

Performance Characteristics

The broker is designed to handle substantial load with modest resources. The API layer uses Axum’s async request handling to efficiently multiplex many concurrent connections onto a small number of OS threads. Connection pooling minimizes database connection overhead.

The figures below are illustrative design targets, not measured guarantees; real numbers depend on hardware, dataset size, and load.

API Performance: The async request path and connection pooling are designed for low-latency reads and high request throughput on a single instance, with authentication lookups served from partial indexes.

Database Performance: Most queries are indexed for fast lookups, and PAK authentication uses partial indexes on pak_hash. The connection pool defaults to 50 connections.

Agent Performance: Reconciliation applies resources via server-side apply and is designed to converge a poll’s worth of changes promptly, with low-latency event reporting back to the broker.

For larger deployments, the broker can be horizontally scaled behind a load balancer since it maintains no in-memory state beyond caches—all persistent state lives in PostgreSQL. One caveat: each replica mints its own ephemeral UI PAK for the embedded operator console, so console traffic needs sticky sessions (see Horizontal Broker Scaling).

Scaling Patterns

Horizontal Broker Scaling

The broker’s stateless design enables straightforward horizontal scaling. Multiple broker instances can run behind a load balancer, all connecting to the same PostgreSQL database. Each instance runs its own background tasks, but these are designed to be safe for concurrent execution through database-level coordination (e.g., work order claiming uses atomic updates to prevent double-claiming).

The API is fully stateless across replicas, but the embedded operator console is not: each broker process mints its own ephemeral read-only UI PAK at startup, and the console page served by one replica carries a token that other replicas reject. When multiple replicas serve console traffic behind a load balancer, enable sticky sessions (session affinity) so a browser stays with the replica that issued its token. Agent, generator, and admin PAK authentication is unaffected — those credentials verify identically on every replica.

C4Container
    title Horizontal Scaling Pattern (C4 Container)

    System_Ext(lb, "Load Balancer", "Distributes API traffic")

    Container_Boundary(brokers, "Broker Instances") {
        Container(b1, "Broker 1", "Rust/Axum", "Stateless API + background tasks")
        Container(b2, "Broker 2", "Rust/Axum", "Stateless API + background tasks")
        Container(b3, "Broker 3", "Rust/Axum", "Stateless API + background tasks")
    }

    ContainerDb(db, "PostgreSQL Primary", "PostgreSQL", "All persistent state")
    ContainerDb(dbr, "PostgreSQL Replica", "PostgreSQL", "Read replicas (optional)")

    Rel(lb, b1, "Routes requests", "HTTPS")
    Rel(lb, b2, "Routes requests", "HTTPS")
    Rel(lb, b3, "Routes requests", "HTTPS")
    Rel(b1, db, "Read/Write", "SQL :5432")
    Rel(b2, db, "Read/Write", "SQL :5432")
    Rel(b3, db, "Read/Write", "SQL :5432")
    Rel(db, dbr, "Replication", "Streaming")

Agent Deployment

Each Kubernetes cluster runs one agent instance. The agent handles all deployments for that cluster, with no need for multiple agents per cluster. Agents operate independently with no inter-agent communication, simplifying the operational model.

For very large clusters or high deployment volumes, the agent’s polling interval can be tuned to balance responsiveness against API load.

Resource Requirements

The Helm charts ship conservative CPU and memory defaults suitable for small to medium deployments; see the per-component tables in the installation guide or the chart values.yaml files for the current values. Production deployments handling thousands of deployment objects or hundreds of agents should increase these limits based on observed resource utilization.

Multi-Tenancy

The broker supports multi-tenant deployments through PostgreSQL schema isolation. When configured with a schema name, the broker creates a dedicated schema and sets the connection’s search_path to use it for all queries. This provides data isolation between tenants sharing a PostgreSQL instance without requiring separate databases.

Schema names are validated to prevent SQL injection, accepting only alphanumeric characters and underscores, and requiring the name to start with a letter.

Schema-per-tenant is a deployment-level isolation mechanism and is distinct from generator registration, which provides application-level isolation within a single broker. Schema isolation separates tenants that each share nothing in the database; generator registration lets multiple applications share one broker while agents opt into the application scopes they serve, preventing accidental cross-application targeting. The two are complementary, not alternatives — see Generator Registration and Application Scopes and the multi-tenancy reference.