Monitoring and Observability

This reference catalogs the Prometheus metrics exposed by the broker and agent and the shipped Grafana dashboards. For scrape configuration, alerting rules, dashboard import, and integration with external monitoring systems, see Setting Up Monitoring.

Metrics Endpoints

Both broker and agent expose Prometheus metrics in standard text exposition format.

Broker Metrics

Endpoint: http://<broker-host>:3000/metrics

Agent Metrics

Endpoint: http://<agent-host>:8080/metrics (8080 is the default; the port is configurable via agent.health_port / BROKKR__AGENT__HEALTH_PORT)

Endpoint Label Normalization

The endpoint label on the broker’s HTTP metrics carries a normalized path, not the raw request path. Every path segment that is a UUID or is entirely numeric is replaced with :id before the metric is recorded, which keeps label cardinality bounded as resources are created.

Request path	endpoint label value
/api/v1/stacks	/api/v1/stacks
/api/v1/stacks/6f1b0f2e-1c3a-4a1e-9c2b-9a7f1d5e3b04	/api/v1/stacks/:id
/api/v1/agents/42/events	/api/v1/agents/:id/events

PromQL that matches on a literal resource path containing a UUID or number therefore never matches; match on the normalized form instead.

Broker Metrics Catalog

HTTP Request Metrics

brokkr_http_requests_total

• Type: Counter
• Description: Total number of HTTP requests by endpoint and status
• Labels:
  • endpoint - Normalized API endpoint path (see Endpoint Label Normalization)
  • method - HTTP method (GET, POST, PUT, DELETE)
  • status - HTTP status code (200, 404, 500, etc.)

Example PromQL:

# Request rate by endpoint
rate(brokkr_http_requests_total[5m])

# Error rate (4xx and 5xx)
sum(rate(brokkr_http_requests_total{status=~"[45].."}[5m])) by (endpoint)

# Success rate percentage
100 * sum(rate(brokkr_http_requests_total{status=~"2.."}[5m])) by (endpoint)
  / sum(rate(brokkr_http_requests_total[5m])) by (endpoint)

brokkr_http_request_duration_seconds

• Type: Histogram
• Description: HTTP request latency distribution in seconds
• Labels:
  • endpoint - Normalized API endpoint path (see Endpoint Label Normalization)
  • method - HTTP method
• Buckets: 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 seconds

Example PromQL:

# 99th percentile latency
histogram_quantile(0.99,
  sum(rate(brokkr_http_request_duration_seconds_bucket[5m])) by (le, endpoint)
)

# Average latency
rate(brokkr_http_request_duration_seconds_sum[5m])
  / rate(brokkr_http_request_duration_seconds_count[5m])

System State Metrics

brokkr_default_admin_pak_hash_in_use

• Type: Gauge
• Description: 1 when the broker’s admin credential is still the publicly-known development PAK hash shipped in the binary, 0 otherwise. The gauge is always published, so a healthy broker reports 0 rather than omitting the series — an == 1 alert is therefore meaningful rather than silently absent.
• Labels: None

The check covers both the configured hash and the one actually stored on the admin role. Those can differ: an install that first started with the default and only later set BROKKR__BROKER__PAK_HASH keeps the public hash in the database and keeps accepting the public PAK, because the admin bootstrap runs only on first startup or an explicit rotation. In that case a restart does not help — rotate the credential.

Example PromQL:

# Alert while the public development admin credential is accepted
brokkr_default_admin_pak_hash_in_use == 1

See Replace the Default Admin PAK for the remediation.

brokkr_active_agents

• Type: Gauge
• Description: Number of currently active agents
• Labels: None

Example PromQL:

# Current active agents
brokkr_active_agents

# Alert if no agents connected
brokkr_active_agents == 0

brokkr_agent_heartbeat_age_seconds

• Type: Gauge
• Description: Time since last heartbeat per agent in seconds
• Labels:
  • agent_id - Agent UUID
  • agent_name - Human-readable agent name
• Freshness: A broker background task refreshes this gauge (and brokkr_active_agents) from the database every ~30 seconds, so the values stay correct without depending on GET /agents traffic.

Example PromQL:

# Agents with stale heartbeats (>5 minutes)
brokkr_agent_heartbeat_age_seconds > 300

# Maximum heartbeat age across all agents
max(brokkr_agent_heartbeat_age_seconds)

brokkr_stacks_total

• Type: Gauge
• Description: Total number of stacks
• Labels: None

brokkr_deployment_objects_total

• Type: Gauge
• Description: Total number of deployment objects
• Labels: None

WebSocket Channel Metrics

These metrics cover the internal broker↔agent WebSocket channel and the live telemetry fan-out (see Internal Broker↔Agent WS Channel). Labels are deliberately low-cardinality; per-connection detail is available from GET /api/v1/admin/ws/connections instead.

brokkr_ws_connected_agents

• Type: Gauge
• Description: Number of agents currently connected on /internal/ws/agent
• Labels: None

brokkr_ws_messages_total

• Type: Counter
• Description: WebSocket messages processed by the broker
• Labels:
  • direction - in (from agents) or out (to agents)
  • type - Wire message variant (e.g. pod_log_line, k8s_event, heartbeat, work_order)

Example PromQL:

# Telemetry ingest rate by message type
sum(rate(brokkr_ws_messages_total{direction="in"}[5m])) by (type)

brokkr_ws_live_subscribers

• Type: Gauge
• Description: Total live-tail subscribers across all stacks (GET /api/v1/stacks/{id}/live)
• Labels: None

brokkr_fleet_live_subscribers

• Type: Gauge
• Description: Consumer-facing fleet live-push subscribers (GET /api/v1/fleet/live)
• Labels: None

brokkr_ws_log_eviction_runs_total

• Type: Counter
• Description: Telemetry eviction passes executed by the retention worker
• Labels: None

brokkr_ws_telemetry_evicted_total

• Type: Counter
• Description: Telemetry rows deleted by the retention worker (6-hour hard ceiling)
• Labels:
  • table - agent_k8s_events or agent_pod_logs

Database Metrics

These metrics show the load of the broker on its PostgreSQL database. A hook on each connection in the broker’s pool records each query. No label holds SQL text or a parameter value.

What counts as a query: each database round trip that the hook sees. This includes the SELECT 1 that the pool sends to test a connection each time the broker takes one from the pool. It also includes SET search_path on a pool that uses a schema. These queries are real load on the database, so the broker counts them.

brokkr_database_queries_total

• Type: Counter
• Description: Database queries that finished, by result
• Labels:
  • result - ok or error (error when the database driver reports an error for the query)

The broker sends both series with the value 0 from start-up. Thus a broker with no queries shows 0, and an older broker shows no series.

brokkr_database_query_duration_seconds

• Type: Histogram
• Description: Time from the start to the end of a query on one connection, in seconds
• Labels: None
• Buckets: 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5

Most broker queries take less than 10 ms, so the small buckets are finer than the buckets of brokkr_http_request_duration_seconds. The 5 s bucket shows lock waits and slow scans. If the hook does not see the start of a query, the counter counts the query, but the histogram does not.

brokkr_database_pool_connections

• Type: Gauge
• Description: Connections in the broker’s database pool, by state
• Labels:
  • state - idle or in_use

brokkr_database_pool_max_connections

• Type: Gauge
• Description: The maximum number of connections that the pool can open
• Labels: None

The broker sets the two pool gauges when /metrics is read. Thus the values are correct at the time of the read.

Example PromQL:

# Queries per second
sum(rate(brokkr_database_queries_total[5m]))

# Failed queries per second
rate(brokkr_database_queries_total{result="error"}[5m])

# 95th percentile query time
histogram_quantile(0.95, rate(brokkr_database_query_duration_seconds_bucket[5m]))

# Pool use as a fraction of the maximum
brokkr_database_pool_connections{state="in_use"} / brokkr_database_pool_max_connections

Agent Metrics Catalog

Broker Polling Metrics

brokkr_agent_poll_requests_total

• Type: Counter
• Description: Total number of broker poll requests
• Labels:
  • status - Request status (success, error)

Example PromQL:

# Poll request rate
rate(brokkr_agent_poll_requests_total[5m])

# Error rate
rate(brokkr_agent_poll_requests_total{status="error"}[5m])

# Success rate percentage
100 * sum(rate(brokkr_agent_poll_requests_total{status="success"}[5m]))
  / sum(rate(brokkr_agent_poll_requests_total[5m]))

brokkr_agent_poll_duration_seconds

• Type: Histogram
• Description: Broker poll request latency distribution in seconds
• Labels: None
• Buckets: 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0 seconds

Example PromQL:

# 99th percentile poll latency
histogram_quantile(0.99,
  sum(rate(brokkr_agent_poll_duration_seconds_bucket[5m])) by (le)
)

Kubernetes Operation Metrics

brokkr_agent_kubernetes_operations_total

• Type: Counter
• Description: Total number of Kubernetes API operations by type
• Labels:
  • operation - Operation type (apply, delete, get, list)

Example PromQL:

# K8s operation rate by type
rate(brokkr_agent_kubernetes_operations_total[5m])

# Total K8s operations per second
sum(rate(brokkr_agent_kubernetes_operations_total[5m]))

brokkr_agent_kubernetes_operation_duration_seconds

• Type: Histogram
• Description: Kubernetes operation latency distribution in seconds
• Labels:
  • operation - Operation type
• Buckets: 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 seconds

Example PromQL:

# 95th percentile K8s operation latency
histogram_quantile(0.95,
  sum(rate(brokkr_agent_kubernetes_operation_duration_seconds_bucket[5m])) by (le, operation)
)

Agent Health Metrics

brokkr_agent_heartbeat_sent_total

• Type: Counter
• Description: Total number of heartbeats sent to broker
• Labels: None

Example PromQL:

# Heartbeat send rate
rate(brokkr_agent_heartbeat_sent_total[5m])

brokkr_agent_last_successful_poll_timestamp

• Type: Gauge
• Description: Unix timestamp of last successful broker poll
• Labels: None

Example PromQL:

# Time since last successful poll
time() - brokkr_agent_last_successful_poll_timestamp

# Alert if no successful poll in 5 minutes
time() - brokkr_agent_last_successful_poll_timestamp > 300

Grafana Dashboards

Brokkr ships pre-built Grafana dashboards in the repository under docs/grafana/:

Dashboard	File
Broker	brokkr-broker-dashboard.json
Agent	brokkr-agent-dashboard.json
WebSocket channel	brokkr-ws-channel-dashboard.json

Prometheus alert rules for the WebSocket channel ship alongside the dashboards as docs/grafana/brokkr-ws-channel.rules.yml. For download and import steps, see Setting Up Monitoring.

Broker Dashboard Features

The broker dashboard includes:

• Active Agents - Current count of connected agents
• Total Stacks - Number of managed stacks
• Deployment Objects - Total deployment objects
• HTTP Request Rate - Requests per second by endpoint
• HTTP Request Latency - p50, p95, p99 latencies by endpoint
• Agent Heartbeat Age - Time since last heartbeat per agent

Agent Dashboard Features

The agent dashboard includes:

• Broker Poll Request Rate - Success/error poll rates
• Broker Poll Latency - p50, p95, p99 poll latencies
• Kubernetes Operations Rate - Operations per second by type
• Kubernetes Operation Latency - p50, p95, p99 operation latencies
• Heartbeat Send Rate - Heartbeats sent per second
• Time Since Last Successful Poll - Gauge showing polling health

Related Documentation

• Setting Up Monitoring - Prometheus scrape configuration, alerting rules, dashboard import, external integrations
• Installation Guide - Helm chart installation
• Health Check Endpoints - Liveness and readiness probes
• Configuration Guide - Configuration options