Using the Operator Console

The Operator Console is the read-only web view of a Brokkr deployment. It is served by the broker itself, at the broker’s own root URL, and it needs no credential from you — open the address in a browser and you are looking at the fleet.

That convenience is also the thing to understand before you expose it. The page hands every visitor a working read-only administrative credential. Whoever can reach the broker’s port can read the whole deployment. This guide covers how to open the console, what each view shows, how to scope it to one tenant, and — first — what its authentication model means for you.

Before You Start

A broker built with the console. The published broker image includes it. A broker you compiled yourself from a plain cargo build does not: it serves a short placeholder page saying the console was not compiled in, and its JSON API works normally. See Local Development Environment if you want the console in a locally built broker.
Network access to the broker’s HTTP port (3000 by default) from your browser. That is the same port the API listens on; there is no separate UI port and no separate UI container.
A modern browser. The console is a WebAssembly application and will not run with scripting disabled.

Step 1: Open the Console

Browse to the broker’s root URL:

http://<broker-host>:3000/

If the broker is inside a cluster and not exposed, a port-forward is the safest way to reach it:

kubectl port-forward -n brokkr svc/brokkr-broker 3000:3000

Then open http://localhost:3000/.

You are not asked to sign in or to paste a key. The console loads and immediately begins reading the API.

Every path the API does not own falls back to the console, so bookmarking any console path works. Paths under /api and /internal always belong to the API and return a normal 404 when they do not exist — the console never shadows them.

Step 2: Know What the Console Is Authenticating As

This is the part worth reading before anyone else gets the URL.

Each time a broker process starts, it mints one random read-only credential and holds it in memory. It is never written to the database, never logged, and cannot be rotated or retrieved through the CLI; restarting the broker simply replaces it. When the broker serves the console page, it embeds that credential in the HTML. The console reads it out of the page and uses it for every API call it makes.

The consequences follow directly:

Network reachability of the broker URL is the console’s entire authentication boundary. Anyone who can fetch the page holds a working read-only admin credential. There is no second factor, no login, and no per-user identity.
That credential sees everything readable, not just what the console draws. It satisfies admin-level read checks across all tenants — the fleet rollup, every tenant’s stacks, telemetry, work orders, and the audit log, which the console itself has no screen for. Someone who copies the token out of the page can query any of it directly.
It cannot change anything. Reads (GET/HEAD) are allowed anywhere. Exactly two POST routes are allowlisted because they do not alter desired state: credential introspection, and requesting a diagnostic for a deployment object. Every other mutation is refused with 403 Forbidden. If you see a 403 from the console, that is the design working, not a bug.
Builds without the embedded console carry no such credential at all — the placeholder page has nothing to hand out.

Mitigate this at the network layer. Restrict who can reach port 3000 with a NetworkPolicy or firewall rules; if the console must be available beyond the cluster, put authentication in front of it at the ingress (OIDC or basic auth on the ingress controller, for example). Treat the broker port as sensitive for read access, the same way you would treat /metrics. The Security Hardening checklist carries this item, and the Security Model explains the credential class and the threat model behind the decision.

The embedded read-only credential is the only identity a console session has. Earlier versions also honoured a write-capable PAK placed in the browser’s local storage under brokkr_pak, and preferred it over the embedded one — so setting it silently made the whole console write-capable. That override was removed in 0.9.0. The one action that needs more than read access, creating a tenant, asks for an admin PAK at the moment you use it and does not keep it.

Step 3: Find Your Way Around

The sidebar groups seven views:

Monitor

Overview — the at-a-glance landing view: headline counts, a fleet-health bar, a broker-throughput sparkline drawn from the broker’s Prometheus metrics, and a recent activity feed of agent events.
Fleet — every agent, grouped into a panel per cluster, with counts of active, degraded, and failing agents across the top. Each row shows the agent’s status and health pills, whether it holds an internal WebSocket connection to the broker, and how long ago it last checked in. Clicking a row opens the agent detail panel, which is also where diagnostics live (Step 5).
Deployments — the stacks this broker knows about. Clicking one opens its detail, including the per-deployment-object health rollup for that stack.
Telemetry — three tabs. Agent events lists the agent lifecycle events (applies, heartbeats, reconciles) that the broker keeps; click one to see its detail. Kube events and Pod logs show one stack: select the stack above the tabs. Kube events lists the Kubernetes events of the objects of that stack. Pod logs is a live tail of the pods of that stack (see The Pod Logs Tab Is Live). The broker keeps each stream for a short window only, and the view shows that window: this is a view of recent activity, not a log archive. To keep logs for longer, send them on as Streaming Pod Logs & Live Tail tells.

Operations

Work orders — active work orders above the completed history; clicking a history row opens its detail. The active list is admin-gated, and the panel says so plainly if the broker declines to serve it.
Webhooks — the webhook subscriptions and their configured events. The API deliberately redacts subscription URLs, so the console shows only whether one is set. Clicking a subscription lists its recent delivery attempts. Below the subscriptions, the Recent deliveries panel shows the newest deliveries of all subscriptions (see The Recent Deliveries Panel).

System

Broker health — headline broker metrics (active agents, connected agents, HTTP request volume, live subscribers, stack and deployment-object counts) drawn from the broker’s /metrics endpoint, alongside the current internal WebSocket connections. Three cards show the database (see The Database Cards).

The Recent Deliveries Panel

The Recent deliveries panel is on the Webhooks view, below the subscription cards. It shows the 20 newest deliveries of all subscriptions, newest first. It reads them from GET /api/v1/webhook-deliveries (see the delivery feed reference). The panel reads the feed again every 10 s.

Each row shows:

Column	Meaning
Subscription	The name of the subscription. “(subscription removed)” shows when the subscription is gone.
Status	`pending`, `acquired`, `success`, `failed` or `dead`.
Event	The event type, for example `deployment.failed`.
Attempts	The number of delivery attempts.
Age	The time since the broker made the delivery.
Last error	The error of the last attempt, if there is one.

Click a row to open the detail of its subscription. If the panel cannot read the feed, it shows an error and a retry control. The subscription cards above it continue to show.

The Database Cards

The Broker health view has three database cards. They come from the database metrics on /metrics (see Database Metrics). The view reads /metrics every 5 s.

Card	Value	Text below the value
DB queries / min	The query rate, from the change in `brokkr_database_queries_total` between the two newest reads. Until the view has two reads, the value is “collecting” and the text is “the rate needs two samples”.	The metric name.
DB query p95	The 95th percentile query time, from `brokkr_database_query_duration_seconds`. Until the broker has a measured query, the value is “collecting” and the text is “no queries yet”.	“last n s”: the queries of the last minute or less. “since broker start”: the window has no queries, or the broker restarted in the window, so the value is for all queries since the broker started.
DB pool in use	Connections in use, and the pool maximum, for example `2/10`.	The metric name.

A card shows “—” and “not sent by this broker” when the broker does not send that metric. An older broker does not send the database metrics. Upgrade the broker to get the cards.

How the Views Refresh

Most views read the REST API again on a 5 s timer, so the page stays current without a reload. The Pod logs tab is the only view that also opens a WebSocket (see below).

The header carries a wall clock. It once also carried a Live/Paused control that was wired to nothing; it was removed in 0.9.0 rather than left as decoration, because in a deployment tool a global “Paused” reads as the fleet is paused — which it never was. Pausing is now a real, per-agent action: see Pause or Resume an Agent.

There is no way to stop the console’s periodic re-reads. They are read-only and cheap; close the tab if you need them to stop.

If you want a genuine live stream of fleet state, that is an API capability rather than a console one — see Monitoring Your Agent Fleet.

The Pod Logs Tab Is Live

When the Pod logs tab shows a stack, the console opens the live stream of that stack (/api/v1/stacks/{id}/live, the stream that Streaming Pod Logs & Live Tail describes). The console first shows the lines that the broker keeps. Then it adds each new line when the agent sends it, usually in less than 1 s. The view shows each line one time, also when a line comes from the stream and from a history read. The view keeps the newest 2000 lines.

The console closes the stream when you go to a different tab, select a different stack, or go to a different view.

A state shows above the lines:

State	Meaning
connecting	The console opens the stream.
live	The stream is open. New lines show when the pod writes them. The console does not read the history again while the stream is open.
reconnecting	The stream closed. The console tries again after the delay that it shows (1 s, then 2 s, 4 s and more, to a maximum of 30 s). Until the stream is open again, the console reads the history every 5 s, so you see new lines at most 5 s late. When the stream opens, the console reads the history one time to get the lines that it did not get.
polling	The browser cannot open the stream. The console reads the history every 5 s.

A line with a gap label tells you that lines are missing at that position, and how many. The agent sends a gap when it drops lines (for example, when a pod writes more lines than its rate limit). The broker sends a gap when the console reads the stream too slowly. The broker keeps the lines that it got, so a reload can show lines that the gap replaced, if the agent did not drop them.

The console does not use the stream to find out if its token is valid. If the broker refuses the stream, the tab shows reconnecting, and the REST reads show the session state as usual.

Step 4: Scope the View to One Tenant

On a broker shared between tenants, the sidebar shows a Tenant selector below the navigation. It lists the tenants that the broker knows about, plus All. Choosing one narrows the Overview, Fleet, Deployments, and Telemetry views to that tenant’s resources; the choice is remembered in your browser between visits, and falls back to All if the tenant it named has since been removed.

On a single-tenant install there is nothing to choose, so the selector is hidden entirely. It is also hidden if the tenant listing cannot be read — the views still work, unscoped.

The tenant selector is a view filter, not an authorization boundary. It trims what the console displays; it does not restrict what the console’s credential is permitted to read, and switching back to All is a click away for anyone using the page. Real isolation comes from tenant ownership, described in Setting Up Multi-Tenant Operation and the Multi-Tenancy reference. Never treat the selector as a way to show one tenant’s operators only their own data.

Step 5: Run a Diagnostic on an Agent

Diagnostics are the console’s one action, and they are read-shaped: you are asking an agent to go look at its cluster and report back, not changing any desired state.

Open the Fleet view and click the agent you want to inspect.
In the agent detail panel, find the deployment object picker. It lists everything currently targeted at that agent. If the agent has no deployment objects, the panel says so instead of offering a dead button — there is nothing to diagnose.
Pick an object and choose Run diagnostic. The console tells you the request was accepted and begins waiting for it.
The result appears in the same panel once the agent submits it: pod statuses for that deployment object, recent events from the namespaces it searched, and log tails. An empty pod list is a legitimate answer for an object that applies no workloads, and the panel labels it as such rather than as an error.

Collection is not instantaneous — the agent picks up pending requests on its own timer and then has to query the Kubernetes API. The console waits a bounded amount of time and then stops, offering a Check again button. Stopping is not evidence that collection failed; it only means no result had arrived yet.

For the request lifecycle, retention, and how to drive the same capability from the API or CLI, see Running On-Demand Diagnostics and the Diagnostics reference.

Pause or Resume an Agent

Open Fleet, click the agent, and use the control under Agent state. Like tenant creation, it asks for an admin PAK for that one request — the console’s own credential is read-only and cannot change agent state.

What pausing does: the agent stops fetching deployment objects and stops picking up work orders. It also stops health checks, diagnostics and webhook deliveries, so a diagnostic that you request for a paused agent stays pending. It keeps heartbeating, so it stays visible and healthy in the console rather than looking offline.

What it does not do: it does not roll anything back. Resources the agent has already applied stay in the cluster exactly as they are. Pausing stops new work being taken up; it is not an undo.

The change takes effect within a poll cycle — the agent re-reads its own record on each heartbeat. Resuming is the same control, which flips to Resume while the agent is paused.

This is the agent honouring its own status, not the broker enforcing it. The broker will still serve deployment objects to a paused agent that asks for them; the shipped agent simply does not ask. For most purposes that distinction does not matter, but do not rely on paused as a containment measure for an agent you do not trust — use network policy or credential rotation for that.

The equivalent outside the console is PUT /api/v1/agents/{id} with {"status": "INACTIVE"}. A newly created agent starts paused (INACTIVE) and must be activated before it applies anything — the same control does that.

Step 6: Create a Tenant

Tenants lists the tenants on this broker, and is the one place the console writes anything. Because a tenant’s PAK is a credential, the console will not mint one on the strength of the read-only identity it carries — you supply an admin PAK for that single request.

Open Tenants in the sidebar and choose + New tenant.
Give the tenant a name, and a description if it helps whoever inherits it.
Paste an admin PAK. This is the only step in the console that asks for a credential.
Choose Create tenant.

The new tenant’s PAK is then shown once, with a Copy button.

Copy it before you close the dialog. The broker stores only a hash, so a PAK that is not captured cannot be recovered — the only remedy is rotating it. This is the same one-shot behaviour as brokkr-broker create generator.

The admin PAK you pasted is held in memory for that one request and cleared as soon as it finishes, whether it succeeded or not. It is never written to browser storage and never logged, so creating a second tenant asks for it again — deliberately, so that a browser left open is not a standing admin credential.

If the PAK is rejected, the dialog says whether it was not an admin credential (403) or the name was already taken (409). Nothing is created in either case.

Outside the console, use POST /api/v1/generators or brokkr-broker create generator (the API calls a tenant a generator); see Generators. Creating agents is not available here — only tenants.

Running More Than One Broker Replica

Each broker process mints its own console credential, and a credential minted by one replica is rejected by the others. A browser served the console page by replica A and then load-balanced onto replica B will find its API calls refused.

Enable session affinity (sticky sessions) on whatever fronts your brokers whenever more than one replica serves console traffic, so a browser keeps talking to the replica that served it the page. Reloading the page also fixes an individual session, since the reload picks up the current replica’s credential — but affinity is what makes the console usable rather than intermittent. Agent, tenant, and admin PAK authentication is unaffected; those credentials verify identically on every replica. See Horizontal Broker Scaling.

The Console Is Not `examples/ui-slim`

The repository also contains examples/ui-slim, a small React application. It is a demonstration of what you can build against the API — not a supported product, and not the console. The differences that matter:

	Operator Console	`examples/ui-slim`
Shipped in the broker image	Yes, served at the broker’s root URL	No; run separately
Credential	Embedded automatically, read-only	You supply an admin PAK by hand
Capabilities	Read-only, plus diagnostics	Demonstrates writes as well
Supported	Yes	No

ui-slim is also the browser consumer of the fleet live-tail WebSocket; the console does not use it. If you are evaluating Brokkr and want the supported view, use the console.

Troubleshooting

The root URL shows a short “compiled without the bundled operator console” page. The broker was built without the console. Use the published image, or build the broker with the console enabled as described in Local Development Environment.

Views show an error where data should be. The console reports API failures inline with a retry control. Confirm the broker is healthy (/healthz, /readyz) and check the broker logs; the console is only as available as the API behind it.

The console says the session expired. See The Console Says the Session Expired below.

The console says the broker refused its token. See The Console Says the Broker Refused Its Token below.

An action you expected is refused with 403. The console authenticates read-only by design. Perform the change with an admin PAK through the CLI, an SDK, or the API.

The Console Says the Session Expired

Symptom. The console worked, and then a banner shows above the view:

The broker restarted, or this page reached a different replica. Reload to get a new session.

The indicator in the top bar shows session expired. The panels below the banner show “Waiting for a new session.” and do not show their data. A panel that fails for a different reason (for example, the broker answers 500) shows its own error and a Retry button.

Cause. The page holds the read-only credential of the broker process that served it. The broker keeps that credential in memory only. When the broker restarts, it makes a new credential, and the broker refuses the old one with 401 Unauthorized. If you run more than one broker replica, each replica has its own credential. A request that a load balancer sends to a different replica is refused in the same way.

Fix.

Click Reload in the banner, or reload the page in the browser. The page then gets the credential of the broker that serves it.
If the banner comes back when the broker did not restart, you have more than one replica behind a load balancer without session affinity. Enable session affinity (sticky sessions) on the load balancer, or send console traffic to one replica. See Running More Than One Broker Replica.

The banner shows only after a request succeeded. If the first requests are refused, see The Console Says the Broker Refused Its Token.

The Console Says the Broker Refused Its Token

Symptom. The console loads, but no data shows. A banner shows above the view:

The broker refused the token of this console. Reload the page. If this does not help, open the console directly at the address of one broker. Each broker process gives a different token to the page that it serves.

The indicator in the top bar shows token refused. The panels below the banner show “Waiting for a new session.” and do not show their data. A panel that fails for a different reason (for example, the broker answers 500) shows its own error and a Retry button.

Cause. The broker answered the first requests of the console with 401 Unauthorized or 403 Forbidden, and no request succeeded. The broker puts the read-only credential of its process into the page that it serves. These are the usual causes:

A load balancer sent the page request to one replica and the API requests to a different replica. The other replica refuses the credential.
The page did not come from a broker. For example, a different web server or a development server served it. Then the page has no credential, and the broker refuses the requests with 401 Unauthorized.
A proxy between the browser and the broker removes the Authorization header. The broker refuses a request without that header with 401 Unauthorized.