Skip to content
Brokkr

Tutorial: Multi-Cluster Targeting

In this tutorial, you’ll deploy different configurations to different clusters using Brokkr’s label and annotation targeting system. You’ll register two agents representing two environments (staging and production), then direct deployments to each one selectively.

What you’ll learn:

  • How shared labels associate stacks with agents automatically
  • How annotations attach key-value metadata that also participates in matching
  • How explicit agent targets bind a stack to an agent unconditionally
  • How one stack can reach multiple clusters while another targets only one

Prerequisites:

  • A broker you can reach at http://localhost:3000 — either a Helm install with kubectl port-forward svc/brokkr-broker 3000:3000 running, or the local development environment (angreal local up)

  • Your admin PAK for that broker (see Adapting the commands to your install)

  • curl and jq installed

  • The brokkr CLI on your PATH, for the label and target steps. Download it from the GitHub Release, or build it with cargo build --release -p brokkr-cli. Each step that uses it also shows the curl form. Point it at your broker:

    Terminal window
    export BROKKR_BROKER_URL=http://localhost:3000
    export BROKKR_PAK=<your-admin-pak>

  • Recommended: Deploy Your First Application, for stacks, deployment objects, and targets. Nothing here depends on the resources it created.

No running agent is required. This tutorial creates its own two agents through the API and works with them as broker-side records only, so it behaves identically on a Helm install and in the development environment. It never touches the pre-created brokkr-integration-test-agent, and nothing is applied to a real cluster.

Step 1: Create Two Agents

Section titled “Step 1: Create Two Agents”

In a real deployment, each agent runs in a different Kubernetes cluster. For this tutorial, we’ll create two agent records in the broker to simulate a multi-cluster setup. (You can also create agents with the brokkr-broker CLI — see the CLI Reference. The CLI ships inside the broker image, so it runs wherever the broker does: kubectl exec deploy/brokkr-broker -- brokkr-broker create agent --name staging-agent --cluster-name staging-cluster on a Helm install, or docker compose exec broker brokkr-broker create agent ... in the development environment.)

We pass generator_ids so each agent is registered with the admin-generator at creation time. Registration is the agent’s opt-in consent boundary: before you can create an explicit target (Step 4) for a stack, the agent must be registered with that stack’s owning generator. (Every agent is also auto-registered with the system generator for fleet-wide stacks.) See agent registration for the full operational picture.

Terminal window
# The admin-generator owns the stacks we'll create in Step 3
GEN_ID=$(curl -s http://localhost:3000/api/v1/generators \
  -H "Authorization: Bearer <your-admin-pak>" \
  | jq -r '.[] | select(.name=="admin-generator") | .id')


STAGING=$(curl -s -X POST http://localhost:3000/api/v1/agents \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"name\": \"staging-agent\", \"cluster_name\": \"staging-cluster\", \"generator_ids\": [\"${GEN_ID}\"]}" | jq -r '.agent.id')


PROD=$(curl -s -X POST http://localhost:3000/api/v1/agents \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"name\": \"prod-agent\", \"cluster_name\": \"prod-cluster\", \"generator_ids\": [\"${GEN_ID}\"]}" | jq -r '.agent.id')


echo "Staging agent: $STAGING"
echo "Production agent: $PROD"

These two agents are database records only — no agent process is running for them, so nothing will be applied to a real cluster. That’s exactly what we want here: this tutorial is about how the broker decides which agent receives which deployment objects.

Step 2: Label the Agents

Section titled “Step 2: Label the Agents”

Labels are simple tags (e.g., env:staging, tier:web). Annotations are key-value pairs (e.g., region=us-east-1). Both do more than organize and filter: they drive dynamic matching. When the broker computes which stacks an agent is responsible for, a stack that shares any label string — or any annotation key and value — with the agent is associated with it automatically. Labels are typically used for broad categories, while annotations carry more specific values.

Add environment labels to each agent. A label has the form key:value:

Terminal window
brokkr agent label staging-agent env:staging
brokkr agent label prod-agent env:production
The same step with curl

Agent labels take an object, unlike stack labels. The body repeats the agent id.

Terminal window
curl -s -X POST "http://localhost:3000/api/v1/agents/${STAGING}/labels" \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"agent_id\": \"${STAGING}\", \"label\": \"env:staging\"}" | jq .


curl -s -X POST "http://localhost:3000/api/v1/agents/${PROD}/labels" \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"agent_id\": \"${PROD}\", \"label\": \"env:production\"}" | jq .

Add a region annotation to the production agent:

Terminal window
curl -s -X POST "http://localhost:3000/api/v1/agents/${PROD}/annotations" \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"agent_id\": \"${PROD}\", \"key\": \"region\", \"value\": \"us-east-1\"}" | jq .

Verify the labels are set. The LABELS column shows them:

Terminal window
brokkr agent list
The same step with curl
Terminal window
curl -s "http://localhost:3000/api/v1/agents/${STAGING}/labels" \
  -H "Authorization: Bearer <your-admin-pak>" | jq '.[].label'


curl -s "http://localhost:3000/api/v1/agents/${PROD}/labels" \
  -H "Authorization: Bearer <your-admin-pak>" | jq '.[].label'

Step 3: Create Stacks with Matching Labels

Section titled “Step 3: Create Stacks with Matching Labels”

Now create two stacks — one for staging, one for production — and label them to match the agents.

How association works: Agents are associated with stacks two ways, OR-ed together: (1) dynamic matching — a stack that shares any label string or any annotation key+value with the agent is associated at query time (no target records are created), and (2) explicit agent targets (Step 4) — unconditional bindings that hold regardless of labels. Once the env:staging label below is in place, the staging agent is already associated with the staging stack.

Note that creating an explicit target (Step 4) additionally requires the agent to be registered with the stack’s owning generator — we satisfied that in Step 1 via generator_ids. An existing agent can be registered later with the registration API; see agent registration.

Stacks need an owning generator; we already captured the admin-generator ID as GEN_ID in Step 1:

Terminal window
# Create staging stack
STAGING_STACK=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"name\": \"myapp-staging\", \"description\": \"My app - staging environment\", \"generator_id\": \"${GEN_ID}\"}" \
  | jq -r '.id')


# Create production stack
PROD_STACK=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"name\": \"myapp-production\", \"description\": \"My app - production environment\", \"generator_id\": \"${GEN_ID}\"}" \
  | jq -r '.id')

Label each stack to match its agent:

Terminal window
brokkr stack label myapp-staging env:staging
brokkr stack label myapp-production env:production
The same step with curl

A stack label is a JSON string, not an object.

Terminal window
curl -s -X POST "http://localhost:3000/api/v1/stacks/${STAGING_STACK}/labels" \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '"env:staging"' | jq .


curl -s -X POST "http://localhost:3000/api/v1/stacks/${PROD_STACK}/labels" \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '"env:production"' | jq .

Step 4: Target Agents to Stacks

Section titled “Step 4: Target Agents to Stacks”

The matching env:* labels already associate each agent with its stack. You can additionally pin each pairing with an explicit agent target — an unconditional binding that keeps working even if labels are later changed or removed. To create a target, the agent must be registered with the stack’s owning generator (which we ensured by passing the admin-generator ID in generator_ids when creating the agents in Step 1); otherwise the broker rejects the request with an agent_not_registered error. The first argument is the stack, the second is the agent:

Terminal window
brokkr stack target myapp-staging staging-agent
brokkr stack target myapp-production prod-agent
The same step with curl

The body requires both agent_id and stack_id:

Terminal window
curl -s -X POST "http://localhost:3000/api/v1/agents/${STAGING}/targets" \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"agent_id\": \"${STAGING}\", \"stack_id\": \"${STAGING_STACK}\"}" | jq .


curl -s -X POST "http://localhost:3000/api/v1/agents/${PROD}/targets" \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"agent_id\": \"${PROD}\", \"stack_id\": \"${PROD_STACK}\"}" | jq .

Step 5: Deploy to Staging Only

Section titled “Step 5: Deploy to Staging Only”

Push a deployment to the staging stack. Only the staging agent will receive it:

Terminal window
curl -s -X POST "http://localhost:3000/api/v1/stacks/${STAGING_STACK}/deployment-objects" \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: myapp-staging\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: myapp\n  namespace: myapp-staging\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: myapp\n  template:\n    metadata:\n      labels:\n        app: myapp\n    spec:\n      containers:\n      - name: myapp\n        image: nginx:1.27\n        env:\n        - name: ENVIRONMENT\n          value: staging"
  }' | jq .

Because these agents are database records with no running process, no deployment events will ever appear — events are reported by live agents after they apply resources. Instead, verify the association by asking the broker what each agent would receive. The new deployment object shows up only in the staging agent’s target state:

Terminal window
# Staging agent's target state includes the new object
curl -s "http://localhost:3000/api/v1/agents/${STAGING}/target-state" \
  -H "Authorization: Bearer <your-admin-pak>" | jq '.[] | {stack_id, sequence_id}'


# Production agent's target state is still empty
curl -s "http://localhost:3000/api/v1/agents/${PROD}/target-state" \
  -H "Authorization: Bearer <your-admin-pak>" | jq '.[] | {stack_id, sequence_id}'

Step 6: Deploy to Production

Section titled “Step 6: Deploy to Production”

Now push to production with a different replica count:

Terminal window
curl -s -X POST "http://localhost:3000/api/v1/stacks/${PROD_STACK}/deployment-objects" \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d '{
    "yaml_content": "apiVersion: v1\nkind: Namespace\nmetadata:\n  name: myapp-production\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: myapp\n  namespace: myapp-production\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: myapp\n  template:\n    metadata:\n      labels:\n        app: myapp\n    spec:\n      containers:\n      - name: myapp\n        image: nginx:1.27\n        env:\n        - name: ENVIRONMENT\n          value: production"
  }' | jq .

Re-run the target-state checks from Step 5: the production agent’s target state now contains the 3-replica object, while staging still sees only its own 1-replica object — each environment gets exactly what it needs.

Step 7: Create a Shared Stack

Section titled “Step 7: Create a Shared Stack”

What if you have infrastructure (like monitoring) that should go to both environments? This stack carries no labels, so dynamic matching won’t associate it with anything — explicit targets are how it reaches both agents:

Terminal window
# Create shared stack
SHARED_STACK=$(curl -s -X POST http://localhost:3000/api/v1/stacks \
  -H "Authorization: Bearer <your-admin-pak>" \
  -H "Content-Type: application/json" \
  -d "{\"name\": \"monitoring-shared\", \"description\": \"Monitoring stack for all clusters\", \"generator_id\": \"${GEN_ID}\"}" \
  | jq -r '.id')


# Both agents target the shared stack
brokkr stack target monitoring-shared staging-agent
brokkr stack target monitoring-shared prod-agent

With curl, these are the same POST /agents/{id}/targets calls as in Step 4, with ${SHARED_STACK} as the stack_id.

Now any deployment object pushed to monitoring-shared will be applied by both agents.

Step 8: Verify Targeting

Section titled “Step 8: Verify Targeting”

Review each agent’s explicit target list:

Terminal window
echo "=== Staging Agent Targets ==="
curl -s "http://localhost:3000/api/v1/agents/${STAGING}/targets" \
  -H "Authorization: Bearer <your-admin-pak>" | jq '.[].stack_id'


echo "=== Production Agent Targets ==="
curl -s "http://localhost:3000/api/v1/agents/${PROD}/targets" \
  -H "Authorization: Bearer <your-admin-pak>" | jq '.[].stack_id'

Staging should show two stacks (its own + shared), and production should also show two (its own + shared). To see the full association — explicit targets and label/annotation matches combined — ask for the agent’s associated stacks instead:

Terminal window
curl -s "http://localhost:3000/api/v1/agents/${STAGING}/stacks" \
  -H "Authorization: Bearer <your-admin-pak>" | jq '.[].name'

Clean Up

Section titled “Clean Up”

Remove the test resources:

Terminal window
# Delete stacks (soft delete)
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${STAGING_STACK}" \
  -H "Authorization: Bearer <your-admin-pak>"
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${PROD_STACK}" \
  -H "Authorization: Bearer <your-admin-pak>"
curl -s -X DELETE "http://localhost:3000/api/v1/stacks/${SHARED_STACK}" \
  -H "Authorization: Bearer <your-admin-pak>"


# Delete agents (soft delete)
curl -s -X DELETE "http://localhost:3000/api/v1/agents/${STAGING}" \
  -H "Authorization: Bearer <your-admin-pak>"
curl -s -X DELETE "http://localhost:3000/api/v1/agents/${PROD}" \
  -H "Authorization: Bearer <your-admin-pak>"

What You’ve Learned

Section titled “What You’ve Learned”
  • Labels like env:staging do double duty: they categorize resources and dynamically associate any stack sharing a label with the agent
  • Annotations like region=us-east-1 participate in matching the same way, on key+value
  • Agent targets create explicit, unconditional bindings — the only mechanism for unlabeled stacks like the shared one
  • An agent’s effective stack set is the union of label matches, annotation matches, and explicit targets
  • Multiple agents can be associated with the same stack for shared infrastructure
  • Use GET /agents/{id}/stacks or GET /agents/{id}/target-state to see what an agent will receive

Next Steps

Section titled “Next Steps”