Webhooks Reference

This reference documents Brokkr’s webhook system for receiving real-time event notifications via HTTP callbacks.

Overview

Webhooks enable external systems to receive notifications when events occur in Brokkr. The system supports:

• Subscription-based event filtering
• Broker or agent-side delivery
• Automatic retries with exponential backoff
• Encrypted URL and authentication storage

Event Types

Agent Events

Stack Events

Deployment Events

Work Order Events

Wildcard Patterns

API Reference

Subscription Endpoints

List Subscriptions

GET /api/v1/webhooks
Authorization: Bearer <admin_pak>

Response:

[
  {
    "id": "uuid",
    "name": "string",
    "has_url": true,
    "has_auth_header": false,
    "event_types": ["deployment.*"],
    "filters": null,
    "target_labels": null,
    "enabled": true,
    "max_retries": 5,
    "timeout_seconds": 30,
    "created_at": "2025-01-02T10:00:00Z",
    "updated_at": "2025-01-02T10:00:00Z",
    "created_by": "admin"
  }
]

Create Subscription

POST /api/v1/webhooks
Authorization: Bearer <admin_pak>
Content-Type: application/json

Request body:

{
  "name": "string (required)",
  "url": "string (required, http:// or https://)",
  "auth_header": "string (optional)",
  "event_types": ["string (required, at least one)"],
  "filters": {
    "agent_id": "uuid (optional)",
    "stack_id": "uuid (optional)",
    "labels": {"key": "value"}
  },
  "target_labels": ["string (optional)"],
  "max_retries": 5,
  "timeout_seconds": 30,
  "validate": false
}

Response: 201 Created with subscription object

Get Subscription

GET /api/v1/webhooks/{id}
Authorization: Bearer <admin_pak>

Response: 200 OK with subscription object

Update Subscription

PUT /api/v1/webhooks/{id}
Authorization: Bearer <admin_pak>
Content-Type: application/json

Request body (all fields optional):

{
  "name": "string",
  "url": "string",
  "auth_header": "string or null",
  "event_types": ["string"],
  "filters": {},
  "target_labels": ["string"] or null,
  "enabled": true,
  "max_retries": 5,
  "timeout_seconds": 30
}

Response: 200 OK with updated subscription object

Delete Subscription

DELETE /api/v1/webhooks/{id}
Authorization: Bearer <admin_pak>

Response: 204 No Content

Test Subscription

POST /api/v1/webhooks/{id}/test
Authorization: Bearer <admin_pak>

Sends a test event to the webhook endpoint.

Response:

{
  "success": true,
  "status_code": 200,
  "message": "Test delivery successful"
}

List Event Types

GET /api/v1/webhooks/event-types
Authorization: Bearer <admin_pak>

Response:

[
  "agent.registered",
  "agent.deregistered",
  "stack.created",
  "stack.deleted",
  "deployment.created",
  "deployment.applied",
  "deployment.failed",
  "deployment.deleted",
  "workorder.created",
  "workorder.claimed",
  "workorder.completed",
  "workorder.failed"
]

Delivery Endpoints

List Deliveries

GET /api/v1/webhooks/{id}/deliveries
Authorization: Bearer <admin_pak>

Query parameters:

Response:

[
  {
    "id": "uuid",
    "subscription_id": "uuid",
    "event_type": "deployment.applied",
    "event_id": "uuid",
    "payload": "{}",
    "target_labels": null,
    "status": "success",
    "acquired_by": null,
    "acquired_until": null,
    "attempts": 1,
    "last_attempt_at": "2025-01-02T10:00:00Z",
    "next_retry_at": null,
    "last_error": null,
    "created_at": "2025-01-02T10:00:00Z",
    "completed_at": "2025-01-02T10:00:01Z"
  }
]

Delivery Modes

Broker Delivery (Default)

When target_labels is null or empty, the broker delivers webhooks directly:

1. Event occurs and is emitted
2. Broker matches event to subscriptions
3. Broker creates delivery records
4. Background task claims and delivers via HTTP POST
5. Success/failure is recorded

Use for external endpoints accessible from the broker.

Agent Delivery

When target_labels is set, matching agents deliver webhooks:

1. Event occurs and is emitted
2. Broker creates delivery with target_labels
3. Agent polls for pending deliveries during heartbeat loop
4. Agent claims deliveries matching its labels
5. Agent delivers via HTTP POST from inside cluster
6. Agent reports result back to broker

Use for in-cluster endpoints (e.g., *.svc.cluster.local) that the broker cannot reach.

Label Matching

An agent can claim a delivery only if it has ALL the specified target labels:

Delivery Status

State Transitions

pending → acquired → success
                  → failed → pending (after backoff)
                          → dead (if max_retries exceeded)

Retry Behavior

• Exponential backoff: 2^attempts seconds (2s, 4s, 8s, 16s…)
• Retryable errors: HTTP 5xx, timeouts, connection failures
• Non-retryable errors: HTTP 4xx (except 429)
• TTL: Acquired deliveries expire after 60 seconds if no result reported

Webhook Payload Format

HTTP Headers

Content-Type: application/json
X-Brokkr-Event-Type: deployment.applied
X-Brokkr-Delivery-Id: a1b2c3d4-e5f6-7890-abcd-ef1234567890
Authorization: <configured auth_header>

Body Structure

{
  "id": "event-uuid",
  "event_type": "deployment.applied",
  "timestamp": "2025-01-02T10:00:00Z",
  "data": {
    // Event-specific fields
  }
}

Example Payloads

deployment.applied

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "event_type": "deployment.applied",
  "timestamp": "2025-01-02T10:00:00Z",
  "data": {
    "deployment_object_id": "a1b2c3d4-...",
    "agent_id": "e5f6g7h8-...",
    "status": "SUCCESS"
  }
}

workorder.completed

{
  "id": "550e8400-e29b-41d4-a716-446655440001",
  "event_type": "workorder.completed",
  "timestamp": "2025-01-02T10:05:00Z",
  "data": {
    "work_order_log_id": "b2c3d4e5-...",
    "work_type": "custom",
    "success": true,
    "result_message": "Applied 3 resources successfully",
    "agent_id": "e5f6g7h8-...",
    "completed_at": "2025-01-02T10:05:00Z"
  }
}

workorder.failed

{
  "id": "550e8400-e29b-41d4-a716-446655440002",
  "event_type": "workorder.failed",
  "timestamp": "2025-01-02T10:05:00Z",
  "data": {
    "work_order_log_id": "c3d4e5f6-...",
    "work_type": "build",
    "success": false,
    "result_message": "Build failed: Dockerfile not found",
    "agent_id": "e5f6g7h8-...",
    "completed_at": "2025-01-02T10:05:00Z"
  }
}

Database Schema

webhook_subscriptions

webhook_deliveries

Security Considerations

Encryption at Rest

Webhook URLs and authentication headers contain sensitive information and are encrypted before storage:

• Algorithm: AES-256-GCM (Authenticated Encryption with Associated Data)
• Key management: Encryption key configured via BROKKR__WEBHOOKS__ENCRYPTION_KEY environment variable
• Fields encrypted: url_encrypted, auth_header_encrypted
• Response handling: API responses show has_url: true and has_auth_header: true/false rather than the actual values

When updating a subscription, provide the URL or auth header to re-encrypt with the current key.

Access Control

• Admin-only access: All webhook endpoints require admin PAK authentication
• Agent authentication: Agents use their PAK to fetch and report deliveries
• TLS recommended: Use HTTPS endpoints in production
• Secret rotation: Rotate auth headers by updating the subscription

Data Retention

The webhook system automatically cleans up old delivery records to prevent unbounded database growth:

• Retention period: 7 days
• Cleanup frequency: Every hour
• Scope: All deliveries older than 7 days are permanently deleted, regardless of status
• Subscriptions: Deleted when explicitly removed; delivery history is cleaned up by the retention policy

Deliveries in terminal states (success, dead) are retained for the full 7-day period to support troubleshooting and audit requirements. Adjust retention by modifying the cleanup background task configuration if needed.

Performance Characteristics

Broker Delivery

• Background task polls every 5 seconds
• Batch size: 10 deliveries per cycle
• Concurrent delivery: single-threaded per broker instance

Agent Delivery

• Polling interval: 10 seconds
• Batch size: 10 deliveries per poll
• Concurrent delivery: single-threaded per agent
• TTL: 60 seconds for acquired deliveries

Scaling Considerations

• Multiple broker instances share the delivery workload
• Agent delivery scales with number of matching agents
• Delivery latency: typically < 15 seconds from event to delivery

Related Documentation

• How to Configure Webhooks - Step-by-step setup guide
• Architecture - System architecture overview
• Data Flows - Event flow through the system