inflowDocsLearning path

Optional technical reference. This material is for developers, integration specialists and platform administrators. For the consultant lesson, read the practical guide.

Preserved from source baseline d4587c7c0e073e58c2d430342f65ac55dd701bf4; source/static examples retain their stated scope. They are not new runtime verification or required consultant setup steps.

Current-source role audit (2026-11-13): the role catalog and dispatch mapping below were rechecked against origin/master 4082f871954bd160e160a1b3dc2a40e4f7275c8d / tree e92f6851c1575615341f442853e1ecd192c2b675 (latest product-source commit 4082f871954bd160e160a1b3dc2a40e4f7275c8d). The six role/dispatch source files named in the lesson were unchanged since the prior 951ca2cad286924e8214e80329ab46430ce4fb9c audit. This is source inspection only; it is not a deployment or runtime qualification.

Active-Active Deployment Manual

Practical guide and worked examples

Read the topology as separate responsibilities

A node identity identifies one JVM/container; node roles decide which worker responsibilities it may undertake; a canonical worker-start setting controls whether its loop starts. Broker connectivity, current leases, database health and application readiness are additional conditions. A running API page or an active heartbeat is not proof that an inbox dispatcher, service-task worker or AI worker is executing the intended queue.

For an admitted design, document each actual node ID, origin/zone, role set, canonical configuration and build identity. Keep identity unique across nodes and retain the same intended identity across a controlled restart. Current source offers roles such as api, outbox-publisher, kafka-consumer, inbox-dispatcher, service-task-worker, ai-worker and escalation-worker; use the actual configured source/role implementation, not a generic “worker=true” label. The checked-in node profile is API-only unless deliberately configured otherwise.

Layer Evidence to inspect Typical misleading shortcut
Node identity/role Actual node record and deployment configuration Two processes sharing an ID because a config file was copied.
Worker loop Current worker status on the intended node Calling /current through a load balancer and assuming which node answered.
Transport Reachable bootstrap and advertised addresses A host using a container-only DNS name.
Queue claim Current owner, expiry and due state Treating lease expiry as rollback of a remote request.
Rolling change Exact target node, drain state and remaining capacity Stopping a random API or clearing leases directly in SQL.

Node administrative endpoints require *; they are not ordinary end-user permissions. The procedures are for separately admitted operator work. No node was started, drained, stopped, reconfigured or failed over during this documentation program.

The examples below use synthetic teaching data. They are grounded in the cited source contracts; this documentation change did not execute these operations against a running Inflow deployment. Use normal authenticated sessions and the actual returned identifiers, revisions and versions from an authorized environment.

Example ROOT-AA-01 — Design distinct API and worker identities

Scenario and inputs. The operator is planning two admitted development nodes for People Operations: one serving HTTP and another processing specific background queues. They share compatible durable services but must not impersonate the same node. The plan must identify which work is required and which runtime/provider features are actually enabled in this environment.

Procedure. Read conf/node.properties and the actual deployment overrides. Assign distinct stable INFLOW_NODE_ID values and explicit INFLOW_NODE_ROLES for each intended process. Keep the API node's role limited to api when it should not claim worker jobs. Give the worker only the required queue roles, then review the canonical platform.worker.autoStart setting and other module prerequisites. Document broker/database addresses reachable from each process rather than assuming container and host networking are identical.

Expected result. The plan separates identity, role eligibility and loop activation. Current heartbeats/readiness can later be compared against the intended topology in an admitted run. A role string or a true feature flag alone does not prove the worker actually started, claimed a job or completed an external business effect.

Failure and recovery. If two active processes report one identity, correct the deployment ownership/configuration before relying on their lease/readiness evidence. Do not rotate another lane's identity, start a second unowned worker or delete heartbeat/lease rows to make the plan appear healthy. Preserve current resource, runtime, provider and source-custody holds while qualifying the owned topology.

Sources. node.properties:1, PlatformNodeProperties.java:1, DefaultWorkerRuntimeService.java:1.

Example ROOT-AA-02 — Diagnose a healthy API with an idle queue

Scenario and inputs. The API answers normal reads, yet the expected leave notification or service task remains pending. The operator has the original work identity and allowed operational access. The problem may be a missing role, stopped loop, disabled module, unavailable dependency, due-time constraint or active lease; simply restarting the API would hide which condition actually mattered.

Procedure. Read the intended node record and active-node list through /api/nodes/<node-id> and /api/nodes/active. For current-loop details, address the intended node and read /api/nodes/current/worker; through a load balancer, retain the answering node identity rather than guessing. Compare roles and canonical worker activation with the due work, its current owner/expiry and the owning worker's safe failure metadata. Check the actual broker/provider boundary only within its separate permitted diagnostic scope.

Expected result. The investigation can distinguish serving HTTP from claiming a particular queue. A pending job with a live owner is different from an unclaimed job or an unsupported role. A worker-status read is operational evidence for that node, not proof of completed user-visible business work or permission to run the same job elsewhere.

Failure and recovery. Do not claim an arbitrary batch, invoke run-once on a shared node or start competing loops merely to force progress. Preserve the original work and uncertain effect receipt. Correct the specific admitted configuration/dependency with its owner, then use the native domain outcome to establish whether the intended work actually completed.

Sources. NodeController.java:61, NodeController.java:48, DefaultWorkerRuntimeService.java:1.

Example ROOT-AA-03 — Drain a known node for a controlled rolling change

Scenario and inputs. A separately approved rolling change will remove one identified node while preserving the required active capacity. The operator has *, the exact target node ID and the environment's existing availability/resource plan. A load-balanced “current node” route cannot reliably select which instance should leave service.

Procedure. Read the target node and remaining active capacity first. When the approved drain step is due, use the explicit target route:

POST /api/nodes/<actual-node-id>/drain

Observe the target's state/readiness and existing in-flight work under the admitted operating plan before stopping its process. To return that draining node to active eligibility after the approved change, use /api/nodes/<actual-node-id>/resume. This node resume is not a workflow case activation or token-continuation API.

Expected result. The native drain moves the identified node to DRAINING, lowers its readiness and stops new worker claims from the next relevant cycle. It records the native drain audit. Existing dispatched remote requests are not recalled or undone by that state change. The source has specific not-found/not-drainable/last-active/not-draining refusals rather than an unconditional toggle.

Failure and recovery. A NODE_LAST_ACTIVE or other state refusal must be resolved through the legitimate capacity/target plan; do not edit the node table or globally kill services. Preserve outstanding work and reconcile original receipts before replacement processing. If the intended target is unclear, stop the change and identify it rather than draining whichever backend answered a convenient URL.

Sources. NodeController.java:90, NodeController.java:101, NodeController.java:115.

Example ROOT-AA-04 — Choose broker addresses for the actual network boundary

Scenario and inputs. A host-run API cannot resolve the Compose-internal name used by container workers, or a broker advertises an address unreachable from one client. The operator has the admitted Compose/network configuration and is diagnosing infrastructure routing, not changing a registered product feature flag.

Procedure. Inspect both bootstrap and advertised listener addresses from the owned deployment. MessagingProperties.bootstrapServers() resolves the JVM infrastructure property messaging.kafka.bootstrapServers, then MESSAGING_KAFKA_BOOTSTRAP_SERVERS, then the messaging properties file, then its container default. Prepare the reachable host/port for that specific process through the allowed deployment mechanism. Keep internal broker names for clients inside the appropriate network and use the actual admitted published listener for host clients; do not assume a universal port or copy another lane's endpoint.

Expected result. Infrastructure broker routing remains distinct from canonical feature switches: the retirement of feature-value environment aliases does not remove this documented infrastructure override. A corrected bootstrap address still needs compatible advertised listeners, credentials/trust and the actual worker role/loop before delivery can be qualified.

Failure and recovery. Do not edit tracked shared defaults to accommodate one host, disable transport security or reconfigure another lane's broker. If startup fails while constructing messaging dependencies, retain that failure and trace the actual reachable listener rather than describing the rest of the platform as healthy. A source configuration comparison is not an executed broker-delivery proof.

Sources. MessagingProperties.java:49, docker-compose.yml:1, messaging.properties:1.

Inflow is designed so many API and worker nodes can run at the same time. API nodes should stay stateless. Durable state belongs in the database, and broker delivery is coordinated through the outbox/inbox tables.

Target Topology

flowchart TD
    USERS["Users / External Systems"] --> LB["Load Balancer"]
    LB --> API1["API Node 1"]
    LB --> API2["API Node 2"]
    LB --> APIN["API Node N"]

    API1 --> DB[("HA Database")]
    API2 --> DB
    APIN --> DB

    API1 --> OUTBOX[("message_outbox")]
    API2 --> OUTBOX
    APIN --> OUTBOX

    OUTBOX --> PUB["Outbox Publisher Nodes"]
    PUB --> BROKER["Kafka / Redpanda Cluster"]
    BROKER --> CONSUMER["Kafka Consumer Nodes"]
    CONSUMER --> INBOX[("message_inbox")]
    INBOX --> DISPATCH["Inbox Dispatcher Nodes"]
    DISPATCH --> SERVICES["Runtime / Task / AI / Integration / Notification Services"]
    SERVICES --> DB

Node Heartbeats

Every JVM runs a node heartbeat lifecycle. It registers itself in platform_node_heartbeats at startup and refreshes the row on platform.node.heartbeatSeconds, regardless of whether the node is API-only or also running workers. The row contains:

  • node_id: stable node identity
  • roles_csv: API or worker roles
  • host_name, process_id, region, zone_name
  • status: STARTING, ACTIVE, DRAINING, STOPPED, UNHEALTHY
  • started_at, last_seen_at

Stale STARTING or ACTIVE nodes are marked UNHEALTHY after platform.node.staleAfterSeconds.

Node Roles

Configure roles in conf/node.properties or environment variables. A non-blank environment variable wins over the file, so one checked-in file can serve every node while each container is given its own identity and duties:

Environment variable Overrides
INFLOW_NODE_ID platform.node.id
INFLOW_NODE_ROLES platform.node.roles
INFLOW_NODE_REGION platform.node.region
INFLOW_NODE_ZONE platform.node.zone

Process timers (intermediate and event-gateway timers, activity and user-task timeouts, retry backoff) are claimed only by a node with the timer-worker role whose worker loop is running. At least one such node must exist in every deployment that runs processes; otherwise every timer row stays PENDING with zero attempts. The single-node Docker Compose stack (api-dev) sets INFLOW_NODE_ROLES with its complete single-node worker role set, and its explicit canonical development profile sets platform.worker.autoStart=true for this reason. Worker startup is a restart-only feature in custom_config.properties; node identity/roles retain the environment precedence above. See Feature switches.

In production, set a stable unique node id through platform.node.id or INFLOW_NODE_ID. When left blank, the platform falls back to the detected host/container name, which is convenient for local development but should not be treated as an explicit deployment identity.

API node:

platform.node.roles=api
platform.worker.autoStart=false

Outbox publisher node:

platform.node.roles=outbox-publisher
platform.worker.autoStart=true

Kafka consumer plus inbox dispatcher node:

platform.node.roles=kafka-consumer,inbox-dispatcher
platform.worker.autoStart=true

Combined worker node:

platform.node.roles=outbox-publisher,kafka-consumer,inbox-dispatcher
platform.worker.autoStart=true

Timer worker node:

platform.node.roles=timer-worker
platform.worker.autoStart=true

Service-task worker node:

platform.node.roles=service-task-worker
platform.worker.autoStart=true

Escalation worker node:

platform.node.roles=escalation-worker
platform.worker.autoStart=true

The escalation-worker role drives the SLA-breach escalation sweep (EscalationSweepWorker) across every active workspace. It is gated by the process.module.enabled feature flag: with the flag off, the role is a no-op (each pass returns immediately without querying any workspace), so it is safe to assign ahead of enabling the Process module. Run it on a single node — the sweep's per-task claim is a conditional database update, so multiple escalation nodes stay correct, but co-locating the role on one node avoids duplicate work.

Worker Run Loop

When platform.worker.autoStart=true, Spring starts WorkerRuntimeLifecycle. The lifecycle delegates to WorkerRuntimeLoop, which repeatedly calls WorkerRuntimeService.runOnce() for the node's configured roles.

Run-loop controls live in conf/node.properties:

platform.worker.batchSize=100
platform.worker.loopIntervalMillis=1000
platform.worker.errorBackoffMillis=5000
platform.worker.stopTimeoutMillis=2000
platform.worker.kafkaPollTimeoutMillis=1000
  • platform.worker.loopIntervalMillis: idle delay after a successful cycle
  • platform.worker.errorBackoffMillis: delay after a failed cycle before retry
  • platform.worker.stopTimeoutMillis: graceful shutdown join time after the lifecycle interrupts the worker thread
  • platform.worker.batchSize: maximum rows each worker phase claims per cycle
  • platform.worker.kafkaPollTimeoutMillis: broker poll timeout for Kafka / Redpanda consumer nodes

On shutdown, the lifecycle marks the node DRAINING, interrupts the worker thread, waits up to platform.worker.stopTimeoutMillis, then marks it STOPPED once the loop exits. If a worker cycle throws, the loop records the failure through logs and waits for platform.worker.errorBackoffMillis instead of hot-looping.

Node Control API

Operators can inspect node state and control the current JVM's worker runtime through the node API:

GET  /api/nodes/current
GET  /api/nodes/active
GET  /api/nodes/{nodeId}
GET  /api/nodes/current/worker
POST /api/nodes/current/worker/run-once
POST /api/nodes/current/worker/start
POST /api/nodes/current/worker/stop
POST /api/nodes/{nodeId}/drain
POST /api/nodes/{nodeId}/resume

GET /api/nodes/current/worker returns the current node id, roles, node status, whether the worker lifecycle is running, and the effective worker configuration. POST /api/nodes/current/worker/run-once executes one worker cycle synchronously and returns the processed counts from that cycle. The start and stop endpoints delegate to the same lifecycle used by Spring auto-start, so they mark node state consistently with normal deployment startup and shutdown.

Draining a node for a rolling deploy

POST /api/nodes/{nodeId}/drain (SCRUM-699) marks any node's heartbeat row DRAINING, by id: behind a load balancer you cannot choose which node answers, so the row is the signal and the target node reads it on its next heartbeat. A draining node:

  • keeps DRAINING on every heartbeat. The heartbeat is one conditional upsert, so a drain or a resume that lands between two heartbeats is never lost (SCRUM-4001). A node that stops heartbeating for longer than staleAfter is marked UNHEALTHY and returns ACTIVE on its next heartbeat, and a drained node that its orchestrator restarts quickly comes back still DRAINING;
  • reports down on /actuator/health/platformNodeReadiness and leaves GET /api/nodes/active, so the load balancer stops routing to it;
  • claims no new work: each worker cycle heartbeats and then runs no claim phase (outbox, broker poll, timers, service tasks and their timeout recovery, agent runs and their timeout recovery, record audit outbox, inbox dispatch, escalations). A cycle already running finishes the work it claimed.

Rolling deploy order:

  1. Drain the node.
  2. Confirm it: GET /api/nodes/{nodeId} shows DRAINING. A node that runs the worker runtime also logs once, on its next worker cycle, "Node … is DRAINING: it claims no new work until POST /api/nodes/{nodeId}/resume" (an API-only node has no worker cycle and logs nothing).
  3. Wait one platform.worker.loopIntervalMillis plus the longest cycle (or watch its worker metrics go quiet).
  4. Stop it; the shutdown lifecycle marks it STOPPED. Drain is allowed from ACTIVE or STARTING; draining a node that is already DRAINING returns it unchanged. A STOPPED or UNHEALTHY node answers 409 NODE_NOT_DRAINABLE, and an unknown id (or a row deleted while the drain runs) 404 NODE_NOT_FOUND. Drain and resume change the status in one conditional statement: a node that moved to another status in between is refused, never overwritten, and an operator transition does not refresh the node's last-seen time.

Never drain the last active node. A drain that would leave no other ACTIVE node answers 409 NODE_LAST_ACTIVE (SCRUM-4001). The check and the change are one serialized step, so two operators draining the last two nodes at the same moment cannot both succeed. The guard counts any ACTIVE node, whatever its roles: keep at least one node that serves the API in rotation, or /resume must be sent to a node directly. Draining every node would drop readiness everywhere, so /resume could not be reached through the load balancer. On a single-node deployment draining is therefore refused; to take that node out, stop it (its outbox, inbox and timer processing stop with it) and restart it. If a node is somehow left DRAINING behind a readiness-gated load balancer, send /resume to the node directly (bypassing the balancer).

POST /api/nodes/{nodeId}/resume returns a DRAINING node to ACTIVE (a drain made by mistake, or a node restarted with the same INFLOW_NODE_ID before its row went stale, which otherwise keeps it draining). An ACTIVE node is returned unchanged; any other status answers 409 NODE_NOT_DRAINING. Both endpoints need the node administrator permission, like the rest of this API. Other background lifecycles (document extraction, data and Fabric sync, retention jobs, the inbound mail consumer, cache refresh) are not gated by DRAINING; stopping the node stops them.

Worker control commands are audit-recorded in audit_events:

  • WORKER_RUN_ONCE_REQUESTED
  • WORKER_START_REQUESTED
  • WORKER_STOP_REQUESTED
  • NODE_DRAIN_REQUESTED and NODE_DRAIN_RESUMED (entity id: the drained node, metadata previousStatus and status); a drain or resume that changes nothing records nothing

Each event uses entity_type = platformNode, the current node id as entity_id, the authenticated operator as actor, and metadata containing the effective worker state or processed counts.

Operators can search those events through the audit API:

GET /api/audit-events?type=WORKER_RUN_ONCE_REQUESTED&entityType=platformNode&page=0&size=50
GET /api/audit-events?entityType=platformNode&entityId={nodeId}

The response is a paged result with items, page, size, and totalItems. Each item includes the event type, entity reference, actor, metadata, and timestamp.

Load Balancing

Put a load balancer in front of API nodes. API nodes must not store workflow, task, or authentication session state in local memory. Use tokens, API keys, or shared identity storage so any API node can handle the next request.

Built-in role grants are seeded by every API node when it starts. A grant a newer build has withdrawn from a built-in role is deleted on that build's start and ignored when it reads role permissions. For example, SCRUM-3763 withdrew PROCESS_INSTANCE_APPROVE from worker. An older node in the same fleet re-seeds such a grant when it starts and holds it while it runs (for SCRUM-3763 this confers nothing in practice, because builds before it have no working approval route); newer nodes ignore and later delete the row. Finish rolling upgrades promptly, and expect a rollback to restore the old build's behaviour on the old nodes.

Failover

Work claiming is database-coordinated:

  • outbox publishers claim rows from message_outbox
  • inbox dispatchers claim rows from message_inbox
  • claims use locks with locked_by, locked_until, attempts, and retry status
  • if a node dies, its locks expire and another node can claim the work
  • worker completion/failure updates verify the current locked_by owner and a non-expired locked_until before changing message state
  • long-running owners can explicitly renew a lease before it expires or release it back to the queue without recording a failure
  • expired PROCESSING messages are claimable, so a node crash does not leave work stuck forever

Outbox publishers and inbox dispatchers expose owner-checked lease operations:

  • MessageOutboxService.renewPublishLease(...)
  • MessageOutboxService.releasePublishLease(...)
  • MessageInboxService.renewProcessingLease(...)
  • MessageInboxService.releaseProcessingLease(...)

Renewal extends locked_until only when the caller still owns a non-expired PROCESSING row. Release clears locked_by and locked_until, moves the row back to PENDING, and sets available_at so another node can claim it without counting it as a failed attempt.

Runtime timer jobs use the same pattern. Due rows in runtime_timer_jobs are claimed with locked_by, locked_until, attempts, and status. A timer worker can only mark a timer COMPLETED, FAILED, or DEAD_LETTER while it still owns the non-expired lease. Expired PROCESSING timers can be reclaimed by another timer worker.

Timer workers can also call RuntimeTimerJobService.renewLease(...) while a timer callback is still resuming a process, or RuntimeTimerJobService.releaseLease(...) during clean shutdown. The service delegates to the owner-checked repository operations. Release moves the job back to PENDING, clears lease ownership, and sets due_at so a different timer worker can claim it without recording a failed attempt.

The worker HTTP API exposes the same timer lease operations:

POST /api/runtime-timer-jobs/{timerJobId}/renew-lease
POST /api/runtime-timer-jobs/{timerJobId}/release-lease

These endpoints require RUNTIME_TIMER_LEASE_MANAGE.

Service task runs use the same lease model in service_task_runs. Platform service-task workers claim only local Java-backed implementations, while external workers can claim generic standard-work-item runs through the worker HTTP API. This keeps built-in workers and external connector/document/RPA workers from competing for the wrong work.

Long-running service-task workers can call ServiceTaskExecutionService.renewLease(...) while they are still processing or ServiceTaskExecutionService.releaseLease(...) when they need to give the work back without marking it failed. Release moves the run back to PENDING and logs the release, keeping the attempt history visible while allowing another active node to continue.

Service-task worker nodes also run timeout recovery before each processing loop. A stale PROCESSING service-task run whose current attempt has exceeded its configured timeoutSeconds is released, marked FAILED or DEAD_LETTER according to maxAttempts, and logged with the recovering node id. This makes service-task failover active/active friendly: another node can continue after the retry backoff instead of waiting for a manual cleanup.

Agent runs use the same pattern in agent_runs. AI worker nodes claim pending agent runs with row-level leases and FOR UPDATE SKIP LOCKED; completion and failure require the current lease owner. Agent workers only claim work when an AgentRunExecutor is available, so deployments can add local-model, AI Gateway, tool-calling, or human-handoff executors independently.

Long-running AI workers can call AgentExecutionService.renewLease(...) to extend ownership while a model, tool, or agent loop is still running. They can call AgentExecutionService.releaseLease(...) to return the run to PENDING without recording a failed attempt. Both operations require the current non-expired locked_by owner and write lease logs.

AI worker nodes also recover stale RUNNING agent runs before each processing loop. If a current attempt exceeds timeoutSeconds, the run is released, closed as a failed attempt, and moved back to PENDING or finally to FAILED based on maxAttempts. The recovery log includes the node id that performed the sweep, which keeps agent execution recoverable across active/active nodes.

Runtime execution state is also becoming durable. Active/waiting/consumed tokens are stored in runtime_execution_tokens, and process variables are stored in runtime_variables. Before resume, signal, or cancel, a restorable engine can reload the BPMN model and variables so a fresh node can continue work started by another node.

Runtime resume also has an atomic token-consumption guard. A token can only be claimed or consumed from ACTIVE or WAITING. Claims write locked_by and locked_until; a different worker cannot consume the token until the lease expires. Resume claims and consumes the token in one transaction, so the claim is an internal resume guard and no lease stays held afterwards. The token-lease renew/release HTTP routes were retired by SCRUM-3764 (they answer 404); the PROCESS_INSTANCE_LEASE_MANAGE permission is kept for stored scopes and grants but no route checks it. Replayed resume requests against the same token are rejected. In JDBC deployments wired with a PlatformTransactionManager, runtime state changes commit as one transaction, so a failed engine resume rolls back token consumption and work item completion.

sequenceDiagram
    participant W1 as Worker Node 1
    participant W2 as Worker Node 2
    participant DB as Database

    W1->>DB: claim pending inbox row
    DB-->>W1: row locked_by=node-1
    W2->>DB: claim pending inbox row
    DB-->>W2: different row or none
    W1--xDB: node dies before processing
    Note over DB: lock expires
    W2->>DB: claim expired row
    DB-->>W2: row locked_by=node-2

Health And Readiness

Actuator is enabled for load balancer and orchestrator checks. The platform node health indicator reports down when the current node does not have a fresh ACTIVE heartbeat.

Recommended probes:

GET /actuator/health
GET /actuator/health/platformNodeReadiness

Operations Dashboard API

Admins can read the platform operations snapshot from:

GET /api/operations/dashboard

This endpoint requires admin access (Permission.ALL). It is read-only and does not claim, retry, or mutate work. The response combines:

  • service-task catalog counts and metadata-quality signals
  • connector catalog readiness and protocol/category counts
  • service_task_runs totals, dispatchable count, locked count, and status counts
  • message_outbox totals, dispatchable count, locked count, and status counts
  • message_inbox totals, dispatchable count, locked count, and status counts
  • runtime_execution_tokens active/waiting token backlog, locked count, and status counts
  • runtime_timer_jobs retryable timer backlog, locked count, and status counts
  • agent_runs AI/agent execution backlog, locked count, and status counts
  • connector execution workload totals, success/failure counts, status counts, recency, and duration signals
  • Fabric materialization workload totals, success/failure counts, status counts, recency, and duration signals
  • active platform nodes, status counts, and role counts

For retry queues, dispatchable means the row is retryable, its due/available time has arrived, and it is not currently locked by a worker. For runtime execution tokens, dispatchable means the token is ACTIVE or WAITING and has no live lease. For runtime timer jobs, expired PROCESSING leases are also counted as dispatchable because another timer worker can reclaim them. For agent runs, dispatchable means a PENDING or retryable FAILED run whose due time has arrived and whose lease is free. This is the operator's first signal for worker backlog. locked means a worker currently owns the row's lease.

Connector executions and Fabric materializations are not lease queues. They are append-only workload/audit streams, so operations reports them as execution health metrics instead of backlog metrics.

Example shape:

{
  "generatedAt": "2026-07-02T04:35:17Z",
  "serviceTaskCatalog": {
    "serviceTaskCount": 241,
    "byImplementation": {
      "connector-event-stream": 4
    }
  },
  "connectorCatalog": {
    "connectorCount": 37,
    "byProtocol": {
      "EVENT_STREAM": 2
    }
  },
  "serviceTaskRuns": {
    "total": 12,
    "dispatchable": 4,
    "locked": 2,
    "byStatus": [
      {"status": "PENDING", "count": 4}
    ]
  },
  "messageOutbox": {
    "total": 20,
    "dispatchable": 8,
    "locked": 1
  },
  "messageInbox": {
    "total": 15,
    "dispatchable": 3,
    "locked": 0
  },
  "runtimeExecutionTokens": {
    "total": 30,
    "dispatchable": 7,
    "locked": 4
  },
  "runtimeTimerJobs": {
    "total": 9,
    "dispatchable": 2,
    "locked": 1
  },
  "agentRuns": {
    "total": 16,
    "dispatchable": 5,
    "locked": 3
  },
  "connectorExecutions": {
    "total": 240,
    "succeeded": 230,
    "failed": 10,
    "averageDurationMillis": 180
  },
  "aiFabricMaterializations": {
    "total": 36,
    "succeeded": 33,
    "failed": 3,
    "averageDurationMillis": 850
  },
  "nodes": {
    "activeCount": 3,
    "currentWindowCount": 3,
    "byRole": {
      "OUTBOX_PUBLISHER": 1,
      "SERVICE_TASK_WORKER": 2
    }
  }
}

Operations Readiness API

Admins can also read a single deployment gate from:

GET /api/operations/readiness

This endpoint requires admin access (Permission.ALL). It does not replace the detailed readiness endpoints for service tasks, connectors, Fabric, queues, or nodes. Instead, it combines them into one operator-facing answer:

  • READY: no blocking errors were found
  • BLOCKED: one or more components has an error that should stop deployment
  • component status:
    • READY: no issues for the component
    • ATTENTION: warnings exist, but no blocking error
    • BLOCKED: at least one blocking error exists

The current aggregate checks include:

  • service-task extension governance and implementation coverage
  • connector readiness, setup actions, rollout warnings, and operational status
  • Fabric catalog readiness issues
  • active platform node visibility in the current heartbeat window
  • failed queue items in service-task, outbox, inbox, runtime-token, runtime-timer, and agent-run queues
  • failed or slow connector/materialization workload streams

Example shape:

{
  "generatedAt": "2026-07-02T07:08:11Z",
  "productionReady": false,
  "status": "BLOCKED",
  "errorCount": 1,
  "warningCount": 4,
  "componentCount": 6,
  "components": {
    "serviceTasks": {
      "status": "BLOCKED",
      "errorCount": 1,
      "warningCount": 2
    },
    "connectors": {
      "status": "READY",
      "errorCount": 0,
      "warningCount": 0
    },
    "aiFabric": {
      "status": "ATTENTION",
      "errorCount": 0,
      "warningCount": 2
    },
    "nodes": {
      "status": "READY",
      "errorCount": 0,
      "warningCount": 0
    },
    "queues": {
      "status": "ATTENTION",
      "errorCount": 0,
      "warningCount": 1
    },
    "workloads": {
      "status": "ATTENTION",
      "errorCount": 0,
      "warningCount": 1
    }
  },
  "issues": [
    {
      "severity": "ERROR",
      "component": "serviceTasks",
      "code": "SERVICE_TASK_EXECUTOR_UNSUPPORTED",
      "itemId": "service.integration.example",
      "message": "Service task has no executor or extension provider."
    }
  ]
}

Operations Metrics API

Admins can read queue and heartbeat pressure metrics from:

GET /api/operations/metrics

This endpoint requires admin access (Permission.ALL) and is read-only. It is designed for dashboards, alert rules, and deployment checks that need numeric signals instead of full work-item payloads.

Queue metrics are returned for:

  • serviceTaskRuns
  • messageOutbox
  • messageInbox
  • runtimeExecutionTokens
  • runtimeTimerJobs
  • agentRuns

Each queue metric includes:

  • total: all rows in the queue table
  • dispatchable: due/available, not locked, retryable rows
  • locked: rows currently leased by a worker
  • failed: rows with FAILED status
  • deadLetter: rows with DEAD_LETTER status
  • retryExhausted: rows whose attempts reached max_attempts, plus failed or dead-letter rows
  • oldestReadyAgeSeconds: age of the oldest dispatchable row
  • averageReadyAgeSeconds: average age of dispatchable rows
  • averageAttempts: average attempts across the table

runtimeExecutionTokens uses the same metric shape so dashboards can render it beside other queues, but token rows are not retry-attempt records. Its dispatchable count is active/waiting unlocked tokens, failed is the number of failed tokens, retryExhausted mirrors failed tokens, and attempt-related values stay 0.

runtimeTimerJobs is a retry queue. Its dispatchable count includes PENDING, FAILED, and expired PROCESSING timers with attempts remaining, because timer workers can reclaim expired processing leases.

agentRuns is the AI worker queue. Its dispatchable count includes PENDING and retryable FAILED runs whose due time has arrived. Agent runs do not use a separate DEAD_LETTER status; an exhausted run remains FAILED, so deadLetter stays 0 and retryExhausted counts terminal failed or max-attempt rows.

Workload metrics are returned for:

  • connectorExecutions
  • aiFabricMaterializations

Each workload metric includes:

  • total: all rows in the workload table
  • succeeded: successful terminal rows
  • failed: failed terminal rows
  • latestAgeSeconds: age of the latest completed row
  • averageDurationMillis: average execution duration
  • maxDurationMillis: slowest execution duration
  • byStatus: status counts

These are intentionally separate from queue metrics. A failed connector execution or materialization can be a historical fact, not necessarily something a worker should reclaim.

Audit metrics are returned in auditEvents. Audit events are append-only operator and platform activity records, so they are reported as an activity stream rather than a queue or success/failure workload.

Audit metrics include:

  • total: all rows in audit_events
  • recentCount: events written during the metrics window
  • distinctActors: unique non-empty actor ids seen in the audit table
  • latestAgeSeconds: age of the most recent audit event
  • byType: count by audit event type
  • byEntityType: count by audited entity type

Node metrics include:

  • registeredNodes: nodes known in the heartbeat table
  • activeNodes: nodes currently marked ACTIVE
  • unhealthyNodes: nodes marked UNHEALTHY
  • oldestHeartbeatAgeSeconds: oldest heartbeat age
  • newestHeartbeatAgeSeconds: newest heartbeat age

Worker metrics are aggregated from append-only samples in platform_worker_metrics. Every active-active node can write its own samples without coordinating with other nodes. The API returns the most recent 15-minute window grouped by workerName and phaseName.

Worker metrics include:

  • sampleCount: number of worker phase samples in the window
  • succeededCount: number of phase samples that completed without throwing
  • failedCount: number of phase samples that threw before returning
  • totalClaimed: total work count reported by the phase
  • totalSucceeded: total successful work count reported by the phase
  • totalFailed: total failed work count reported by the phase
  • averageDurationMillis: average phase duration
  • maxDurationMillis: slowest phase duration in the window
  • throughputPerMinute: successful work count divided by the 15-minute window
  • errorRate: failed work count divided by total completed/failed work count

The response also includes alerts, calculated from the operations policy in conf/custom_config.properties. Alerts are returned as WARN or ERROR and are also folded into GET /api/operations/readiness as operations readiness issues.

Built-in worker phases include:

  • outbox-publisher / drain-outbox
  • kafka-consumer / poll-broker
  • timer-worker / run-due-timers
  • service-task-worker / recover-timeouts
  • service-task-worker / process-due-runs
  • ai-worker / recover-agent-timeouts
  • ai-worker / process-agent-runs
  • inbox-dispatcher / dispatch-inbox

Example shape:

{
  "generatedAt": "2026-07-02T07:15:38Z",
  "serviceTaskRuns": {
    "total": 120,
    "dispatchable": 14,
    "locked": 6,
    "failed": 3,
    "deadLetter": 1,
    "retryExhausted": 2,
    "oldestReadyAgeSeconds": 95,
    "averageReadyAgeSeconds": 31,
    "averageAttempts": 0.45
  },
  "messageOutbox": {
    "total": 42,
    "dispatchable": 8,
    "locked": 2,
    "failed": 1,
    "deadLetter": 0,
    "retryExhausted": 1,
    "oldestReadyAgeSeconds": 21,
    "averageReadyAgeSeconds": 9,
    "averageAttempts": 0.2
  },
  "messageInbox": {
    "total": 18,
    "dispatchable": 4,
    "locked": 1,
    "failed": 0,
    "deadLetter": 0,
    "retryExhausted": 0,
    "oldestReadyAgeSeconds": 12,
    "averageReadyAgeSeconds": 6,
    "averageAttempts": 0.1
  },
  "runtimeExecutionTokens": {
    "total": 30,
    "dispatchable": 7,
    "locked": 4,
    "failed": 1,
    "deadLetter": 0,
    "retryExhausted": 1,
    "oldestReadyAgeSeconds": 42,
    "averageReadyAgeSeconds": 13,
    "averageAttempts": 0
  },
  "runtimeTimerJobs": {
    "total": 9,
    "dispatchable": 2,
    "locked": 1,
    "failed": 1,
    "deadLetter": 1,
    "retryExhausted": 1,
    "oldestReadyAgeSeconds": 120,
    "averageReadyAgeSeconds": 33,
    "averageAttempts": 1.2
  },
  "agentRuns": {
    "total": 16,
    "dispatchable": 5,
    "locked": 3,
    "failed": 1,
    "deadLetter": 0,
    "retryExhausted": 1,
    "oldestReadyAgeSeconds": 75,
    "averageReadyAgeSeconds": 22,
    "averageAttempts": 0.8
  },
  "connectorExecutions": {
    "total": 240,
    "succeeded": 230,
    "failed": 10,
    "latestAgeSeconds": 6,
    "averageDurationMillis": 180,
    "maxDurationMillis": 4200,
    "byStatus": [
      {"status": "FAILED", "count": 10},
      {"status": "SUCCEEDED", "count": 230}
    ]
  },
  "aiFabricMaterializations": {
    "total": 36,
    "succeeded": 33,
    "failed": 3,
    "latestAgeSeconds": 22,
    "averageDurationMillis": 850,
    "maxDurationMillis": 7600,
    "byStatus": [
      {"status": "FAILED", "count": 3},
      {"status": "MATERIALIZED", "count": 33}
    ]
  },
  "auditEvents": {
    "total": 840,
    "recentCount": 42,
    "distinctActors": 12,
    "latestAgeSeconds": 4,
    "byType": [
      {"status": "WORKER_RUN_ONCE_REQUESTED", "count": 7}
    ],
    "byEntityType": [
      {"status": "platformNode", "count": 9}
    ]
  },
  "nodes": {
    "registeredNodes": 3,
    "activeNodes": 3,
    "unhealthyNodes": 0,
    "oldestHeartbeatAgeSeconds": 8,
    "newestHeartbeatAgeSeconds": 1
  },
  "workers": [
    {
      "workerName": "service-task-worker",
      "phaseName": "process-due-runs",
      "sampleCount": 12,
      "succeededCount": 11,
      "failedCount": 1,
      "totalClaimed": 78,
      "totalSucceeded": 76,
      "totalFailed": 2,
      "averageDurationMillis": 241,
      "maxDurationMillis": 900,
      "throughputPerMinute": 5.07,
      "errorRate": 0.03
    }
  ],
  "alerts": [
    {
      "severity": "WARN",
      "component": "queues",
      "metric": "deadLetter",
      "itemId": "serviceTaskRuns",
      "actualValue": 1,
      "thresholdValue": 1,
      "message": "serviceTaskRuns has 1 dead-letter item(s)."
    }
  ]
}

Operations Alert Policy

Operations alert thresholds live in conf/custom_config.properties:

operations.metrics.worker.windowSeconds=900
operations.metrics.worker.retentionDays=30
operations.metrics.retention.autoStart=true
operations.metrics.retention.cleanupIntervalSeconds=86400

operations.alerts.queue.dispatchable.warn=1000
operations.alerts.queue.dispatchable.error=10000
operations.alerts.queue.oldestReadyAgeSeconds.warn=300
operations.alerts.queue.oldestReadyAgeSeconds.error=900
operations.alerts.queue.deadLetter.warn=1
operations.alerts.queue.deadLetter.error=10

operations.alerts.worker.errorRate.warn=0.05
operations.alerts.worker.errorRate.error=0.20
operations.alerts.worker.averageDurationMillis.warn=5000
operations.alerts.worker.averageDurationMillis.error=30000
operations.alerts.worker.throughputPerMinute.warnBelow=0.10

operations.alerts.workload.failed.warn=1
operations.alerts.workload.failed.error=10
operations.alerts.workload.averageDurationMillis.warn=5000
operations.alerts.workload.averageDurationMillis.error=30000

operations.alerts.audit.recentEvents.warn=5000
operations.alerts.audit.recentEvents.error=20000
operations.alerts.audit.latestAgeSeconds.warn=86400
operations.alerts.audit.latestAgeSeconds.error=259200

operations.alerts.node.heartbeatAgeSeconds.warn=60
operations.alerts.node.heartbeatAgeSeconds.error=180

Set an individual threshold to 0 to disable that alert. The worker metrics window controls aggregation, throughput calculations, and the audit recentCount window. Retention controls how long raw worker metric samples should be kept.

operations.metrics.retention.autoStart=true starts a lightweight platform maintenance lifecycle when the Spring application starts. The lifecycle deletes expired platform_worker_metrics rows immediately, then sleeps for operations.metrics.retention.cleanupIntervalSeconds before running again. Disable it in deployments that use an external database job or Kubernetes CronJob for retention.

Operations Metrics Cleanup

Worker metric samples are append-only. The built-in retention lifecycle handles cleanup for normal platform nodes, and the scripts below are useful for manual maintenance, dry runs, or deployments that disable the built-in lifecycle.

Admins can also run and inspect cleanup through the operations API:

POST /api/operations/maintenance/metrics-retention
X-Actor-Id: root
GET /api/operations/maintenance/metrics-retention
X-Actor-Id: root

The POST endpoint deletes expired platform_worker_metrics rows using the configured retention window and returns a persisted maintenance run. The GET endpoint returns the latest run across all active-active nodes. Maintenance runs are stored in operations_maintenance_runs with the maintenance type, status, node id, start and finish timestamps, cutoff timestamp, removed row count, and message.

.\scripts\operations-metrics-clean.ps1 -RetentionDays 30 -DryRun
.\scripts\operations-metrics-clean.ps1 -RetentionDays 30
./scripts/operations-metrics-clean.sh 30 --dry-run
./scripts/operations-metrics-clean.sh 30

The scripts delete platform_worker_metrics rows older than the retention window.

Log Archive Cleanup

The logging subsystem also has an operations maintenance API:

GET /api/operations/maintenance/log-retention/policy
POST /api/operations/maintenance/log-retention?dryRun=true
POST /api/operations/maintenance/log-retention?dryRun=false
GET /api/operations/maintenance/log-retention

dryRun=true is the default and counts expired archive files without deleting them. dryRun=false deletes expired .gz and .log files under logging.dir/archive. The service refuses paths outside the workspace log directory and records runs in operations_maintenance_runs as log-archive-retention.

Database Connection Pooling

Each platform node uses a bounded Hikari connection pool for its runtime database access. Pool settings live in conf/custom_config.properties:

platform.database.url=postgres://inflow:inflow@postgres-dev:5432/inflow?sslmode=disable
platform.database.jdbcUrl=
platform.database.username=
platform.database.password=
platform.database.driverClassName=org.postgresql.Driver
platform.database.pool.name=inflow-platform-db
platform.database.pool.maximumPoolSize=40
platform.database.pool.minimumIdle=5
platform.database.pool.connectionTimeoutMillis=10000
platform.database.pool.idleTimeoutMillis=600000
platform.database.pool.maxLifetimeMillis=1800000
platform.database.pool.validationTimeoutMillis=5000

DATABASE_URL is supported for container deployment and overrides platform.database.url, including the checked-in postgres-dev development default. If platform.database.jdbcUrl is set, it remains the highest-priority explicit override and can point at another JDBC engine, for example MySQL, SQL Server, or Oracle, as long as the matching JDBC driver is available to the runtime. Explicit platform.database.username, password, and driverClassName values continue to override values inferred from the selected URL.

Identity statements have their own time bound (SCRUM-3917). This covers the login-session, user, group, role-permission, password-policy, identity API-key, MFA-factor and SCIM-token reads and writes. Each statement waits at most 5 seconds, and the bound holds inside a longer transaction too. A shorter transaction remainder or a shorter Lifecycle data deadline still wins. A request that meets a stalled identity table is answered 503 with code ACCESS_UNCHECKED and Retry-After: 5, and the person's session is kept. When the pool is exhausted, the wait for a connection (connectionTimeoutMillis) comes first, so one request waits at most about the connection timeout plus 5 seconds. The bound does not cover the workspace and grant tables, the rule and managed API-key stores, or the MCP OAuth token store (follow-up FU-1). It does not help when the database cannot be reached over the network: a cancel cannot reach it. Use a driver socketTimeout or a per-role statement_timeout as the operations backstop for that case (FU-2).

Credential Vault Master Key

Every production deployment must provide a stable INFLOW_VAULT_MASTER_KEY secret to every API node. The credential vault uses this key to encrypt and decrypt stored vault: connector credentials. If the variable is missing, the API creates a random development-only key for that process; credentials saved under that fallback key become unreadable after the process or container restarts.

Production rules:

  • Generate one 32-byte random key per environment and base64-encode it.
  • Store it in the deployment secret manager, not in Git, images, XML catalogs, or config maps.
  • Inject the same value into every API node in the environment.
  • Back it up using the organization's secret-recovery procedure.
  • Treat missing INFLOW_VAULT_MASTER_KEY as a deployment blocker outside local development.
  • Rotate it only through the vault master-key rotation flow, which re-encrypts existing credential_vault rows before the old key is retired.

Generation examples:

openssl rand -base64 32
$bytes = New-Object byte[] 32
$rng = [System.Security.Cryptography.RandomNumberGenerator]::Create()
$rng.GetBytes($bytes)
$rng.Dispose()
[Convert]::ToBase64String($bytes)

For local Docker Compose development, docker-compose.yml includes a stable dev-only fallback key for java-dev and api-dev so local connected-system credentials survive container restarts when no .env file is present. Keep a generated value in an ignored .env file when you need a team- or machine-specific local key, and pass it through as INFLOW_VAULT_MASTER_KEY. Do not reuse the committed local-development fallback or any local .env key for staging or production.

For active-active deployments, size pools as a cluster budget, not only as a single-node setting:

total possible database connections =
  node count * platform.database.pool.maximumPoolSize

Keep that total below the database serve

Per-instance execution serialization (transactional token operations)

Concurrent resume/signal/cancel calls for the SAME process instance — from any number of API or worker nodes — are serialized through a pessimistic per-instance lock: ProcessInstanceRepository.lockById(...) takes SELECT ... FOR UPDATE on the instance row as the first statement of the runtime transaction, BEFORE engine state is restored from the database.

Consequences:

  • Engine decisions (parallel-join firing, instance completion, variable merges) are always made against fresh, uncontended state. Two nodes completing the two branches of a parallel fork at the same instant cannot double-fire or lose the join — proven by ActiveActiveJoinRaceTest, which races two fully independent runtime nodes (separate engines, separate connections, shared database only) over repeated iterations and asserts the downstream task is created exactly once.
  • A single lock acquired first in every mutating runtime call means no lock-ordering deadlocks.
  • Different instances proceed fully in parallel; the serialization is per instance, which matches how BPM engines scale in practice.
  • The lock is a no-op for in-memory adapters (demo/test environments) and requires the transactional runtime wiring (the API configuration always supplies a transaction manager).

Document Management encryption key

Every production node that reads or writes document blobs must receive the same environment-specific DMS encryption key. DMS_ENCRYPTION_KEY is the recommended secret-manager injection path and overrides dms.encryption.key in conf/custom_config.properties. The effective value is a Base64-encoded AES key of 16, 24, or 32 random bytes. Startup deliberately fails when neither source supplies a valid key. Never copy the development Compose fallback into staging or production or commit a production key to the configuration file.

Keep the key stable across rolling deployments and back it up using the protected secret-recovery procedure. Existing document blobs cannot be read after an unplanned key change, and the current provider has no online re-encryption/keyring rotation flow. Before enabling uploads, prove that all nodes can start with the injected key and that a node with both key sources removed fails closed. Operational details are in docs/manuals/document-management.md.

Records opaque-handle key

Every API node that serves the ordinary Records workspace must receive the same INFLOW_RECORD_HANDLE_KEY. Generate an independent 32-byte random value, base64-encode it, and inject it from the deployment secret manager. Do not reuse the credential-vault master key.

Record handles are AES-256-GCM encrypted, expire after eight hours, and are bound to the workspace, actor, model, applied model version, and record type. Changing the key immediately invalidates all outstanding handles; users can recover by refreshing Records. During a rolling deployment, keep the key stable across old and new nodes. The process-local fallback is for direct local Java execution only, logs a warning, and invalidates handles on restart. Docker Compose supplies a stable dev-only fallback that must be overridden in staging and production.

Published Data Action record-reference key

Every API node that issues or consumes a published Data Action record result must receive the same INFLOW_DATA_ACTION_RECORD_REFERENCE_KEY. Generate an independent 32-byte random value, base64-encode it, and inject it from the deployment secret manager. Do not reuse the credential-vault, ordinary record-handle, or audit-receipt keys.

Published record references are AES-256-GCM encrypted, bounded, expire after eight hours, and bind the inner opaque record handle and optimistic baseline to the workspace, actor, screen version, producer action/result, model version, immutable publication, and record type. Changing the key invalidates outstanding uncommitted Data Action references; the interface must reload its named record before attempting a new mutation. A proven committed replay is resolved from the durable receipt before reference-expiry validation.

Keep the key stable across rolling deployments and all active-active nodes. For rotation, deploy the new current key and retain each still-valid old key in the comma-separated INFLOW_DATA_ACTION_RECORD_REFERENCE_PREVIOUS_KEYS keyring until the maximum reference lifetime and any deployment overlap have elapsed. The versioned reference carries a non-secret key fingerprint so a node chooses one exact decryption key rather than attempting every key. New references use only the current key. Remove a previous key only after no valid reference can remain under it.

The process-local fallback is development-only, logs a warning, and invalidates references on restart. Docker Compose supplies a separate stable dev-only fallback that must be overridden in staging and production.

Archive recovery locator key

Every API node must receive the same INFLOW_DATA_ACTION_ARCHIVE_RECOVERY_KEY (an independent 32-byte random value, base64-encoded). The key seals the opaque locator of every Archive recovery episode. Those episodes are written by the Archive Data Action, by a delete in a workspace that forbids permanent removal, and by a generated delete-record rule. The administrative Archived view lists an archived record, and Restore accepts it, only while its locator still authenticates. Locators are long-lived, so never replace this key outright. Deploy the new current key and keep every earlier key in the comma-separated INFLOW_DATA_ACTION_ARCHIVE_RECOVERY_PREVIOUS_KEYS keyring for as long as archived records sealed under it must remain restorable.

Without the variable, the API refuses to start unless development mode is explicitly on. In development mode it uses a process-local key and logs a warning. Every archive made before a restart then disappears from the Archived view and cannot be restored (SCRUM-4366). The listing logs archiveRecoveryEpisodesNotRestorable with a count when it skips such episodes. Docker Compose supplies a stable dev-only fallback that must be overridden in staging and production.

Integration live-test confirmation key

Every API node that issues or verifies a consultant's live Integration test confirmation must receive the same independent INFLOW_INTEGRATION_LIVE_TEST_INTENT_KEY. Generate 32 random bytes, encode them with standard base64, and inject the value from the deployment secret manager. Do not reuse vault, evidence, cursor, record-handle, Data Action, or audit keys.

For rotation, deploy the new current key and retain every still-valid old key in the comma-separated INFLOW_INTEGRATION_LIVE_TEST_INTENT_PREVIOUS_KEYS keyring on every node. New confirmations use only the current key. Keep an old key through the maximum 15-minute confirmation lifetime and the full rolling deployment overlap, then remove it. A missing or malformed current/previous key fails startup outside explicit development mode. Development mode uses a process-local random key unless Docker Compose supplies its stable dev-only fallback; either behavior is intentionally unsuitable for production.

Integration evidence equality key

Every API node that captures or replays sanitized Integration evidence must receive the same independent INFLOW_CONNECTOR_EVIDENCE_HMAC_KEY. Do not reuse the credential-vault master key: rotating vault encryption must not invalidate saved Integration evidence. Keep the evidence key stable across rolling deployments and active-active nodes.

To rotate it, deploy the new current key while retaining the old value in the comma-separated INFLOW_CONNECTOR_EVIDENCE_HMAC_PREVIOUS_KEYS keyring. New evidence uses only the current key; existing evidence continues to verify by its non-secret key identifier. Remove an old key only after its evidence has expired or been deliberately recaptured. A missing old key makes that evidence non-restorable and consultants must run the Integration again. Docker Compose supplies a separate stable development value; production must override it.

Integration-history cursor key

Every API node that serves published Integration history must receive the same INFLOW_INTEGRATION_HISTORY_CURSOR_KEY. Generate an independent 32-byte random value, base64-encode it, and inject it from the deployment secret manager. Do not reuse a credential-vault, record-handle, Data Action, or audit-receipt key.

Integration-history cursors are AES-256-GCM encrypted, expire after 15 minutes, and are bound to the requesting workspace. Keep the key stable across rolling deployments and all active-active nodes; changing it invalidates outstanding cursors, and consultants can recover by reloading the Integration catalog. Startup fails closed outside explicit development mode when the key is absent or malformed. Development mode uses a process-local random key, so cursors are intentionally invalidated when the API restarts.

Target-side record audit receipts

Modeled Records mutations span two independent databases: the publication target and the platform audit store. They therefore do not rely on a Spring transaction or XA coordinator. Each create, update, or delete first verifies a per-model audit-outbox table in the target schema, then commits the business DML and one redacted receipt in the same target JDBC transaction. If receipt insertion fails, the business mutation rolls back. If the platform audit store is unavailable after target commit, the receipt remains pending and the API can still report the correctly committed outcome.

Nodes with the integration-worker role scan applied publications and relay pending receipts. The receipt event id is preserved as the audit_events primary key, so concurrent nodes and crash-after-central-insert retries are idempotent. A replay with the same id but a different payload fails closed. Workers write the platform audit event before marking the target receipt delivered.

Every production node that writes or relays Records must receive the same independent INFLOW_AUDIT_RECEIPT_KEY: a base64-encoded 32-byte random secret. The HMAC is verified before target data becomes platform audit evidence. Do not reuse the vault or record-handle keys. During rotation, provide old 32-byte keys through the comma-separated INFLOW_AUDIT_RECEIPT_PREVIOUS_KEYS setting until their pending receipts drain. A receipt whose key id is temporarily unavailable remains pending with HMAC_KEY_UNAVAILABLE; it is not misclassified as a forged receipt or permanently quarantined. A bad signature under a known key still fails closed and is quarantined. The process-local fallback is development-only; Docker Compose supplies a stable local fallback.

Operational monitoring must alert on pending receipt age, repeated delivery attempts, invalid signatures, unreachable publication targets, and undrained backlog. DataRecordAuditOutboxWorker.statuses(limit) exposes bounded point-in-time snapshots for applied publications, including pending, claimed, failed-attempt, quarantined, and oldest-pending receipt state. These counts are operational telemetry, not deletion-gap proof: a target DBA can delete a not-yet-relayed receipt so it vanishes from every target-side bucket. Strict production deployments also require restricted DDL/DML ownership plus an independent platform intent/gap detector; the remaining hardening work is tracked in the database certification bug ledger.

Application-user access runtime (inflow.application-user-access.enabled)

Set inflow.application-user-access.enabled=true in each node's canonical conf/custom_config.properties and plan a restart on every API node so group application runtime access (application-runtime-access), group home destinations (group-destinations), direct Studio admission policy (studio-policy), and /application-entry routing are active across the cluster. The local Compose launcher prepares a writable canonical copy from its explicit development profile, which enables this flag. Separate deployment/bootstrap owners must stage canonical values before upgrading; old feature-value environment overrides no longer activate it. See Feature switches and migration. Keep this flag consistent across all active-active API nodes so session capabilities (applicationUserAccessEnabled), runtime access previews/grants, and published interface admission behave identically regardless of which node serves the request.

Current role-to-work inventory

The role enum is broader than the short deployment examples above. Use this inventory to decide which role belongs on each process. It describes the current WorkerRuntimeService.runOnce() dispatch mapping; it does not mean that every listed worker is composed, enabled, connected to its dependency or available in every deployment profile.

Configured role Worker phase selected by the current runtime Operational interpretation
api Selects no phase by itself in WorkerRuntimeService.runOnce(). If the record-audit outbox worker is composed, it also selects record-audit-worker draining in the worker cycle. The role label does not provision an HTTP listener. The default role is not proof that the worker loop starts; confirm the canonical worker-start setting and the composed audit worker before expecting this background phase.
outbox-publisher outbox-publisher / drain-outbox Selects the outbox publishing phase. Confirm its transport and downstream broker separately.
kafka-consumer kafka-consumer / poll-broker Selects one broker polling phase; it is not the same phase as dispatching messages already stored in the platform inbox.
inbox-dispatcher inbox-dispatcher / dispatch-inbox Selects the inbox dispatch phase.
runtime-worker inbox-dispatcher / dispatch-inbox Also selects inbox dispatch; the role name does not establish a separate runtime queue in this dispatch table.
notification-worker inbox-dispatcher / dispatch-inbox Also selects inbox dispatch; notification behavior still depends on the message handler and its configured dependencies.
integration-worker record-audit-worker when composed, and inbox-dispatcher / dispatch-inbox Selects both shown phases. It is not a substitute for the outbox-publisher or kafka-consumer roles.
timer-worker timer-worker / run-due-timers Selects due process timers and timeout work. A process deployment needs at least one eligible, running timer worker.
service-task-worker service-task-worker / recover-timeouts and process-due-runs Selects service-task timeout recovery and due service-task processing.
ai-worker ai-worker / recover-agent-timeouts and process-agent-runs, plus inbox dispatch Selects agent-run processing and recovery as well as the inbox dispatch phase. Provider readiness and effect/result checks remain separate.
escalation-worker escalation-worker / run-due-escalations Selects the SLA escalation sweep. It is gated by process.module.enabled; with that module disabled, the phase is a no-op.

PlatformNodeRole contains exactly these eleven role identifiers at the referenced source revision. Role configuration is a routing/eligibility choice, not a user authorization grant and not evidence that a phase ran. The normal worker lifecycle must be enabled, and its dependencies and module gates must be ready. For example, a node with INFLOW_NODE_ROLES=timer-worker but with platform.worker.autoStart=false has timer eligibility configured but no automatic worker loop. Conversely, starting the loop on an API-only node does not select the timer, broker-consumer or service-task phases.

Document extraction is a separate DocumentExtractionWorkerLifecycle thread, not one of the roles in PlatformNodeRole and not a phase in this role table. When that lifecycle is composed, it auto-starts on a node whose roles contain api or integration-worker; this selection is separate from the platform.worker.autoStart switch for the shared worker loop. A configured role still does not prove that the lifecycle or extraction dependency is healthy. Do not treat a service-task-worker label as selection for document extraction.

Worked configuration. For a deliberately separated API and timer layout, an operator's process manager could provide the following independent values. Each block belongs only to the named process; do not combine them into one shared environment file. This is a configuration illustration, not an Inflow deployment CLI.

# Inject only into each API process; assign a different ID per instance.
INFLOW_NODE_ID=peopleops-api-a
INFLOW_NODE_ROLES=api

API process conf/custom_config.properties:

platform.worker.autoStart=false
# Inject only into each timer process; use a different ID for every instance.
INFLOW_NODE_ID=peopleops-timer-a
INFLOW_NODE_ROLES=timer-worker

Timer process conf/custom_config.properties:

platform.worker.autoStart=true

The timer process must also be configured to reach the approved durable services. Keep credentials in the approved secret store; do not copy them into either process's public environment manifest. Use stable, unique IDs per running process, and confirm the effective role set and worker state on the intended node using the protected Node Control API. A load-balanced /current request may answer from another node; retain the returned node ID and use an approved direct operator route when a check must address one specific instance.

Source references: role enum, role dispatch, environment/file precedence, node routes and permission guard, separate document-extraction lifecycle, and focused property tests / controller tests.

Inflow documentation · Published 19 November 2026

Screenshot captions retain their recorded build and capture context.

Type to search articles and their contents.