Optional technical reference. This material is for developers, integration specialists and platform administrators. For the consultant lesson, read the practical guide.
Preserved from source baseline
d4587c7c0e073e58c2d430342f65ac55dd701bf4; source/static examples retain their stated scope. They are not new runtime verification or required consultant setup steps.
Current-source role audit (2026-11-13): the role catalog and dispatch mapping below were rechecked against
origin/master4082f871954bd160e160a1b3dc2a40e4f7275c8d/ treee92f6851c1575615341f442853e1ecd192c2b675(latest product-source commit4082f871954bd160e160a1b3dc2a40e4f7275c8d). The six role/dispatch source files named in the lesson were unchanged since the prior951ca2cad286924e8214e80329ab46430ce4fb9caudit. This is source inspection only; it is not a deployment or runtime qualification.
Active-Active Deployment Manual
Practical guide and worked examples
Read the topology as separate responsibilities
A node identity identifies one JVM/container; node roles decide which worker responsibilities it may undertake; a canonical worker-start setting controls whether its loop starts. Broker connectivity, current leases, database health and application readiness are additional conditions. A running API page or an active heartbeat is not proof that an inbox dispatcher, service-task worker or AI worker is executing the intended queue.
For an admitted design, document each actual node ID, origin/zone, role set, canonical configuration and build identity. Keep identity unique across nodes and retain the same intended identity across a controlled restart. Current source offers roles such as api, outbox-publisher, kafka-consumer, inbox-dispatcher, service-task-worker, ai-worker and escalation-worker; use the actual configured source/role implementation, not a generic “worker=true” label. The checked-in node profile is API-only unless deliberately configured otherwise.
| Layer | Evidence to inspect | Typical misleading shortcut |
|---|---|---|
| Node identity/role | Actual node record and deployment configuration | Two processes sharing an ID because a config file was copied. |
| Worker loop | Current worker status on the intended node | Calling /current through a load balancer and assuming which node answered. |
| Transport | Reachable bootstrap and advertised addresses | A host using a container-only DNS name. |
| Queue claim | Current owner, expiry and due state | Treating lease expiry as rollback of a remote request. |
| Rolling change | Exact target node, drain state and remaining capacity | Stopping a random API or clearing leases directly in SQL. |
Node administrative endpoints require *; they are not ordinary end-user permissions. The procedures are for separately admitted operator work. No node was started, drained, stopped, reconfigured or failed over during this documentation program.
The examples below use synthetic teaching data. They are grounded in the cited source contracts; this documentation change did not execute these operations against a running Inflow deployment. Use normal authenticated sessions and the actual returned identifiers, revisions and versions from an authorized environment.
Example ROOT-AA-01 — Design distinct API and worker identities
Scenario and inputs. The operator is planning two admitted development nodes for People Operations: one serving HTTP and another processing specific background queues. They share compatible durable services but must not impersonate the same node. The plan must identify which work is required and which runtime/provider features are actually enabled in this environment.
Procedure. Read conf/node.properties and the actual deployment overrides. Assign distinct stable INFLOW_NODE_ID values and explicit INFLOW_NODE_ROLES for each intended process. Keep the API node's role limited to api when it should not claim worker jobs. Give the worker only the required queue roles, then review the canonical platform.worker.autoStart setting and other module prerequisites. Document broker/database addresses reachable from each process rather than assuming container and host networking are identical.
Expected result. The plan separates identity, role eligibility and loop activation. Current heartbeats/readiness can later be compared against the intended topology in an admitted run. A role string or a true feature flag alone does not prove the worker actually started, claimed a job or completed an external business effect.
Failure and recovery. If two active processes report one identity, correct the deployment ownership/configuration before relying on their lease/readiness evidence. Do not rotate another lane's identity, start a second unowned worker or delete heartbeat/lease rows to make the plan appear healthy. Preserve current resource, runtime, provider and source-custody holds while qualifying the owned topology.
Sources. node.properties:1, PlatformNodeProperties.java:1, DefaultWorkerRuntimeService.java:1.
Example ROOT-AA-02 — Diagnose a healthy API with an idle queue
Scenario and inputs. The API answers normal reads, yet the expected leave notification or service task remains pending. The operator has the original work identity and allowed operational access. The problem may be a missing role, stopped loop, disabled module, unavailable dependency, due-time constraint or active lease; simply restarting the API would hide which condition actually mattered.
Procedure. Read the intended node record and active-node list through /api/nodes/<node-id> and /api/nodes/active. For current-loop details, address the intended node and read /api/nodes/current/worker; through a load balancer, retain the answering node identity rather than guessing. Compare roles and canonical worker activation with the due work, its current owner/expiry and the owning worker's safe failure metadata. Check the actual broker/provider boundary only within its separate permitted diagnostic scope.
Expected result. The investigation can distinguish serving HTTP from claiming a particular queue. A pending job with a live owner is different from an unclaimed job or an unsupported role. A worker-status read is operational evidence for that node, not proof of completed user-visible business work or permission to run the same job elsewhere.
Failure and recovery. Do not claim an arbitrary batch, invoke run-once on a shared node or start competing loops merely to force progress. Preserve the original work and uncertain effect receipt. Correct the specific admitted configuration/dependency with its owner, then use the native domain outcome to establish whether the intended work actually completed.
Sources. NodeController.java:61, NodeController.java:48, DefaultWorkerRuntimeService.java:1.
Example ROOT-AA-03 — Drain a known node for a controlled rolling change
Scenario and inputs. A separately approved rolling change will remove one identified node while preserving the required active capacity. The operator has *, the exact target node ID and the environment's existing availability/resource plan. A load-balanced “current node” route cannot reliably select which instance should leave service.
Procedure. Read the target node and remaining active capacity first. When the approved drain step is due, use the explicit target route:
POST /api/nodes/<actual-node-id>/drain
Observe the target's state/readiness and existing in-flight work under the admitted operating plan before stopping its process. To return that draining node to active eligibility after the approved change, use /api/nodes/<actual-node-id>/resume. This node resume is not a workflow case activation or token-continuation API.
Expected result. The native drain moves the identified node to DRAINING, lowers its readiness and stops new worker claims from the next relevant cycle. It records the native drain audit. Existing dispatched remote requests are not recalled or undone by that state change. The source has specific not-found/not-drainable/last-active/not-draining refusals rather than an unconditional toggle.
Failure and recovery. A NODE_LAST_ACTIVE or other state refusal must be resolved through the legitimate capacity/target plan; do not edit the node table or globally kill services. Preserve outstanding work and reconcile original receipts before replacement processing. If the intended target is unclear, stop the change and identify it rather than draining whichever backend answered a convenient URL.
Sources. NodeController.java:90, NodeController.java:101, NodeController.java:115.
Example ROOT-AA-04 — Choose broker addresses for the actual network boundary
Scenario and inputs. A host-run API cannot resolve the Compose-internal name used by container workers, or a broker advertises an address unreachable from one client. The operator has the admitted Compose/network configuration and is diagnosing infrastructure routing, not changing a registered product feature flag.
Procedure. Inspect both bootstrap and advertised listener addresses from the owned deployment. MessagingProperties.bootstrapServers() resolves the JVM infrastructure property messaging.kafka.bootstrapServers, then MESSAGING_KAFKA_BOOTSTRAP_SERVERS, then the messaging properties file, then its container default. Prepare the reachable host/port for that specific process through the allowed deployment mechanism. Keep internal broker names for clients inside the appropriate network and use the actual admitted published listener for host clients; do not assume a universal port or copy another lane's endpoint.
Expected result. Infrastructure broker routing remains distinct from canonical feature switches: the retirement of feature-value environment aliases does not remove this documented infrastructure override. A corrected bootstrap address still needs compatible advertised listeners, credentials/trust and the actual worker role/loop before delivery can be qualified.
Failure and recovery. Do not edit tracked shared defaults to accommodate one host, disable transport security or reconfigure another lane's broker. If startup fails while constructing messaging dependencies, retain that failure and trace the actual reachable listener rather than describing the rest of the platform as healthy. A source configuration comparison is not an executed broker-delivery proof.
Sources. MessagingProperties.java:49, docker-compose.yml:1, messaging.properties:1.
Inflow is designed so many API and worker nodes can run at the same time. API nodes should stay stateless. Durable state belongs in the database, and broker delivery is coordinated through the outbox/inbox tables.
Target Topology
flowchart TD
USERS["Users / External Systems"] --> LB["Load Balancer"]
LB --> API1["API Node 1"]
LB --> API2["API Node 2"]
LB --> APIN["API Node N"]
API1 --> DB[("HA Database")]
API2 --> DB
APIN --> DB
API1 --> OUTBOX[("message_outbox")]
API2 --> OUTBOX
APIN --> OUTBOX
OUTBOX --> PUB["Outbox Publisher Nodes"]
PUB --> BROKER["Kafka / Redpanda Cluster"]
BROKER --> CONSUMER["Kafka Consumer Nodes"]
CONSUMER --> INBOX[("message_inbox")]
INBOX --> DISPATCH["Inbox Dispatcher Nodes"]
DISPATCH --> SERVICES["Runtime / Task / AI / Integration / Notification Services"]
SERVICES --> DB
Node Heartbeats
Every JVM runs a node heartbeat lifecycle. It registers itself in
platform_node_heartbeats at startup and refreshes the row on
platform.node.heartbeatSeconds, regardless of whether the node is API-only or
also running workers. The row contains:
node_id: stable node identityroles_csv: API or worker roleshost_name,process_id,region,zone_namestatus:STARTING,ACTIVE,DRAINING,STOPPED,UNHEALTHYstarted_at,last_seen_at
Stale STARTING or ACTIVE nodes are marked UNHEALTHY after
platform.node.staleAfterSeconds.
Node Roles
Configure roles in conf/node.properties or environment variables. A non-blank
environment variable wins over the file, so one checked-in file can serve every
node while each container is given its own identity and duties:
| Environment variable | Overrides |
|---|---|
INFLOW_NODE_ID |
platform.node.id |
INFLOW_NODE_ROLES |
platform.node.roles |
INFLOW_NODE_REGION |
platform.node.region |
INFLOW_NODE_ZONE |
platform.node.zone |
Process timers (intermediate and event-gateway timers, activity and user-task
timeouts, retry backoff) are claimed only by a node with the timer-worker role
whose worker loop is running. At least one such node must exist in every
deployment that runs processes; otherwise every timer row stays PENDING with
zero attempts. The single-node Docker Compose stack (api-dev) sets
INFLOW_NODE_ROLES with its complete single-node worker role set, and its explicit
canonical development profile sets platform.worker.autoStart=true for this reason.
Worker startup is a restart-only feature in custom_config.properties; node identity/roles
retain the environment precedence above. See Feature switches.
In production, set a stable unique node id through platform.node.id or
INFLOW_NODE_ID. When left blank, the platform falls back to the detected
host/container name, which is convenient for local development but should not be
treated as an explicit deployment identity.
API node:
platform.node.roles=api
platform.worker.autoStart=false
Outbox publisher node:
platform.node.roles=outbox-publisher
platform.worker.autoStart=true
Kafka consumer plus inbox dispatcher node:
platform.node.roles=kafka-consumer,inbox-dispatcher
platform.worker.autoStart=true
Combined worker node:
platform.node.roles=outbox-publisher,kafka-consumer,inbox-dispatcher
platform.worker.autoStart=true
Timer worker node:
platform.node.roles=timer-worker
platform.worker.autoStart=true
Service-task worker node:
platform.node.roles=service-task-worker
platform.worker.autoStart=true
Escalation worker node:
platform.node.roles=escalation-worker
platform.worker.autoStart=true
The escalation-worker role drives the SLA-breach escalation sweep
(EscalationSweepWorker) across every active workspace. It is gated by the
process.module.enabled feature flag: with the flag off, the role is a no-op
(each pass returns immediately without querying any workspace), so it is safe to
assign ahead of enabling the Process module. Run it on a single node — the
sweep's per-task claim is a conditional database update, so multiple escalation
nodes stay correct, but co-locating the role on one node avoids duplicate work.
Worker Run Loop
When platform.worker.autoStart=true, Spring starts WorkerRuntimeLifecycle.
The lifecycle delegates to WorkerRuntimeLoop, which repeatedly calls
WorkerRuntimeService.runOnce() for the node's configured roles.
Run-loop controls live in conf/node.properties:
platform.worker.batchSize=100
platform.worker.loopIntervalMillis=1000
platform.worker.errorBackoffMillis=5000
platform.worker.stopTimeoutMillis=2000
platform.worker.kafkaPollTimeoutMillis=1000
platform.worker.loopIntervalMillis: idle delay after a successful cycleplatform.worker.errorBackoffMillis: delay after a failed cycle before retryplatform.worker.stopTimeoutMillis: graceful shutdown join time after the lifecycle interrupts the worker threadplatform.worker.batchSize: maximum rows each worker phase claims per cycleplatform.worker.kafkaPollTimeoutMillis: broker poll timeout for Kafka / Redpanda consumer nodes
On shutdown, the lifecycle marks the node DRAINING, interrupts the worker
thread, waits up to platform.worker.stopTimeoutMillis, then marks it
STOPPED once the loop exits. If a worker cycle throws, the loop records the
failure through logs and waits for platform.worker.errorBackoffMillis instead
of hot-looping.
Node Control API
Operators can inspect node state and control the current JVM's worker runtime through the node API:
GET /api/nodes/current
GET /api/nodes/active
GET /api/nodes/{nodeId}
GET /api/nodes/current/worker
POST /api/nodes/current/worker/run-once
POST /api/nodes/current/worker/start
POST /api/nodes/current/worker/stop
POST /api/nodes/{nodeId}/drain
POST /api/nodes/{nodeId}/resume
GET /api/nodes/current/worker returns the current node id, roles, node
status, whether the worker lifecycle is running, and the effective worker
configuration. POST /api/nodes/current/worker/run-once executes one worker
cycle synchronously and returns the processed counts from that cycle. The
start and stop endpoints delegate to the same lifecycle used by Spring
auto-start, so they mark node state consistently with normal deployment
startup and shutdown.
Draining a node for a rolling deploy
POST /api/nodes/{nodeId}/drain (SCRUM-699) marks any node's heartbeat row DRAINING, by id: behind a load balancer
you cannot choose which node answers, so the row is the signal and the target node reads it on its next heartbeat. A
draining node:
- keeps
DRAININGon every heartbeat. The heartbeat is one conditional upsert, so a drain or a resume that lands between two heartbeats is never lost (SCRUM-4001). A node that stops heartbeating for longer thanstaleAfteris markedUNHEALTHYand returnsACTIVEon its next heartbeat, and a drained node that its orchestrator restarts quickly comes back stillDRAINING; - reports down on
/actuator/health/platformNodeReadinessand leavesGET /api/nodes/active, so the load balancer stops routing to it; - claims no new work: each worker cycle heartbeats and then runs no claim phase (outbox, broker poll, timers, service tasks and their timeout recovery, agent runs and their timeout recovery, record audit outbox, inbox dispatch, escalations). A cycle already running finishes the work it claimed.
Rolling deploy order:
- Drain the node.
- Confirm it:
GET /api/nodes/{nodeId}showsDRAINING. A node that runs the worker runtime also logs once, on its next worker cycle, "Node … is DRAINING: it claims no new work until POST /api/nodes/{nodeId}/resume" (an API-only node has no worker cycle and logs nothing). - Wait one
platform.worker.loopIntervalMillisplus the longest cycle (or watch its worker metrics go quiet). - Stop it; the shutdown lifecycle marks it
STOPPED. Drain is allowed fromACTIVEorSTARTING; draining a node that is alreadyDRAININGreturns it unchanged. ASTOPPEDorUNHEALTHYnode answers 409NODE_NOT_DRAINABLE, and an unknown id (or a row deleted while the drain runs) 404NODE_NOT_FOUND. Drain and resume change the status in one conditional statement: a node that moved to another status in between is refused, never overwritten, and an operator transition does not refresh the node's last-seen time.
Never drain the last active node. A drain that would leave no other ACTIVE node answers 409 NODE_LAST_ACTIVE
(SCRUM-4001). The check and the change are one serialized step, so two operators draining the last two nodes at the same
moment cannot both succeed. The guard counts any ACTIVE node, whatever its roles: keep at least one node that serves
the API in rotation, or /resume must be sent to a node directly. Draining every node would drop readiness everywhere, so /resume could not be reached through the load
balancer. On a single-node deployment draining is therefore refused; to take that node out, stop it (its
outbox, inbox and timer processing stop with it) and restart it. If a node is somehow left DRAINING behind a
readiness-gated load balancer, send /resume to the node directly (bypassing the balancer).
POST /api/nodes/{nodeId}/resume returns a DRAINING node to ACTIVE (a drain made by mistake, or a node restarted
with the same INFLOW_NODE_ID before its row went stale, which otherwise keeps it draining). An ACTIVE node is
returned unchanged; any other status answers 409 NODE_NOT_DRAINING. Both endpoints need the node administrator
permission, like the rest of this API. Other background lifecycles (document extraction, data and Fabric sync,
retention jobs, the inbound mail consumer, cache refresh) are not gated by DRAINING; stopping the node stops them.
Worker control commands are audit-recorded in audit_events:
WORKER_RUN_ONCE_REQUESTEDWORKER_START_REQUESTEDWORKER_STOP_REQUESTEDNODE_DRAIN_REQUESTEDandNODE_DRAIN_RESUMED(entity id: the drained node, metadatapreviousStatusandstatus); a drain or resume that changes nothing records nothing
Each event uses entity_type = platformNode, the current node id as
entity_id, the authenticated operator as actor, and metadata containing the
effective worker state or processed counts.
Operators can search those events through the audit API:
GET /api/audit-events?type=WORKER_RUN_ONCE_REQUESTED&entityType=platformNode&page=0&size=50
GET /api/audit-events?entityType=platformNode&entityId={nodeId}
The response is a paged result with items, page, size, and totalItems.
Each item includes the event type, entity reference, actor, metadata, and
timestamp.
Load Balancing
Put a load balancer in front of API nodes. API nodes must not store workflow, task, or authentication session state in local memory. Use tokens, API keys, or shared identity storage so any API node can handle the next request.
Built-in role grants are seeded by every API node when it starts. A grant a newer build has withdrawn from a built-in
role is deleted on that build's start and ignored when it reads role permissions. For example, SCRUM-3763 withdrew
PROCESS_INSTANCE_APPROVE from worker. An older node in the same fleet re-seeds such a grant when it starts and
holds it while it runs (for SCRUM-3763 this confers nothing in practice, because builds before it have no working
approval route); newer nodes ignore and later delete the row. Finish rolling upgrades promptly, and expect a
rollback to restore the old build's behaviour on the old nodes.
Failover
Work claiming is database-coordinated:
- outbox publishers claim rows from
message_outbox - inbox dispatchers claim rows from
message_inbox - claims use locks with
locked_by,locked_until, attempts, and retry status - if a node dies, its locks expire and another node can claim the work
- worker completion/failure updates verify the current
locked_byowner and a non-expiredlocked_untilbefore changing message state - long-running owners can explicitly renew a lease before it expires or release it back to the queue without recording a failure
- expired
PROCESSINGmessages are claimable, so a node crash does not leave work stuck forever
Outbox publishers and inbox dispatchers expose owner-checked lease operations:
MessageOutboxService.renewPublishLease(...)MessageOutboxService.releasePublishLease(...)MessageInboxService.renewProcessingLease(...)MessageInboxService.releaseProcessingLease(...)
Renewal extends locked_until only when the caller still owns a non-expired
PROCESSING row. Release clears locked_by and locked_until, moves the row
back to PENDING, and sets available_at so another node can claim it without
counting it as a failed attempt.
Runtime timer jobs use the same pattern. Due rows in runtime_timer_jobs are
claimed with locked_by, locked_until, attempts, and status. A timer
worker can only mark a timer COMPLETED, FAILED, or DEAD_LETTER while it
still owns the non-expired lease. Expired PROCESSING timers can be reclaimed
by another timer worker.
Timer workers can also call RuntimeTimerJobService.renewLease(...) while a
timer callback is still resuming a process, or
RuntimeTimerJobService.releaseLease(...) during clean shutdown. The service
delegates to the owner-checked repository operations. Release moves the job
back to PENDING, clears lease ownership, and sets due_at so a different
timer worker can claim it without recording a failed attempt.
The worker HTTP API exposes the same timer lease operations:
POST /api/runtime-timer-jobs/{timerJobId}/renew-lease
POST /api/runtime-timer-jobs/{timerJobId}/release-lease
These endpoints require RUNTIME_TIMER_LEASE_MANAGE.
Service task runs use the same lease model in service_task_runs. Platform
service-task workers claim only local Java-backed implementations, while
external workers can claim generic standard-work-item runs through the worker
HTTP API. This keeps built-in workers and external connector/document/RPA
workers from competing for the wrong work.
Long-running service-task workers can call
ServiceTaskExecutionService.renewLease(...) while they are still processing
or ServiceTaskExecutionService.releaseLease(...) when they need to give the
work back without marking it failed. Release moves the run back to PENDING
and logs the release, keeping the attempt history visible while allowing
another active node to continue.
Service-task worker nodes also run timeout recovery before each processing
loop. A stale PROCESSING service-task run whose current attempt has exceeded
its configured timeoutSeconds is released, marked FAILED or DEAD_LETTER
according to maxAttempts, and logged with the recovering node id. This makes
service-task failover active/active friendly: another node can continue after
the retry backoff instead of waiting for a manual cleanup.
Agent runs use the same pattern in agent_runs. AI worker nodes claim pending
agent runs with row-level leases and FOR UPDATE SKIP LOCKED; completion and
failure require the current lease owner. Agent workers only claim work when an
AgentRunExecutor is available, so deployments can add local-model, AI Gateway,
tool-calling, or human-handoff executors independently.
Long-running AI workers can call AgentExecutionService.renewLease(...) to
extend ownership while a model, tool, or agent loop is still running. They can
call AgentExecutionService.releaseLease(...) to return the run to PENDING
without recording a failed attempt. Both operations require the current
non-expired locked_by owner and write lease logs.
AI worker nodes also recover stale RUNNING agent runs before each processing
loop. If a current attempt exceeds timeoutSeconds, the run is released,
closed as a failed attempt, and moved back to PENDING or finally to FAILED
based on maxAttempts. The recovery log includes the node id that performed
the sweep, which keeps agent execution recoverable across active/active nodes.
Runtime execution state is also becoming durable. Active/waiting/consumed tokens
are stored in runtime_execution_tokens, and process variables are stored in
runtime_variables. Before resume, signal, or cancel, a restorable engine can
reload the BPMN model and variables so a fresh node can continue work started by
another node.
Runtime resume also has an atomic token-consumption guard. A token can only be
claimed or consumed from ACTIVE or WAITING. Claims write locked_by and
locked_until; a different worker cannot consume the token until the lease
expires. Resume claims and consumes the token in one transaction, so the claim is
an internal resume guard and no lease stays held afterwards. The token-lease
renew/release HTTP routes were retired by SCRUM-3764 (they answer 404); the
PROCESS_INSTANCE_LEASE_MANAGE permission is kept for stored scopes and grants
but no route checks it. Replayed resume
requests against the same token are rejected. In JDBC deployments wired with a
PlatformTransactionManager, runtime state changes commit as one transaction,
so a failed engine resume rolls back token consumption and work item completion.
sequenceDiagram
participant W1 as Worker Node 1
participant W2 as Worker Node 2
participant DB as Database
W1->>DB: claim pending inbox row
DB-->>W1: row locked_by=node-1
W2->>DB: claim pending inbox row
DB-->>W2: different row or none
W1--xDB: node dies before processing
Note over DB: lock expires
W2->>DB: claim expired row
DB-->>W2: row locked_by=node-2
Health And Readiness
Actuator is enabled for load balancer and orchestrator checks. The platform node
health indicator reports down when the current node does not have a fresh
ACTIVE heartbeat.
Recommended probes:
GET /actuator/health
GET /actuator/health/platformNodeReadiness
Operations Dashboard API
Admins can read the platform operations snapshot from:
GET /api/operations/dashboard
This endpoint requires admin access (Permission.ALL). It is read-only and
does not claim, retry, or mutate work. The response combines:
- service-task catalog counts and metadata-quality signals
- connector catalog readiness and protocol/category counts
service_task_runstotals, dispatchable count, locked count, and status countsmessage_outboxtotals, dispatchable count, locked count, and status countsmessage_inboxtotals, dispatchable count, locked count, and status countsruntime_execution_tokensactive/waiting token backlog, locked count, and status countsruntime_timer_jobsretryable timer backlog, locked count, and status countsagent_runsAI/agent execution backlog, locked count, and status counts- connector execution workload totals, success/failure counts, status counts, recency, and duration signals
- Fabric materialization workload totals, success/failure counts, status counts, recency, and duration signals
- active platform nodes, status counts, and role counts
For retry queues, dispatchable means the row is retryable, its due/available
time has arrived, and it is not currently locked by a worker. For runtime
execution tokens, dispatchable means the token is ACTIVE or WAITING and
has no live lease. For runtime timer jobs, expired PROCESSING leases are also
counted as dispatchable because another timer worker can reclaim them. For
agent runs, dispatchable means a PENDING or retryable FAILED run whose
due time has arrived and whose lease is free. This is the operator's first
signal for worker backlog. locked means a worker currently owns the row's
lease.
Connector executions and Fabric materializations are not lease queues. They are append-only workload/audit streams, so operations reports them as execution health metrics instead of backlog metrics.
Example shape:
{
"generatedAt": "2026-07-02T04:35:17Z",
"serviceTaskCatalog": {
"serviceTaskCount": 241,
"byImplementation": {
"connector-event-stream": 4
}
},
"connectorCatalog": {
"connectorCount": 37,
"byProtocol": {
"EVENT_STREAM": 2
}
},
"serviceTaskRuns": {
"total": 12,
"dispatchable": 4,
"locked": 2,
"byStatus": [
{"status": "PENDING", "count": 4}
]
},
"messageOutbox": {
"total": 20,
"dispatchable": 8,
"locked": 1
},
"messageInbox": {
"total": 15,
"dispatchable": 3,
"locked": 0
},
"runtimeExecutionTokens": {
"total": 30,
"dispatchable": 7,
"locked": 4
},
"runtimeTimerJobs": {
"total": 9,
"dispatchable": 2,
"locked": 1
},
"agentRuns": {
"total": 16,
"dispatchable": 5,
"locked": 3
},
"connectorExecutions": {
"total": 240,
"succeeded": 230,
"failed": 10,
"averageDurationMillis": 180
},
"aiFabricMaterializations": {
"total": 36,
"succeeded": 33,
"failed": 3,
"averageDurationMillis": 850
},
"nodes": {
"activeCount": 3,
"currentWindowCount": 3,
"byRole": {
"OUTBOX_PUBLISHER": 1,
"SERVICE_TASK_WORKER": 2
}
}
}
Operations Readiness API
Admins can also read a single deployment gate from:
GET /api/operations/readiness
This endpoint requires admin access (Permission.ALL). It does not replace the
detailed readiness endpoints for service tasks, connectors, Fabric, queues,
or nodes. Instead, it combines them into one operator-facing answer:
READY: no blocking errors were foundBLOCKED: one or more components has an error that should stop deployment- component status:
READY: no issues for the componentATTENTION: warnings exist, but no blocking errorBLOCKED: at least one blocking error exists
The current aggregate checks include:
- service-task extension governance and implementation coverage
- connector readiness, setup actions, rollout warnings, and operational status
- Fabric catalog readiness issues
- active platform node visibility in the current heartbeat window
- failed queue items in service-task, outbox, inbox, runtime-token, runtime-timer, and agent-run queues
- failed or slow connector/materialization workload streams
Example shape:
{
"generatedAt": "2026-07-02T07:08:11Z",
"productionReady": false,
"status": "BLOCKED",
"errorCount": 1,
"warningCount": 4,
"componentCount": 6,
"components": {
"serviceTasks": {
"status": "BLOCKED",
"errorCount": 1,
"warningCount": 2
},
"connectors": {
"status": "READY",
"errorCount": 0,
"warningCount": 0
},
"aiFabric": {
"status": "ATTENTION",
"errorCount": 0,
"warningCount": 2
},
"nodes": {
"status": "READY",
"errorCount": 0,
"warningCount": 0
},
"queues": {
"status": "ATTENTION",
"errorCount": 0,
"warningCount": 1
},
"workloads": {
"status": "ATTENTION",
"errorCount": 0,
"warningCount": 1
}
},
"issues": [
{
"severity": "ERROR",
"component": "serviceTasks",
"code": "SERVICE_TASK_EXECUTOR_UNSUPPORTED",
"itemId": "service.integration.example",
"message": "Service task has no executor or extension provider."
}
]
}
Operations Metrics API
Admins can read queue and heartbeat pressure metrics from:
GET /api/operations/metrics
This endpoint requires admin access (Permission.ALL) and is read-only. It is
designed for dashboards, alert rules, and deployment checks that need numeric
signals instead of full work-item payloads.
Queue metrics are returned for:
serviceTaskRunsmessageOutboxmessageInboxruntimeExecutionTokensruntimeTimerJobsagentRuns
Each queue metric includes:
total: all rows in the queue tabledispatchable: due/available, not locked, retryable rowslocked: rows currently leased by a workerfailed: rows withFAILEDstatusdeadLetter: rows withDEAD_LETTERstatusretryExhausted: rows whose attempts reachedmax_attempts, plus failed or dead-letter rowsoldestReadyAgeSeconds: age of the oldest dispatchable rowaverageReadyAgeSeconds: average age of dispatchable rowsaverageAttempts: average attempts across the table
runtimeExecutionTokens uses the same metric shape so dashboards can render it
beside other queues, but token rows are not retry-attempt records. Its
dispatchable count is active/waiting unlocked tokens, failed is the number
of failed tokens, retryExhausted mirrors failed tokens, and attempt-related
values stay 0.
runtimeTimerJobs is a retry queue. Its dispatchable count includes
PENDING, FAILED, and expired PROCESSING timers with attempts remaining,
because timer workers can reclaim expired processing leases.
agentRuns is the AI worker queue. Its dispatchable count includes PENDING
and retryable FAILED runs whose due time has arrived. Agent runs do not use a
separate DEAD_LETTER status; an exhausted run remains FAILED, so
deadLetter stays 0 and retryExhausted counts terminal failed or
max-attempt rows.
Workload metrics are returned for:
connectorExecutionsaiFabricMaterializations
Each workload metric includes:
total: all rows in the workload tablesucceeded: successful terminal rowsfailed: failed terminal rowslatestAgeSeconds: age of the latest completed rowaverageDurationMillis: average execution durationmaxDurationMillis: slowest execution durationbyStatus: status counts
These are intentionally separate from queue metrics. A failed connector execution or materialization can be a historical fact, not necessarily something a worker should reclaim.
Audit metrics are returned in auditEvents. Audit events are append-only
operator and platform activity records, so they are reported as an activity
stream rather than a queue or success/failure workload.
Audit metrics include:
total: all rows inaudit_eventsrecentCount: events written during the metrics windowdistinctActors: unique non-empty actor ids seen in the audit tablelatestAgeSeconds: age of the most recent audit eventbyType: count by audit event typebyEntityType: count by audited entity type
Node metrics include:
registeredNodes: nodes known in the heartbeat tableactiveNodes: nodes currently markedACTIVEunhealthyNodes: nodes markedUNHEALTHYoldestHeartbeatAgeSeconds: oldest heartbeat agenewestHeartbeatAgeSeconds: newest heartbeat age
Worker metrics are aggregated from append-only samples in
platform_worker_metrics. Every active-active node can write its own samples
without coordinating with other nodes. The API returns the most recent 15-minute
window grouped by workerName and phaseName.
Worker metrics include:
sampleCount: number of worker phase samples in the windowsucceededCount: number of phase samples that completed without throwingfailedCount: number of phase samples that threw before returningtotalClaimed: total work count reported by the phasetotalSucceeded: total successful work count reported by the phasetotalFailed: total failed work count reported by the phaseaverageDurationMillis: average phase durationmaxDurationMillis: slowest phase duration in the windowthroughputPerMinute: successful work count divided by the 15-minute windowerrorRate: failed work count divided by total completed/failed work count
The response also includes alerts, calculated from the operations policy in
conf/custom_config.properties. Alerts are returned as WARN or ERROR and
are also folded into GET /api/operations/readiness as operations readiness
issues.
Built-in worker phases include:
outbox-publisher/drain-outboxkafka-consumer/poll-brokertimer-worker/run-due-timersservice-task-worker/recover-timeoutsservice-task-worker/process-due-runsai-worker/recover-agent-timeoutsai-worker/process-agent-runsinbox-dispatcher/dispatch-inbox
Example shape:
{
"generatedAt": "2026-07-02T07:15:38Z",
"serviceTaskRuns": {
"total": 120,
"dispatchable": 14,
"locked": 6,
"failed": 3,
"deadLetter": 1,
"retryExhausted": 2,
"oldestReadyAgeSeconds": 95,
"averageReadyAgeSeconds": 31,
"averageAttempts": 0.45
},
"messageOutbox": {
"total": 42,
"dispatchable": 8,
"locked": 2,
"failed": 1,
"deadLetter": 0,
"retryExhausted": 1,
"oldestReadyAgeSeconds": 21,
"averageReadyAgeSeconds": 9,
"averageAttempts": 0.2
},
"messageInbox": {
"total": 18,
"dispatchable": 4,
"locked": 1,
"failed": 0,
"deadLetter": 0,
"retryExhausted": 0,
"oldestReadyAgeSeconds": 12,
"averageReadyAgeSeconds": 6,
"averageAttempts": 0.1
},
"runtimeExecutionTokens": {
"total": 30,
"dispatchable": 7,
"locked": 4,
"failed": 1,
"deadLetter": 0,
"retryExhausted": 1,
"oldestReadyAgeSeconds": 42,
"averageReadyAgeSeconds": 13,
"averageAttempts": 0
},
"runtimeTimerJobs": {
"total": 9,
"dispatchable": 2,
"locked": 1,
"failed": 1,
"deadLetter": 1,
"retryExhausted": 1,
"oldestReadyAgeSeconds": 120,
"averageReadyAgeSeconds": 33,
"averageAttempts": 1.2
},
"agentRuns": {
"total": 16,
"dispatchable": 5,
"locked": 3,
"failed": 1,
"deadLetter": 0,
"retryExhausted": 1,
"oldestReadyAgeSeconds": 75,
"averageReadyAgeSeconds": 22,
"averageAttempts": 0.8
},
"connectorExecutions": {
"total": 240,
"succeeded": 230,
"failed": 10,
"latestAgeSeconds": 6,
"averageDurationMillis": 180,
"maxDurationMillis": 4200,
"byStatus": [
{"status": "FAILED", "count": 10},
{"status": "SUCCEEDED", "count": 230}
]
},
"aiFabricMaterializations": {
"total": 36,
"succeeded": 33,
"failed": 3,
"latestAgeSeconds": 22,
"averageDurationMillis": 850,
"maxDurationMillis": 7600,
"byStatus": [
{"status": "FAILED", "count": 3},
{"status": "MATERIALIZED", "count": 33}
]
},
"auditEvents": {
"total": 840,
"recentCount": 42,
"distinctActors": 12,
"latestAgeSeconds": 4,
"byType": [
{"status": "WORKER_RUN_ONCE_REQUESTED", "count": 7}
],
"byEntityType": [
{"status": "platformNode", "count": 9}
]
},
"nodes": {
"registeredNodes": 3,
"activeNodes": 3,
"unhealthyNodes": 0,
"oldestHeartbeatAgeSeconds": 8,
"newestHeartbeatAgeSeconds": 1
},
"workers": [
{
"workerName": "service-task-worker",
"phaseName": "process-due-runs",
"sampleCount": 12,
"succeededCount": 11,
"failedCount": 1,
"totalClaimed": 78,
"totalSucceeded": 76,
"totalFailed": 2,
"averageDurationMillis": 241,
"maxDurationMillis": 900,
"throughputPerMinute": 5.07,
"errorRate": 0.03
}
],
"alerts": [
{
"severity": "WARN",
"component": "queues",
"metric": "deadLetter",
"itemId": "serviceTaskRuns",
"actualValue": 1,
"thresholdValue": 1,
"message": "serviceTaskRuns has 1 dead-letter item(s)."
}
]
}
Operations Alert Policy
Operations alert thresholds live in conf/custom_config.properties:
operations.metrics.worker.windowSeconds=900
operations.metrics.worker.retentionDays=30
operations.metrics.retention.autoStart=true
operations.metrics.retention.cleanupIntervalSeconds=86400
operations.alerts.queue.dispatchable.warn=1000
operations.alerts.queue.dispatchable.error=10000
operations.alerts.queue.oldestReadyAgeSeconds.warn=300
operations.alerts.queue.oldestReadyAgeSeconds.error=900
operations.alerts.queue.deadLetter.warn=1
operations.alerts.queue.deadLetter.error=10
operations.alerts.worker.errorRate.warn=0.05
operations.alerts.worker.errorRate.error=0.20
operations.alerts.worker.averageDurationMillis.warn=5000
operations.alerts.worker.averageDurationMillis.error=30000
operations.alerts.worker.throughputPerMinute.warnBelow=0.10
operations.alerts.workload.failed.warn=1
operations.alerts.workload.failed.error=10
operations.alerts.workload.averageDurationMillis.warn=5000
operations.alerts.workload.averageDurationMillis.error=30000
operations.alerts.audit.recentEvents.warn=5000
operations.alerts.audit.recentEvents.error=20000
operations.alerts.audit.latestAgeSeconds.warn=86400
operations.alerts.audit.latestAgeSeconds.error=259200
operations.alerts.node.heartbeatAgeSeconds.warn=60
operations.alerts.node.heartbeatAgeSeconds.error=180
Set an individual threshold to 0 to disable that alert. The worker metrics
window controls aggregation, throughput calculations, and the audit
recentCount window. Retention controls how long raw worker metric samples
should be kept.
operations.metrics.retention.autoStart=true starts a lightweight platform
maintenance lifecycle when the Spring application starts. The lifecycle deletes
expired platform_worker_metrics rows immediately, then sleeps for
operations.metrics.retention.cleanupIntervalSeconds before running again.
Disable it in deployments that use an external database job or Kubernetes
CronJob for retention.
Operations Metrics Cleanup
Worker metric samples are append-only. The built-in retention lifecycle handles cleanup for normal platform nodes, and the scripts below are useful for manual maintenance, dry runs, or deployments that disable the built-in lifecycle.
Admins can also run and inspect cleanup through the operations API:
POST /api/operations/maintenance/metrics-retention
X-Actor-Id: root
GET /api/operations/maintenance/metrics-retention
X-Actor-Id: root
The POST endpoint deletes expired platform_worker_metrics rows using the
configured retention window and returns a persisted maintenance run. The GET
endpoint returns the latest run across all active-active nodes. Maintenance
runs are stored in operations_maintenance_runs with the maintenance type,
status, node id, start and finish timestamps, cutoff timestamp, removed row
count, and message.
.\scripts\operations-metrics-clean.ps1 -RetentionDays 30 -DryRun
.\scripts\operations-metrics-clean.ps1 -RetentionDays 30
./scripts/operations-metrics-clean.sh 30 --dry-run
./scripts/operations-metrics-clean.sh 30
The scripts delete platform_worker_metrics rows older than the retention
window.
Log Archive Cleanup
The logging subsystem also has an operations maintenance API:
GET /api/operations/maintenance/log-retention/policy
POST /api/operations/maintenance/log-retention?dryRun=true
POST /api/operations/maintenance/log-retention?dryRun=false
GET /api/operations/maintenance/log-retention
dryRun=true is the default and counts expired archive files without deleting
them. dryRun=false deletes expired .gz and .log files under
logging.dir/archive. The service refuses paths outside the workspace log
directory and records runs in operations_maintenance_runs as
log-archive-retention.
Database Connection Pooling
Each platform node uses a bounded Hikari connection pool for its runtime
database access. Pool settings live in conf/custom_config.properties:
platform.database.url=postgres://inflow:inflow@postgres-dev:5432/inflow?sslmode=disable
platform.database.jdbcUrl=
platform.database.username=
platform.database.password=
platform.database.driverClassName=org.postgresql.Driver
platform.database.pool.name=inflow-platform-db
platform.database.pool.maximumPoolSize=40
platform.database.pool.minimumIdle=5
platform.database.pool.connectionTimeoutMillis=10000
platform.database.pool.idleTimeoutMillis=600000
platform.database.pool.maxLifetimeMillis=1800000
platform.database.pool.validationTimeoutMillis=5000
DATABASE_URL is supported for container deployment and overrides
platform.database.url, including the checked-in postgres-dev development
default. If platform.database.jdbcUrl is set, it remains the highest-priority
explicit override and can point at another JDBC engine, for example MySQL, SQL
Server, or Oracle, as long as the matching JDBC driver is available to the
runtime. Explicit platform.database.username, password, and
driverClassName values continue to override values inferred from the selected
URL.
Identity statements have their own time bound (SCRUM-3917). This covers the login-session, user, group,
role-permission, password-policy, identity API-key, MFA-factor and SCIM-token reads and writes. Each statement waits at most 5 seconds, and the
bound holds inside a longer transaction too. A shorter transaction remainder or a shorter Lifecycle data deadline
still wins. A request that meets a stalled identity table is answered 503 with code ACCESS_UNCHECKED and
Retry-After: 5, and the person's session is kept. When the pool is exhausted, the wait for a connection
(connectionTimeoutMillis) comes first, so one request waits at most about the connection timeout plus 5 seconds.
The bound does not cover the workspace and grant tables, the rule and managed API-key stores, or the MCP OAuth
token store (follow-up FU-1). It does not help when the database cannot be reached over the network: a cancel
cannot reach it. Use a driver socketTimeout or a per-role statement_timeout as the operations backstop for that
case (FU-2).
Credential Vault Master Key
Every production deployment must provide a stable
INFLOW_VAULT_MASTER_KEY secret to every API node. The credential vault uses
this key to encrypt and decrypt stored vault: connector credentials. If the
variable is missing, the API creates a random development-only key for that
process; credentials saved under that fallback key become unreadable after the
process or container restarts.
Production rules:
- Generate one 32-byte random key per environment and base64-encode it.
- Store it in the deployment secret manager, not in Git, images, XML catalogs, or config maps.
- Inject the same value into every API node in the environment.
- Back it up using the organization's secret-recovery procedure.
- Treat missing
INFLOW_VAULT_MASTER_KEYas a deployment blocker outside local development. - Rotate it only through the vault master-key rotation flow, which
re-encrypts existing
credential_vaultrows before the old key is retired.
Generation examples:
openssl rand -base64 32
$bytes = New-Object byte[] 32
$rng = [System.Security.Cryptography.RandomNumberGenerator]::Create()
$rng.GetBytes($bytes)
$rng.Dispose()
[Convert]::ToBase64String($bytes)
For local Docker Compose development, docker-compose.yml includes a stable
dev-only fallback key for java-dev and api-dev so local connected-system
credentials survive container restarts when no .env file is present. Keep a
generated value in an ignored .env file when you need a team- or
machine-specific local key, and pass it through as INFLOW_VAULT_MASTER_KEY.
Do not reuse the committed local-development fallback or any local .env key
for staging or production.
For active-active deployments, size pools as a cluster budget, not only as a single-node setting:
total possible database connections =
node count * platform.database.pool.maximumPoolSize
Keep that total below the database serve
Per-instance execution serialization (transactional token operations)
Concurrent resume/signal/cancel calls for the SAME process instance — from
any number of API or worker nodes — are serialized through a pessimistic
per-instance lock: ProcessInstanceRepository.lockById(...) takes
SELECT ... FOR UPDATE on the instance row as the first statement of the
runtime transaction, BEFORE engine state is restored from the database.
Consequences:
- Engine decisions (parallel-join firing, instance completion, variable
merges) are always made against fresh, uncontended state. Two nodes
completing the two branches of a parallel fork at the same instant cannot
double-fire or lose the join — proven by
ActiveActiveJoinRaceTest, which races two fully independent runtime nodes (separate engines, separate connections, shared database only) over repeated iterations and asserts the downstream task is created exactly once. - A single lock acquired first in every mutating runtime call means no lock-ordering deadlocks.
- Different instances proceed fully in parallel; the serialization is per instance, which matches how BPM engines scale in practice.
- The lock is a no-op for in-memory adapters (demo/test environments) and requires the transactional runtime wiring (the API configuration always supplies a transaction manager).
Document Management encryption key
Every production node that reads or writes document blobs must receive the
same environment-specific DMS encryption key. DMS_ENCRYPTION_KEY is the
recommended secret-manager injection path and overrides
dms.encryption.key in conf/custom_config.properties. The effective value is
a Base64-encoded AES key of 16, 24, or 32 random bytes. Startup deliberately
fails when neither source supplies a valid key. Never copy the development
Compose fallback into staging or production or commit a production key to the
configuration file.
Keep the key stable across rolling deployments and back it up using the
protected secret-recovery procedure. Existing document blobs cannot be read
after an unplanned key change, and the current provider has no online
re-encryption/keyring rotation flow. Before enabling uploads, prove that all
nodes can start with the injected key and that a node with both key sources
removed fails closed.
Operational details are in docs/manuals/document-management.md.
Records opaque-handle key
Every API node that serves the ordinary Records workspace must receive the
same INFLOW_RECORD_HANDLE_KEY. Generate an independent 32-byte random value,
base64-encode it, and inject it from the deployment secret manager. Do not
reuse the credential-vault master key.
Record handles are AES-256-GCM encrypted, expire after eight hours, and are bound to the workspace, actor, model, applied model version, and record type. Changing the key immediately invalidates all outstanding handles; users can recover by refreshing Records. During a rolling deployment, keep the key stable across old and new nodes. The process-local fallback is for direct local Java execution only, logs a warning, and invalidates handles on restart. Docker Compose supplies a stable dev-only fallback that must be overridden in staging and production.
Published Data Action record-reference key
Every API node that issues or consumes a published Data Action record result
must receive the same INFLOW_DATA_ACTION_RECORD_REFERENCE_KEY. Generate an
independent 32-byte random value, base64-encode it, and inject it from the
deployment secret manager. Do not reuse the credential-vault, ordinary
record-handle, or audit-receipt keys.
Published record references are AES-256-GCM encrypted, bounded, expire after eight hours, and bind the inner opaque record handle and optimistic baseline to the workspace, actor, screen version, producer action/result, model version, immutable publication, and record type. Changing the key invalidates outstanding uncommitted Data Action references; the interface must reload its named record before attempting a new mutation. A proven committed replay is resolved from the durable receipt before reference-expiry validation.
Keep the key stable across rolling deployments and all active-active nodes.
For rotation, deploy the new current key and retain each still-valid old key in
the comma-separated INFLOW_DATA_ACTION_RECORD_REFERENCE_PREVIOUS_KEYS keyring
until the maximum reference lifetime and any deployment overlap have elapsed.
The versioned reference carries a non-secret key fingerprint so a node chooses
one exact decryption key rather than attempting every key. New references use
only the current key. Remove a previous key only after no valid reference can
remain under it.
The process-local fallback is development-only, logs a warning, and invalidates references on restart. Docker Compose supplies a separate stable dev-only fallback that must be overridden in staging and production.
Archive recovery locator key
Every API node must receive the same INFLOW_DATA_ACTION_ARCHIVE_RECOVERY_KEY
(an independent 32-byte random value, base64-encoded). The key seals the opaque
locator of every Archive recovery episode. Those episodes are written by the
Archive Data Action, by a delete in a workspace that forbids permanent removal,
and by a generated delete-record rule. The administrative Archived view lists
an archived record, and Restore accepts it, only while its locator still
authenticates. Locators are long-lived, so never replace this key outright.
Deploy the new current key and keep every earlier key in the comma-separated
INFLOW_DATA_ACTION_ARCHIVE_RECOVERY_PREVIOUS_KEYS keyring for as long as
archived records sealed under it must remain restorable.
Without the variable, the API refuses to start unless development mode is
explicitly on. In development mode it uses a process-local key and logs a
warning. Every archive made before a restart then disappears from the Archived
view and cannot be restored (SCRUM-4366). The listing logs
archiveRecoveryEpisodesNotRestorable with a count when it skips such episodes.
Docker Compose supplies a stable dev-only fallback that must be overridden in
staging and production.
Integration live-test confirmation key
Every API node that issues or verifies a consultant's live Integration test
confirmation must receive the same independent
INFLOW_INTEGRATION_LIVE_TEST_INTENT_KEY. Generate 32 random bytes, encode them
with standard base64, and inject the value from the deployment secret manager.
Do not reuse vault, evidence, cursor, record-handle, Data Action, or audit keys.
For rotation, deploy the new current key and retain every still-valid old key
in the comma-separated INFLOW_INTEGRATION_LIVE_TEST_INTENT_PREVIOUS_KEYS
keyring on every node. New confirmations use only the current key. Keep an old
key through the maximum 15-minute confirmation lifetime and the full rolling
deployment overlap, then remove it. A missing or malformed current/previous
key fails startup outside explicit development mode. Development mode uses a
process-local random key unless Docker Compose supplies its stable dev-only
fallback; either behavior is intentionally unsuitable for production.
Integration evidence equality key
Every API node that captures or replays sanitized Integration evidence must
receive the same independent INFLOW_CONNECTOR_EVIDENCE_HMAC_KEY. Do not reuse
the credential-vault master key: rotating vault encryption must not invalidate
saved Integration evidence. Keep the evidence key stable across rolling
deployments and active-active nodes.
To rotate it, deploy the new current key while retaining the old value in the
comma-separated INFLOW_CONNECTOR_EVIDENCE_HMAC_PREVIOUS_KEYS keyring. New
evidence uses only the current key; existing evidence continues to verify by
its non-secret key identifier. Remove an old key only after its evidence has
expired or been deliberately recaptured. A missing old key makes that evidence
non-restorable and consultants must run the Integration again. Docker Compose
supplies a separate stable development value; production must override it.
Integration-history cursor key
Every API node that serves published Integration history must receive the same
INFLOW_INTEGRATION_HISTORY_CURSOR_KEY. Generate an independent 32-byte random
value, base64-encode it, and inject it from the deployment secret manager. Do
not reuse a credential-vault, record-handle, Data Action, or audit-receipt key.
Integration-history cursors are AES-256-GCM encrypted, expire after 15 minutes, and are bound to the requesting workspace. Keep the key stable across rolling deployments and all active-active nodes; changing it invalidates outstanding cursors, and consultants can recover by reloading the Integration catalog. Startup fails closed outside explicit development mode when the key is absent or malformed. Development mode uses a process-local random key, so cursors are intentionally invalidated when the API restarts.
Target-side record audit receipts
Modeled Records mutations span two independent databases: the publication target and the platform audit store. They therefore do not rely on a Spring transaction or XA coordinator. Each create, update, or delete first verifies a per-model audit-outbox table in the target schema, then commits the business DML and one redacted receipt in the same target JDBC transaction. If receipt insertion fails, the business mutation rolls back. If the platform audit store is unavailable after target commit, the receipt remains pending and the API can still report the correctly committed outcome.
Nodes with the integration-worker role scan applied publications and relay
pending receipts. The receipt event id is preserved as the audit_events
primary key, so concurrent nodes and crash-after-central-insert retries are
idempotent. A replay with the same id but a different payload fails closed.
Workers write the platform audit event before marking the target receipt
delivered.
Every production node that writes or relays Records must receive the same
independent INFLOW_AUDIT_RECEIPT_KEY: a base64-encoded 32-byte random secret.
The HMAC is verified before target data becomes platform audit evidence. Do not
reuse the vault or record-handle keys. During rotation, provide old 32-byte keys
through the comma-separated INFLOW_AUDIT_RECEIPT_PREVIOUS_KEYS setting until
their pending receipts drain. A receipt whose key id is temporarily unavailable
remains pending with HMAC_KEY_UNAVAILABLE; it is not misclassified as a forged
receipt or permanently quarantined. A bad signature under a known key still
fails closed and is quarantined. The process-local fallback is development-only;
Docker Compose supplies a stable local fallback.
Operational monitoring must alert on pending receipt age, repeated delivery
attempts, invalid signatures, unreachable publication targets, and undrained
backlog. DataRecordAuditOutboxWorker.statuses(limit) exposes bounded
point-in-time snapshots for applied publications, including pending, claimed,
failed-attempt, quarantined, and oldest-pending receipt state. These counts are
operational telemetry, not deletion-gap proof: a target DBA can delete a
not-yet-relayed receipt so it vanishes from every target-side bucket. Strict
production deployments also require restricted DDL/DML ownership plus an
independent platform intent/gap detector; the remaining hardening work is
tracked in the database certification bug ledger.
Application-user access runtime (inflow.application-user-access.enabled)
Set inflow.application-user-access.enabled=true in each node's canonical
conf/custom_config.properties and plan a restart on every API node so group application runtime access (application-runtime-access),
group home destinations (group-destinations), direct Studio admission policy
(studio-policy), and /application-entry routing are active across the cluster.
The local Compose launcher prepares a writable canonical copy from its explicit
development profile, which enables this flag. Separate deployment/bootstrap owners
must stage canonical values before upgrading; old feature-value environment overrides
no longer activate it. See Feature switches and migration. Keep this flag consistent across all active-active API nodes so session
capabilities (applicationUserAccessEnabled), runtime access previews/grants, and
published interface admission behave identically regardless of which node serves
the request.
Current role-to-work inventory
The role enum is broader than the short deployment examples above. Use this
inventory to decide which role belongs on each process. It describes the
current WorkerRuntimeService.runOnce() dispatch mapping; it does not mean that
every listed worker is composed, enabled, connected to its dependency or
available in every deployment profile.
| Configured role | Worker phase selected by the current runtime | Operational interpretation |
|---|---|---|
api |
Selects no phase by itself in WorkerRuntimeService.runOnce(). If the record-audit outbox worker is composed, it also selects record-audit-worker draining in the worker cycle. |
The role label does not provision an HTTP listener. The default role is not proof that the worker loop starts; confirm the canonical worker-start setting and the composed audit worker before expecting this background phase. |
outbox-publisher |
outbox-publisher / drain-outbox |
Selects the outbox publishing phase. Confirm its transport and downstream broker separately. |
kafka-consumer |
kafka-consumer / poll-broker |
Selects one broker polling phase; it is not the same phase as dispatching messages already stored in the platform inbox. |
inbox-dispatcher |
inbox-dispatcher / dispatch-inbox |
Selects the inbox dispatch phase. |
runtime-worker |
inbox-dispatcher / dispatch-inbox |
Also selects inbox dispatch; the role name does not establish a separate runtime queue in this dispatch table. |
notification-worker |
inbox-dispatcher / dispatch-inbox |
Also selects inbox dispatch; notification behavior still depends on the message handler and its configured dependencies. |
integration-worker |
record-audit-worker when composed, and inbox-dispatcher / dispatch-inbox |
Selects both shown phases. It is not a substitute for the outbox-publisher or kafka-consumer roles. |
timer-worker |
timer-worker / run-due-timers |
Selects due process timers and timeout work. A process deployment needs at least one eligible, running timer worker. |
service-task-worker |
service-task-worker / recover-timeouts and process-due-runs |
Selects service-task timeout recovery and due service-task processing. |
ai-worker |
ai-worker / recover-agent-timeouts and process-agent-runs, plus inbox dispatch |
Selects agent-run processing and recovery as well as the inbox dispatch phase. Provider readiness and effect/result checks remain separate. |
escalation-worker |
escalation-worker / run-due-escalations |
Selects the SLA escalation sweep. It is gated by process.module.enabled; with that module disabled, the phase is a no-op. |
PlatformNodeRole contains exactly these eleven role identifiers at the
referenced source revision. Role configuration is a routing/eligibility choice,
not a user authorization grant and not evidence that a phase ran. The normal
worker lifecycle must be enabled, and its dependencies and module gates must be
ready. For example, a node with INFLOW_NODE_ROLES=timer-worker but with
platform.worker.autoStart=false has timer eligibility configured but no
automatic worker loop. Conversely, starting the loop on an API-only node does
not select the timer, broker-consumer or service-task phases.
Document extraction is a separate DocumentExtractionWorkerLifecycle thread,
not one of the roles in PlatformNodeRole and not a phase in this role table.
When that lifecycle is composed, it auto-starts on a node whose roles contain
api or integration-worker; this selection is separate from the
platform.worker.autoStart switch for the shared worker loop. A configured
role still does not prove that the lifecycle or extraction dependency is
healthy. Do not treat a service-task-worker label as selection for document
extraction.
Worked configuration. For a deliberately separated API and timer layout, an operator's process manager could provide the following independent values. Each block belongs only to the named process; do not combine them into one shared environment file. This is a configuration illustration, not an Inflow deployment CLI.
# Inject only into each API process; assign a different ID per instance.
INFLOW_NODE_ID=peopleops-api-a
INFLOW_NODE_ROLES=api
API process conf/custom_config.properties:
platform.worker.autoStart=false
# Inject only into each timer process; use a different ID for every instance.
INFLOW_NODE_ID=peopleops-timer-a
INFLOW_NODE_ROLES=timer-worker
Timer process conf/custom_config.properties:
platform.worker.autoStart=true
The timer process must also be configured to reach the approved durable
services. Keep credentials in the approved secret store; do not copy them into
either process's public environment manifest. Use stable, unique IDs per
running process, and confirm the effective role set and worker state on the
intended node using the protected Node Control API. A load-balanced /current
request may answer from another node; retain the returned node ID and use an
approved direct operator route when a check must address one specific instance.
Source references: role enum, role dispatch, environment/file precedence, node routes and permission guard, separate document-extraction lifecycle, and focused property tests / controller tests.