Optional technical reference. This material is for developers, integration specialists and platform administrators. For the consultant lesson, read the practical guide.
Preserved from source baseline
d4587c7c0e073e58c2d430342f65ac55dd701bf4; source/static examples retain their stated scope. They are not new runtime verification or required consultant setup steps.
AI Gateway
Prepare the route before sending business content
The gateway separates administrator-managed connections, vendor model selection, platform model IDs, purpose defaults and per-workspace policy. A consultant's model choice must be one the environment actually exposes. There is no built-in provider/model catalog to assume, and a catalog read does not qualify reachability or disclosure.
| Layer | Example question | Owning evidence |
|---|---|---|
| Connection | Which endpoint and protected credential reference are admitted? | Administrator connection record. |
| Selected vendor model | Which model on that connection was chosen? | Actual discovered/selected vendor ID. |
| Platform model | What ID can rules/agents request? | Current gateway catalog response. |
| Purpose default | What should studio-agent choose when unspecified? | Actual AI defaults record. |
| Workspace policy | Which route and token usage are allowed here? | Current workspace policy and usage. |
Use synthetic content and an admitted endpoint. Connection testing, model discovery and task runs may spend actual provider attempts; the documentation did not perform them. The cases below distinguish preparation from execution and show supported recovery rather than automatic repeated calls.
Example UIAI-AI_GATEWAY-01 — Prepare a connection without exposing a secret
Scenario and inputs. An authorized AI administrator is preparing the people-ops demonstration. They own the admitted endpoint and protected credential reference. Nadia's consultant permission does not authorize creating connections, and a sample URL or placeholder reference is not an executable credential.
Procedure. Review the connection type, endpoint, policy and credential lifecycle. Create or update through /api/admin/ai-providers using its actual fields; use the authorized secret-material or protected-reference mechanism instead of putting a token in a manual, screenshot or agent catalog. Retain the returned connection ID separately from connectionKey. Perform test-connection/model discovery only when that external attempt is authorized.
Expected result. The connection record stores nonsecret settings and an opaque credential reference; it is not a model response or automatic proof of healthy execution. The controller's request includes connectionKey/name/type/endpoint/credentialReference/policy and optional secretMaterial, while the public connection object never contains the secret itself.
Failure and recovery. An invalid endpoint, refused secret or missing administrator authority requires the accountable owner to correct the configuration. Do not replace it with another agent's credentials, relax network admission or assume a stored endpoint is reachable. Keep the feature unavailable until its actual prerequisites are satisfied.
Sources. Connection controller and request, nonsecret connection record.
Example UIAI-AI_GATEWAY-02 — Select a real model and inspect its runtime plan
Scenario and inputs. The administrator has a real connection ID and may list that provider's available models. The teaching fixture uses people-ai as its illustrative connection key; it is not a built-in connection. The actual vendor model name must come from the admitted provider catalog.
Procedure. Use the connection's /models discovery surface, select an actual returned vendor model with /selected-model, and enable the connection through the supported administration flow. Inspect /api/ai-gateway/models and retain the actual platform ID, typically people-ai-chat for this fixture. Read /models/{modelId}/runtime-plan before configuring a rule or an agent preference; inspect enabled/runtime/load settings rather than guessing a provider name.
Expected result. The platform model links to a connection-backed provider selection. A runtime plan describes intended deployment/load metadata; it is not evidence that the endpoint executed a prompt successfully. A purpose default may select this model only while it is actually valid for the request and workspace policy.
Failure and recovery. An empty discovery result, missing vendor model or disabled connection must stay visible as a preparation issue. Have the connection owner repair or choose a real available model. Copying an old seed ID, editing a prompt or setting an arbitrary agent preferredModelId cannot provision the missing model.
Sources. Discovery and selection, gateway model/runtime APIs, runtime plan fields.
Example UIAI-AI_GATEWAY-03 — Understand workspace token limits
Scenario and inputs. A workspace administrator is reviewing why an otherwise valid demonstration AI task cannot run. The current policy uses a monthly token limit, not a dollar budget. The workspace key people-ops and any usage values are synthetic teaching coordinates, not permission to alter a shared production limit.
Procedure. Read the actual workspace /ai-policy and /ai-usage before considering a change. Compare enabled state, allowedModels, allowedProviders and monthlyTokenLimit with the chosen platform model and current month-to-date usage. Retain the actual usage report rather than estimating provider prices from a token count. If an authorized policy change is appropriate, preserve the owner's intended allowlists and explicit token units.
Expected result. The explanation separates a disabled route, disallowed model/provider and exhausted token allowance. The request contract contains monthlyTokenLimit. The legacy monthlyBudgetUsd field is deliberately refused rather than silently accepted without enforcing a limit; usage reporting records tokens instead of invented cost estimates.
Failure and recovery. Do not remove a limit or widen allowlists merely to make a tutorial pass. Ask the workspace AI-policy owner for the needed decision, or reduce the admitted workload. A failed policy/readiness request is not evidence of zero usage or unlimited budget, and it must not trigger repeated provider attempts in the background.
Sources. Actual policy and usage DTOs, legacy-field refusal.
Example UIAI-AI_GATEWAY-04 — Diagnose an unavailable provider safely
Scenario and inputs. A synthetic leave-summary task has a known chosen platform model but its request fails. The operator may inspect gateway/model information and safe audit identifiers. They must not print prompts containing private records or expose credential material while diagnosing the problem.
Procedure. Check the actual model and provider connection state, then the model catalog/report and runtime-plan metadata. Distinguish a missing model from a provider that was selected but unreachable, and a transport deadline from output-contract validation. Use the governed error and safe attempt/audit identifier returned by the actual task surface; do not invent a universal receipt field or treat every failure as the same retryable condition.
Expected result. The operator can identify which preparation or execution boundary failed. The failed response is not a successful empty model answer. Redaction, workspace accounting and native disclosure policy remain part of the request path even when a provider is unavailable or an operator reads its settings.
Failure and recovery. Repeated attempts may spend provider capacity or duplicate an uncertain effect. Correct the specific admitted endpoint/model/credential/deadline issue, then retry only through the original surface when authorized. A UI Stop ends the browser wait but does not prove the provider cancelled its request; preserve that uncertainty and its native usage/audit evidence.
Sources. Model catalog and runtime plan APIs, transport deadline tests, unreachable provider test.
The AI Gateway is the platform abstraction for cloud AI providers and on-prem model runtimes.
flowchart TD
A["Workflow AI Task"] --> B["AI Gateway"]
S["BPMN service.ai.* task"] --> T["AiGatewayServiceTaskExecutor"]
T --> B
C["Fabric"] --> B
D["AI Agent Task"] --> B
B --> E["Provider Registry"]
E --> F["OpenAI / Anthropic / Gemini / Azure / Bedrock"]
E --> G["Mistral / Cohere / Hugging Face / Others"]
E --> H["Ollama / vLLM / TGI / llama.cpp / ONNX"]
B --> I["Routing Policy"]
AI comes only from the AI provider connections an administrator creates in
Authorization -> AI providers (database table
platform_ai_provider_connections). Each enabled connection with a chosen model
("Use this model") contributes one model, <connectionKey>-chat: the enabled
model. There are no built-in providers, models or load profiles. The former XML
catalog (conf/ai-providers/platform-ai-providers.xml) and its loader were
removed by SCRUM-4817 (2026-11-05).
Authenticated API
The AI Gateway catalog APIs are authenticated and read-only. They list connection data only: the connections and their enabled models (runtime provider connections are administered separately — see AI Provider Connections):
GET /api/ai-gateway/providers
GET /api/ai-gateway/providers/{providerId}
GET /api/ai-gateway/models
GET /api/ai-gateway/models/search?q=local
GET /api/ai-gateway/models/{modelId}
GET /api/ai-gateway/model-load-profiles
GET /api/ai-gateway/models/{modelId}/load-profile
GET /api/ai-gateway/runtime-plans
GET /api/ai-gateway/models/{modelId}/runtime-plan
Required permission:
AI_GATEWAY_VIEW
The development designer and worker roles receive this permission. root
has *.
These APIs do not execute AI calls. AI execution still flows through
AiGatewayService, BPMN service.ai.* tasks, and future agent/runtime workers.
The API is for configuration visibility, runtime planning, admin consoles, and
UI model selection.
Core Concepts
AiProviderDefinition: cloud provider or local runtime endpoint, built from an administrator connection.AiModelDefinition: model id, provider model name, runtime type, capabilities, context window, and data classification.AiModelLoadProfile: how a local/on-prem model is loaded, warmed, scaled, and health-checked.AiCapability: chat, structured output, tools, embeddings, image, audio, code, and agentic reasoning.AiGatewayRequest: provider-agnostic AI request with purpose, input, constraints, and actor.AiRoutingPolicyService: chooses a provider/model based on preferred model or required capabilities.AiModelRuntimeService: creates runtime plans from model definitions and load profiles.
BPMN Service Task Usage
AI service tasks in conf/service-tasks/platform-service-tasks.xml use the
ai-gateway implementation. At runtime, AiGatewayServiceTaskExecutor maps the
BPMN activity configuration and workflow variables into an AiGatewayRequest.
Supported seed task types include:
service.ai.extractservice.ai.summarizeservice.ai.classifyservice.ai.translateservice.ai.generate.emailservice.ai.agent.run
Example BPMN activity configuration:
{
"activityType": "service.ai.summarize",
"modelId": "my-openai-chat",
"prompt": "Summarize this workflow input for an operations user.",
"input": {
"content": "Customer requested renewal pricing."
},
"constraints": {
"maxOutputTokens": 250
}
}
The task completes with workflow variables such as aiText,
aiStructuredOutput, modelId, providerId, usage, and auditId.
Centralized routing, redaction, token-usage recording, and audit behavior remain in
the AI Gateway rather than inside individual BPMN tasks.
First Routing Behavior
The default routing policy:
- uses
preferredModelIdwhen provided — but still validates the pinned model against allrequiredCapabilities(AI_MODEL_CAPABILITY_NOT_FOUNDon mismatch, before any provider call). While another model is enabled, a pin that names a model that is not an enabled connection model fails withAI_MODEL_NOT_FOUND(the plain "choose a model" outcome); with no enabled model it fails withAI_PROVIDER_NOT_CONNECTED; - otherwise finds enabled models that satisfy required capabilities;
- prefers on-prem models when multiple candidates match;
- requires on-prem models for
dataClass=restricted; - prefers on-prem models for
dataClass=confidential.
Routing only ever picks enabled models. With no enabled model at all, AI calls
fail with AI_PROVIDER_NOT_CONNECTED ("No AI provider is connected..."). When
enabled models exist but none fits the request, the call fails with
AI_MODEL_CAPABILITY_NOT_FOUND. Restricted data still requires an on-premise
enabled model.
If a restricted request explicitly asks for a cloud model, the router rejects it
with AI_MODEL_DATA_RESIDENCY_VIOLATION.
Requests carrying binary attachments (AI task IMAGE inputs) are additionally
restricted to on-premise models: binary content cannot be scanned by the text
redaction policy, so automatic routing filters to on-premise candidates and
any external resolution — pinned or otherwise — is refused before provider
invocation with AI_TASK_ATTACHMENT_EXTERNAL_BLOCKED, enforced both in the
routing policy and fail-closed in the gateway itself.
Removed: XML provider format and extension pack
The XML provider format (conf/ai-providers/*.xml) and the AI gateway
extension-pack API (POST /api/ai-gateway/extension-pack/preview, and its MCP
declaration) were removed by SCRUM-4817 (2026-11-05). Add providers and models
as connections in Authorization -> AI providers. The other extension-pack
APIs (connectors, service tasks, AI fabric) are unchanged.
AI Provider Connections (Database-Backed Administration)
Provider connections are managed at runtime from the Studio (AI Providers screen) or the admin API, without restarting:
GET /api/admin/ai-providers
GET /api/admin/ai-providers/{id}
POST /api/admin/ai-providers
PUT /api/admin/ai-providers/{id}
DELETE /api/admin/ai-providers/{id}
POST /api/admin/ai-providers/{id}/enable
POST /api/admin/ai-providers/{id}/disable
POST /api/admin/ai-providers/{id}/test-connection
GET /api/admin/ai-providers/{id}/models
POST /api/admin/ai-providers/{id}/selected-model
Required permissions:
reads (list/get): AI_GATEWAY_VIEW
writes, test, discovery: AI_PROVIDER_MANAGE
Connections persist in platform_ai_provider_connections (migration
000116). The table stores an opaque credentialReference only — never
secret material. When a create/update carries secretMaterial, the service
stores it in the credential vault (purpose AI_PROVIDER_RUNTIME) and keeps
the returned vault:<id> reference; an update with new material always
stores a brand-new vault entry and repoints the connection (never rotates in
place), because vault: references can be shared across connections. Secret
material is never returned by any API, written to logs, or included in audit
metadata (create audits record the new field values, update audits record
old/new field maps, and delete audits record the full old field map, each
with a credentialStored marker where applicable).
Routing semantics (no restart, read-through on every lookup so active-active nodes never serve stale data):
- only enabled connections take part; a disabled connection contributes nothing;
- an enabled connection with a chosen model (
policy.selectedModelId, set with "Use this model") contributes the model{connectionKey}-chat(provider id = the connection key, vendor name = the picked id). Ask Inflow (studio-agent) and the assist features route to it. A connection without a chosen model contributes no model; - discovery (below) is an administrative pick-list aid;
- blank/clear removes the model. The choice lives in the existing JSON
policycolumn.
Credential reference resolution is fail-closed: env: and secret: schemes
resolve from the environment, vault: resolves through the credential vault,
and any other reference resolves to empty (it is never used verbatim as the
credential).
Upgrade note (2026-07-31): EnvironmentAiCredentialResolver is now
fail-closed — only env: and secret: references resolve (vault: resolves
upstream through the vault). Before this change, an unrecognized reference
(for example a typo, an oauth: form, or a literal secret) was silently used
verbatim as the credential. Any pre-existing verbatim or foreign-scheme
credential references — on AI providers, AI provider connections, connectors,
or connected systems — must be migrated to a supported scheme before upgrade,
or resolution returns empty and calls fail closed.
POST /{id}/test-connection performs a real minimal call through the
resolved provider client and always returns HTTP 200 with:
{ "ok": true, "latencyMs": 132, "providerType": "OLLAMA", "modelCount": 7 }
Failures return ok: false with an explicit errorCode
(AI_PROVIDER_PROBE_UNSUPPORTED, AI_PROVIDER_CREDENTIAL_RESOLUTION_FAILED,
AI_PROVIDER_REQUEST_FAILED, AI_PROVIDER_UNREACHABLE, or AI_PROVIDER_PROBE_FAILED).
AI_PROVIDER_UNREACHABLE (spec 4798) means the provider gave no answer at all
(unknown host, refused connection, I/O error or the attempt deadline). Its
message is the plain "<Provider> could not be reached."; the address goes to
the server log only. Model discovery (GET /{id}/models) answers
supported: false with the same errorCode for such a failure, so Studio says
"The provider could not be reached" and keeps "not supported" for providers
without a listing API. The probe is a
models-listing call per client: OpenAI-compatible GET /models (OpenAI,
Groq, Together, DeepSeek, OpenRouter, Perplexity, vLLM, Mistral), Anthropic
GET /v1/models, Gemini GET /v1beta/models, Ollama GET /api/tags,
Bedrock GET https://bedrock.{region}.amazonaws.com/foundation-models with
a Bedrock API key as Authorization: Bearer (IAM SigV4 is not implemented;
access-key-only connections report listing unsupported). Region comes from
policy.region or the runtime endpoint host (us-east-2 from
https://bedrock-runtime.us-east-2.amazonaws.com). Provider types without a
listing adapter report AI_PROVIDER_PROBE_UNSUPPORTED (Azure OpenAI unless
openaiCompatible is set, Cohere, Hugging Face, TGI, llama.cpp, ONNX,
CUSTOM).
GET /{id}/models runs the same listing call and returns
{ "supported": true, "modelIds": ["x-ai/grok-4.6"],
"models": [{ "id": "x-ai/grok-4.6", "label": "xAI: Grok 4.6", "kind": "CHAT" }] }
for the Studio model pick list. kind is CHAT, EMBEDDING, BATCH, or
OTHER. Unsupported providers return supported: false so the UI falls
back to typing a vendor id. Responses never contain credential material.
POST /{id}/selected-model with { "modelId": "..." } writes
policy.selectedModelId (max 256 characters; blank clears) and pins the
studio-agent AI default to {connectionKey}-chat. It requires
AI_PROVIDER_MANAGE, audits AI_PROVIDER_CONNECTION_UPDATED with the
previous and new vendor ids only, and never accepts or returns secret
material.
Deleting a connection removes the row only; the vault entry is deliberately
kept because vault: references can be shared (revoke is an explicit
vault-administration operation).
Model Load Profiles
Model load profiles are optional runtime instructions for local and on-prem
models. They are separate from AiModelDefinition so the stable platform model
id can stay the same while operations changes deployment details such as the
container image, artifact URI, device, replica counts, warmup prompt, or health
check.
Supported load modes:
STARTUP: load and warm the model when the runtime starts.LAZY: load the model the first time it is needed.ON_DEMAND: load for a specific request or job and allow scale-to-zero.EXTERNAL_MANAGED: the platform calls a provider endpoint but does not manage the model lifecycle.DISABLED: keep the model listed but do not load it.
There are no shipped load profiles. Load profiles and runtime plans exist only
for on-premise connection models (see below). AiModelRuntimeService exposes
the Java runtime plan; a connection model without an explicit profile defaults
to LAZY.
An on-premise AI provider connection created in AI providers (Ollama, vLLM,
TGI, llama.cpp or ONNX runtime on a private endpoint) has a load profile for its
<connectionKey>-chat model while it is enabled and has a selected model
(spec 4319). A LAZY profile is derived from the connection: the provider's runtime type, the
health check /api/tags (Ollama) or /health, a 180-second warmup, deployment
target ai-provider-connection:<connectionKey> and the selected model as the
artifact.
An on-premise connection without a selected model offers no model; the catalog
report warns "on-premise connection has no selected model".
A connection counts as on-premise only when its type is self-hosted and its
endpoint is on a private network. IP literals are checked as addresses, and an
all-digits or hex host such as 134744072 or 0x8080808 (which resolvers read
as 8.8.8.8) is never private. A private-looking name (single label, .local,
.internal, .lan, .localhost) must resolve only to loopback, RFC 1918,
link-local or IPv6 unique-local addresses; otherwise the model is treated as a
cloud model, restricted data and attachments are not routed to it, and the
model catalog says why. The check runs on every registry lookup, so routing uses
an answer at most one cache period old. Each lookup or capture resolves a
distinct endpoint once; concurrent callers share one lookup per host; finished
answers are cached (30 s resolved, 10 s not) and a caller stops waiting after
1.5 s, which counts as not private. Several distinct uncached slow endpoints
add up to 1.5 s each. A private reverse proxy that forwards to a cloud LLM still
counts as on-premise: by configuring the endpoint the administrator attests to
where it sends data.
An on-premise connection model is READY only after a live runtime check. For
Ollama the catalog probes /api/tags (host reachable, model pulled). Other
runtimes stay NEEDS_ATTENTION with "on-prem runtime reachability is not
verified", because a connection does not record its Test result.
Model Catalog Operations (connection data only)
The model catalog is built from the administrator's connections only; the
endpoints below (/model-catalog, /models, /providers,
/model-load-profiles, /runtime-plans) keep their response shapes. Raw
provider, model, and runtime-plan APIs are useful for configuration. For
operators and AI workflow designers, the composed model catalog is usually the
better starting point:
GET /api/ai-gateway/model-catalog
X-Actor-Id: worker
GET /api/ai-gateway/model-catalog/my-ollama-chat
X-Actor-Id: worker
GET /api/ai-gateway/model-catalog/report
X-Actor-Id: worker
AiModelCatalogService joins:
AiProviderDefinitionAiModelDefinition- configured or generated
AiModelRuntimePlan - explicit
AiModelLoadProfilestate
Each catalog entry includes provider type, endpoint, credential reference, provider model name, runtime type, load mode, capabilities, context window, data classification, local runtime deployment details, replica counts, device, restricted-data allowance, operational status, readiness warnings, provider policy, and runtime metadata.
The report endpoint summarizes:
- provider count and model count
- cloud versus on-prem model count
- configured load-profile count
- enabled versus disabled models
- counts by provider type, runtime type, and capability
- on-prem models missing explicit load profiles
- models allowed for restricted data
- readiness warnings
This gives administrators one quick view of whether cloud providers and
on-prem model loaders are ready before routing BPMN service tasks or agents to
them. These endpoints require AI_GATEWAY_VIEW.
For Ollama-backed models, readiness is not computed from static config alone:
the catalog probes the provider endpoint's GET /api/tags (1s connect / 2s
request timeouts, memoized per endpoint for 10s). A model whose
providerModelName is absent from the host's installed list reports
NEEDS_ATTENTION with model not pulled on Ollama host (run: ollama pull <name>), and an unreachable Ollama host also reports NEEDS_ATTENTION —
a catalog row never claims READY while its runtime is down. The local
Compose stack no longer includes Ollama (ollama-dev and ollama-pull were
removed). A local Ollama is optional: run one yourself and connect it in AI
providers like any other provider, then pull the model you choose.
No built-in catalog
There is no seed provider catalog and no startup catalog import. Every provider,
model and load profile comes from an administrator connection, so a fresh system
has no AI until a connection is created, enabled and given a model. Assist
features (data-model, expression and rule assist, process-design assist) and the
Studio Agent have no configured model: they use the purpose default when it
names an existing model, otherwise the enabled model. Without one, assist falls
back to basic rules with a plain reason and Ask Inflow is hidden. The settings
platform.assist.model, platform.assist.provider, platform.agent.model,
platform.agent.provider and platform.process.assist.model were removed.
Provider Clients
Provider clients send provider-agnostic AiGatewayRequest objects to a concrete
provider API and normalize the response into AiGatewayResponse.
Attempt deadline
Every provider call (OpenAI-compatible including OpenRouter, Anthropic, Gemini,
Ollama, Bedrock) runs on the shared platform HTTP factory. That factory bounds
connect and the wait for response headers (platform.http.*, 30 s each; see
the connectors manual). On top of it, an AI provider call has a whole-attempt
deadline:
- Default 60 seconds. Set
INFLOW_AI_ATTEMPT_TIMEOUT_SECONDS(whole seconds,1–3600) in the API environment to change it. Any other value, for example0, a negative number or a non-integer, stops the API from starting, so a typo can never mean "no limit". - Once response headers have arrived, the deadline stops the call for real:
the thread reading the answer is woken and the connection is dropped.
Before headers arrive, only the shared connect and header bounds apply, so
with a deadline below 60 s a provider that withholds headers can hold the
call for up to 30 s connect plus 30 s header wait before it fails. The
effective bound is therefore the larger of the deadline and those shared
bounds. Either way the task fails with the usual provider failure
(
AI_PROVIDER_REQUEST_FAILED). When the deadline itself ends the call, Technical details name it ("AI provider attempt exceeded 60 s."); when a shared connect or header bound ends it first, they show that timeout instead. - Some gateways, OpenRouter among them, send response headers early and keep a slow answer alive while the model generates. Such answers were previously unbounded; they now end at the deadline. Raise the deadline if long reasoning or long outputs are expected. Each call holds one request thread for up to the deadline, so very high values (up to an hour) have a matching resource cost.
- The deadline covers one provider call. It is not a deadline for the whole AI task, and it does not cancel work the provider is still doing; stopping a test run in Studio only stops the browser request.
OpenAI-Compatible Client
OpenAiCompatibleProviderClient sends requests to /chat/completions endpoints
and parses text, JSON structured output, usage, and provider response id.
It supports:
- OpenAI
- Groq
- Together AI
- DeepSeek
- OpenRouter
- Perplexity
- vLLM
- any provider with policy
"openaiCompatible": true
The client maps:
AiModelDefinition.providerModelNameto the providermodelAiGatewayRequest.prompt,purpose,input, andconstraintsinto a user messageSTRUCTURED_OUTPUTorresponseFormat=jsonintoresponse_format: {"type":"json_object"}temperature,maxTokens, andtopPconstraints into compatible request fields
Anthropic Client
AnthropicProviderClient supports providers with type ANTHROPIC.
It maps:
- endpoint to
/v1/messages - credentials to the
x-api-keyheader - provider policy
anthropicVersionto theanthropic-versionheader, defaulting to2023-06-01 AiModelDefinition.providerModelNametomodelmaxTokenstomax_tokens, defaulting to1024temperatureandtopPinto compatible request fields- response
content[].text,usage, andidintoAiGatewayResponse
For structured output, the client appends a strict JSON instruction to the normalized prompt and parses JSON text when the provider returns it.
Gemini Client
GeminiProviderClient supports providers with type GOOGLE_GEMINI.
It maps:
- endpoint to
/v1beta/models/{providerModelName}:generateContent - credentials to the Gemini API
keyquery parameter - prompt content into
contents[].parts[].text temperature,maxTokens, andtopPintogenerationConfigSTRUCTURED_OUTPUTorresponseFormat=jsonintoresponseMimeType=application/json- response candidate text,
usageMetadata, andresponseIdintoAiGatewayResponse
Ollama Client
OllamaProviderClient supports providers with type OLLAMA and sends native
non-streaming chat requests to:
/api/chat
It maps:
- endpoint to
/api/chat AiModelDefinition.providerModelNameto Ollamamodel- normalized prompt content into
messages[0].content STRUCTURED_OUTPUTorresponseFormat=jsonintoformat=jsontemperature,topP, andmaxTokensinto Ollamaoptions- response
message.contentintoAiGatewayResponse.text prompt_eval_countandeval_countinto normalized usage keys:prompt_tokenscompletion_tokenstotal_tokens
This allows an Ollama connection to run through the AI Gateway without requiring an OpenAI-compatible adapter.
DefaultAiGatewayService performs the full execution path:
sequenceDiagram
participant Caller
participant Gateway as "DefaultAiGatewayService"
participant Router as "AiRoutingPolicyService"
participant Registry as "AiProviderRegistryService"
participant Client as "AiProviderClient"
participant Provider as "AI Provider"
Caller->>Gateway: execute(request)
Gateway->>Router: chooseModel(request)
Router-->>Gateway: providerId/modelId
Gateway->>Registry: load provider/model
Gateway->>Client: execute(provider, model, request)
Client->>Provider: POST provider-native endpoint
Provider-->>Client: provider response
Client-->>Gateway: AiGatewayResponse
Gateway-->>Caller: AiGatewayResponse
AI Call Audit And Metering
Every DefaultAiGatewayService.execute call records an audit event when an
AuditService is configured.
Successful calls use:
eventType: AI_GATEWAY_CALL_SUCCEEDED
entityType: AI_GATEWAY_CALL
entityId: generated call id
Failed calls use:
eventType: AI_GATEWAY_CALL_FAILED
entityType: AI_GATEWAY_CALL
entityId: generated call id
The audit metadata includes:
- generated
callId status- request
purpose preferredModelId- required capabilities
- constraints, including values such as
dataClass - routing reason
- provider id and provider type
- model id and provider model name
- on-premise flag
- duration in milliseconds
- provider response id when available
- usage metadata when available
tokenUsagewithinputTokens,outputTokens,cachedInputTokensandtotalTokenswhen usage is available- redaction metadata when the AI redaction policy is configured
- error type, message, and platform error code for failures
This gives the platform the first durable foundation for:
- per-user AI usage history
- provider/model usage dashboards
- token metering
- compliance review for restricted/confidential data
- incident review when provider calls fail
Pricing and cost estimation (removed)
Pricing and cost estimation were removed by SCRUM-4817 (2026-11-05):
conf/ai-providers/platform-ai-pricing.xml, the cost estimator and every cost
field are gone. Audit metadata records tokenUsage (input, output, cached input
and total tokens) and Studio shows token counts, not cost.
Workspace token limit
Workspace AI budgets are token based. The Workspace AI policy has
monthlyTokenLimit (integer; empty = no limit) instead of monthlyBudgetUsd. A PUT /api/workspaces/{idOrKey}/ai-policy that still sends
monthlyBudgetUsd is refused with 400 AI_POLICY_FIELD_REMOVED ("monthlyBudgetUsd was
removed; set monthlyTokenLimit instead.") and nothing is saved.
Before each call and after recording its usage, the gateway compares
month-to-date total tokens with the limit; when it is reached the call fails with
AI_WORKSPACE_BUDGET_EXCEEDED ("monthly AI token limit"). Studio shows
"Monthly token limit". GET /api/workspaces/{idOrKey}/ai-usage returns
monthToDateTokens and per-call totalTokens. Migration 000416 adds
workspace_ai_policies.monthly_token_limit and
workspace_ai_usage.total_tokens. The old monthly_budget_usd and
estimated_cost_usd columns stay in the database unused (forward-only
migrations) and existing USD budgets are not converted: the token limit starts
empty.
Operator rollout runbook (SCRUM-4817)
Existing data is not migrated. Saved ids of removed catalog models no longer resolve, so prepare before deploying:
- Create or enable an AI provider connection, choose a model ("Use this model") and pass Test connection. Restricted-data AI Tasks need an on-premise connection.
- Re-pick models in AI Tasks and process steps that named removed models. A pin to a removed model fails with the plain "choose a model" outcome; a missing backup model is reported at publish ("This backup model is no longer available. Choose a connected AI model.").
- Fix workspace allow-lists that name removed ids. Allow-lists stay fail-closed and are never widened automatically.
- Purpose defaults that name a missing model (including the old
studio-agentseed) are never re-routed: while no AI provider is connected they give "No AI provider is connected"; once one is connected they fail with the plain "choose a model" outcome. Re-pick or clear them in AI defaults. (The Studio Agent itself uses the enabled model it finds.) A default that cannot be read refuses the call (AI_MODEL_DEFAULTS_UNAVAILABLE). - USD budgets are lifted. Workspaces that had a monthly USD budget have no limit until a monthly token limit is set: the old column is no longer read. Set token limits for them (query below). A call whose provider reports no token usage counts as 0 tokens.
- If you used the compose
ollama-dev, run your own Ollama and connect it.
Read-only checks (PostgreSQL):
-- Workspaces with allow-lists that may name removed ids
SELECT workspace_id, allowed_models, allowed_providers
FROM workspace_ai_policies
WHERE COALESCE(allowed_models, '') <> '' OR COALESCE(allowed_providers, '') <> '';
-- Workspaces whose USD budget is no longer enforced (no token limit set yet)
SELECT workspace_id, monthly_budget_usd
FROM workspace_ai_policies
WHERE monthly_budget_usd IS NOT NULL AND monthly_token_limit IS NULL;
-- Purpose defaults that do not name a connection model
SELECT purpose, model_id, agent_definition_id
FROM platform_ai_defaults
WHERE model_id NOT LIKE '%-chat';
Prompt Redaction Policy
AI request redaction is configured outside Java code under:
conf/ai-providers/platform-ai-redaction.xml
The gateway applies redaction after routing and before provider execution, so
all provider clients receive the sanitized request. If a blocking rule matches,
the gateway stops before provider execution and records a failed audit event
with error code AI_REQUEST_BLOCKED_BY_REDACTION_POLICY.
<aiRedactionPolicy version="1" enabled="true">
<rule id="email-address">
<description>Redact email addresses before AI provider submission.</description>
<pattern>[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}</pattern>
<replacement>[REDACTED:email]</replacement>
<action>REDACT</action>
<enabled>true</enabled>
</rule>
</aiRedactionPolicy>
Supported actions:
REDACT: replace matching text and continue execution.BLOCK: stop the AI call before provider execution.
The default policy seeds rules for:
- email addresses
- bearer/API token shaped values
- credit-card-like number sequences
- private key material, which is blocked
Redaction is recursive across:
- prompt text
- input map values
- constraint map values
- nested maps and lists
Audit metadata records only safe redaction information:
- whether redaction was enabled
- whether the request was blocked
- replacement count
- matched rule ids
- blocked rule ids
It does not record raw matched values.
Provider coverage is defined by the current client and connection sections above. Bedrock already has an API-key client; IAM SigV4 remains unsupported. An OpenAI-compatible endpoint does not establish a distinct qualified native client or model-loader lifecycle for every server product. Provider-specific admission, residency and production certification still need the deployment's own evidence.
Delivered (2026-07-31): database-backed provider connections with Studio and admin-API management, vault-stored credentials, test-connection probes, model discovery, and the read-through routing overlay — see AI Provider Connections.
Current-source provider instructions and reasoning
Purpose-scoped model defaults
The selected-model operation keeps its legacy behavior when no purpose is supplied: it sets the shared studio-agent default. New callers may explicitly target studio-agent, app-chat, or home-app-chat; an explicit purpose changes only that purpose. Home provisioning should target home-app-chat so it leaves Ask Inflow and published App Shell Chat defaults unchanged. Explicit purpose selection requires an enabled connection, a nonblank model, and both provider-management and AI-default-management capabilities. The server refuses a selection that would redirect another purpose already using the same connection-backed model.
This addendum records behavior added after the preserved reference snapshot above.
OpenAI-compatible provider requests receive systemInstructions as a separate system message, not repeated inside
the user-message JSON. If a compatible endpoint has no system role, the same instruction text is prepended to the
prompt under an Instructions: heading.
Provider connections may set policy.reasoningEffort to off, low, medium or high; unsupported values are
rejected. When a connection omits the setting, App Chat requests medium reasoning effort and other purposes send
no reasoning parameter. OpenRouter receives reasoning.effort; OpenAI and Azure OpenAI reasoning models receive
reasoning_effort. Reasoning requests omit temperature and top_p. The provider adapter determines whether its
model qualifies for reasoning by model name or the connection's policy.reasoningModel declaration. Configure the
policy on the connection; do not assume a Studio picker or a model name alone enables provider-side reasoning.
For App Chat's record/task tool catalog, source and field checks, per-turn limits, and fallback path, read the App Chat tool-calling technical reference. These notes describe source behavior and configuration; they are not a provider execution or deployment verification.