Inflow.aiDocs

Optional technical reference. This material is for developers, integration specialists and platform administrators. For the consultant lesson, read the practical guide.

Preserved from source baseline d4587c7c0e073e58c2d430342f65ac55dd701bf4; source/static examples retain their stated scope. They are not new runtime verification or required consultant setup steps.

AI Gateway

Prepare the route before sending business content

The gateway separates administrator-managed connections, vendor model selection, platform model IDs, purpose defaults and per-workspace policy. A consultant's model choice must be one the environment actually exposes. There is no built-in provider/model catalog to assume, and a catalog read does not qualify reachability or disclosure.

Layer Example question Owning evidence
Connection Which endpoint and protected credential reference are admitted? Administrator connection record.
Selected vendor model Which model on that connection was chosen? Actual discovered/selected vendor ID.
Platform model What ID can rules/agents request? Current gateway catalog response.
Purpose default What should studio-agent choose when unspecified? Actual AI defaults record.
Workspace policy Which route and token usage are allowed here? Current workspace policy and usage.

Use synthetic content and an admitted endpoint. Connection testing, model discovery and task runs may spend actual provider attempts; the documentation did not perform them. The cases below distinguish preparation from execution and show supported recovery rather than automatic repeated calls.

Example UIAI-AI_GATEWAY-01 — Prepare a connection without exposing a secret

Scenario and inputs. An authorized AI administrator is preparing the people-ops demonstration. They own the admitted endpoint and protected credential reference. Nadia's consultant permission does not authorize creating connections, and a sample URL or placeholder reference is not an executable credential.

Procedure. Review the connection type, endpoint, policy and credential lifecycle. Create or update through /api/admin/ai-providers using its actual fields; use the authorized secret-material or protected-reference mechanism instead of putting a token in a manual, screenshot or agent catalog. Retain the returned connection ID separately from connectionKey. Perform test-connection/model discovery only when that external attempt is authorized.

Expected result. The connection record stores nonsecret settings and an opaque credential reference; it is not a model response or automatic proof of healthy execution. The controller's request includes connectionKey/name/type/endpoint/credentialReference/policy and optional secretMaterial, while the public connection object never contains the secret itself.

Failure and recovery. An invalid endpoint, refused secret or missing administrator authority requires the accountable owner to correct the configuration. Do not replace it with another agent's credentials, relax network admission or assume a stored endpoint is reachable. Keep the feature unavailable until its actual prerequisites are satisfied.

Sources. Connection controller and request, nonsecret connection record.

Example UIAI-AI_GATEWAY-02 — Select a real model and inspect its runtime plan

Scenario and inputs. The administrator has a real connection ID and may list that provider's available models. The teaching fixture uses people-ai as its illustrative connection key; it is not a built-in connection. The actual vendor model name must come from the admitted provider catalog.

Procedure. Use the connection's /models discovery surface, select an actual returned vendor model with /selected-model, and enable the connection through the supported administration flow. Inspect /api/ai-gateway/models and retain the actual platform ID, typically people-ai-chat for this fixture. Read /models/{modelId}/runtime-plan before configuring a rule or an agent preference; inspect enabled/runtime/load settings rather than guessing a provider name.

Expected result. The platform model links to a connection-backed provider selection. A runtime plan describes intended deployment/load metadata; it is not evidence that the endpoint executed a prompt successfully. A purpose default may select this model only while it is actually valid for the request and workspace policy.

Failure and recovery. An empty discovery result, missing vendor model or disabled connection must stay visible as a preparation issue. Have the connection owner repair or choose a real available model. Copying an old seed ID, editing a prompt or setting an arbitrary agent preferredModelId cannot provision the missing model.

Sources. Discovery and selection, gateway model/runtime APIs, runtime plan fields.

Example UIAI-AI_GATEWAY-03 — Understand workspace token limits

Scenario and inputs. A workspace administrator is reviewing why an otherwise valid demonstration AI task cannot run. The current policy uses a monthly token limit, not a dollar budget. The workspace key people-ops and any usage values are synthetic teaching coordinates, not permission to alter a shared production limit.

Procedure. Read the actual workspace /ai-policy and /ai-usage before considering a change. Compare enabled state, allowedModels, allowedProviders and monthlyTokenLimit with the chosen platform model and current month-to-date usage. Retain the actual usage report rather than estimating provider prices from a token count. If an authorized policy change is appropriate, preserve the owner's intended allowlists and explicit token units.

Expected result. The explanation separates a disabled route, disallowed model/provider and exhausted token allowance. The request contract contains monthlyTokenLimit. The legacy monthlyBudgetUsd field is deliberately refused rather than silently accepted without enforcing a limit; usage reporting records tokens instead of invented cost estimates.

Failure and recovery. Do not remove a limit or widen allowlists merely to make a tutorial pass. Ask the workspace AI-policy owner for the needed decision, or reduce the admitted workload. A failed policy/readiness request is not evidence of zero usage or unlimited budget, and it must not trigger repeated provider attempts in the background.

Sources. Actual policy and usage DTOs, legacy-field refusal.

Example UIAI-AI_GATEWAY-04 — Diagnose an unavailable provider safely

Scenario and inputs. A synthetic leave-summary task has a known chosen platform model but its request fails. The operator may inspect gateway/model information and safe audit identifiers. They must not print prompts containing private records or expose credential material while diagnosing the problem.

Procedure. Check the actual model and provider connection state, then the model catalog/report and runtime-plan metadata. Distinguish a missing model from a provider that was selected but unreachable, and a transport deadline from output-contract validation. Use the governed error and safe attempt/audit identifier returned by the actual task surface; do not invent a universal receipt field or treat every failure as the same retryable condition.

Expected result. The operator can identify which preparation or execution boundary failed. The failed response is not a successful empty model answer. Redaction, workspace accounting and native disclosure policy remain part of the request path even when a provider is unavailable or an operator reads its settings.

Failure and recovery. Repeated attempts may spend provider capacity or duplicate an uncertain effect. Correct the specific admitted endpoint/model/credential/deadline issue, then retry only through the original surface when authorized. A UI Stop ends the browser wait but does not prove the provider cancelled its request; preserve that uncertainty and its native usage/audit evidence.

Sources. Model catalog and runtime plan APIs, transport deadline tests, unreachable provider test.

The AI Gateway is the platform abstraction for cloud AI providers and on-prem model runtimes.

flowchart TD
    A["Workflow AI Task"] --> B["AI Gateway"]
    S["BPMN service.ai.* task"] --> T["AiGatewayServiceTaskExecutor"]
    T --> B
    C["Fabric"] --> B
    D["AI Agent Task"] --> B
    B --> E["Provider Registry"]
    E --> F["OpenAI / Anthropic / Gemini / Azure / Bedrock"]
    E --> G["Mistral / Cohere / Hugging Face / Others"]
    E --> H["Ollama / vLLM / TGI / llama.cpp / ONNX"]
    B --> I["Routing Policy"]

AI comes only from the AI provider connections an administrator creates in Authorization -> AI providers (database table platform_ai_provider_connections). Each enabled connection with a chosen model ("Use this model") contributes one model, <connectionKey>-chat: the enabled model. There are no built-in providers, models or load profiles. The former XML catalog (conf/ai-providers/platform-ai-providers.xml) and its loader were removed by SCRUM-4817 (2026-11-05).

Authenticated API

The AI Gateway catalog APIs are authenticated and read-only. They list connection data only: the connections and their enabled models (runtime provider connections are administered separately — see AI Provider Connections):

GET /api/ai-gateway/providers
GET /api/ai-gateway/providers/{providerId}
GET /api/ai-gateway/models
GET /api/ai-gateway/models/search?q=local
GET /api/ai-gateway/models/{modelId}
GET /api/ai-gateway/model-load-profiles
GET /api/ai-gateway/models/{modelId}/load-profile
GET /api/ai-gateway/runtime-plans
GET /api/ai-gateway/models/{modelId}/runtime-plan

Required permission:

AI_GATEWAY_VIEW

The development designer and worker roles receive this permission. root has *.

These APIs do not execute AI calls. AI execution still flows through AiGatewayService, BPMN service.ai.* tasks, and future agent/runtime workers. The API is for configuration visibility, runtime planning, admin consoles, and UI model selection.

Core Concepts

  • AiProviderDefinition: cloud provider or local runtime endpoint, built from an administrator connection.
  • AiModelDefinition: model id, provider model name, runtime type, capabilities, context window, and data classification.
  • AiModelLoadProfile: how a local/on-prem model is loaded, warmed, scaled, and health-checked.
  • AiCapability: chat, structured output, tools, embeddings, image, audio, code, and agentic reasoning.
  • AiGatewayRequest: provider-agnostic AI request with purpose, input, constraints, and actor.
  • AiRoutingPolicyService: chooses a provider/model based on preferred model or required capabilities.
  • AiModelRuntimeService: creates runtime plans from model definitions and load profiles.

BPMN Service Task Usage

AI service tasks in conf/service-tasks/platform-service-tasks.xml use the ai-gateway implementation. At runtime, AiGatewayServiceTaskExecutor maps the BPMN activity configuration and workflow variables into an AiGatewayRequest.

Supported seed task types include:

  • service.ai.extract
  • service.ai.summarize
  • service.ai.classify
  • service.ai.translate
  • service.ai.generate.email
  • service.ai.agent.run

Example BPMN activity configuration:

{
  "activityType": "service.ai.summarize",
  "modelId": "my-openai-chat",
  "prompt": "Summarize this workflow input for an operations user.",
  "input": {
    "content": "Customer requested renewal pricing."
  },
  "constraints": {
    "maxOutputTokens": 250
  }
}

The task completes with workflow variables such as aiText, aiStructuredOutput, modelId, providerId, usage, and auditId. Centralized routing, redaction, token-usage recording, and audit behavior remain in the AI Gateway rather than inside individual BPMN tasks.

First Routing Behavior

The default routing policy:

  1. uses preferredModelId when provided — but still validates the pinned model against all requiredCapabilities (AI_MODEL_CAPABILITY_NOT_FOUND on mismatch, before any provider call). While another model is enabled, a pin that names a model that is not an enabled connection model fails with AI_MODEL_NOT_FOUND (the plain "choose a model" outcome); with no enabled model it fails with AI_PROVIDER_NOT_CONNECTED;
  2. otherwise finds enabled models that satisfy required capabilities;
  3. prefers on-prem models when multiple candidates match;
  4. requires on-prem models for dataClass=restricted;
  5. prefers on-prem models for dataClass=confidential.

Routing only ever picks enabled models. With no enabled model at all, AI calls fail with AI_PROVIDER_NOT_CONNECTED ("No AI provider is connected..."). When enabled models exist but none fits the request, the call fails with AI_MODEL_CAPABILITY_NOT_FOUND. Restricted data still requires an on-premise enabled model.

If a restricted request explicitly asks for a cloud model, the router rejects it with AI_MODEL_DATA_RESIDENCY_VIOLATION.

Requests carrying binary attachments (AI task IMAGE inputs) are additionally restricted to on-premise models: binary content cannot be scanned by the text redaction policy, so automatic routing filters to on-premise candidates and any external resolution — pinned or otherwise — is refused before provider invocation with AI_TASK_ATTACHMENT_EXTERNAL_BLOCKED, enforced both in the routing policy and fail-closed in the gateway itself.

Removed: XML provider format and extension pack

The XML provider format (conf/ai-providers/*.xml) and the AI gateway extension-pack API (POST /api/ai-gateway/extension-pack/preview, and its MCP declaration) were removed by SCRUM-4817 (2026-11-05). Add providers and models as connections in Authorization -> AI providers. The other extension-pack APIs (connectors, service tasks, AI fabric) are unchanged.

AI Provider Connections (Database-Backed Administration)

Provider connections are managed at runtime from the Studio (AI Providers screen) or the admin API, without restarting:

GET    /api/admin/ai-providers
GET    /api/admin/ai-providers/{id}
POST   /api/admin/ai-providers
PUT    /api/admin/ai-providers/{id}
DELETE /api/admin/ai-providers/{id}
POST   /api/admin/ai-providers/{id}/enable
POST   /api/admin/ai-providers/{id}/disable
POST   /api/admin/ai-providers/{id}/test-connection
GET    /api/admin/ai-providers/{id}/models
POST   /api/admin/ai-providers/{id}/selected-model

Required permissions:

reads (list/get):            AI_GATEWAY_VIEW
writes, test, discovery:     AI_PROVIDER_MANAGE

Connections persist in platform_ai_provider_connections (migration 000116). The table stores an opaque credentialReference only — never secret material. When a create/update carries secretMaterial, the service stores it in the credential vault (purpose AI_PROVIDER_RUNTIME) and keeps the returned vault:<id> reference; an update with new material always stores a brand-new vault entry and repoints the connection (never rotates in place), because vault: references can be shared across connections. Secret material is never returned by any API, written to logs, or included in audit metadata (create audits record the new field values, update audits record old/new field maps, and delete audits record the full old field map, each with a credentialStored marker where applicable).

Routing semantics (no restart, read-through on every lookup so active-active nodes never serve stale data):

  • only enabled connections take part; a disabled connection contributes nothing;
  • an enabled connection with a chosen model (policy.selectedModelId, set with "Use this model") contributes the model {connectionKey}-chat (provider id = the connection key, vendor name = the picked id). Ask Inflow (studio-agent) and the assist features route to it. A connection without a chosen model contributes no model;
  • discovery (below) is an administrative pick-list aid;
  • blank/clear removes the model. The choice lives in the existing JSON policy column.

Credential reference resolution is fail-closed: env: and secret: schemes resolve from the environment, vault: resolves through the credential vault, and any other reference resolves to empty (it is never used verbatim as the credential).

Upgrade note (2026-07-31): EnvironmentAiCredentialResolver is now fail-closed — only env: and secret: references resolve (vault: resolves upstream through the vault). Before this change, an unrecognized reference (for example a typo, an oauth: form, or a literal secret) was silently used verbatim as the credential. Any pre-existing verbatim or foreign-scheme credential references — on AI providers, AI provider connections, connectors, or connected systems — must be migrated to a supported scheme before upgrade, or resolution returns empty and calls fail closed.

POST /{id}/test-connection performs a real minimal call through the resolved provider client and always returns HTTP 200 with:

{ "ok": true, "latencyMs": 132, "providerType": "OLLAMA", "modelCount": 7 }

Failures return ok: false with an explicit errorCode (AI_PROVIDER_PROBE_UNSUPPORTED, AI_PROVIDER_CREDENTIAL_RESOLUTION_FAILED, AI_PROVIDER_REQUEST_FAILED, AI_PROVIDER_UNREACHABLE, or AI_PROVIDER_PROBE_FAILED). AI_PROVIDER_UNREACHABLE (spec 4798) means the provider gave no answer at all (unknown host, refused connection, I/O error or the attempt deadline). Its message is the plain "<Provider> could not be reached."; the address goes to the server log only. Model discovery (GET /{id}/models) answers supported: false with the same errorCode for such a failure, so Studio says "The provider could not be reached" and keeps "not supported" for providers without a listing API. The probe is a models-listing call per client: OpenAI-compatible GET /models (OpenAI, Groq, Together, DeepSeek, OpenRouter, Perplexity, vLLM, Mistral), Anthropic GET /v1/models, Gemini GET /v1beta/models, Ollama GET /api/tags, Bedrock GET https://bedrock.{region}.amazonaws.com/foundation-models with a Bedrock API key as Authorization: Bearer (IAM SigV4 is not implemented; access-key-only connections report listing unsupported). Region comes from policy.region or the runtime endpoint host (us-east-2 from https://bedrock-runtime.us-east-2.amazonaws.com). Provider types without a listing adapter report AI_PROVIDER_PROBE_UNSUPPORTED (Azure OpenAI unless openaiCompatible is set, Cohere, Hugging Face, TGI, llama.cpp, ONNX, CUSTOM).

GET /{id}/models runs the same listing call and returns

{ "supported": true, "modelIds": ["x-ai/grok-4.6"],
  "models": [{ "id": "x-ai/grok-4.6", "label": "xAI: Grok 4.6", "kind": "CHAT" }] }

for the Studio model pick list. kind is CHAT, EMBEDDING, BATCH, or OTHER. Unsupported providers return supported: false so the UI falls back to typing a vendor id. Responses never contain credential material.

POST /{id}/selected-model with { "modelId": "..." } writes policy.selectedModelId (max 256 characters; blank clears) and pins the studio-agent AI default to {connectionKey}-chat. It requires AI_PROVIDER_MANAGE, audits AI_PROVIDER_CONNECTION_UPDATED with the previous and new vendor ids only, and never accepts or returns secret material.

Deleting a connection removes the row only; the vault entry is deliberately kept because vault: references can be shared (revoke is an explicit vault-administration operation).

Model Load Profiles

Model load profiles are optional runtime instructions for local and on-prem models. They are separate from AiModelDefinition so the stable platform model id can stay the same while operations changes deployment details such as the container image, artifact URI, device, replica counts, warmup prompt, or health check.

Supported load modes:

  • STARTUP: load and warm the model when the runtime starts.
  • LAZY: load the model the first time it is needed.
  • ON_DEMAND: load for a specific request or job and allow scale-to-zero.
  • EXTERNAL_MANAGED: the platform calls a provider endpoint but does not manage the model lifecycle.
  • DISABLED: keep the model listed but do not load it.

There are no shipped load profiles. Load profiles and runtime plans exist only for on-premise connection models (see below). AiModelRuntimeService exposes the Java runtime plan; a connection model without an explicit profile defaults to LAZY.

An on-premise AI provider connection created in AI providers (Ollama, vLLM, TGI, llama.cpp or ONNX runtime on a private endpoint) has a load profile for its <connectionKey>-chat model while it is enabled and has a selected model (spec 4319). A LAZY profile is derived from the connection: the provider's runtime type, the health check /api/tags (Ollama) or /health, a 180-second warmup, deployment target ai-provider-connection:<connectionKey> and the selected model as the artifact. An on-premise connection without a selected model offers no model; the catalog report warns "on-premise connection has no selected model".

A connection counts as on-premise only when its type is self-hosted and its endpoint is on a private network. IP literals are checked as addresses, and an all-digits or hex host such as 134744072 or 0x8080808 (which resolvers read as 8.8.8.8) is never private. A private-looking name (single label, .local, .internal, .lan, .localhost) must resolve only to loopback, RFC 1918, link-local or IPv6 unique-local addresses; otherwise the model is treated as a cloud model, restricted data and attachments are not routed to it, and the model catalog says why. The check runs on every registry lookup, so routing uses an answer at most one cache period old. Each lookup or capture resolves a distinct endpoint once; concurrent callers share one lookup per host; finished answers are cached (30 s resolved, 10 s not) and a caller stops waiting after 1.5 s, which counts as not private. Several distinct uncached slow endpoints add up to 1.5 s each. A private reverse proxy that forwards to a cloud LLM still counts as on-premise: by configuring the endpoint the administrator attests to where it sends data.

An on-premise connection model is READY only after a live runtime check. For Ollama the catalog probes /api/tags (host reachable, model pulled). Other runtimes stay NEEDS_ATTENTION with "on-prem runtime reachability is not verified", because a connection does not record its Test result.

Model Catalog Operations (connection data only)

The model catalog is built from the administrator's connections only; the endpoints below (/model-catalog, /models, /providers, /model-load-profiles, /runtime-plans) keep their response shapes. Raw provider, model, and runtime-plan APIs are useful for configuration. For operators and AI workflow designers, the composed model catalog is usually the better starting point:

GET /api/ai-gateway/model-catalog
X-Actor-Id: worker
GET /api/ai-gateway/model-catalog/my-ollama-chat
X-Actor-Id: worker
GET /api/ai-gateway/model-catalog/report
X-Actor-Id: worker

AiModelCatalogService joins:

  • AiProviderDefinition
  • AiModelDefinition
  • configured or generated AiModelRuntimePlan
  • explicit AiModelLoadProfile state

Each catalog entry includes provider type, endpoint, credential reference, provider model name, runtime type, load mode, capabilities, context window, data classification, local runtime deployment details, replica counts, device, restricted-data allowance, operational status, readiness warnings, provider policy, and runtime metadata.

The report endpoint summarizes:

  • provider count and model count
  • cloud versus on-prem model count
  • configured load-profile count
  • enabled versus disabled models
  • counts by provider type, runtime type, and capability
  • on-prem models missing explicit load profiles
  • models allowed for restricted data
  • readiness warnings

This gives administrators one quick view of whether cloud providers and on-prem model loaders are ready before routing BPMN service tasks or agents to them. These endpoints require AI_GATEWAY_VIEW.

For Ollama-backed models, readiness is not computed from static config alone: the catalog probes the provider endpoint's GET /api/tags (1s connect / 2s request timeouts, memoized per endpoint for 10s). A model whose providerModelName is absent from the host's installed list reports NEEDS_ATTENTION with model not pulled on Ollama host (run: ollama pull <name>), and an unreachable Ollama host also reports NEEDS_ATTENTION — a catalog row never claims READY while its runtime is down. The local Compose stack no longer includes Ollama (ollama-dev and ollama-pull were removed). A local Ollama is optional: run one yourself and connect it in AI providers like any other provider, then pull the model you choose.

No built-in catalog

There is no seed provider catalog and no startup catalog import. Every provider, model and load profile comes from an administrator connection, so a fresh system has no AI until a connection is created, enabled and given a model. Assist features (data-model, expression and rule assist, process-design assist) and the Studio Agent have no configured model: they use the purpose default when it names an existing model, otherwise the enabled model. Without one, assist falls back to basic rules with a plain reason and Ask Inflow is hidden. The settings platform.assist.model, platform.assist.provider, platform.agent.model, platform.agent.provider and platform.process.assist.model were removed.

Provider Clients

Provider clients send provider-agnostic AiGatewayRequest objects to a concrete provider API and normalize the response into AiGatewayResponse.

Attempt deadline

Every provider call (OpenAI-compatible including OpenRouter, Anthropic, Gemini, Ollama, Bedrock) runs on the shared platform HTTP factory. That factory bounds connect and the wait for response headers (platform.http.*, 30 s each; see the connectors manual). On top of it, an AI provider call has a whole-attempt deadline:

  • Default 60 seconds. Set INFLOW_AI_ATTEMPT_TIMEOUT_SECONDS (whole seconds, 1–3600) in the API environment to change it. Any other value, for example 0, a negative number or a non-integer, stops the API from starting, so a typo can never mean "no limit".
  • Once response headers have arrived, the deadline stops the call for real: the thread reading the answer is woken and the connection is dropped. Before headers arrive, only the shared connect and header bounds apply, so with a deadline below 60 s a provider that withholds headers can hold the call for up to 30 s connect plus 30 s header wait before it fails. The effective bound is therefore the larger of the deadline and those shared bounds. Either way the task fails with the usual provider failure (AI_PROVIDER_REQUEST_FAILED). When the deadline itself ends the call, Technical details name it ("AI provider attempt exceeded 60 s."); when a shared connect or header bound ends it first, they show that timeout instead.
  • Some gateways, OpenRouter among them, send response headers early and keep a slow answer alive while the model generates. Such answers were previously unbounded; they now end at the deadline. Raise the deadline if long reasoning or long outputs are expected. Each call holds one request thread for up to the deadline, so very high values (up to an hour) have a matching resource cost.
  • The deadline covers one provider call. It is not a deadline for the whole AI task, and it does not cancel work the provider is still doing; stopping a test run in Studio only stops the browser request.

OpenAI-Compatible Client

OpenAiCompatibleProviderClient sends requests to /chat/completions endpoints and parses text, JSON structured output, usage, and provider response id.

It supports:

  • OpenAI
  • Groq
  • Together AI
  • DeepSeek
  • OpenRouter
  • Perplexity
  • vLLM
  • any provider with policy "openaiCompatible": true

The client maps:

  • AiModelDefinition.providerModelName to the provider model
  • AiGatewayRequest.prompt, purpose, input, and constraints into a user message
  • STRUCTURED_OUTPUT or responseFormat=json into response_format: {"type":"json_object"}
  • temperature, maxTokens, and topP constraints into compatible request fields

Anthropic Client

AnthropicProviderClient supports providers with type ANTHROPIC.

It maps:

  • endpoint to /v1/messages
  • credentials to the x-api-key header
  • provider policy anthropicVersion to the anthropic-version header, defaulting to 2023-06-01
  • AiModelDefinition.providerModelName to model
  • maxTokens to max_tokens, defaulting to 1024
  • temperature and topP into compatible request fields
  • response content[].text, usage, and id into AiGatewayResponse

For structured output, the client appends a strict JSON instruction to the normalized prompt and parses JSON text when the provider returns it.

Gemini Client

GeminiProviderClient supports providers with type GOOGLE_GEMINI.

It maps:

  • endpoint to /v1beta/models/{providerModelName}:generateContent
  • credentials to the Gemini API key query parameter
  • prompt content into contents[].parts[].text
  • temperature, maxTokens, and topP into generationConfig
  • STRUCTURED_OUTPUT or responseFormat=json into responseMimeType=application/json
  • response candidate text, usageMetadata, and responseId into AiGatewayResponse

Ollama Client

OllamaProviderClient supports providers with type OLLAMA and sends native non-streaming chat requests to:

/api/chat

It maps:

  • endpoint to /api/chat
  • AiModelDefinition.providerModelName to Ollama model
  • normalized prompt content into messages[0].content
  • STRUCTURED_OUTPUT or responseFormat=json into format=json
  • temperature, topP, and maxTokens into Ollama options
  • response message.content into AiGatewayResponse.text
  • prompt_eval_count and eval_count into normalized usage keys:
    • prompt_tokens
    • completion_tokens
    • total_tokens

This allows an Ollama connection to run through the AI Gateway without requiring an OpenAI-compatible adapter.

DefaultAiGatewayService performs the full execution path:

sequenceDiagram
    participant Caller
    participant Gateway as "DefaultAiGatewayService"
    participant Router as "AiRoutingPolicyService"
    participant Registry as "AiProviderRegistryService"
    participant Client as "AiProviderClient"
    participant Provider as "AI Provider"

    Caller->>Gateway: execute(request)
    Gateway->>Router: chooseModel(request)
    Router-->>Gateway: providerId/modelId
    Gateway->>Registry: load provider/model
    Gateway->>Client: execute(provider, model, request)
    Client->>Provider: POST provider-native endpoint
    Provider-->>Client: provider response
    Client-->>Gateway: AiGatewayResponse
    Gateway-->>Caller: AiGatewayResponse

AI Call Audit And Metering

Every DefaultAiGatewayService.execute call records an audit event when an AuditService is configured.

Successful calls use:

eventType: AI_GATEWAY_CALL_SUCCEEDED
entityType: AI_GATEWAY_CALL
entityId: generated call id

Failed calls use:

eventType: AI_GATEWAY_CALL_FAILED
entityType: AI_GATEWAY_CALL
entityId: generated call id

The audit metadata includes:

  • generated callId
  • status
  • request purpose
  • preferredModelId
  • required capabilities
  • constraints, including values such as dataClass
  • routing reason
  • provider id and provider type
  • model id and provider model name
  • on-premise flag
  • duration in milliseconds
  • provider response id when available
  • usage metadata when available
  • tokenUsage with inputTokens, outputTokens, cachedInputTokens and totalTokens when usage is available
  • redaction metadata when the AI redaction policy is configured
  • error type, message, and platform error code for failures

This gives the platform the first durable foundation for:

  • per-user AI usage history
  • provider/model usage dashboards
  • token metering
  • compliance review for restricted/confidential data
  • incident review when provider calls fail

Pricing and cost estimation (removed)

Pricing and cost estimation were removed by SCRUM-4817 (2026-11-05): conf/ai-providers/platform-ai-pricing.xml, the cost estimator and every cost field are gone. Audit metadata records tokenUsage (input, output, cached input and total tokens) and Studio shows token counts, not cost.

Workspace token limit

Workspace AI budgets are token based. The Workspace AI policy has monthlyTokenLimit (integer; empty = no limit) instead of monthlyBudgetUsd. A PUT /api/workspaces/{idOrKey}/ai-policy that still sends monthlyBudgetUsd is refused with 400 AI_POLICY_FIELD_REMOVED ("monthlyBudgetUsd was removed; set monthlyTokenLimit instead.") and nothing is saved. Before each call and after recording its usage, the gateway compares month-to-date total tokens with the limit; when it is reached the call fails with AI_WORKSPACE_BUDGET_EXCEEDED ("monthly AI token limit"). Studio shows "Monthly token limit". GET /api/workspaces/{idOrKey}/ai-usage returns monthToDateTokens and per-call totalTokens. Migration 000416 adds workspace_ai_policies.monthly_token_limit and workspace_ai_usage.total_tokens. The old monthly_budget_usd and estimated_cost_usd columns stay in the database unused (forward-only migrations) and existing USD budgets are not converted: the token limit starts empty.

Operator rollout runbook (SCRUM-4817)

Existing data is not migrated. Saved ids of removed catalog models no longer resolve, so prepare before deploying:

  1. Create or enable an AI provider connection, choose a model ("Use this model") and pass Test connection. Restricted-data AI Tasks need an on-premise connection.
  2. Re-pick models in AI Tasks and process steps that named removed models. A pin to a removed model fails with the plain "choose a model" outcome; a missing backup model is reported at publish ("This backup model is no longer available. Choose a connected AI model.").
  3. Fix workspace allow-lists that name removed ids. Allow-lists stay fail-closed and are never widened automatically.
  4. Purpose defaults that name a missing model (including the old studio-agent seed) are never re-routed: while no AI provider is connected they give "No AI provider is connected"; once one is connected they fail with the plain "choose a model" outcome. Re-pick or clear them in AI defaults. (The Studio Agent itself uses the enabled model it finds.) A default that cannot be read refuses the call (AI_MODEL_DEFAULTS_UNAVAILABLE).
  5. USD budgets are lifted. Workspaces that had a monthly USD budget have no limit until a monthly token limit is set: the old column is no longer read. Set token limits for them (query below). A call whose provider reports no token usage counts as 0 tokens.
  6. If you used the compose ollama-dev, run your own Ollama and connect it.

Read-only checks (PostgreSQL):

-- Workspaces with allow-lists that may name removed ids
SELECT workspace_id, allowed_models, allowed_providers
FROM workspace_ai_policies
WHERE COALESCE(allowed_models, '') <> '' OR COALESCE(allowed_providers, '') <> '';

-- Workspaces whose USD budget is no longer enforced (no token limit set yet)
SELECT workspace_id, monthly_budget_usd
FROM workspace_ai_policies
WHERE monthly_budget_usd IS NOT NULL AND monthly_token_limit IS NULL;

-- Purpose defaults that do not name a connection model
SELECT purpose, model_id, agent_definition_id
FROM platform_ai_defaults
WHERE model_id NOT LIKE '%-chat';

Prompt Redaction Policy

AI request redaction is configured outside Java code under:

conf/ai-providers/platform-ai-redaction.xml

The gateway applies redaction after routing and before provider execution, so all provider clients receive the sanitized request. If a blocking rule matches, the gateway stops before provider execution and records a failed audit event with error code AI_REQUEST_BLOCKED_BY_REDACTION_POLICY.

<aiRedactionPolicy version="1" enabled="true">
  <rule id="email-address">
    <description>Redact email addresses before AI provider submission.</description>
    <pattern>[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}</pattern>
    <replacement>[REDACTED:email]</replacement>
    <action>REDACT</action>
    <enabled>true</enabled>
  </rule>
</aiRedactionPolicy>

Supported actions:

  • REDACT: replace matching text and continue execution.
  • BLOCK: stop the AI call before provider execution.

The default policy seeds rules for:

  • email addresses
  • bearer/API token shaped values
  • credit-card-like number sequences
  • private key material, which is blocked

Redaction is recursive across:

  • prompt text
  • input map values
  • constraint map values
  • nested maps and lists

Audit metadata records only safe redaction information:

  • whether redaction was enabled
  • whether the request was blocked
  • replacement count
  • matched rule ids
  • blocked rule ids

It does not record raw matched values.

Provider coverage is defined by the current client and connection sections above. Bedrock already has an API-key client; IAM SigV4 remains unsupported. An OpenAI-compatible endpoint does not establish a distinct qualified native client or model-loader lifecycle for every server product. Provider-specific admission, residency and production certification still need the deployment's own evidence.

Delivered (2026-07-31): database-backed provider connections with Studio and admin-API management, vault-stored credentials, test-connection probes, model discovery, and the read-through routing overlay — see AI Provider Connections.

Current-source provider instructions and reasoning

Purpose-scoped model defaults

The selected-model operation keeps its legacy behavior when no purpose is supplied: it sets the shared studio-agent default. New callers may explicitly target studio-agent, app-chat, or home-app-chat; an explicit purpose changes only that purpose. Home provisioning should target home-app-chat so it leaves Ask Inflow and published App Shell Chat defaults unchanged. Explicit purpose selection requires an enabled connection, a nonblank model, and both provider-management and AI-default-management capabilities. The server refuses a selection that would redirect another purpose already using the same connection-backed model.

This addendum records behavior added after the preserved reference snapshot above.

OpenAI-compatible provider requests receive systemInstructions as a separate system message, not repeated inside the user-message JSON. If a compatible endpoint has no system role, the same instruction text is prepended to the prompt under an Instructions: heading.

Provider connections may set policy.reasoningEffort to off, low, medium or high; unsupported values are rejected. When a connection omits the setting, App Chat requests medium reasoning effort and other purposes send no reasoning parameter. OpenRouter receives reasoning.effort; OpenAI and Azure OpenAI reasoning models receive reasoning_effort. Reasoning requests omit temperature and top_p. The provider adapter determines whether its model qualifies for reasoning by model name or the connection's policy.reasoningModel declaration. Configure the policy on the connection; do not assume a Studio picker or a model name alone enables provider-side reasoning.

For App Chat's record/task tool catalog, source and field checks, per-turn limits, and fallback path, read the App Chat tool-calling technical reference. These notes describe source behavior and configuration; they are not a provider execution or deployment verification.

Try “document generation”, “integrations” or “first form”.