AI Integrations

AI integration is a system design problem, not a model call.

A practical map for LLM applications, data, retrieval, agents, tools, evaluation, security, human review, observability, cost, latency and fallback.

Definition

An AI integration is the architectural connection between an application, its data, business workflows and one or more AI capabilities, with explicit controls for context, quality, security, cost and human oversight. AI is not a complete architecture by itself. A model API does not solve domain boundaries, data access, workflow design or operational risk, and not every product problem requires probabilistic behavior.

Why AI integration architecture matters

AI features become useful when they are connected to real workflows, trusted data, clear permissions, measurable quality and fallback paths. Without architecture, a convincing demo can become an expensive, unpredictable or unsafe production feature.

Problems AI integration architecture helps clarify

A workflow needs deterministic automation, but a model is added because it feels modern.
Prompts are scattered through the codebase without ownership, versioning or tests.
Model output is accepted as correct because it has a JSON shape.
Retrieval is added before source quality, permissions and freshness are understood.
Tools are exposed to a model without clear authorization, idempotency or audit boundaries.
Evaluation depends on demos instead of task-specific criteria and production feedback.
Costs and latency are invisible until usage increases.
There is no fallback when the model, retrieval or provider fails.

When AI creates value

AI can help when a workflow involves natural language, ambiguous classification, extraction, summarization, assisted generation, knowledge retrieval or decision support where variability is acceptable and review is possible. Rules, search and deterministic automation should still be considered first when they solve the problem clearly.

Use AI where interpretation or generation creates value.
Keep simple rules when the desired result is exact.
Define quality criteria before production rollout.

When not to use AI

Avoid AI when a simple rule works, the output must be exact, quality cannot be evaluated, sensitive data cannot be handled safely, latency is incompatible, cost exceeds value, the workflow is unclear or there is no fallback. This is not anti-AI; it is architectural judgment.

Do not automate a broken workflow with a model.
Do not use AI when deterministic validation is enough.
Do not ship high-risk behavior without review and recovery paths.

Deterministic automation vs AI

Deterministic automation uses rules, conditions, validations, integrations and predictable workflow steps. AI adds interpretation, generation, semantic retrieval or probabilistic classification. Strong systems often combine both: rules control the workflow, AI interprets content, validation checks the result and people review risky cases.

Keep workflow orchestration explicit.
Use AI for uncertain content, not every step.
Treat model output as an input to validate.

System boundaries

AI capabilities should sit behind clear application boundaries: domain workflows, an AI gateway, provider adapters, prompt management, retrieval, tools, validation, observability and fallback. Business rules should not disappear into scattered prompts.

Separate domain logic from model behavior.
Centralize provider access and prompt ownership where practical.
Keep permission and fallback boundaries visible.

Models and provider strategy

Model decisions include capability, context window, latency, cost, structured output support, tool support, availability, data policy and version changes. Abstraction can help, but a generic layer can also hide important model capabilities. Portability is a trade-off, not a guarantee.

Choose models by task and constraints.
Track provider dependency deliberately.
Avoid provider-neutral abstractions that hide real requirements.

Prompts and context

Prompts include system instructions, user input, application context, retrieved context, history, constraints, examples and output format. They should be versioned, reviewed, tested and observed as part of the system, not treated as magic text.

Version prompts and model settings.
Keep token limits and context construction explicit.
Do not publish internal prompts as documentation.

Structured outputs

Typed JSON, schemas, parsers, retries and validators help connect model output to software systems, but formatted output is not automatically correct. Semantic validation and domain mapping are still required.

Validate structure and meaning.
Handle invalid, partial and inconsistent responses.
Map model output into domain concepts deliberately.

Tools and function execution

Tool calling requires definitions, permissions, input validation, idempotency, authorization, side-effect control, confirmations, audit logs and transaction boundaries. A model should not receive unrestricted action access.

Distinguish suggesting, preparing and executing an action.
Use minimum required permissions.
Require confirmation for critical side effects.

Retrieval and knowledge access

Retrieval connects documents, chunking, indexing, metadata, permissions, search, ranking, freshness and context construction. It can improve grounding, but it does not fix poor source material or replace data architecture.

Respect document permissions and freshness.
Cite or expose sources when the workflow needs traceability.
Do not add RAG when a small curated context is enough.

Agents and orchestration

An agent combines a model with instructions, context or memory, tools and a decision loop. Agents can help open-ended tasks, but explicit workflows are usually more predictable. Agents need boundaries, observability, permissions and stop conditions.

Prefer explicit orchestration for linear workflows.
Use agents only when the task needs adaptive steps.
Avoid loops, unlimited tools and vague goals.

Data architecture

AI quality depends on source data, ownership, access, permissions, freshness, lineage, retention, sensitive data handling, anonymization, context, feedback and evaluation datasets. Connecting documents is not the same as designing data flow.

Separate retrieval from training concepts.
Protect sensitive and tenant context.
Use feedback data carefully and intentionally.

Security and privacy

Inputs and outputs should be treated as untrusted. Architecture must consider data minimization, prompt injection, output injection, access control, tool permissions, secrets, logging, retention, isolation, provider policies and exfiltration risk.

Do not log sensitive data indiscriminately.
Treat model output as untrusted until validated.
Do not imply automatic compliance or zero risk.

Human review and fallback

Human review should match risk. Some outputs can be accepted automatically after validation; others need queues, approvals, escalation, manual workflow, user correction or degraded mode. Review is a control, not a universal solution.

Classify risk before choosing review depth.
Design deterministic fallback paths.
Let users recover when AI assistance fails.

Evaluation and quality

Evaluation needs task-specific criteria, test datasets, expected outputs, correctness, completeness, relevance, safety, latency, cost, human judgment, automated checks and production feedback. A good demo is not a reliability signal.

Evaluate the task, not the model brand.
Run regression checks when prompts or models change.
Use multiple quality signals instead of one universal score.

Observability

AI workflows need tracing across request, model, prompt version, latency, token usage, cost, tool calls, retrieval sources, errors, refusals, retries, validation and user feedback. Observability should connect the AI step to the whole workflow.

Correlate AI events with product workflows.
Track cost and latency per use case.
Avoid storing sensitive prompts or outputs unnecessarily.

Cost and latency

Cost and latency are shaped by input tokens, output tokens, context size, retrieval, multiple calls, retries, model selection, caching, batching and user expectations. Larger models are not automatically better for operational value.

Budget by usage pattern, not static provider prices.
Keep context as small as the task allows.
Choose model size in proportion to risk and value.

Reliability and failure modes

AI integrations can fail through outages, timeouts, rate limits, invalid output, hallucination, tool failure, retrieval failure, stale context, partial completion, loops, prompt injection or cost spikes. Design limited retries, timeouts, fallback, validation and manual recovery.

Plan for provider and retrieval failure.
Use bounded retries and circuit-breaking concepts.
Communicate degraded behavior clearly.

Governance and change management

Governance defines ownership, approved use cases, data policies, provider review, model updates, prompt changes, evaluation, access, incident review, documentation, rollout and deprecation. Good governance enables adoption with traceability.

Document who owns prompts, tools and evaluation.
Review changes before production impact.
Keep governance proportional to risk.

When not to overengineer

A useful first AI integration may be a controlled call, small context, structured output, validation, human review and basic metrics. Avoid agents for linear workflows, vector databases for tiny static context, multi-model pipelines without criteria and universal provider abstractions before a use case is real.

Start with the smallest safe architecture.
Add retrieval, agents or memory only when the task demands it.
Do not confuse prototype speed with production readiness.

Principles

Use AI where variability creates value

AI should help where interpretation, generation or ambiguity matters.

Do not replace exact rules with probabilistic behavior.

Keep deterministic workflows explicit

Business rules, approvals and integrations should remain understandable outside the prompt.

Let models assist; do not let prompts become hidden workflow engines.

Treat model input and output as untrusted data

Inputs may be hostile and outputs may be wrong even when they look structured.

Validate, sanitize and authorize around the model.

Evaluate the task, not the demo

Quality must be measured against the work the product expects to perform.

Use examples, edge cases, human review and production feedback.

Design human review according to risk

Review should focus on consequential, uncertain or irreversible cases.

Do not review everything by default or nothing by default.

Give tools the minimum required permissions

Model-driven actions need narrow permissions and audited side effects.

Confirm critical actions before execution.

Make cost and latency observable

Operational value depends on knowing how each AI workflow behaves in production.

Track tokens, retries, model choices and response time.

Design fallback before rollout

AI failure should not leave the workflow without a safe next step.

Use manual, deterministic or degraded paths.

Key architectural decisions

Is AI necessary?

Question: does the task require interpretation or generation? Context: rules, search or automation may be simpler. Risk: adding cost and uncertainty without value.

Compare AI against deterministic alternatives.

Workflow shape

Question: should the model assist a step or lead the workflow? Trade-off: adaptability versus predictability. Risk: opaque behavior in business-critical flows.

Prefer explicit orchestration when possible.

Model and provider selection

Question: which capability fits the task constraints? Trade-off: quality, latency, cost, data policy and availability. Risk: dependency that cannot evolve.

Document provider assumptions and model versions.

Context construction

Question: what information should the model receive? Trade-off: relevance versus cost, latency and exposure. Risk: leaking data or overwhelming the task.

Keep context intentional and permission-aware.

Prompt ownership and versioning

Question: who owns instructions and changes? Trade-off: speed versus control. Risk: silent quality regressions.

Version prompts with evaluation criteria.

Structured output strategy

Question: how does output become application data? Trade-off: flexibility versus validation. Risk: formatted but incorrect responses.

Validate syntax and domain meaning.

Retrieval strategy

Question: what knowledge should be retrieved and how? Trade-off: freshness and relevance versus complexity. Risk: wrong or unauthorized context.

Use metadata, permissions and source quality controls.

Tool permission model

Question: what can the model cause the system to do? Trade-off: usefulness versus safety. Risk: unauthorized or irreversible side effects.

Limit tools, validate inputs and audit execution.

Human review threshold

Question: which outputs need people? Trade-off: speed versus risk. Risk: false trust or review overload.

Review by consequence and uncertainty.

Evaluation strategy

Question: how is quality known over time? Trade-off: manual judgment versus automated checks. Risk: demo success becoming production failure.

Combine tests, feedback and regression checks.

Fallback and cost limits

Question: what happens when AI fails or usage spikes? Trade-off: resilience versus complexity. Risk: degraded service, surprise spend or blocked users.

Set timeouts, budget controls and recovery paths.

Future insights being developed

These topics are planned under the AI Integrations hub. They are intentionally not public routes yet.

When not to use AI in a software product.
AI agents vs explicit workflow orchestration.
How to evaluate an LLM feature before production.
How to design fallback for an AI integration.
Prompt injection is an architecture problem.
How to control cost and latency in LLM applications.