Anatomy of a production AI agent in 2026

It is 2026, and the phrase "AI agent" has stopped meaning a single thing. In one room it means a chatbot with function calling. In the next, it means a multi-step orchestrator routing across seven systems and three departments. The difference between the two — and the difference between a successful production deployment and a slow-motion incident — is architectural.

This piece is the architecture we deploy at Mindlytic when a client tells us they want "an agent." It is opinionated, and it has been load-bearing for systems handling tens of millions of decisions a year. We will walk through the five layers, the boundaries between them, and where most teams accidentally cross those boundaries and pay for it later.

Five layers, in order

A production agent is not one model call. It is a pipeline with five distinct concerns, and each one needs to be observable independently. Conflate them — for example, putting your tool router inside your prompt — and you lose the ability to debug, evaluate, or rate-limit anything.

1. Intake. Normalizes the request: parses the user's input, attaches identity, applies policy, and returns a structured intent. Cheap. No model required for the routine cases.
2. Plan. Decomposes the intent into a sequence of steps. This is where the LLM earns its keep. Plans should be explicit and inspectable — never implicit in a function-calling loop.
3. Execute. Calls tools. We treat tools like microservices: typed, versioned, and tested independently of the agent. The agent does not know how a tool works; it only knows the contract.
4. Verify. Checks the result against the plan. Did the tool succeed? Is the answer well-formed? Did we cite sources we said we'd cite? This is where the kill switch lives.
5. Persist. Writes the run to memory: short-term for the conversation, long-term for the user, episodic for evaluation. Most teams forget this layer. Most teams cannot answer "what did the agent decide for customer 4928 last Tuesday?"

What "plan" actually means

There is a stylistic gap between two ways of using LLMs as planners. The first is the chatty agent: it thinks, it acts, it observes, and it loops, all in one stream. The second is the explicit planner: it emits a structured plan first, then a separate executor consumes it.

We default to the second. The chatty agent is impressive in demos and unrunnable in production. You cannot rate-limit it, you cannot resume it, you cannot run it through an evaluator that checks the plan independently of the execution. The explicit planner gives you a JSON object — steps, tools, expected types — that you can validate, log, and replay.

Tools are microservices, not prompt fragments

The most common architectural error we see: tools defined as docstrings inside the prompt, with the agent inferring how to use them. This is fine for prototypes. It collapses in production for three reasons. First, you cannot version the contract independently of the model. Second, you cannot test the tool without invoking the model. Third, you cannot revoke a tool — meaning you cannot deprecate, restrict, or A/B-test it — without redeploying the agent.

Treat tools as you would treat any backend service. Each tool has an OpenAPI-style contract, an owner, an SLA, and a permission scope. The agent gets a manifest of available tools at request time; the manifest is generated, not handwritten. When you remove a tool from the manifest, the agent stops using it within the latency of one request — not after a fine-tune.

Verification is not a vibe check

"Verify" is the layer that separates serious systems from unserious ones. The verification layer is a deterministic check, not an LLM judge in a trench coat. We use LLM judges where we have to (open-ended generation), but the moment a check can be deterministic — schema validation, citation presence, numeric bounds, allowlists — we make it deterministic and we test it like any other unit.

For an underwriting agent, verification means: does the decision cite at least two source rules? Are the cited rules currently active? Does the recommended decision match the policy table for the cited rules? If any check fails, the run halts and routes to a human, with the failed check named in the audit log.

The kill switch

Every agent must have a kill switch. The kill switch is not a flag in code; it is a feature in your control plane that an on-call engineer can flip in less than 60 seconds. When flipped, the agent (a) stops accepting new requests, (b) drains in-flight runs to a queue, and (c) routes new traffic to the fallback path — usually a deterministic system or a human queue.

You will use the kill switch. We have used it. The first time you use it, you will be glad you spent the week building it. The relevant industry guidance, including the NIST AI Risk Management Framework, formalizes this under "manage" — but you do not need a framework to know that anything autonomous needs a stop button.

Memory: short, long, episodic

Memory is the most common feature to underbuild. Three kinds, and you need all three:

• Short-term: the current conversation. Bounded, in-memory, scoped to one session.
• Long-term: what the agent has learned about this user — preferences, account state, past decisions. Stored in a database with proper retention policies. Not in the vector store.
• Episodic: a complete log of every run — input, plan, tools called, outputs, verification result. This is what you replay during evals and what regulators ask for. Treat it as immutable.

Latency budgets, not latency targets

Set a budget for the full pipeline (say, 4 seconds for a synchronous call) and decompose it across layers: 100ms intake, 800ms plan, 2.5s execute, 300ms verify, 300ms persist. Measure each layer separately. When you blow the budget, you'll know which layer is responsible — and the answer will not always be the model.

What we'd build differently in 2026

Three things have changed since we started shipping agents in earnest. First, models are good enough that the planning layer is rarely the bottleneck — the verification layer is. Second, the cost of a tool call is low enough that we now preflight tools (dry-run them with the agent's intended arguments) instead of guessing. Third, we run two agents in parallel for high-stakes decisions and fail to a human if they disagree. It costs more. It catches a lot.

None of this is novel. The novelty is the discipline of treating each layer as a separate engineering problem with its own SLA, owner, and tests. That is the architecture. The rest is detail.