LLM infrastructure is the set of services and operating controls that lets an application use large language models reliably. For an AI agent, the model endpoint is one dependency within a larger system: the agent also needs an execution environment, permitted information, tools, and a way to report outcomes.
A useful starting point is to trace one task from the user to a checked result. This reveals who operates each dependency and where a failure could prevent useful work.
The model endpoint is one dependency; the runtime, data, and tools complete the system.
Separate model serving from agent execution
Model serving runs the model and returns generated output. Agent execution runs the application logic that assembles context, calls the model, invokes permitted tools, and decides what to do next. These can run in different environments with different owners.
A gateway can provide a common point of access to configured model endpoints. It does not by itself define the agent's job or decide which business actions a person must approve. For the broader system, see AI agent architecture.
Map the six parts of an LLM infrastructure solution
A long list of available models tells you little about whether a supplier can run your work. Trace one complete task instead and ask, for each of the six parts below, who runs it and what happens when it fails.
- Model access: sends requests to an approved model. Usually run by the platform team and the model provider. Watch for: the model is unavailable, or the only alternative is not good enough for the task.
- Agent runtime: carries out the task step by step. Usually run by the application operator. Watch for: a task that stops halfway, leaving nobody sure whether it finished.
- Knowledge and tools: the information the agent may read and the actions it may take. Usually run by each source owner. Watch for: missing, outdated, or blocked information.
- Work in progress and results: where drafts and finished results are kept. Usually run by the application's data owner. Watch for: a half-finished draft that looks complete.
- Recovery rules: when to try again, switch to another model, or stop. Usually set by the workflow owner. Watch for: repeating an action, such as a message or a record update, that may already have happened.
- Usage and cost: what each task costs and which budget applies. Usually owned by the budget owner. Watch for: costs nobody can trace, or a request that no budget checks.
Ask the agent to draft the worksheet from the task description and the systems it will connect to. Your team then checks each answer with the person who actually runs that part. Keep the list of models separate from the agent's permissions: adding a model should never quietly give the agent access to new records or new actions.
Infrastructure requirements worksheet (blank)
- Model access: who runs it, what happens when it fails, how we will test it: ___
- Agent runtime: who runs it, what happens when it fails, how we will test it: ___
- Knowledge and tools: who owns each source, what happens when one is unavailable, how we will test it: ___
- Work in progress and results: where drafts are kept, how a half-finished draft is marked, how we will test it: ___
- Recovery rules: when to retry, when to switch, when to stop, and who is told: ___
- Usage and cost: who owns the budget, which budget applies to this task, how we will confirm it: ___
Example: a service request assistant
In this fictional pilot, a facilities team wants an assistant to classify incoming maintenance requests and prepare a suggested response. The model interprets free text; the application checks permitted categories and displays a draft. An operator approves any change to the maintenance record.
The team writes a small infrastructure brief:
- Input: Request text and the approved category guide; exclude unrelated personnel records.
- Execution: A named application owner maintains the runtime and its tool connection.
- Model choice: Test configured candidates against ambiguous requests and missing information.
- Output: A category recommendation, supporting evidence, and a draft response.
- Failure behavior: Show a pending review state when model access or the category guide is unavailable.
- Operating review: Inspect failed requests and spending alongside accepted recommendations.
Success means the operator can check the recommendation and understand any failure. A reply from the model on its own does not mean the task succeeded.
The completed worksheet
The facilities team's version reads like this. Each test is something the pilot has to pass before real requests go through.
Infrastructure requirements worksheet: facilities pilot (fictional)
- Model access: The platform team runs one approved model with no untested backup. If the model is unavailable, the request waits as “pending” and the facilities operator handles it by hand. Test: switch the model off in a test environment and confirm the request shows as pending.
- Agent runtime: The application operator runs it. The agent checks the request against approved categories and prepares a draft; it never changes the maintenance record. If a task stops halfway, it stays marked incomplete until the operator looks at it. Test: interrupt a task and confirm it cannot pass for finished.
- Knowledge and tools: The facilities team owns the category guide, and the agent can read only the approved version. If the guide is missing, out of date, or blocked, the agent stops and asks for manual handling. Test: try an old version, a missing guide, and blocked access.
- Work in progress and results: Drafts stay “pending” until the operator accepts them, and only the facilities team can see them. Test: interrupt a save, then sign in as a second user and check what they can see.
- Recovery rules: The agent retries once when a model call clearly fails. If it cannot tell whether a step finished, it stops and the operator checks before anything is repeated. Test: simulate a slow response and confirm the agent stops rather than repeats.
- Usage and cost: The pilot budget owner tracks model charges, other service charges, and staff review time separately. Missing usage is reported as missing, never as zero. Test: send a request that should trigger the budget warning and confirm that it does.
One part working does not make the whole task work. Keep the approved worksheet with the pilot results, so the next operator can see what was intended and what was actually tested.
Plan a pilot around observable outcomes
Choose representative cases before tuning infrastructure: a clear request, a request spanning two categories, a missing location, and an unavailable dependency. Record completion time, whether the draft is usable, and the cost of the whole task. Keep assumptions visible when usage records cover only part of the route.
Then decide whether your next gap is provider failover, deployment ownership, or the difference between a gateway and an agent platform.
When you compare running costs, count failed attempts and the time people spend fixing results, not only model prices. The cost-per-completed-task worksheet shows how. Budgets are a separate question: they cap spending, while this worksheet tells you what a useful result really costs.
In AgentShelf
The infrastructure overview describes managed services and configured model access. Provider availability depends on configuration, credentials, region, plan, and catalog. Review the developer runtime for supported execution options and governance and cost controls for monetary budgets and their coverage boundaries.