An LLM gateway sits between applications and model endpoints. An AI agent platform coordinates the work around model calls. The two can be part of the same product, but they address different operating responsibilities.
Start with the missing capability in your workflow. A common model access point and a controlled business action are different needs.
A gateway manages model access; the agent layer decides what the task may do.
Compare responsibilities at the boundary
Gateway implementations may centralize routing, authentication, usage records, and request policies. Agent platforms may provide execution, context, tool connections, and human review. Check what each product actually includes; the category name alone does not tell you.
Scroll horizontally to see all columns.
| Responsibility | Gateway question | Agent platform question |
|---|---|---|
| Model request | Which configured endpoint receives this request? | When does this task need a model call? |
| Identity | May this application call the endpoint? | May this user run this agent and use these tools? |
| Context | Which request data passes through the gateway? | Which approved evidence belongs in the task? |
| Failure | What happens when a model route is unavailable? | What state is the task in after the failure? |
| Cost | Which model usage can be attributed? | What did the complete task cost across its steps? |
| Review | Which request policies apply? | Who approves a consequential action? |
For example, a gateway confirming that an application may call a model does not mean an employee may update a customer record. That decision also depends on application permissions and the connected system.
AI gateway, LLM gateway, and agent gateway
Vendors use these names loosely. LLM gateway and AI gateway usually describe the same layer: one entry point for model requests that handles credentials, routing, usage records, and request policies. Some AI gateways also front other AI services. An agent gateway usually governs a different path: traffic from agents to tools and to other agents, such as Model Context Protocol (MCP) tool calls or agent-to-agent requests. Some products combine these roles.
None of the three runs the task itself. Before comparing feature lists, ask which traffic a product governs (model calls, tool calls, or agent-to-agent requests) and where its authority ends.
Clarify what “router” means
A model router chooses a model endpoint for a request. An agent router chooses which agent or workflow should handle a job. A tool selector chooses a permitted operation during execution. These choices may appear close together in a diagram, so document each one's inputs, authority, and output.
Ask whether a selection changes the data destination, the set of allowed actions, or only the model used for an existing step. Test those changes independently.
Separate provider management from business permissions
Managing model providers is one job: which models you can use, whose credentials pay for them, how requests are routed, and what usage is recorded. Deciding what an agent may do with your information is a different job, and the list of available models answers none of it. A model being available does not make it approved for your customer records.
Settle these questions with the gateway's owner and write the answers down:
Provider management decisions (blank)
- Credentials: who owns them, what they may be used for, and how they are replaced (never paste the keys themselves into the record): ___
- Approved models: which ones, where their data may be processed, and which tests they passed: ___
- Routing changes: who approves a change, and which tests are rerun first: ___
- Visibility: which usage and error records operators can see, what is missing, and who follows up on anything unexplained: ___
- Budget: which spending policy applies to this request, confirmed by test rather than assumed because traffic passes through the gateway: ___
In AgentShelf, the infrastructure overview covers configured providers and how requests reach them. Governance and cost controls covers which requests a budget checks.
Example: a returns assistant
A fictional retailer already has an application that finds an order, checks return eligibility, and prepares a response. The team wants consistent model access across several applications. A gateway may address that shared dependency while the existing application continues to own the return rules.
Later, the team wants reusable agents with defined tools and approval before issuing a refund. That requires decisions at the agent execution layer as well. Changing the model route cannot substitute for refund authorization.
Write two acceptance statements:
- Model access: An approved application can use its configured model route, and an unavailable route produces an observable outcome.
- Business action: A refund cannot proceed without the required authorization, including after a model failure or change.
Use provider failover planning to test the first boundary and an agent harness to understand the second.
Choose a layer with an ownership map
Before adding infrastructure, name the owner of model credentials, routing changes, runtime updates, tool permissions, and incident response. Decide where operators can trace one task across the layers. A new gateway also needs monitoring and an owner; adding it can introduce another dependency.
Pilot one complete task and deliberately interrupt a model call. Check whether the team can explain both the model outcome and the business outcome. If those answers are scattered across systems, agree on the evidence needed before broadening use.
In AgentShelf
AgentShelf combines managed infrastructure with an agent harness and supported developer runtime options. Review these documented surfaces against your ownership map. Model access, agent execution, and business permissions remain separate questions even when one platform supports several layers.
Review gateway-held credentials and release provenance together using software supply chain security with AI. Routing requests and governing an agent do not settle the supplier-access question.