LiteLLM: what "agent" means
Part of Agent Definition Research. Produced by running RESEARCH_PROMPT. Evidence from unversioned live documentation at docs.litellm.ai, fetched as markdown by appending .md to each doc path (for example https://docs.litellm.ai/docs/a2a.md), enumerated from the llms.txt page index, with the GitHub release feed of BerriAI/litellm used only to date the current release. LiteLLM is an open source AI gateway (a proxy server plus a Python SDK) that fronts LLM providers, MCP servers, and A2A agents behind one set of virtual keys. Retrieved 2026-10-13. The pin is weak; see Source anchors.
The agent noun (primary-source quotes)
- The primary noun is a registered gateway record that points at an agent LiteLLM does not run. The Agent Gateway page frames the whole feature as routing and access: "Add A2A Agents on LiteLLM AI Gateway, Invoke agents in A2A Protocol, track request/response logs in LiteLLM Logs. Manage which Teams, Keys can access which Agents onboarded" (Agent Gateway). Registration takes a name and a URL: "The URL should be the invocation URL for your A2A agent". The card page restates the value proposition as proxying: LiteLLM exposes A2A agents "to your clients through LiteLLM with virtual keys, team scoping, observability, and a unified agent card" (A2A Agent Card).
- It is persistent and identified. The record has an
agent_id(a UUID) and anagent_name, is created throughPOST /v1/agentsor the Admin UI under Agentic > Agents, and is addressable by either: "{agent_id}may be the agent UUID or the registered agent name" (Agent Gateway). The name is "A unique identifier for your agent (used in API calls)" (A2A Agent Cost Tracking). The storage table is named in the audit-log description: a kill switch fire "writes akill_switch_firedrow to the audit log against theLiteLLM_AgentsTabletable with the agent id as the object" (Agent Kill Switch). - The record carries no persona and no loop. Every documented field is either the upstream's published card (
agent_card_params: name, description, url, version, capabilities, skills, protocol version), upstream connection settings (litellm_params: provider, credentials, card path, trace-id flags, session budgets), header forwarding (static_headers,extra_headers), akill_switchwebhook, rate limits, and anobject_permissionblock. No page documents a system prompt, instructions, tool list, or default model field on the gateway agent; we infer from that absence that the definition of what the agent is lives entirely upstream. Skill routing confirms it: "LiteLLM forwards the entire message envelope, including metadata, to the upstream agent unchanged. The upstream agent is responsible for readingskillIdand routing internally" (A2A Agent Card). - The same record is a policy principal. A virtual key can be bound to an agent (
agent_idpassed to/key/generate), and a JWT can be bound to one by claim: "Provision agent identities in your identity provider (for example, one Microsoft Entra ID app registration per agent) and let LiteLLM enforce the agent's policies on every JWT that agent presents" (OIDC, JWT-based Auth). The record is therefore both a callee (something clients invoke at/a2a/{agent_id}) and a caller identity (something whose outbound model and MCP calls are narrowed and attributed). - The word "agent" names other, unrelated things in the same product.
litellm.agent(), an SDK function: "Run an agent withlitellm.agent()to drive Claude Code, Codex, OpenCode, Deep Agents or Tool Loop from Python with one API." It is flagged "litellm.agent()is in beta. The API may change between releases" (Agents (litellm.agent)). Nothing on the harness pages references a registered gateway agent; attribution is by request tag (harness,<name>), notagent_id.- Google managed agents, a pass-through to another vendor's resource at
/v1beta/agents: "The agent lives entirely on Google's side. LiteLLM is just the auth + routing layer" (Google AI Studio, Managed Agents). - Agents as model routes: LangGraph, Bedrock Agents, Vertex AI Agent Engine, Azure AI Foundry Agents, and Databricks agents are reachable as
modelstrings (for example the provider routelanggraph/{agent_id}, LangGraph), and any registered agent is reachable from/chat/completionswith thea2a/prefix (Invoking A2A Agents).
- Conceptual model (our inference): agent-as-policy-principal over a proxied endpoint. The gateway agent is closest to the corpus's agent-as-network-endpoint (the A2A card), plus an identity that keys and tokens can be bound to. It is not agent-as-config in the corpus's usual sense, because the config it holds governs access to the agent, not the agent's behaviour.
Subagents
- No subagent noun on the gateway record. The record has no field naming other agents, no roster, and no parent reference. The only documented sub-agent concept is a calling convention: "When the backend agent itself calls LiteLLM (for chat completions or to invoke a sub-agent), LiteLLM forwards two headers to maintain trace continuity", namely
X-LiteLLM-Trace-IdandX-LiteLLM-Agent-Id(Agent Gateway). - Delegation is the upstream agent's choice; the gateway only observes it. An agent invoked through LiteLLM that calls another registered agent does so as an ordinary A2A client of the proxy. The propagation is cooperative: "To enable these features, your A2A server must forward these headers to any LLM calls it makes back to LiteLLM." What does not cross: "The caller's virtual key and end-user ID are not automatically forwarded" (Agent Gateway).
- A sub-agent role can be enforced, not declared. The
require_trace_id_on_calls_to_agentflag exists for agents meant only to be called inside someone else's trace: "Use when the agent should only be called as a sub-agent with a trace context. Returns 400 if missing" (Agent Iteration Budgets). The gateway cannot tell which agent is the parent; it only checks that a trace id is present. - Inheritance between agents is not modelled. Each agent's permissions, limits, and credentials stand alone, and outbound headers are isolated: "Each agent invocation uses an isolated HTTP connection" (A2A Agent Authentication Headers). No depth or fan-out limit is documented for agent-to-agent calls; the nearest bound is the per-session iteration and dollar cap, which counts LLM calls, not delegations.
- The SDK harness passes subagents through to the wrapped runtime. For
Harness.DEEPAGENTS,DeepAgentsOptions.subagentsholds "deepagents SubAgent dicts, passed through", and the page recommends "a cheaper group for subagents" (Deep Agents). Subagent semantics are LangChain's, not LiteLLM's.
Configuration surface (what, where, why)
- Fields on the gateway agent, grouped by what they govern (assembled from Agent Gateway, A2A Agent Card, A2A Agent Authentication Headers, Agent Kill Switch, Agent Iteration Budgets, Budgets, Rate Limits, and MCP Permission Management):
- Identity and discovery:
agent_name, generatedagent_id, andagent_card_params(the A2A card LiteLLM republishes, including a pinnedprotocolVersionof0.3or1.0, curatedskills, and a free-formversionstring). - Upstream connection:
agent_card_params.urlfor A2A servers, orlitellm_params.custom_llm_providerplusmodeland credentials for "completion-bridge" agents (LangGraph, Bedrock AgentCore, Databricks, and others);agent_card_pathfor non-standard card locations. - Outbound credentials:
static_headers,extra_headers(client header names eligible for forwarding), provider credentials inlitellm_params, and minted credentials (Databricks OAuth, Microsoft Entra). - Resource limits:
tpm_limitandrpm_limit"on the agent itself",session_tpm_limitandsession_rpm_limitper session, andmax_iterationsplusmax_budget_per_sessioninlitellm_params. - Trace enforcement:
require_trace_id_on_calls_to_agent(inbound) andrequire_trace_id_on_calls_by_agent(outbound, required for session budgets). - Outbound authority:
object_permissionwithmcp_serversandmcp_tool_permissions, which narrows what the agent's own calls can reach. - Cost: flat cost per query or per-token input and output cost, used to price calls to the agent (A2A Agent Cost Tracking).
- Operations: a
kill_switchwebhook, and agent access group tags.
- Identity and discovery:
- Absent knobs, each a gap rather than a denial: system prompt, instructions, tool list, default model, memory, sandbox or environment, schedule, and triggers. No page states that these are out of scope; the configuration model simply has no place for them, because the upstream agent owns them.
- Where configuration lives. The Admin UI, the
/v1/agentsREST API (POST,GET,PATCH,PUT, plusmake_publicandkill_switchactions), and the top-levelagents:key inconfig.yaml. The config path exists for immutable deployments: "Agents can also be declared inconfig.yamlunder the top-levelagentskey, which is useful when the gateway is deployed from a ConfigMap or another read-only source" (Agent Gateway). Some settings are reachable only from one surface: "Tagging an agent with access groups is currently a dashboard-only operation" (Agent Permission Management). - Who may grant access to an agent sits on keys and teams, not on the agent. "Agent Permission Management lets you restrict which agents a LiteLLM Virtual Key or Team can access", with stated rationale "Multi-tenant environments: Give different teams access to different agents", security, and compliance. The default is open: "Open access by default when no restrictions are set" (Agent Permission Management). Access Groups bundle agents with models and MCP servers: "One group can grant access to all three resource types" (Access Groups).
- Stated rationale for the remaining knobs, quoted.
- Session budgets exist for cost: "When agents run agentic loops, they can make unbounded LLM calls, causing unexpected costs" (Agent Iteration Budgets).
- Agent-level MCP grants exist to bound autonomy independently of the caller: "Useful for capping what an autonomous agent can do regardless of which key originally invoked it" (MCP Permission Management).
- Static header precedence exists so callers cannot impersonate the gateway to the upstream: "This ensures admin-controlled credentials cannot be overridden by client requests" (A2A Agent Authentication Headers).
- The AI Hub exists for reuse: "Share pre-built agents (A2A spec) across your organization. Users can discover and use agents without rebuilding them" (AI Hub).
- JWT binding exists so an identity provider, not a LiteLLM key, can carry the agent's identity; the agent's policies then apply because "Everything that already applies to a virtual key bound to an agent then applies to the JWT caller too" (OIDC, JWT-based Auth).
- Credential handling is not uniform across the record. Provider credentials are hidden: "Any credential-bearing field in
litellm_params(aws_access_key_id,aws_secret_access_key,aws_session_token,api_key, and similar) is write-only" (Bedrock AgentCore). Static headers are not: "static_headersvalues are stored in the database and returned by the API" (A2A Agent Authentication Headers). Kill switch auth is redacted on read and visible only to proxy admins. - The SDK harness is configured per call, not per record.
litellm.agent()takes aHarnessenum value, a prompt, a requiredsandbox, amodel, and optionaltools,permissions,options, andmetadata; "Options you pass toagent_session()apply to every turn" (Sessions). Its stated rationale is outsourcing the loop: "Withlitellm.completionyou get one model call and write the loop yourself. With a harness you get a finished loop that somebody else maintains" (Agents (litellm.agent)).
Binding time
- Registration time binds the card and the upstream. LiteLLM fetches the upstream card once, the admin curates it, and "LiteLLM saves the curated card and serves it at"
/a2a/{agent_id}/.well-known/agent.json(A2A Agent Card). The served card does not track the upstream automatically; an explicit operation rebinds it: "Use the Re-sync from upstream button to pick up new skills or capabilities the upstream agent has added since registration; it shows a diff and lets you accept changes selectively." Invalid protocol versions are rejected at registration with HTTP 400. - Database records are mutable at any time; config records only at restart.
PATCHandPUTupdate a database agent in place, including limits, headers, and kill switch. Config-declared agents "are not stored in the database, so they cannot be edited or deleted from the Admin UI. Change the config file and restart the gateway instead" (Agent Gateway). Name collisions resolve toward the database: "If an agent created in the UI already uses a name you then declare inconfig.yaml, the database record wins and the config entry with that name is skipped, so the name always resolves to the record you can edit." - Per-request binding is where the policy is evaluated. Each call to
/a2a/{agent_id}resolves the key and team allow-lists at request time; each outbound MCP call by a bound key intersects the agent's grants on every request ("the agent's list is intersected with the key's on every request the key makes", MCP Permission Management). Changes to access groups apply to every holder at once. Unpinned protocol versions bind per request from client signals (method name or ana2a-versionheader). - Session state is bound by a caller-supplied id. The "session" that budgets and session rate limits count is whatever the caller names: a "Dollar cap per session (identified by
x-litellm-trace-id)", with the id also accepted asmetadata.session_id(Agent Iteration Budgets). Counters are ephemeral: "Counters expire after 1 hour by default". - Derived state is cached until the card changes. "Agent embeddings are computed once per process and reused until the agent card changes" (Agent Gateway).
- Versioning: none for the gateway record. No page documents revisions, history, or pinning of the agent record.
agent_card_params.versionis the card's own free-form string, republished to clients. The pass-through to Google exposes Google's versions (/v1beta/agents/{name}/versions), which LiteLLM neither stores nor manages (Google AI Studio, Managed Agents). - What happens to in-flight work on edit is undocumented. Because the gateway does not run the loop, an edit can only change routing, limits, and grants applied to subsequent requests; we infer an in-flight upstream task is unaffected until its next call through the proxy, but no page states this.
- SDK harness binding is per call or per session: options bind at
agent_session()creation, and resumable state "holds the harness, its native session id, the working directory and the model. It never holds credentials" (Sessions).
Relationships between nouns
- Agent to virtual key: N:M for invocation, 1:N for ownership. Any number of keys may be allowed to invoke an agent through
object_permissionlists or agent access groups. Separately, a key may be bound to one agent (agent_idat/key/generate), which makes the key the agent's own credential for outbound calls; the iteration budget walkthrough ends with "Proceed to create a new key for the agent", and the gateway page calls it "a key owned by an agent" (Agent Iteration Budgets; Agent Gateway). We infer an agent can hold several bound keys; no page caps it. - Agent to team: allow-list only. "A2A permission resolution operates over two levels: Key and Team." A key with no list inherits the team's ("The key inherits agent permissions from the team"), and when both have lists the result is the "Intersection of both lists (most restrictive wins)" (Agent Permission Management). No page places an agent inside a team or organization as its owner; the record shows a
created_byuser. - Agent to organization: only through MCP. Organizations do not appear in A2A invocation permissions. They appear in the MCP permission hierarchy, where the organization "Acts as a ceiling" over key, team, end user, agent, and internal user lists (MCP Permission Management).
- Agent to MCP server: the agent is a grantee, never a host. Agents hold MCP server and per-tool grants in
object_permission; those grants narrow the outbound calls of keys and JWTs bound to the agent, and of requests that name the agent inx-litellm-agent-id. "Granting a server to the key alone is not enough; the agent must also hold the grant". MCP clients can also find agents: a key with tool search enabled sees anagent_searchvirtual tool that ranks the registry (MCP Tool Search). - Agent to model: none on the A2A record, a route for bridge agents. An A2A agent references no gateway model group; the models it calls are whatever its bound key may call. Completion-bridge agents carry a provider
modelinlitellm_params. Conversely, every agent is reachable as a model viamodel: a2a/<agent>, and some agent platforms are configured as ordinarymodel_listentries. Search uses a configured embedding model (agent_search_embedding_model). - Agent to session: 1:N, but the session is an id, not a resource. The gateway's session is a caller-supplied trace id used to key counters, with a default one-hour TTL. A2A tasks and contexts belong to the upstream agent: task methods are "forwarded to the upstream agent unchanged" (Agent Gateway).
- Agent to memory: by naming convention only. The
/memorystore is "CRUD endpoints for storing and retrieving user/team-scoped memory entries on the LiteLLM proxy", and suggests the key patternagent:{agent_id}:memoryunder the rule "Keys are globally unique. Use namespaced keys to organize entries" (/memory). Memory is scoped to users and teams; the agent prefix is a convention the store does not enforce. - Agent to sandbox: none on the gateway. The SDK harness alone has a sandbox noun, and there "A sandbox is required by every call" (Sandboxes). The harness
Session("a live runtime with its sandbox, working directory and history") never becomes a gateway record. - Agent to agent: no edge. Calls between registered agents are invisible as relationships; they surface only as shared trace ids and per-agent spend.
Lifecycle
- Create: Admin UI,
POST /v1/agents, or aconfig.yamlentry read at startup. Config entries missingagent_nameoragent_card_paramsare skipped: "agent_nameandagent_card_paramsare both required; entries missing either one are skipped at startup" (Agent Gateway). - Publish: optionally made public on the AI Hub (
make_public), which lists the curated card at/public/agent_hub(AI Hub). - Invoke:
POST /a2a/{agent_id}with JSON-RPC, or/chat/completionswithmodel: a2a/<agent>. Only messaging runs through LiteLLM's own client: "message/sendandmessage/streamgo through LiteLLM's A2A client (logging, guardrails, spend). All other methods are forwarded to the upstream URL inagent_card_params.url" (Agent Gateway). - Who owns the loop: never LiteLLM, on the gateway path. The registered agent runs elsewhere, and the gateway cannot stop it except by asking. The kill switch is an outbound webhook to the upstream runtime: "LiteLLM only makes the webhook call; the agent stays registered and enabled on the LiteLLM side, so you decide separately whether to delete it." The UI warns "that firing stops the agent's upstream runtime and can cause an outage" (Agent Kill Switch). Budgets and rate limits stop a loop indirectly, by answering its model calls with 429.
- Pause and resume: no gateway equivalent. The record has no running state to pause; disabling access means editing key or team lists, and stopping work means firing the kill switch.
- Delete: database agents are deleted through the UI or API; config agents only by editing the file and restarting. Deleting a database agent that shadows a config entry hands the name back: "Delete the database agent and the config entry takes the name back on the next reload."
- Persists: the record, its card, spend logs and Agent Usage analytics, and audit rows for kill switch fires. Session counters expire. Conversation history, tasks, and memory are the upstream agent's.
- The SDK harness has its own, client-side lifecycle.
litellm.agent()opens a session for one turn and closes it;agent_session()keeps it open; sessions end withclose(),detach()(resumable, sandbox left running), orstop()(resumable, runtime stopped), and resume across processes from serializedState(Sessions). The loop is the wrapped runtime's (Claude Code, Codex, OpenCode, Deep Agents) except forHarness.TOOL_LOOP, which is LiteLLM's own minimal loop aroundlitellm.acompletion(). Setup is the caller's job: "Do any setup in your own code. There are no lifecycle hooks" (Sandboxes). - Server-side reruns exist, but not as an agent. The Agentic Loop Hook page says to "Build a
CustomLoggercallback that intercepts a model response, fulfills tool calls server-side, and reruns the model, all transparently to the caller" (Agentic Loop Hook), with a default of three reruns. It attaches to model requests, not to registered agents.
What makes it "an agent" here (our inference)
Our inference: in LiteLLM an agent is a registered, named gateway record that points at someone else's A2A endpoint, republishes its curated card, and serves as a policy principal that keys and tokens can be bound to, while the loop it names always runs outside the gateway. Registered traces to "the registered agent name" and POST /v1/agents; someone else's A2A endpoint to "The URL should be the invocation URL for your A2A agent" and the forwarding of task methods upstream; curated card to "LiteLLM saves the curated card and serves it at"; policy principal to key and JWT binding ("enforce the agent's policies on every JWT that agent presents"), agent-level MCP grants, limits, and spend attribution; and outside the gateway to the kill switch, which "stops the agent's upstream runtime" and for which "LiteLLM only makes the webhook call".
The distinctive datum for the study is the split between identity and definition. Every other product in the corpus that keeps an agent record keeps something that defines or deploys the agent (a prompt, a model, tools, code, or an image) and attaches authority to it. LiteLLM stores what the agent may do and spend and nothing that determines how it behaves; the card it keeps is a description the upstream published. The record is an authorization and accounting subject, with persona and loop left wholly to the upstream. The word is also overloaded inside one product: litellm.agent() is a client-side call that drives third-party harnesses and never touches the record, and the /v1beta/agents route forwards to Google's resource without storing it. And the gateway's "session" is only a caller-asserted trace id with a one-hour counter, so session-level budgets bound cost per trace rather than per conversation.
Open questions
- Where session budgets are stored. The same page says both "Budget controls (
max_iterations,max_budget_per_session) are stored in the virtual key's metadata" and "Budget controls are set on the agent'slitellm_params(not on individual keys), so they apply across all keys for the agent" (Agent Iteration Budgets). Budgets, Rate Limits sides with the agent'slitellm_params. Recorded as an upstream contradiction, not resolved. - Who may assert
x-litellm-agent-id. MCP grants for "the agent identified byx-litellm-agent-id" are intersection-only, so asserting the header can only narrow tool access. The same header "Attributes spend to the correct agent". No page states whether an arbitrary key can attribute its spend to, or consume the session budget of, an agent it is not bound to. - Whether
max_iterationsand session budgets apply to A2A invocations or only to the bound key's outbound LLM calls. The prose says "Each LLM call for a session increments a counter", and enforcement requires the outbound trace flag; the treatment ofmessage/sendcalls to the agent is not stated. - Ownership of the record. The API response carries
created_by, but no page says whether a team admin (rather than a proxy admin) can create, edit, or delete an agent, or whether agents are scoped to a team or organization at all. - Several keys bound to one agent. Key binding is documented one key at a time; whether limits such as
tpm_limitaggregate across all bound keys is implied by "on the agent itself" but never shown. - Org-level control over A2A access. Organizations bound MCP access as a ceiling but do not appear in A2A permission resolution; whether an organization can restrict agent invocation is undocumented.
- Edits during in-flight tasks. No page states what happens to a running upstream task, or to pending
tasks/getpolls, when the agent's URL, credentials, or protocol version change mid-task. - Version history of the record. No revision or audit trail of card or policy edits is documented beyond kill switch fires; whether general audit logging covers
PATCH /v1/agentsis not stated. - The
agents:config key's release status. The gateway page says "Theagentskey is read correctly starting in the next release (afterv1.95.0)", while the current release at retrieval isv1.105.0; the note reads as stale, and the release that fixed it is not named. - Whether
litellm.agent()will ever bind to a registered agent. The harness pages attribute spend by tag and never mentionagent_id. The SDK is "in beta," so the separation may be deliberate or merely early.
Source anchors
The docs are unversioned live pages. Every page fetched carries a last_updated: "2026-10-12" field in its markdown frontmatter, and none states the LiteLLM version it describes. The current GitHub release at retrieval was v1.105.0, published 2026-10-11 (https://github.com/BerriAI/litellm/releases/tag/v1.105.0). Feature-floor statements on individual pages give more version signals: the AI Hub notes "Agents are only available in v1.79.4-stable and above", and the gateway page references v1.95.0 for the agents: config key. The pin is therefore weak: retrieval date 2026-10-13, page last_updated 2026-10-12, and the release current on that date. The site publishes no archive URL and no content digest, and many of these pages have no counterpart at the same path in the repository's docs/my-website/docs/ tree, so the served .md form was used throughout.
- Agent Gateway: overview, agent card, invoking, authentication headers, cost tracking, permissions, iteration budgets, kill switch, and Databricks agents.
- Identity and access: Access Groups, AI Hub, Budgets, Rate Limits, OIDC, JWT-based Auth, MCP Permission Management, and MCP Tool Search.
- Providers reached as agents: Bedrock AgentCore, LangGraph, and Google AI Studio, Managed Agents.
- SDK harness: Agents (litellm.agent), Using with LiteLLM AI Gateway, Sessions, Sandboxes, Events, and Deep Agents.
- Adjacent surfaces: /memory and Agentic Loop Hook.
Mechanical adaptations to quoted passages: markdown emphasis inside quotes (**...**) is kept as LiteLLM wrote it, so it renders as bold here. Quotes that span a markdown link in the source are split at the link. Page titles containing a dash separator in the source (for example "OIDC - JWT-based Auth") are written with a comma in link text, and the Managed Agents page title is shortened the same way. No quoted passage required removing an em dash or en dash; passages containing one were not quoted.
