How API Design Changes When Your Client Is an LLM

APIs have historically been designed for two types of consumers:

Humans indirectly using them through an application.
Deterministic programs consuming them directly.

LLMs are kind of a hybrid of those two. They are programs, but they have limited context windows, lossy memory, and probabilistic reasoning. Earlier models probably operated more like a deterministic program consuming it - everything had to be setup just right. Later model improvements and more advanced harness programming have lead them to interact more like how a human might consume an api - much more flexible. It can still be slow, expensive, and unreliable if things are not setup right.

The conventional API model

A normal API interaction might look like:

Frontend
   ↓
Backend
   ↓
GET /customers/123
   ↓
Customer service
   ↓
JSON

Suppose the endpoint returns a customer with 200 fields:

{
  "id": "123",
  "name": "Alice",
  "email": "alice@example.com",
  "plan": "basic",
  "...": "196 more fields"
}

But the frontend only needs:

name
email
plan

You will still just send it all over, since the cost of sending extra data over the wire is generally less than the cost of having to go through the pomp and circumstance of another BE change and deployment. Now the FE can cheaply do customer["address"] if another field is used. So you have bandwidth and latency concerns, but the program itself is not confused by returning so many fields.

An agent consumes an API differently

An agent interaction looks more like:

LLM
 ↓
generates a tool request
 ↓
Agent runtime
 ↓
Your API
 ↓
Agent runtime
 ↓
serialize + tokenize response
 ↓
LLM

The API response becomes input to another inference step. If your endpoint returns 200 fields but there are only 3 that matter, the other 197 pollute the context window. And one of the core tenets of engineering these days is context management. Those extra fields now cost:

tokens
attention
context-window space
inference time

Every tool call crosses a runtime boundary, and the result has to be serialized back into model context before the model can reason about it. That creates some API design pressures that conventional clients do not have.

1. Optimize for context transfer, not API completeness

A traditional API often returns the canonical representation of a resource:

get_customer(id)
       ↓
entire customer object

This makes sense. There is one well-understood shape, and every application can select the fields it needs. For an agent, it can be better to support projection:

get_customer(
    id="123",
    fields=["name", "plan", "balance"]
)

Or filtering:

list_orders(
    customer_id="123",
    status="open"
)

Or a purpose-built summary:

{
  "customer_id": "123",
  "plan": "basic",
  "outstanding_balance": 74.22,
  "open_orders": 2
}

The goal is not to create 50 slightly different agent endpoints. The goal is to let the agent communicate:

This is the information I actually need.

2. Return enough state to re-ground the model after mutations

Suppose the model reads:

{
  "customer_id": "123",
  "plan": "basic"
}

Then it requests:

update_customer(
    id="123",
    plan="pro"
)

A conventional API might return:

{
  "success": true
}

That is perfectly reasonable for a deterministic program. The program already knows what it requested. But the agent is about to enter another inference step, and its context now says:

Earlier:
plan = basic

Requested:
change plan to pro

Result:
success = true

The model can probably reconstruct the current state. But why make it? You are simply just increasing the hallucination rate. A better response would be:

{
  "customer_id": "123",
  "before": {
    "plan": "basic"
  },
  "after": {
    "plan": "pro"
  },
  "version": 42
}

Now the most recent authoritative information explicitly says:

plan = pro

The model has been re-grounded in the current state of the world. You do not necessarily need to return the entire customer object either. A semantic delta can be enough:

{
  "changed": {
    "plan": {
      "from": "basic",
      "to": "pro"
    }
  },
  "unchanged": [
    "name",
    "email"
  ],
  "version": 42
}

This is much more useful than:

{
  "status": 200
}

3. Prefer stable references over repeatedly retransmitting large objects

Suppose an agent runs a query that returns 80,000 rows. You probably do not want to put this into model context:

{
  "rows": [
    "...80,000 rows..."
  ]
}

You probably cannot even put that into the model context.

Instead, return a stable reference:

{
  "result_id": "result_84721",
  "row_count": 80137,
  "columns": [
    "customer_id",
    "revenue",
    "region"
  ],
  "summary": "Revenue by customer for Q2 2026."
}

Then expose tools like:

inspect_result(result_id)

query_result(result_id, query)

read_result(result_id, offset, limit)

aggregate_result(result_id, ...)

The data stays in the tool runtime. Only the information needed for reasoning crosses into model context. This also makes the workflow easier to resume. An agent interaction is not one long-running process:

model → tool → model → tool → model

It is a series of separate inference turns. Stable IDs, cursors, and versions let the next turn refer to previous work without relying on hidden process-local state or retransmitting the entire object.

4. Pagination is a must

Pagination is a must for any endpoint that can return a large result set. This is already standard API design, but it matters even more for agents because returning everything at once can pollute or overflow the model context.

{
  "data": ["..."],
  "next_cursor": "cursor_abc",
  "returned": 50,
  "remaining_estimate": 4237,
  "has_more": true
}

5. Expose cheap ways to inspect before fetching

A normal API might expose:

get_document(id)

And return the entire document. That is fine for a 20 KB document. It becomes a problem when get_document returns a 300-page PDF or a 10 MB log file. A better interface might be:

describe_document(id)
        ↓
title
size
sections
metadata

search_document(id, query)
        ↓
matching passages

read_document(id, range)
        ↓
specific content

This lets the model progressively discover information. The agent can inspect the shape of the data, search for the relevant section, and only then fetch the content it needs.

Progressive disclosure also avoids a bad failure mode where one overly broad tool response fills most of the context window before the agent even understands what it is looking for.

Does this mean everyone needs a separate agent API? No.

A sophisticated agent harness can compensate for a mediocre API. Suppose a legacy endpoint returns 5 MB of JSON. Instead of dumping that entire response into the context window, the harness can do this:

API response
     ↓
save result.json
     ↓
tell model:
"Result stored in result.json.
5.1 MB, 83,294 records."
     ↓
model searches or processes the file
     ↓
only relevant data enters context

The harness acts as an adapter between an API designed for deterministic software and a model that reasons through context. As models and harnesses get better, the need for agent specific endpoints decrease. A good API for humans does include compact responses, semantic mutation state, stable references, pagination, etc. It is just all the more important for an agent API. What you are optimizing for slightly changes though:

Traditional API:
developer ergonomics
latency
bandwidth

Agent-facing API:
context pressure
token cost
re-grounding
latency
bandwidth