A Mental Model for LLM Tool Calling

I wanted to dive into how LLM tool calling really works since the first time you use it, it feels like magic. If the model is just predicting token after token, how can it suddenly make an API call? The short answer is: it doesn't. The runtime does though. The runtime being the server that is executing the model.

An LLM generates tokens indicating that it wants a tool called. Software outside the LLM interprets those tokens, runs the tool, and sends the result back to the model. Let's break it down:

1. LLMs predict the next token

At the most basic level, a transformer receives a sequence of tokens:

[x1, x2, x3, ... xn]

And produces a probability distribution for the next token:

P(x[n+1] | x[1:n])

Then it picks a token, appends it to the sequence, and does it again:

prompt
  ↓
predict token
  ↓
append token
  ↓
predict token
  ↓
append token
  ↓
...

Nothing about that primitive says:

HTTP
Bash
Read
Write

It just predicts tokens.

2. So how can predicting tokens cause an API call?

Model trainers give certain outputs special meaning. Conceptually, the model might generate something like:

<tool_call>
{
  "name": "get_weather",
  "arguments": {
    "city": "Tokyo"
  }
}
</tool_call>

The exact representation depends on the model and provider. But the point I want to drive home is from the model's perspective, it just generated another sequence of tokens. But from the runtime's perspective, which has been observing the token outputs, that sequence means something:

Stop generating text and return a tool request.

The model did not make an HTTP request. It predicted that get_weather was the most appropriate thing to generate next. The software around the model is what gives that prediction real-world consequences.

3. Your server actually calls the tool

For a custom tool, the flow is roughly:

Your server
    ↓
sends prompt + tool definitions
    ↓
Model provider
    ↓
LLM generates a tool request
    ↓
Model provider returns the request
    ↓
Your server calls the weather API

Your server might receive something like:

{
  "type": "function_call",
  "name": "get_weather",
  "arguments": {
    "city": "Tokyo"
  }
}

Then ordinary deterministic software takes over:

if tool_call.name == "get_weather":
    result = weather_api.get(
        city=tool_call.arguments["city"]
    )

The LLM proposes the operation. Your application decides whether to run it and performs the actual IO. The model saying:

delete_customer(id=123)

does not mean the customer should be deleted.

Your software still needs to check authentication, authorization, tenant boundaries, input validation, and business rules. The LLM proposes an action. Software decides whether that action is allowed to happen.

4. The tool result becomes input to another inference step

Suppose the weather API returns:

{
  "city": "Tokyo",
  "temperature": 27,
  "conditions": "Rain"
}

Your server adds both the model's tool call and the result to the conversation in a structured form:

{
  "role": "assistant",
  "tool_calls": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "arguments": {"city": "Tokyo"}
    }
  }]
}

{
  "role": "tool",
  "name": "get_weather",
  "content": "Tokyo is 27°C with rain."
}

This is how you square the earlier <tool_call> output with the assistant and tool messages. The model emits whatever tool-call format it was trained to use. The runtime parses that output into a structured tool-call object for your application. After your application runs the tool, it adds the result to the conversation. Before the next inference step, a model-specific chat template turns those structured messages back into the exact format the model expects.

API / tool result
        ↓
structured conversation
        ↓
model-specific chat template
        ↓
serialized text
        ↓
tokenizer
        ↓
token IDs
        ↓
embedding layer
        ↓
vectors for the next inference step

That content gets serialized into the model's expected message format, tokenized, embedded, and processed like the rest of its input. Then a new inference step begins and the model can now predict:

It is currently 27°C and raining in Tokyo.

So the complete loop is:

tokens
  ↓
tool request
  ↓
normal software
  ↓
API response
  ↓
more tokens
  ↓
more inference

The tool result did not get injected into some persistent internal memory. It became context for the next prediction.

What happens to batching while the tool runs?

When reading this, I am sure you thought about the performance implications of this and how it might effect batching. How are model providers able to most efficiently use the gpu, given they do not want the GPU sitting around waiting for your weather API. Once the model finishes generating the tool request, that inference step is done. The provider can use the GPU capacity for other requests while your server is making the API call. When your server sends the tool result back, it becomes another inference request that can be scheduled onto the GPU.

So tool calling is less like pausing a function halfway through:

weather = await get_weather()

And more like ending one model invocation and starting another:

inference
  ↓
external IO
  ↓
new inference

The external API does not necessarily block everyone else using the model. But it absolutely adds latency to your agent's workflow. In an inference engine such as vLLM, they might use the term batching, but under the hood it is really a complicated scheduling algorithm: after each model step, finished requests can leave and new requests can join. Your tool call ends one inference request, and the tool result returns later as another. The weather API adds a gap to your workflow, but it does not reserve an empty seat on the GPU while everyone waits.