Day 4 - How Tool Calling Works: The Truth Behind AI Agent Autonomy
Week 1 · Day 4 · Lecture 18. This is the point where the course explains what tools actually are, the essential idea behind agentic AI. The short version: an LLM never runs anything. It generates tokens, and the code you write reads them and decides what to do.
What "tools" are
LLMs are models that take an input sequence of tokens and generate the most likely tokens to follow it. (If you are unsure how that works, the instructor's YouTube series on LLMs is linked in the course resources.) Tools give an LLM new abilities beyond generating text, such as querying a database, running some SQL, looking up a stock price, or editing a Google Sheet as we did in n8n.
Tool calling is also called function calling. You write a function (a bit of Python), tell OpenAI "as part of answering this question, you may also run this function on my computer", and it sounds spooky: is OpenAI going to reach into my computer, and how do matrix calculations that generate text suddenly decide to run code? The reality is much more mundane. A lot of agentic AI comes down to clever prompting, JSON, and plain if statements.
Tool calling: in theory versus in practice
In theory, it feels magical: the user talks to software, the software calls an LLM in the cloud, and the LLM itself reaches out to tools like searching the internet, querying a database, editing a Google Sheet or looking up a stock price.
In practice, the tools hang off your software, not the LLM:
- The user asks a question and your software calls the LLM with it.
- The LLM only generates tokens. Those tokens can be a reply to the user, or a request such as "don't reply yet, I want to run a SQL query, please use the SQL tool".
- Your software makes the SQL query.
- Your software calls the LLM again: the user said this, we used a tool, here are the results, now what?
- The LLM continues generating tokens.
The thing that calls the tool is your code, not the LLM.
Demo 1: a tool call that is only a prompt
To show that tool use simply comes down to prompting, the instructor pastes a vanilla request into ChatGPT, with no real tool behind it. The prompt says the model is a customer support assistant for an airline, that it answers users' questions, and that it has a special ability to look up ticket prices: respond like "GET_TICKET_PRICE: London" to get the price of a ticket to London. The user question is "How much does it cost to go to Paris?" The model replies GET_TICKET_PRICE: Paris.
That was a tool defined and chosen through nothing but a prompt and a response. A student once asked: if the user wants the price to Paris, why not just call the function directly, why involve an LLM at all? Because the user never typed "GET_TICKET_PRICE: Paris". They said "how much does it cost to go to Paris?" The LLM understands that cost and price mean the same thing, that Paris differs from London, and how to craft the exact call. Turning free-form natural language into specific discrete actions was close to an unsolved problem a few years ago; now LLMs do it in a heartbeat.
Demo 2: autonomy is prompting plus interpreting output
Second prompt: you are exploring a room, you can move in four directions by responding MOVE NORTH, MOVE EAST, MOVE SOUTH or MOVE WEST to learn more about what's there; respond with your choice. ChatGPT answers MOVE NORTH. Asked again, it picks east.
This is another tool-call example, but the bigger point is that it is a real example of what gets packaged up as autonomy. The model chose to move north, and nothing told it to. Attach some code that reads that reply to a little robot and you would say ChatGPT autonomously moved one square. Sounds fancy; it is just how you prompt and how you interpret the output. Tokens in, tokens out.
The agent loop, the same trick
It feels as if an agent loop is software calling an LLM in a loop, with the LLM somehow using tools:
Actually, it is this: your software repeatedly calls the LLM, and based on the response may call some functions, then keeps calling the LLM. Your software owns the loop and calls the tools. The LLM just takes input tokens and gives output tokens.
What you should now understand
- Tool calling (function calling) lets an LLM "use" a function, but it is your software that executes it.
- The LLM signals a tool call through the tokens it generates, guided by your prompt, usually in a parseable format such as JSON.
- The LLM's real value is understanding the meaning and intent of free-form text and mapping it to the right discrete action.
- "Autonomy" is prompting plus code that interprets the output, with if statements around it.
- An agent loop is code that repeatedly calls an LLM and the tools it asks for; the loop and the tools belong to your software.
Resource links
- OpenAI - Function calling guide
- Anthropic - Tool use documentation
- Anthropic - "Building Effective Agents"
- This lecture on Udemy
Next
On to the Day 4 lab (lab 3), where you will build tools and an agent loop yourself, starting with the Digital Twin project: reading a PDF and mastering system prompts.