Day 2 - Agentic AI Risks, Guardrails, Evals and Traps to Avoid
Week 1 · Day 2 · Lecture 11. Before we spend the next six weeks marveling at what agentic AI can do, it's worth being honest about why it's hard to work with — and about two traps that catch almost everyone the first time they try to build an agent for a real business.
The core risk: agentic AI is unpredictable
The same flexibility that makes agents useful — an LLM deciding what to do next rather than following a fixed script — is also their biggest liability. Three things become unpredictable at once:

- Unpredictable path — the route the agent takes through the system can differ run to run.
- Unpredictable output — results aren't deterministic; you can get a different answer each time.
- Unpredictable costs — the number and size of LLM calls can vary, so spend varies with it.
Mitigation 1: monitoring
Know what your agent is actually doing, both in development and in production.
- Observability — traces showing exactly what happened on each LLM call.
- Evals — measured performance checks. The most important is an eval tied to your commercial goal (leads generated, revenue driven) rather than a purely technical metric. A second useful kind is an LLM-as-judge eval — using an LLM to grade the output — which is just the evaluator-optimizer pattern from the previous lecture applied to production monitoring.
Mitigation 2: guardrails
OpenAI defines guardrails as code that "ensures your agents behave safely, consistently, and within your intended boundaries." In practice, guardrails are just tests wrapped around your agent's inputs and outputs — checks you add because an LLM's output is never guaranteed to be something you should act on without verifying it first. You'll build guardrails throughout this course.
Trap #1: leading with the solution instead of the problem

This is a trap almost every technologist building agentic AI has fallen into: a business sponsor calls and says "I need an agent for my business," or more specifically, "I want a Strategy Agent to give me advice for my org." The hype around agentic AI has made people solution-first — they've already decided they want "an agent," sometimes with budget already earmarked for it, before anyone has named the problem it's meant to solve.
The fix is always the same question: "OK — and what business problem is that solving?" Only once you have a real problem can you tell afterward whether the agent actually solved it.
A concrete example: a client asked for a "culture agent" to advise on company culture. Pressed on what problem it would solve, the real issue turned out to be a morale problem, which turned out to be measurable as employee attrition. Once the problem was named as "reduce attrition," an AI agent became one possible option among several — alongside things as basic as running an employee survey — rather than the only conceivable answer.
There's a deeper point buried in this trap. LLMs are trained to generate plausible, believable content — that's the objective function, full stop. A culture agent will cheerfully produce believable-sounding advice about improving culture. Being believable does not make it correct. The only way to know whether an agent's output is actually right is to measure it against a real business outcome and iterate until the metric moves. Without a measurable problem, you can't tell a good agent from a persuasive one.
Trap #2: anthropomorphizing your architecture

The second trap: reaching for a whiteboard and sketching an "agent architecture" with human-sounding job titles before you've built anything. For a stock-trading system, that might look like a Trading Manager Agent coordinating a Market Research Agent, a Trader Agent, and a Risk Manager Agent — an org chart dressed up as a system design.
This is a mistake because it treats LLMs as if they were people with roles, when they are token-generation engines predicting plausible continuations of an input. The org-chart shape feels intuitive because it mirrors how humans divide labor — but that intuition is not evidence that it performs better.
The right approach:
- Start simple. One LLM, one prompt, one objective.
- Measure that single LLM call against your real business outcome (response rate, leads generated, whatever the deliverable is meant to achieve).
- Only then iterate — split into more LLM calls or more agents because it measurably improves performance, never because a multi-agent org chart "sounds right."
It's entirely possible that dividing work along human-team-shaped lines does turn out to perform best — but you should arrive there deliberately, through measured experiments, not by starting there because it mirrors how a human team would be organized.
The three takeaways
- Build an agent system because it solves a business problem in a way you can measure — not because "agent" is the word of the moment.
- Choose your architecture because it improves a business metric, not because it looks like a sensible human org chart.
- LLMs generate believable output, not necessarily accurate output. Aligning next-token prediction with a real business outcome — and adding the guardrails to keep it there — is the AI engineer's actual job. Since you are building the system rather than merely using one, "the model hallucinated" is not an excuse available to you; you're expected to have coded for it.
One more thing: what is an "agentic engineer"?

The term "agentic engineer" is genuinely ambiguous — it can mean either someone who uses agents to engineer (e.g. a developer who relies heavily on coding agents like Claude Code), or someone who engineers agents (builds agentic systems). Both meanings are in circulation. The recommendation is simply to be explicit about which you mean: after an AI-coding-focused course you'd describe yourself as "an engineer skilled at using coding agents"; after this course, you'll be able to say "an AI engineer skilled at building agents."
Resource links
- Anthropic — "Building Effective Agents" (the source of the evaluator-optimizer / LLM-as-judge pattern referenced here)
- OpenAI — Practices for Governing Agentic AI Systems (the source of the guardrails definition used in this lecture)
- Simon Willison — writing on AI agents
- Course resources page
- This lecture on Udemy
Next
That wraps Day 2 — theory only, no labs. Day 3 gets hands-on again, with lots of LLM calls.