Day 3 - LLM as a Judge: Rank Multiple LLMs with an Orchestration Flow
Week 1 · Day 3 · Lecture 16. Lab file: 1_foundations/2_lab2.ipynb. The finale of the Day 3 lab: instead of judging the nine answers yourself, hand them to another LLM — anonymised — and let it rank them. That one step turns a fan-out of LLM calls into a small agentic workflow.
Two Python tricks first
zip— walks several lists in lockstep, sofor competitor, answer in zip(competitors, answers)prints each model next to its own answer with no manual index counting. It accepts any number of lists.enumerate— gives you the index and the item together. Here it is used to build text like "Response from competitor 1", "Response from competitor 2" and so on, one block per answer.
Step 1: anonymise the competitors
The judge shouldn't know which model wrote which answer, so the real names are replaced by competitor 1 … competitor 9 before anything is sent. Knowing that an answer came from a famous flagship (or a tiny local model) would bias the verdict; anonymising keeps the judgement about the writing itself.
Step 2: write the judge prompt
The prompt is built with an f-string and says, in effect:
- You are judging a competition between nine competitors.
- Each model was given this question (the question is included).
- Evaluate each response for clarity and strength of argument, and rank them from best to worst.
- Respond in a specified JSON format (a ranked list of competitor numbers), here are the responses (the anonymised blocks from above), now respond with the JSON.
This is the key idea of the whole lecture: the LLM is being given agency — it decides the ranking — but mechanically it is still just generating tokens. The surrounding code constrains those tokens into a JSON shape that ordinary Python can read, which is how you let an LLM make a decision inside a program. The prompt then goes into a messages list (role: user, content: judge prompt) exactly as in every earlier call.
Step 3: choose a judge that isn't a contestant
Using one of the nine competitors as judge would feel biased even if it can't see the names, so the judge is a model not yet used in the lab: Grok (spelled with a K — xAI's model, not the Groq-with-a-Q inference provider), at version 4.3 in the recording. Grok is marketed as a maximally truth-seeking model, which makes a fitting umpire for this contest. The call is the same grok.chat.completions.create(...) shape as the other providers and took about 16 seconds. If Grok 4.3 is too pricey, a faster, cheaper non-reasoning variant (Grok 4.1 fast) works too.
Step 4: map the ranking back to names
The judge returns only competitor numbers in rank order. A short block of code looks each number up in the competitors list to recover the real model names.
The result
| Rank | Model |
|---|---|
| 1 | Kimi K2.6 (Moonshot AI, via OpenRouter) — open source |
| 2 | Claude Sonnet 4.6 |
| 3 | Gemini Flash Lite |
| 4 | GPT-OSS 20B (run locally) |
| 5 | GPT nano |
| 6 | GPT-OSS 120B (via Groq) |
| 7 | Gemma 4 (local) |
| 8 | DeepSeek |
| 9 | Llama 3.2 1B (local) — still a decent answer, but the blandest |
Your own run will differ — the question, the models and the judge's mood all vary. Things worth noticing in this one:
- Kimi's winning answer ("honesty becomes a weapon") was the one the instructor found a head-scratcher, but it was the most unusual and best-argued. "Most interesting and distinctive" appears to be what this judge rewards — and Grok's brand leans on radical honesty, so it's fair to ask whether it favoured a virtue-vice story about honesty. Judges have tastes; go back to the answers and check whether you agree.
- The ranking is roughly ordered by model size, with one flip (GPT-OSS 20B above the larger 120B). That lines up reasonably with what Artificial Analysis shows.
- Swap in flagship models (the biggest GPT or Claude Opus) and you would expect stronger answers across the board — worth trying.
Which agentic patterns did we just use?
The lab chains several LLM calls: one call generated the question, many calls answered it, and one call judged the answers. The instructor says this is a hybrid of two of the patterns from lecture 1.10 and deliberately leaves the identification to you — there is active debate in the Udemy Q&A about which two. Re-read the pattern diagrams and form your own view before looking at anyone else's. (Candidates worth weighing: the one-question-to-many-models fan-out, and the judge at the end.)
Your exercise
- Decide which pattern(s) the flow uses and why.
- Extend it: add more LLM calls, make the workflow more sophisticated, or bring in another pattern.
- Contribute it as a PR to the repo's community contributions folder (see the Git guide), and optionally post about it on LinkedIn and tag the instructor.
Why this matters commercially
Taking one request, farming it out to several LLMs, and then judging for the best answer is common in real agentic systems and applies to almost any business vertical wherever you want more resilient, higher-quality output. LLM-as-a-judge in particular shows up in agent workflows everywhere.
What you should now understand
zipandenumeratemake parallel-list handling clean.- Anonymise candidates before judging to avoid bias, and never use a contestant as the judge.
- Asking an LLM for a constrained JSON ranking is how you give it a decision to make inside ordinary code — it is still only token generation.
- Judges have preferences too, so sanity-check their verdicts against the raw answers.
- Day 3 in one line: Python code orchestrating multiple LLMs — passing one's output to the next — is the beginning of building agentic systems.
Resource links
- Artificial Analysis — compare model intelligence, speed and cost
- OpenRouter — one key, routed to many providers
- Anthropic — "Building Effective Agents" (the patterns referenced in lecture 1.10)
- Course resources page (the Git guide covers community contributions)
- This lecture on Udemy
Next
That closes Day 3 — about 10% of the course done. Day 4 gets into tools, the foundation of agentic AI, starting with the agent landscape.