AI Engineer Agentic Track: The Complete Agent & MCP

Day 3 - LLM as a Judge: Rank Multiple LLMs with an Orchestration Flow

Beginner 15
Rupinder Singh
6 views
Rank nine LLM answers with a judge model: anonymise the competitors, prompt for a JSON ranking, use Grok as the non-contestant judge, and map results back to names.

Week 1 · Day 3 · Lecture 16. Lab file: 1_foundations/2_lab2.ipynb. The finale of the Day 3 lab: instead of judging the nine answers yourself, hand them to another LLM — anonymised — and let it rank them. That one step turns a fan-out of LLM calls into a small agentic workflow.

Two Python tricks first

  • zip — walks several lists in lockstep, so for competitor, answer in zip(competitors, answers) prints each model next to its own answer with no manual index counting. It accepts any number of lists.
  • enumerate — gives you the index and the item together. Here it is used to build text like "Response from competitor 1", "Response from competitor 2" and so on, one block per answer.

Step 1: anonymise the competitors

The judge shouldn't know which model wrote which answer, so the real names are replaced by competitor 1 … competitor 9 before anything is sent. Knowing that an answer came from a famous flagship (or a tiny local model) would bias the verdict; anonymising keeps the judgement about the writing itself.

Step 2: write the judge prompt

The prompt is built with an f-string and says, in effect:

  • You are judging a competition between nine competitors.
  • Each model was given this question (the question is included).
  • Evaluate each response for clarity and strength of argument, and rank them from best to worst.
  • Respond in a specified JSON format (a ranked list of competitor numbers), here are the responses (the anonymised blocks from above), now respond with the JSON.

This is the key idea of the whole lecture: the LLM is being given agency — it decides the ranking — but mechanically it is still just generating tokens. The surrounding code constrains those tokens into a JSON shape that ordinary Python can read, which is how you let an LLM make a decision inside a program. The prompt then goes into a messages list (role: user, content: judge prompt) exactly as in every earlier call.

Step 3: choose a judge that isn't a contestant

Using one of the nine competitors as judge would feel biased even if it can't see the names, so the judge is a model not yet used in the lab: Grok (spelled with a K — xAI's model, not the Groq-with-a-Q inference provider), at version 4.3 in the recording. Grok is marketed as a maximally truth-seeking model, which makes a fitting umpire for this contest. The call is the same grok.chat.completions.create(...) shape as the other providers and took about 16 seconds. If Grok 4.3 is too pricey, a faster, cheaper non-reasoning variant (Grok 4.1 fast) works too.

Step 4: map the ranking back to names

The judge returns only competitor numbers in rank order. A short block of code looks each number up in the competitors list to recover the real model names.

The result

RankModel
1Kimi K2.6 (Moonshot AI, via OpenRouter) — open source
2Claude Sonnet 4.6
3Gemini Flash Lite
4GPT-OSS 20B (run locally)
5GPT nano
6GPT-OSS 120B (via Groq)
7Gemma 4 (local)
8DeepSeek
9Llama 3.2 1B (local) — still a decent answer, but the blandest

Your own run will differ — the question, the models and the judge's mood all vary. Things worth noticing in this one:

  • Kimi's winning answer ("honesty becomes a weapon") was the one the instructor found a head-scratcher, but it was the most unusual and best-argued. "Most interesting and distinctive" appears to be what this judge rewards — and Grok's brand leans on radical honesty, so it's fair to ask whether it favoured a virtue-vice story about honesty. Judges have tastes; go back to the answers and check whether you agree.
  • The ranking is roughly ordered by model size, with one flip (GPT-OSS 20B above the larger 120B). That lines up reasonably with what Artificial Analysis shows.
  • Swap in flagship models (the biggest GPT or Claude Opus) and you would expect stronger answers across the board — worth trying.

Which agentic patterns did we just use?

The lab chains several LLM calls: one call generated the question, many calls answered it, and one call judged the answers. The instructor says this is a hybrid of two of the patterns from lecture 1.10 and deliberately leaves the identification to you — there is active debate in the Udemy Q&A about which two. Re-read the pattern diagrams and form your own view before looking at anyone else's. (Candidates worth weighing: the one-question-to-many-models fan-out, and the judge at the end.)

Your exercise

  1. Decide which pattern(s) the flow uses and why.
  2. Extend it: add more LLM calls, make the workflow more sophisticated, or bring in another pattern.
  3. Contribute it as a PR to the repo's community contributions folder (see the Git guide), and optionally post about it on LinkedIn and tag the instructor.

Why this matters commercially

Taking one request, farming it out to several LLMs, and then judging for the best answer is common in real agentic systems and applies to almost any business vertical wherever you want more resilient, higher-quality output. LLM-as-a-judge in particular shows up in agent workflows everywhere.

What you should now understand

  • zip and enumerate make parallel-list handling clean.
  • Anonymise candidates before judging to avoid bias, and never use a contestant as the judge.
  • Asking an LLM for a constrained JSON ranking is how you give it a decision to make inside ordinary code — it is still only token generation.
  • Judges have preferences too, so sanity-check their verdicts against the raw answers.
  • Day 3 in one line: Python code orchestrating multiple LLMs — passing one's output to the next — is the beginning of building agentic systems.

Resource links

Next

That closes Day 3 — about 10% of the course done. Day 4 gets into tools, the foundation of agentic AI, starting with the agent landscape.

Frequently Asked Questions (7)

Why do we anonymise the competitors before asking the judge to rank them?

The judge should assess the writing, not the brand. Replacing real model names with competitor 1 to competitor 9 stops a famous flagship, or a tiny local model, from biasing the verdict.

Why use Grok as the judge rather than one of the nine models?

Using a contestant as the judge would feel biased even with anonymised answers. Grok (xAI, with a K) had not yet been used in the lab, so it can act as a neutral umpire. A cheaper option is the faster non-reasoning Grok 4.1 fast.

Is Grok the same as Groq?

No. Grok with a K is xAI's model family. Groq with a Q is a separate inference provider that hosted GPT-OSS 120B earlier in the lab.

How does an LLM make a decision inside ordinary Python code?

It is still only generating tokens. The prompt tells it to respond with a specific JSON format, so the output can be parsed by normal code and used as a ranking, then mapped back to model names.

What do zip and enumerate do in this lab?

zip walks several lists in lockstep, for example competitors and answers together. enumerate gives you an index alongside each item, which is used to label the blocks competitor 1, competitor 2 and so on.

Can I trust the judge's ranking?

Treat it as one opinion. Judges have tastes: in the lecture Kimi K2.6 won with a distinctive honesty-themed answer. Your own run will differ, so read the raw answers and check whether you agree with the verdict.

Which agentic patterns does this workflow combine?

The lecture deliberately leaves it to you. The flow generates a question, fans it out to many models and then judges the results, and is a hybrid of two of the patterns from lecture 1.10. Re-read those patterns and decide for yourself, then try extending it.