# LLL Evaluation Starting with the basic question, *"What is Evaluation ?"* Evaluation refers to the measurement of different properties in system's outputs. What sort of properties are we interested in measuring? * * * ### AI as a tool: We use AI as a tool to accomplish broad set of tasks/requests in a wide set of domains. For example, *drafting a mail, navigation, education*, *writing, coding, summarization*, etc. The important idea is: > **Don't just ask whether the AI's output looks good. Ask whether it actually helps the user accomplish their task.** This is called **user-focused evaluation**. **User-focused evaluation** - *How well does the system help the user accomplish the task the user is interested in?* *Start a Task -> Interact -> Finish Task -> Get Output -> Give Feedback* ### Example Suppose the user asks: > "Find me a restaurant near Tokyo." An answer such as: > "Restaurants are places where people eat food." is grammatically perfect. But it is **useless**. Therefore: **Good language ≠ good system.** The real question is: > **Did the system help accomplish the intended task?** Evaluation can be based on two things: * **Output Quality**: * Following Instructions: Strictly adhere to user's constraints, guidelines, requirements. * Coherence: Logical consistency, clarity, flow and grammatical correctness of the generated text. * Factuality: Truthfulness/Correctness/Accuracy of the information provided by the model. * **System Performance**: * Latency: Time delay between sending a prompt and receiving a response back. Low latency is preferred. * Pricing: Financial costs incurred per API request or token-based costs (input vs. output tokens). * Reliability: Stability, availability and consistency of system under operational workloads i.e. avoiding errors/failures. * * * ### Why is LLM Evaluation Hard ? LLM generates free-form text which is difficult to evaluate. Traditional ML often has a relatively simple output. For example: Classification Task Input: > Image of a cat Expected: > Cat Model: > Cat Easy to compare. But an LLM can generate many different valid answers. For: > "Why is exercise important?" Answer A: > Exercise improves physical fitness and can benefit overall health. Answer B: > Regular physical activity can improve cardiovascular health, strengthen muscles, and support mental well-being. Both may be correct. So what exactly is the **correct answer**? ### Human Evaluation The obvious solution: ask humans. If evaluating an answer automatically is difficult, we could ask humans: "I*s this answer good?*" For example: | Response | Human judgement | | --- | --- | | A | Good | | B | Bad | This seems reasonable. But there are problems. * **Subjectivity -** Different people can judge the same response differently. Suppose: > User: Explain photosynthesis. Response: > "Plants use sunlight to make food." One person might say: > Good enough. Another might say: > Too simplistic. Another might say: > It doesn't mention carbon dioxide or chlorophyll. So evaluation becomes **subjective**. Metric: "*How much better is our actual agreement compared to what we would get purely by chance ?*" $$\kappa = \frac{p_o - p_e}{1 - p_e}$$ po - observed agreement; pe = Expected agreement * **Human evaluation is slow** - If we have: **1 million outputs** and want humans to inspect every answer, that takes a huge amount of time. * **Human evaluation is expensive -** Human annotators need to be paid. So we want ***Automatic evaluation*** given the slow and expensive nature of human evaluation. * * * ### Rule-Based Metrics The basic idea is: Write labels/reference answers once, and use them as references for future automatic evaluation. Instead of asking a human every time: ```plaintext Human → Is this answer good? ``` we create a reference answer: ```plaintext Reference answer ↓ Metric ↓ Model answer ↓ Score ``` Common Rule-based metrics: * **METEOR - Metric for Evaluation of Translation with Explicit ORdering** It was introduced for machine translation evaluation. The basic idea is: Compare the generated sentence with a reference sentence. METEOR considers things such as: Precision, Recall, Matching words, Word ordering, Penalties for fragmented matches A simplified intuition is: $$METEOR \approx F_{\text{mean}}\times(1-\text{penalty})$$ where $$Fmean = \frac{P R}{\alpha P +(1-\alpha)R}$$ $$p = \gamma \cdot \left(\frac{c}{\mu_m}\right)^\beta$$ and a penalty *p* is applied when matching words are poorly ordered. Suppose reference: *The cat is sitting on the mat.* Prediction: *The cat is sitting on the mat.* Excellent match. But: *Mat the on sitting cat is the.* It has many of the same words but terrible ordering. METEOR tries to account for this. * **BLEU** - *BiLingual Evaluation Understudy*. It is another automatic metric originally designed primarily for machine translation. The central idea is: *"How many n-grams in the generated sentence match the reference?".* An n-gram is a sequence of n words. For: "The cat is sleeping" Unigrams - Individual words: The cat is sleeping Bigrams - Two-word sequences: The cat cat is is sleeping Trigrams - The cat is cat is sleeping And so on. Suppose reference: The cat is sleeping. Prediction: The cat is sleeping. There is high n-gram overlap. Therefore: BLEU → high If prediction: A vehicle is moving rapidly. there may be little overlap. Therefore: BLEU → low formula contains a brevity penalty (BP). The purpose is important. Imagine reference: The cat is sleeping on the sofa. Prediction: The cat. There are matching words. But the prediction is extremely short. Without a penalty, the model could generate tiny answers and still receive reasonable overlap scores. So BLEU penalizes excessively short outputs. $$\text{BLEU} = BP \times \exp\left( \frac{1}{N} \sum_{n=1}^{N} \log p_n \right)$$ $$BP = \min \left( 1, \exp\left(1 - \frac{n_{\text{ref}}}{n_{\text{pred}}}\right) \right)$$ $$p_n = \frac{\sum\limits_{\text{n-gram} \in \hat{y}} \max\limits_{k \in [1,m]} \Big[ \text{count}\big(\text{matching n-gram between } \hat{y} \text{ and } y_k\big) \Big]} {\sum\limits_{\text{n-gram} \in \hat{y}} \text{count}(\text{n-gram})}$$ where pn represents modified n-gram precision. * **ROUGE -** Recall-Oriented Understudy for Gisting Evaluation It is especially common in **summarization**. The important variants are: ROUGE-N, ROUGE-L, ROUGE-S / ROUGE-SU, ROUGE-W. **ROUGE-N** measures: ***n-gram overlap*** For example, ROUGE-1: unigram overlap ROUGE-2: bigram overlap So: ```plaintext ROUGE-1 → words ROUGE-2 → two-word sequences ROUGE-3 → three-word sequences ``` **ROUGE-L** uses: ***Longest Common Subsequence (LCS)*** This is important. A subsequence does **not** need to be contiguous. Example Reference: ```plaintext I like machine learning ``` Prediction: ```plaintext I really like deep machine learning ``` Some words occur in the same order even though other words appear between them. ROUGE-L can capture this structure. ROUGE-L captures sentence-level structure and word order without requiring contiguous matches, making it useful when paraphrasing is allowed. **ROUGE-S / ROUGE-SU use** Skip-bigrams. A skip-bigram considers two words that appear in the same order but don't necessarily have to be next to each other. This helps when a summary changes the wording or ordering somewhat. **ROGUE-W -** Weighted Longest Common Subsequence The important idea: Consecutive matches receive greater weight * * * PROBLEM WITH RULE-BASED METRICS Rule-based metrics are useful, but they don't understand language deeply. 1. They don't properly account for stylistic variation. 2. Correlation with human ratings isn't always very strong. 3. Human ratings are still needed. Reference: *The automobile rapidly moved down the street.* Prediction: *The car quickly drove along the road.* A human might say: *Excellent paraphrase.* But a word-overlap metric may say: *Not very similar.* Why? Because: ```plaintext automobile ≠ car rapidly ≠ quickly moved ≠ drove street ≠ road ``` So: ***Semantic similarity can be high while lexical overlap is low.*** This is one of the major reasons we need better evaluation methods. * * * ### LLM-AS-A-JUDGE User prompt : *Explain why the sky is blue.* Model response: *The sky appears blue because shorter wavelengths of sunlight are scattered more strongly by Earth's atmosphere.* We send both to another LLM: ```plaintext Judge LLM Prompt + Model response ↓ Evaluate ↓ Score + rationale ``` The judge is asked to evaluate relevance. Conceptually: ```plaintext Evaluate how relevant the model's answer is to the user's prompt. Prompt: {prompt} Model Response: {model_response} Return: - Rationale - Score uses a binary score: 1 = mostly relevant, 0 = mostly irrelevant and asks for a short rationale. ``` * Advantage 1 — No reference answer required. Traditional metrics often need: ```plaintext Model output + Reference answer ``` But LLM-as-a-Judge can potentially evaluate: ```plaintext Prompt + Model output ``` without a manually written reference i.e. *No need for reference / label* * Advantage 2 — Interpretability through rationales. Instead of merely saying: ```plaintext Score = 0.6 ``` the judge can say: *The answer addresses the question correctly but does not explain the mechanism in sufficient detail.* So we get: ```plaintext Score + Reason ``` This gives us some interpretability. ### Why enforce output format? Suppose your evaluation pipeline expects: ```plaintext { "score": 1, "rationale": "..." } ``` but the LLM returns: *I think the answer is mostly correct, perhaps 0.8, although there are some issues...* A human can understand it, but a program may not. Therefore, we need **structured outputs**. The idea is: ```plaintext LLM ↓ Strict schema ↓ Machine-readable result ``` This is particularly important when the judge is part of an automated evaluation pipeline. ### DIFFERENT TYPES OF LLM-AS-A-JUDGE * Pointwise evaluation : One answer is evaluated independently. ```plaintext Prompt ↓ Response ↓ Judge ↓ Score ``` Example: *Rate this response from 0 to 1 for relevance.* * Pairwise evaluation: Two answers are compared. ```plaintext Prompt ↓ Response A Response B ↓ Judge ↓ A better / B better / tie ``` Example: *Which answer is better?* This is useful when comparing two LLMs. * Listwise evaluation: Multiple responses are presented simultaneously and ranked. For example: A B C D Judge: C > A > D > B * * * ## LAAJ TAXONOMY LLM-as-a-Judge along four dimensions: 1. **Functionality** 2. **Methodology** 3. **Applications** 4. **Meta-evaluation** ### Functionality **A. Performance evaluation** The judge evaluates model outputs. Examples: * Accuracy * Relevance * Coherence * Fluency It can evaluate general responses or dynamic tasks such as: * Coding * Mathematics * Reasoning **B. Model enhancement** The judge isn't merely measuring the model. It can help **improve** the model. Examples shown include: * Reward modeling * Acting as a verifier * Providing feedback for refinement **C. Data construction** The judge can help create datasets. Examples: * Annotating unlabeled data * Generating/synthesizing data * Evaluating generated examples ### Methodology There are different ways of constructing the judging system. **Single-LLM** ```plaintext Input ↓ One judge LLM ↓ Evaluation ``` **Multi-LLM** Several LLMs participate in the evaluation. ```plaintext LLM 1 ──┐ LLM 2 ──┼→ judgement LLM 3 ──┘ ``` The idea can be similar to getting multiple opinions. **Human-AI collaboration** ```plaintext LLM judgement + Human judgement ↓ Final evaluation ``` This can combine the scalability of LLMs with human oversight. ### Applications LLM-as-a-Judge can be used across many domains. **General** * Dialogue generation * Open-ended QA * Summarization * Translation **Medical** * Medical text generation * Medical QA * Medical reasoning * Mental-health-related evaluation **Financial** * Investment risk * Credit scoring **Information retrieval** * Relevance * Ranking * Explanation * RAG systems **Multimodal** * Images * Audio * Video * Real-world scenarios **Legal** * Law LLM performance * Relevance judgement **Education** * Assignment grading * Essay scoring * Mathematical reasoning * Debate performance **Software engineering** * Code quality * Bug reports ### Meta-evaluation Suppose: LLM-A is the judge. How do we know whether **LLM-A is a good judge?** We need to evaluate the evaluator. That is: **Meta-evaluation = evaluating the evaluation system itself.** For example: ```plaintext Human judgement ↓ Compare ↑ LLM judge ``` We can measure how well the LLM judge agrees with humans. The meta-evaluation taxonomy includes benchmark-based evaluation and statistical measures such as accuracy, Pearson, Spearman, Kendall's Tau, Cohen's Kappa, and ICC. * * * ## BIASES IN LLM-AS-A-JUDGE LLM-as-a-Judge sounds great: Model → Judge LLM → Score But the judge itself can be biased. 1. Position bias 2. Verbosity bias 3. Self-enhancement bias ### Position bias Suppose we ask: Which is better? Response A vs. Response B The judge might prefer A simply because it appears first. Then we reverse them: Response B vs. Response A and the judge might now prefer B. That is: Position bias Why is this bad? The judgement should depend on: quality not position Remedy: Take the average For example: A vs B → A wins B vs A → B wins Then average the results. Alternate solution: modifying/tweaking position embeddings. ### Verbosity bias Suppose: Response A : The answer is 4. Response B : Let's carefully examine the problem. First we consider... \[lots of explanation\] ... Therefore, the final answer is 4. Both give the same answer. But the judge might think: B must be better because it is more detailed. That is verbosity bias. Important point : More words do not necessarily mean better answer. An answer can be: Correct, Concise, Relevant and therefore better than a long, distracting answer. Remedies: Explicit guidelines, Few-shot examples, Penalty based on output length ### Self-enhancement bias Suppose an LLM judges: Human answer vs AI-generated answer It might systematically prefer the AI-generated answer. That is: Self-enhancement bias The important academic takeaway isn't literally "never use LLM judges." It is: Do not blindly trust an LLM judge. * * * ## HOW TO MAKE LLM-AS-A-JUDGE BETTER * Use crisp guidelines Don't tell the judge: "Judge whether this is good." Instead: "Judge whether the answer directly addresses the question, contains correct information, and avoids irrelevant content." More precise criteria → more consistent evaluation. * Prefer binary scales when appropriate Instead of: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 sometimes use: 0 = bad 1 = good Why? Because very fine-grained scores can be difficult for judges to apply consistently. * Ask for rationale before score This is an important evaluation trick: First: Explain why. Then: Give score. Instead of: Score = 1 Reason = ... This encourages the judge to explicitly consider the evidence before committing to a score. Write rationale before outputting score. * Mitigate biases We need to explicitly consider: Position bias, Verbosity bias, Self-enhancement bias, Other possible biases * Calibrate with human judgements We shouldn't simply assume: LLM judge = correct judge. Instead: ```plaintext Human evaluations ↓ Compare ↑ LLM evaluations ``` If the two disagree frequently, investigate. * * * ### WHAT EXACTLY SHOULD WE EVALUATE? * Task performance: * Usefulness (Does the response actually help?) * Factuality (Is the information factually correct?) * Relevance (Does it answer the question?) * Alignment: * Tone (Does it use an appropriate tone?) * Style (Does it follow the desired style?) * Safety (Does it behave safely?) * Factuality: The objective is to quantify how factual an output is. Task performance + Alignment -> Both matter ### Why factuality is difficult? Consider this answer: > Teddy bears, first created in the 1920s, were named after President Theodore Roosevelt after he proudly wanted to shoot a captured bear on a hunting trip. There are several individual claims inside one sentence. The question is: > Is the **whole sentence** correct? That's not necessarily the best way to evaluate it. Instead, we can **decompose it into individual facts**. Fact decomposition: Break the response into claims: Fact 1 > Teddy bears were first created in the 1920s. Fact 2 > Teddy bears were named after President Theodore Roosevelt. Fact 3 > Theodore Roosevelt was on a hunting trip where a bear was captured. Fact 4 > Theodore Roosevelt proudly wanted to shoot the captured bear. Now evaluate each fact separately. This is much more fine-grained. | Fact | Correct? | Importance | | --- | --- | --- | | 1 | No | 0.3 | | 2 | Yes | 0.4 | | 3 | Yes | 0.2 | | 4 | No | 0.1 | $$Score = \sum_{i=1}^{n}\alpha_i \times score_i$$ where: * α\_i = importance of fact (i) * score\_i = whether/how well fact (i) is supported Because the correct facts have importance: 0.4 + 0.2 = 0.6, overall score is 0.6 ### Why fact decomposition is useful? Imagine an LLM produces 20 factual claims. Suppose: ```plaintext 18 correct 2 incorrect ``` If we simply label the entire answer: > "Incorrect" we lose information. Fact decomposition tells us: ```plaintext Claim 1 → correct Claim 2 → correct Claim 3 → incorrect ... ``` So we can quantify **how much of the response is factual**. * * * ### What is an agent? An agent typically follows a loop: ```plaintext Observe ↓ Plan ↓ Act ↓ Observe again ↓ Plan ↓ Act ↓ ... ``` **Input → Observe → Plan → Act → Observe → ... → Output** A normal LLM can be thought of as: ```plaintext Input ↓ LLM ↓ Output ``` An agent can interact with the outside world. User: > Find the weather in Delhi and tell me whether I need an umbrella. Agent: Step 1 — Observe Understand the request. Step 2 — Plan Need weather information. Step 3 — Act Call weather tool. Step 4 — Observe Receive: > Rain expected. Step 5 — Plan Recommend carrying an umbrella. Step 6 — Output > Yes, you should probably carry an umbrella. So now evaluation becomes more complicated. We aren't evaluating only: > final answer We need to evaluate the **whole interaction process**. * * * ### AGENT FAILURE MODES: There are essentially failures at multiple stages: ```plaintext User ↓ LLM ↓ Tool selection ↓ Tool call ↓ Tool output ↓ LLM ↓ Final response ``` An error can occur anywhere. ### Failure 1 — Tool prediction error The LLM predicts the wrong tool. User: > Find a bear near me. Available tool: ```plaintext find_teddy_bear() ``` But the model invents or selects the wrong function. The model failed to determine: > **Which tool should I use?** ### Failure 2 — Doesn't use the tool Suppose the user asks: > Find a bear near me. The LLM simply says: > Sorry, I don't know where I can find one. without calling the available tool. Possible cause: > The model doesn't know how to use the tool. ### Failure 3 — Hallucinates a tool Now suppose the available tools are: ```plaintext find_restaurant() find_weather() ``` but the LLM generates: ```plaintext find_teddy_bear() ``` even though that tool doesn't exist. Unclear instructions are one possible cause ### Failure 4 — Uses the wrong tool Maybe the system has: ```plaintext find_restaurant() send_message() find_weather() ``` The user asks: > Find a bear near me. But the model calls: ```plaintext send_message() ``` The tool **exists**, but it is the **wrong tool**. ### Failure 5 — Tool infers wrong argument Suppose the correct tool is: ```plaintext find_teddy_bear(location) ``` The user is somewhere at: ```plaintext (37.42, -122.17) ``` but the model passes: ```plaintext location = (0, -0) ``` The tool itself is correct but the argument is wrong. ### TOOL CALL ERROR: Suppose: ```plaintext LLM ↓ find_teddy_bear(location) ↓ Backend ↓ ERROR ``` Maybe the backend returns an error instead of valid data. Not every failure is the LLM's fault. The tool/backend itself can fail. ### Failure 6 — Wrong tool response ```plaintext LLM ↓ Correct tool ↓ Backend ↓ Incorrect result ``` The tool may: * Return an incorrect value * Return an error instead of a value ### Failure 7 — No response Suppose the tool is called correctly but returns nothing. ```plaintext Tool ↓ nothing ``` This can sometimes be a bug or sometimes a characteristic/"feature" of the tool implementation. An important consequence is that the final LLM may then hallucinate an answer because it has insufficient information. ### Failure 8 — Wrong final response Suppose the tool correctly returns: ```plaintext { "name": "Teddy", ... } ``` but the LLM says: > I didn't find any bear. The tool's information was correct. The LLM's interpretation/output was wrong. ### Failure 9 — Final response does not convey tool response Suppose the tool returns: ```plaintext { "name": "Teddy", ... } ``` The LLM needs to communicate that information. Instead it says: Didn't find any bear! So the model may technically receive the data but fail to properly understand or ground itself in it. * * * THE THREE MAJOR AGENT FAILURE STAGES This is an excellent way to remember the entire section. Stage 1 — Tool selection Which tool should I use? Failures: * Doesn't use tool * Hallucinates tool * Uses wrong tool * Tool prediction error Stage 2 — Tool execution Did the tool call work? Failures: * Wrong argument * Tool call error * Wrong tool response * No response Stage 3 — Response generation Did the LLM correctly interpret and communicate the tool result? Failures: * Wrong response * Hallucinated response * Doesn't convey tool result ***Overall debugging perspective:*** Modeling problems: * Weak reasoning * Weak grounding * Too much information in the context window * Incorrect tool modelling Tool problems: * Tool itself has a problem * Tool output isn't interpretable *Therefore: Debugging an agent requires special care and patience because the failure can occur at multiple stages.* * * * ### BENCHMARKS: A **benchmark** is a standardized evaluation setup. Instead of saying: > "This model seems good." we create: ```plaintext Dataset + Tasks + Evaluation criterion = Benchmark ``` Why do we need benchmarks? Suppose: Model A Accuracy = 80% Model B Accuracy = 85% We can say: B performed better. But only if: * Same task * Same dataset * Same evaluation method Therefore, benchmarks make model comparisons more standardized. | Benchmark | What it evaluates | Evaluation | | --- | --- | --- | | **MMLU** | Broad knowledge | Multiple-choice accuracy | | **AIME** | Mathematical reasoning | Exact answer | | **SWE-bench** | Software engineering | Tests pass | | **HarmBench** | Safety | Attack success rate | MMLU measures a particular capability (knowledge and understanding) Different benchmarks measure different capabilities. Modern LLM systems aren't necessarily text-only. They can process: ```plaintext Text + Images + Other modalities ``` Therefore, safety evaluation also needs to consider multimodal inputs. $$ASR = \frac{\text{successful attacks}} {\text{total attacks}}$$ For a safety benchmark, **lower ASR is generally better**, because successful attacks indicate that the model was induced to produce harmful behavior. * * * SUMMARY ```plaintext LLM EVALUATION │ ┌─────────────────┴─────────────────┐ │ │ WHAT TO EVALUATE HOW TO EVALUATE │ │ ┌─────┴──────┐ ┌─────────┼─────────┐ │ │ │ │ │ Task Performance Alignment Humans Rule-based LLM Judge │ │ │ │ │ │ │ │ BLEU Biases │ │ │ ROUGE Position │ │ │ METEOR Verbosity │ │ │ Self-enhancement │ │ │ ├─ Usefulness │ ├─ Factuality │ └─ Relevance │ AGENT EVALUATION │ ┌──────────┴──────────┐ │ │ Tool stage Response stage │ │ Wrong tool Wrong response No tool No response Fake tool Hallucination Wrong argument Tool error BENCHMARKS │ ┌────────────────┼────────────────┐ │ │ │ MMLU AIME SWE-bench Knowledge Math reasoning Coding HarmBench Safety ```