LLL Evaluation

Starting with the basic question, "What is Evaluation ?"
Evaluation refers to the measurement of different properties in system's outputs.

What sort of properties are we interested in measuring?

---

AI as a tool:

We use AI as a tool to accomplish broad set of tasks/requests in a wide set of domains. For example, drafting a mail, navigation, education, writing, coding, summarization, etc.

The important idea is:

Don't just ask whether the AI's output looks good. Ask whether it actually helps the user accomplish their task.

This is called user-focused evaluation.

User-focused evaluation - How well does the system help the user accomplish the task the user is interested in?

Start a Task -> Interact -> Finish Task -> Get Output -> Give Feedback

Example

Suppose the user asks:

"Find me a restaurant near Tokyo."

An answer such as:

"Restaurants are places where people eat food."

is grammatically perfect.

But it is useless.

Therefore:

Good language ≠ good system.

The real question is:

Did the system help accomplish the intended task?

Evaluation can be based on two things:

• Output Quality:

  • Following Instructions: Strictly adhere to user's constraints, guidelines, requirements.

  • Coherence: Logical consistency, clarity, flow and grammatical correctness of the generated text.

  • Factuality: Truthfulness/Correctness/Accuracy of the information provided by the model.

• System Performance:

  • Latency: Time delay between sending a prompt and receiving a response back. Low latency is preferred.

  • Pricing: Financial costs incurred per API request or token-based costs (input vs. output tokens).

  • Reliability: Stability, availability and consistency of system under operational workloads i.e. avoiding errors/failures.

---

Why is LLM Evaluation Hard ?

LLM generates free-form text which is difficult to evaluate.

Traditional ML often has a relatively simple output.

For example: Classification Task

Input:

Image of a cat

Expected:

Cat

Model:

Cat

Easy to compare.

But an LLM can generate many different valid answers.

For:

"Why is exercise important?"

Answer A:

Exercise improves physical fitness and can benefit overall health.

Answer B:

Regular physical activity can improve cardiovascular health, strengthen muscles, and support mental well-being.

Both may be correct.

So what exactly is the correct answer?

Human Evaluation

The obvious solution: ask humans. If evaluating an answer automatically is difficult, we could ask humans: "Is this answer good?"

For example:

Response	Human judgement
A	Good
B	Bad

This seems reasonable. But there are problems.

• Subjectivity - Different people can judge the same response differently. Suppose:

  User: Explain photosynthesis.

  Response:

  "Plants use sunlight to make food."

  One person might say:

  Good enough.

  Another might say:

  Too simplistic.

  Another might say:

  It doesn't mention carbon dioxide or chlorophyll.

  So evaluation becomes subjective. Metric: "How much better is our actual agreement compared to what we would get purely by chance ?"

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

po - observed agreement; pe = Expected agreement

• Human evaluation is slow - If we have: 1 million outputs and want humans to inspect every answer, that takes a huge amount of time.

• Human evaluation is expensive - Human annotators need to be paid.

So we want Automatic evaluation given the slow and expensive nature of human evaluation.

---

Rule-Based Metrics

The basic idea is: Write labels/reference answers once, and use them as references for future automatic evaluation.

Instead of asking a human every time:

Human → Is this answer good?

we create a reference answer:

Reference answer
       ↓
    Metric
       ↓
 Model answer
       ↓
     Score

Common Rule-based metrics:

• METEOR - Metric for Evaluation of Translation with Explicit ORdering

  It was introduced for machine translation evaluation. The basic idea is: Compare the generated sentence with a reference sentence.

  METEOR considers things such as: Precision, Recall, Matching words, Word ordering, Penalties for fragmented matches

  A simplified intuition is:

  $$METEOR \approx F_{\text{mean}}\times(1-\text{penalty})$$

  where

  $$Fmean = \frac{P R}{\alpha P +(1-\alpha)R}$$

  $$p = \gamma \cdot \left(\frac{c}{\mu_m}\right)^\beta$$

  and a penalty p is applied when matching words are poorly ordered.

  Suppose reference: The cat is sitting on the mat.

  Prediction: The cat is sitting on the mat.

  Excellent match. But: Mat the on sitting cat is the.

  It has many of the same words but terrible ordering. METEOR tries to account for this.

• BLEU - BiLingual Evaluation Understudy. It is another automatic metric originally designed primarily for machine translation. The central idea is: "How many n-grams in the generated sentence match the reference?". An n-gram is a sequence of n words.

  For: "The cat is sleeping"

  Unigrams - Individual words: The cat is sleeping

  Bigrams - Two-word sequences: The cat cat is is sleeping

  Trigrams - The cat is cat is sleeping

  And so on.

  Suppose reference: The cat is sleeping.

  Prediction: The cat is sleeping.

  There is high n-gram overlap. Therefore: BLEU → high

  If prediction: A vehicle is moving rapidly.

  there may be little overlap. Therefore: BLEU → low

  formula contains a brevity penalty (BP). The purpose is important.

  Imagine reference: The cat is sleeping on the sofa.

  Prediction: The cat.

  There are matching words. But the prediction is extremely short.

  Without a penalty, the model could generate tiny answers and still receive reasonable overlap scores. So BLEU penalizes excessively short outputs.

  $$\text{BLEU} = BP \times \exp\left( \frac{1}{N} \sum_{n=1}^{N} \log p_n \right)$$

  $$BP = \min \left( 1, \exp\left(1 - \frac{n_{\text{ref}}}{n_{\text{pred}}}\right) \right)$$

  $$p_n = \frac{\sum\limits_{\text{n-gram} \in \hat{y}} \max\limits_{k \in [1,m]} \Big[ \text{count}\big(\text{matching n-gram between } \hat{y} \text{ and } y_k\big) \Big]} {\sum\limits_{\text{n-gram} \in \hat{y}} \text{count}(\text{n-gram})}$$

  where pn represents modified n-gram precision.

• ROUGE - Recall-Oriented Understudy for Gisting Evaluation

  It is especially common in summarization. The important variants are: ROUGE-N, ROUGE-L, ROUGE-S / ROUGE-SU, ROUGE-W.

  ROUGE-N measures: n-gram overlap

  For example, ROUGE-1: unigram overlap

  ROUGE-2: bigram overlap

  So:

  ROUGE-1 → words
  ROUGE-2 → two-word sequences
  ROUGE-3 → three-word sequences
  

  ROUGE-L uses: Longest Common Subsequence (LCS)

  This is important. A subsequence does not need to be contiguous.

  Example Reference:

  I like machine learning
  

  Prediction:

  I really like deep machine learning
  

  Some words occur in the same order even though other words appear between them. ROUGE-L can capture this structure.

  ROUGE-L captures sentence-level structure and word order without requiring contiguous matches, making it useful when paraphrasing is allowed.

  ROUGE-S / ROUGE-SU use Skip-bigrams.

  A skip-bigram considers two words that appear in the same order but don't necessarily have to be next to each other. This helps when a summary changes the wording or ordering somewhat.

  ROGUE-W - Weighted Longest Common Subsequence

  The important idea: Consecutive matches receive greater weight

---

PROBLEM WITH RULE-BASED METRICS

Rule-based metrics are useful, but they don't understand language deeply.

1. They don't properly account for stylistic variation.

2. Correlation with human ratings isn't always very strong.

3. Human ratings are still needed.

Reference: The automobile rapidly moved down the street.

Prediction: The car quickly drove along the road.

A human might say: Excellent paraphrase.

But a word-overlap metric may say: Not very similar.

Why?

Because:

automobile ≠ car
rapidly ≠ quickly
moved ≠ drove
street ≠ road

So: Semantic similarity can be high while lexical overlap is low.

This is one of the major reasons we need better evaluation methods.

---

LLM-AS-A-JUDGE

User prompt : Explain why the sky is blue.

Model response: The sky appears blue because shorter wavelengths of sunlight are scattered more strongly by Earth's atmosphere.

We send both to another LLM:

Judge LLM

Prompt
   +
Model response
   ↓
Evaluate
   ↓
Score + rationale

The judge is asked to evaluate relevance.

Conceptually:

Evaluate how relevant the model's answer is to the user's prompt.

Prompt: {prompt}

Model Response: {model_response}

Return:
    - Rationale
    - Score

uses a binary score: 1 = mostly relevant, 0 = mostly irrelevant

and asks for a short rationale.

• Advantage 1 — No reference answer required.

  Traditional metrics often need:

  Model output
  +
  Reference answer
  

  But LLM-as-a-Judge can potentially evaluate:

  Prompt
  +
  Model output
  

  without a manually written reference i.e. No need for reference / label

• Advantage 2 — Interpretability through rationales.

  Instead of merely saying:

  Score = 0.6
  

  the judge can say: The answer addresses the question correctly but does not explain the mechanism in sufficient detail.

  So we get:

  Score
  +
  Reason
  

  This gives us some interpretability.

Why enforce output format?

Suppose your evaluation pipeline expects:

{
  "score": 1,
  "rationale": "..."
}

but the LLM returns: I think the answer is mostly correct, perhaps 0.8, although there are some issues...

A human can understand it, but a program may not. Therefore, we need structured outputs. The idea is:

LLM
 ↓
Strict schema
 ↓
Machine-readable result

This is particularly important when the judge is part of an automated evaluation pipeline.

DIFFERENT TYPES OF LLM-AS-A-JUDGE

• Pointwise evaluation : One answer is evaluated independently.

  Prompt 
    ↓ 
  Response 
    ↓ 
   Judge
    ↓
   Score
  

  Example: Rate this response from 0 to 1 for relevance.

• Pairwise evaluation: Two answers are compared.

  Prompt 
    ↓ 
  Response A 
  Response B 
    ↓ 
  Judge
    ↓ 
  A better / B better / tie
  

  Example: Which answer is better?

  This is useful when comparing two LLMs.

• Listwise evaluation: Multiple responses are presented simultaneously and ranked.

  For example: A B C D

  Judge: C > A > D > B

---

LAAJ TAXONOMY

LLM-as-a-Judge along four dimensions:

1. Functionality

2. Methodology

3. Applications

4. Meta-evaluation

Functionality

A. Performance evaluation

The judge evaluates model outputs.

Examples:

• Accuracy

• Relevance

• Coherence

• Fluency

It can evaluate general responses or dynamic tasks such as:

• Coding

• Mathematics

• Reasoning

B. Model enhancement

The judge isn't merely measuring the model. It can help improve the model.

Examples shown include:

• Reward modeling

• Acting as a verifier

• Providing feedback for refinement

C. Data construction

The judge can help create datasets.

Examples:

• Annotating unlabeled data

• Generating/synthesizing data

• Evaluating generated examples

Methodology

There are different ways of constructing the judging system.

Single-LLM

Input
 ↓
One judge LLM
 ↓
Evaluation

Multi-LLM

Several LLMs participate in the evaluation.

LLM 1 ──┐
LLM 2 ──┼→ judgement
LLM 3 ──┘

The idea can be similar to getting multiple opinions.

Human-AI collaboration

LLM judgement
      +
Human judgement
      ↓
Final evaluation

This can combine the scalability of LLMs with human oversight.

Applications

LLM-as-a-Judge can be used across many domains.

General

• Dialogue generation

• Open-ended QA

• Summarization

• Translation

Medical

• Medical text generation

• Medical QA

• Medical reasoning

• Mental-health-related evaluation

Financial

• Investment risk

• Credit scoring

Information retrieval

• Relevance

• Ranking

• Explanation

• RAG systems

Multimodal

• Images

• Audio

• Video

• Real-world scenarios

Legal

• Law LLM performance

• Relevance judgement

Education

• Assignment grading

• Essay scoring

• Mathematical reasoning

• Debate performance

Software engineering

• Code quality

• Bug reports

Meta-evaluation

Suppose: LLM-A is the judge.

How do we know whether LLM-A is a good judge?

We need to evaluate the evaluator.

That is: Meta-evaluation = evaluating the evaluation system itself.

For example:

Human judgement
       ↓
Compare
       ↑
LLM judge

We can measure how well the LLM judge agrees with humans.

The meta-evaluation taxonomy includes benchmark-based evaluation and statistical measures such as accuracy, Pearson, Spearman, Kendall's Tau, Cohen's Kappa, and ICC.

---

BIASES IN LLM-AS-A-JUDGE

LLM-as-a-Judge sounds great: Model → Judge LLM → Score

But the judge itself can be biased.

1. Position bias

2. Verbosity bias

3. Self-enhancement bias

Position bias

Suppose we ask: Which is better? Response A vs. Response B

The judge might prefer A simply because it appears first.

Then we reverse them: Response B vs. Response A

and the judge might now prefer B.

That is: Position bias

Why is this bad? The judgement should depend on: quality not position

Remedy: Take the average

For example:

A vs B → A wins

B vs A → B wins

Then average the results.

Alternate solution: modifying/tweaking position embeddings.

Verbosity bias

Suppose:

Response A : The answer is 4.

Response B : Let's carefully examine the problem. First we consider... [lots of explanation] ... Therefore, the final answer is 4.

Both give the same answer.

But the judge might think: B must be better because it is more detailed.

That is verbosity bias.

Important point : More words do not necessarily mean better answer.

An answer can be: Correct, Concise, Relevant

and therefore better than a long, distracting answer.

Remedies: Explicit guidelines, Few-shot examples, Penalty based on output length

Self-enhancement bias

Suppose an LLM judges: Human answer vs AI-generated answer

It might systematically prefer the AI-generated answer.

That is: Self-enhancement bias

The important academic takeaway isn't literally "never use LLM judges."

It is: Do not blindly trust an LLM judge.

---

HOW TO MAKE LLM-AS-A-JUDGE BETTER

• Use crisp guidelines

Don't tell the judge: "Judge whether this is good."

Instead: "Judge whether the answer directly addresses the question, contains correct information, and avoids irrelevant content."

More precise criteria → more consistent evaluation.

• Prefer binary scales when appropriate

Instead of: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

sometimes use: 0 = bad 1 = good

Why? Because very fine-grained scores can be difficult for judges to apply consistently.

• Ask for rationale before score

This is an important evaluation trick:

First: Explain why.

Then: Give score.

Instead of: Score = 1 Reason = ...

This encourages the judge to explicitly consider the evidence before committing to a score. Write rationale before outputting score.

• Mitigate biases

We need to explicitly consider: Position bias, Verbosity bias, Self-enhancement bias, Other possible biases

• Calibrate with human judgements

We shouldn't simply assume: LLM judge = correct judge.

Instead:

Human evaluations 
      ↓ 
   Compare
      ↑ 
LLM evaluations

If the two disagree frequently, investigate.

---

WHAT EXACTLY SHOULD WE EVALUATE?

• Task performance:

  • Usefulness (Does the response actually help?)

  • Factuality (Is the information factually correct?)

  • Relevance (Does it answer the question?)

• Alignment:

  • Tone (Does it use an appropriate tone?)

  • Style (Does it follow the desired style?)

  • Safety (Does it behave safely?)

• Factuality: The objective is to quantify how factual an output is.

Task performance + Alignment -> Both matter

Why factuality is difficult?

Consider this answer:

Teddy bears, first created in the 1920s, were named after President Theodore Roosevelt after he proudly wanted to shoot a captured bear on a hunting trip.

There are several individual claims inside one sentence.

The question is:

Is the whole sentence correct?

That's not necessarily the best way to evaluate it.

Instead, we can decompose it into individual facts.

Fact decomposition: Break the response into claims:

Fact 1

Teddy bears were first created in the 1920s.

Fact 2

Teddy bears were named after President Theodore Roosevelt.

Fact 3

Theodore Roosevelt was on a hunting trip where a bear was captured.

Fact 4

Theodore Roosevelt proudly wanted to shoot the captured bear.

Now evaluate each fact separately. This is much more fine-grained.

Fact	Correct?	Importance
1	No	0.3
2	Yes	0.4
3	Yes	0.2
4	No	0.1

$$Score = \sum_{i=1}^{n}\alpha_i \times score_i$$

where:

• α_i = importance of fact (i)

• score_i = whether/how well fact (i) is supported

Because the correct facts have importance: 0.4 + 0.2 = 0.6, overall score is 0.6

Why fact decomposition is useful?

Imagine an LLM produces 20 factual claims.

Suppose:

18 correct
2 incorrect

If we simply label the entire answer:

"Incorrect"

we lose information.

Fact decomposition tells us:

Claim 1 → correct
Claim 2 → correct
Claim 3 → incorrect
...

So we can quantify how much of the response is factual.

---

What is an agent?

An agent typically follows a loop:

Observe
   ↓
Plan
   ↓
Act
   ↓
Observe again
   ↓
Plan
   ↓
Act
   ↓
...

Input → Observe → Plan → Act → Observe → ... → Output

A normal LLM can be thought of as:

Input
 ↓
LLM
 ↓
Output

An agent can interact with the outside world.

User:

Find the weather in Delhi and tell me whether I need an umbrella.

Agent:

Step 1 — Observe

Understand the request.

Step 2 — Plan

Need weather information.

Step 3 — Act

Call weather tool.

Step 4 — Observe

Receive:

Rain expected.

Step 5 — Plan

Recommend carrying an umbrella.

Step 6 — Output

Yes, you should probably carry an umbrella.

So now evaluation becomes more complicated.

We aren't evaluating only:

final answer

We need to evaluate the whole interaction process.

---

AGENT FAILURE MODES:

There are essentially failures at multiple stages:

User
 ↓
LLM
 ↓
Tool selection
 ↓
Tool call
 ↓
Tool output
 ↓
LLM
 ↓
Final response

An error can occur anywhere.

Failure 1 — Tool prediction error

The LLM predicts the wrong tool.

User:

Find a bear near me.

Available tool:

find_teddy_bear()

But the model invents or selects the wrong function.

The model failed to determine:

Which tool should I use?

Failure 2 — Doesn't use the tool

Suppose the user asks:

Find a bear near me.

The LLM simply says:

Sorry, I don't know where I can find one.

without calling the available tool.

Possible cause:

The model doesn't know how to use the tool.

Failure 3 — Hallucinates a tool

Now suppose the available tools are:

find_restaurant()
find_weather()

but the LLM generates:

find_teddy_bear()

even though that tool doesn't exist.

Unclear instructions are one possible cause

Failure 4 — Uses the wrong tool

Maybe the system has:

find_restaurant()
send_message()
find_weather()

The user asks:

Find a bear near me.

But the model calls:

send_message()

The tool exists, but it is the wrong tool.

Failure 5 — Tool infers wrong argument

Suppose the correct tool is:

find_teddy_bear(location)

The user is somewhere at:

(37.42, -122.17)

but the model passes:

location = (0, -0)

The tool itself is correct but the argument is wrong.

TOOL CALL ERROR:

Suppose:

LLM
 ↓
find_teddy_bear(location)
 ↓
Backend
 ↓
ERROR

Maybe the backend returns an error instead of valid data.

Not every failure is the LLM's fault. The tool/backend itself can fail.

Failure 6 — Wrong tool response

LLM 
 ↓ 
Correct tool 
 ↓ 
Backend
 ↓ 
Incorrect result

The tool may:

• Return an incorrect value

• Return an error instead of a value

Failure 7 — No response

Suppose the tool is called correctly but returns nothing.

Tool
 ↓
nothing

This can sometimes be a bug or sometimes a characteristic/"feature" of the tool implementation.

An important consequence is that the final LLM may then hallucinate an answer because it has insufficient information.

Failure 8 — Wrong final response

Suppose the tool correctly returns:

{
  "name": "Teddy",
  ...
}

but the LLM says:

I didn't find any bear.

The tool's information was correct. The LLM's interpretation/output was wrong.

Failure 9 — Final response does not convey tool response

Suppose the tool returns:

{
    "name": "Teddy",
    ...
}

The LLM needs to communicate that information.

Instead it says: Didn't find any bear!

So the model may technically receive the data but fail to properly understand or ground itself in it.

---

THE THREE MAJOR AGENT FAILURE STAGES This is an excellent way to remember the entire section.

Stage 1 — Tool selection Which tool should I use?

Failures:

• Doesn't use tool

• Hallucinates tool

• Uses wrong tool

• Tool prediction error

Stage 2 — Tool execution Did the tool call work?

Failures:

• Wrong argument

• Tool call error

• Wrong tool response

• No response

Stage 3 — Response generation Did the LLM correctly interpret and communicate the tool result?

Failures:

• Wrong response

• Hallucinated response

• Doesn't convey tool result

Overall debugging perspective:

Modeling problems:

• Weak reasoning

• Weak grounding

• Too much information in the context window

• Incorrect tool modelling

Tool problems:

• Tool itself has a problem

• Tool output isn't interpretable

Therefore: Debugging an agent requires special care and patience because the failure can occur at multiple stages.

---

BENCHMARKS:

A benchmark is a standardized evaluation setup.

Instead of saying:

"This model seems good."

we create:

Dataset
 +
Tasks
 +
Evaluation criterion
 =
Benchmark

Why do we need benchmarks? Suppose: Model A Accuracy = 80% Model B Accuracy = 85% We can say: B performed better.

But only if:

• Same task

• Same dataset

• Same evaluation method Therefore, benchmarks make model comparisons more standardized.

Benchmark	What it evaluates	Evaluation
MMLU	Broad knowledge	Multiple-choice accuracy
AIME	Mathematical reasoning	Exact answer
SWE-bench	Software engineering	Tests pass
HarmBench	Safety	Attack success rate

MMLU measures a particular capability (knowledge and understanding)

Different benchmarks measure different capabilities.

Modern LLM systems aren't necessarily text-only.

They can process:

Text
+
Images
+
Other modalities

Therefore, safety evaluation also needs to consider multimodal inputs.

$$ASR = \frac{\text{successful attacks}} {\text{total attacks}}$$

For a safety benchmark, lower ASR is generally better, because successful attacks indicate that the model was induced to produce harmful behavior.

---

SUMMARY

                         LLM EVALUATION
                               │
             ┌─────────────────┴─────────────────┐
             │                                   │
        WHAT TO EVALUATE                    HOW TO EVALUATE
             │                                   │
       ┌─────┴──────┐                  ┌─────────┼─────────┐
       │            │                  │         │         │
Task Performance  Alignment         Humans   Rule-based  LLM Judge
       │            │                  │         │         │
       │            │                  │      BLEU      Biases
       │            │                  │      ROUGE     Position
       │            │                  │      METEOR    Verbosity
       │            │                  │                Self-enhancement
       │            │                  │
       ├─ Usefulness                   │
       ├─ Factuality                    │
       └─ Relevance                     │
                                     
                         AGENT EVALUATION
                               │
                    ┌──────────┴──────────┐
                    │                     │
                 Tool stage          Response stage
                    │                     │
             Wrong tool              Wrong response
             No tool                  No response
             Fake tool               Hallucination
             Wrong argument
             Tool error
             
                         BENCHMARKS
                              │
             ┌────────────────┼────────────────┐
             │                │                │
           MMLU             AIME          SWE-bench
         Knowledge       Math reasoning      Coding
                             
                         HarmBench
                            Safety