Evaluating Foundation Models
Proving an FM works: evaluation approaches, metrics like ROUGE and BLEU, evaluating RAG and agent applications, and business alignment.
You have picked a model, grounded it in your data, shaped its behavior with prompts, and maybe fine-tuned it. None of that tells you whether the result is good. This topic closes the largest exam domain with the question that decides whether anything you built should ship: how do you know, with evidence rather than a demo?
The answer has layers. Scores tell you whether the text is close to something acceptable. Judges and human reviewers tell you whether it is genuinely useful. Stage-level metrics tell you which part of a RAG or agent pipeline broke. And business signals tell you whether any of it changed the work people do.
What This Topic Covers
- the three approaches to evaluation: automatic scoring against a dataset, an LLM acting as a judge, and human reviewers, with the task types, built-in datasets, and rating methods each one uses
- benchmark datasets such as MMLU and HELM, and why a leaderboard rank is not a verdict on your use case
- ROUGE, BLEU, and BERTScore worked with real sentences and real numbers, plus the metric AWS actually computes for each evaluation task type
- RAG evaluation in Amazon Bedrock: retrieve-only metrics, retrieve-and-generate metrics, and the boundary between faithfulness and correctness
- agent evaluation with AgentCore Evaluations, including goal success rate and tool selection and parameter accuracy
- judging whether a model meets a business objective through productivity, user engagement, and task engineering, plus acceptance thresholds, ground truth datasets, user feedback, and drift
Why It Matters
Evaluation is where exam questions stop being definitional. You will be handed a scenario with no ground truth, or a system that scores well and behaves badly, and asked which approach or metric fits. Getting those right depends on knowing what each measurement can structurally prove.
The same skill decides real outcomes. Teams that cannot measure their generative AI application cannot defend it, improve it, or safely change the model underneath it. Everything in this topic exists to replace "it looked good when I tried it" with something that survives a review.