AWS Certified AI Practitioner
Evaluating RAG and Agent Applications
A RAG system has two places to fail and an agent has many. This lesson shows how to localize the failure with the metrics AWS provides for knowledge bases and agents, instead of guessing at the model.
Intermediate 20 minutes 5 Learning Objectives
- Separate a retrieval failure from a generation failure in a RAG application
- Describe the built-in metrics for retrieve-only and retrieve-and-generate RAG evaluation jobs in Amazon Bedrock
- Distinguish faithfulness from correctness and explain why both are needed
- Explain why an agent needs its trajectory evaluated, not just its final answer
- Identify the AgentCore evaluation levels and the built-in evaluators that measure tool use