Evaluating AI Systems: Testing LLMs, RAG, and Agents Kindle Edition

The definitive guide to testing AI systems that actually work.Most AI systems ship without meaningful evaluation. Teams eyeball a few responses, declare the system "good enough," and push to production. Then quality degrades, hallucinations appear, and nobody knows why.Evaluating AI Systems is a practical, technical guide to building evaluation frameworks for LLMs, RAG pipelines, and AI agents. Written by Alex Merced, Head of Developer Relations at Dremio and author of multiple technical books, it covers the full evaluation lifecycle from dataset generation to production monitoring.What you will learn:Understand why traditional software testing fails for AI and what to do insteadBuild golden evaluation datasets that accurately measure system qualityImplement prompt testing with tools like DeepEval, RAGAS, and promptfooDesign evaluation metrics for correctness, faithfulness, relevance, and safetyDetect and measure hallucinations with automated pipelinesUse LLM-as-judge patterns with bias mitigation and multi-model consensusBuild regression testing that catches quality degradation before users doDeploy production monitoring with drift detection and quality alertingEvaluate multi-step agent workflows with tool use accuracy metricsManage evaluation costs with tiered strategies from smoke tests to deep expert reviewsWritten with verified specifications for GPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Pro throughout. Every technique is immediately applicable to production AI systems.For AI engineers: Build evaluation pipelines that prevent quality incidents.For QA engineers: Apply testing discipline to the most untestable systems you have ever worked with.For engineering managers: Make informed quality decisions with data, not gut feeling. Read more