Amazon's AWS AI team has introduced RAGChecker, a new research tool to evaluate Retrieval-Augmented Generation (RAG) systems. RAGChecker is a framework that offers a detailed approach to assessing AI systems that combine LLMs with external databases. The tool is currently being used internally by Amazon's researchers and developers.

RAGChecker uses claim-level entailment checking to analyze both the retrieval and generation components of RAG systems. It breaks down responses into individual claims and evaluates their accuracy and relevance based on the context retrieved by the system. The framework provides overall metrics for holistic performance assessment and diagnostic metrics to pinpoint specific weaknesses in either the retrieval or generation phases.

Amazon claims RAGChecker can guide researchers and practitioners in developing more effective RAG systems. The tool was tested on eight different RAG systems using a benchmark dataset spanning 10 distinct domains, including medicine, finance, and law.