rag-eval-kit

A common experiment, across different systems

A retrieval change can improve an average score while making a particular topic worse, increasing latency, or doubling the cost of an answer. I built rag-eval-kit to make those tradeoffs visible in the same experiment.

A pipeline plugs in through one query method. A configuration file defines the labeled questions, parameter sweep, and metrics. The runner applies the same evaluation loop to raw model integrations, LangChain, LlamaIndex, R2R, and Haystack. It records per-question results before producing aggregate comparisons.

Keep the run inspectable

Parallel queries, checkpoints, and configuration-aware resume support make experiments practical to repeat. Reports retain the individual answers, group results by dataset tags, and compare matching configurations across runs. CSV exports and self-contained HTML reports keep the evidence available outside the original execution environment.

The metric set separates answer matching, retrieval quality, model-judged correctness and faithfulness, latency, and estimated token cost. That separation matters: a high retrieval score does not establish that the generated answer is correct, and agreement from another model is a different kind of evidence from an executable check.

Available to reproduce

The package is published on PyPI with source, configuration examples, adapter documentation, and a test suite. The reusable contribution is the experiment infrastructure; any conclusion about a particular RAG system still depends on the dataset, metric choices, and run being examined.