Notes from evaluating a customer support chat agent system: heuristic evaluators give false signal, retrieval bugs masquerade as LLM failures, and the cost/quality Pareto frontier is rarely where you think [D]

Posting some practical findings from a structured audit of a production customer support RAG system. Methodology and caveats up front. Methodology: 6 representative turns from a real production session as the eval set (small, acknowledged limitation) LLM-as-judge using Claude Haiku 4.5, scoring rele