FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts

Large language models (LLMs) are known to hallucinate, producing natural language outputs that are not grounded in the input, reference materials, or real-world knowledge. In enterprise applications where AI features support business decisions, such hallucinations can be particularly detrimental. LLMs that analyze and summarize contact center conversations introduce a unique set of challenges for factuality evaluation, because ground-truth labels often do not exist for analytical interpretations about sentiments captured in the conversation and root causes of the business problems. To remedy this, we first introduce a $\mathbf{\text{3D}}$ – $\mathbf{\text{Decompose,}}\mathbf{\text{Decouple,}}\mathbf{\text{Detach}}$ – paradigm in the human annotation guideline and the LLM-judges’ prompt to ground the factuality labels in linguistically-informed evaluation criteria. We then introduce $\mathbf{\text{FECT}}$, a novel benchmark dataset for $\mathbf{\text{F}}$actuality $\mathbf{\text{E}}$valuation of Interpretive AI-Generated $\mathbf{\text{C}}$laims in Contact Center Conversation $\mathbf{\text{T}}$ranscripts, labeled under our 3D paradigm. Lastly, we report our findings from aligning LLM-judges on the 3D paradigm. Overall, our findings contribute a new approach for automatically evaluating the factuality of outputs generated by an AI system for analyzing contact center conversations.