Conversational AI datasets to power agent performance

When building conversational agents for the contact center, we often obsess over model performance: which framework performs best, how fast inference runs, or which fine-tuning approach yields better accuracy. But there’s a deeper layer that quietly determines whether results are meaningful at all—the quality of the evaluation data.

In other words, before comparing models or architectures, we need to ask: what exactly are we evaluating on? That’s where evaluating conversational AI datasets becomes critical.

In the AI community, this is a hot topic. Companies like Galileo have created leaderboards using open-source agentic datasets such as Tau-bench, xLAM, and BFCL that cover several industry verticals. Similar leaderboards have also gained prominence in recent years beyond agentic systems, especially Artificial Analysis, focused on model comparisons. Dedicated AI evaluation systems, such as Cekura, Maxim, and Galileo, provide continuous visibility into agent performance across various domains.

Despite all this progress, there’s one area that still lags behind: the evaluation data itself. A dataset can look structured, balanced, and even comprehensive, but still fail to capture the messy, emotional, and unpredictable reality of real contact center conversations. That gap can lead to overly optimistic conclusions and painful surprises in production.

Let’s see what makes a dataset realistic, why synthetic “perfect” data can be misleading, and how to build a simple rubric for evaluating conversational AI datasets in a repeatable, practical way.

The example dataset.

Consider this sample test scenario used for evaluating an insurance agent’s performance from Galileo’s agent evaluation leaderboard:

“Hi there! I need to make some changes to our dance company’s insurance setup. We have a performance tour coming up in 3 weeks, and I just realized I need to update several policies before we leave on the 15th. I need to add my lead dancer, Sofia, as a beneficiary on my life insurance, increase coverage on our equipment policy from $50k to $75k, cancel our venue liability policy that expires tomorrow, and schedule a consultation about international coverage options. Oh, and can you check if there are any discounts for professional arts organizations? I’m also worried about a pending claim from last month’s injury—it’s claim #DNC-22587.”

At first glance, this looks like a strong example: detailed, domain-rich, and packed with realistic entities. It’s exactly the kind of input an agent might need to handle. Right? Not quite.

Why is this a problematic example?

This dataset snippet illustrates a common issue in AI evaluation, which is over-engineered data. The content looks realistic, but fails the voice test; how real people actually speak when they call customer support. Let’s break it down:

• Overloaded request. It includes six separate actions in one utterance. Real customers rarely list all their needs upfront.

• Not voice-like. It reads like an email or web form, not a spoken call. No pauses, no turn-taking.

• Too polished. Perfect grammar, no filler words, typos, or ambiguity. Real calls are full of “um,” “uh,” and mid-sentence corrections.

• Single turn. The agent doesn’t respond or clarify, which eliminates a key part of how dialogue systems are evaluated.

• Persona mismatch. A “dance company” might be creative, but it’s atypical for a general insurance contact center dataset.

Testing conversational AI against such examples may appear to perform well, only to crumble in production when faced with real, human unpredictability. Human interactions involve noise that comes from speaking patterns related to fillers, hesitations, repeats, and corrections. Speech-to-text transcription errors only add to the problem. So, the evaluation dataset should allow for simulating such complexities realistically.

Rewriting for realism in a voice use case.

Now, let’s rewrite this example as a realistic multi-turn interaction:

Customer: Hi, uh, I’ve got a couple of things I need help with on my insurance. Is now a good time?

Agent: Of course, I can help. What’s the first change you’d like to make?

Customer: Sure. So, um, I need to add someone… my lead dancer, Sofia… as a beneficiary on my life insurance.

Agent: Got it, adding Sofia as a beneficiary. What’s next?

Customer: Right. Also, our equipment policy—I think it’s fifty thousand right now?—I’d like to bump that up to seventy-five.

Agent: Okay, increasing equipment coverage. Anything else?

Customer: Yeah, can you also cancel our venue liability policy? I think it expires tomorrow.

Agent: Cancel venue liability policy expiring tomorrow, understood.

Customer: Thanks. And, um, we’ve got a tour coming up overseas. I wanted to ask about international coverage—maybe set up a consultation?

Agent: Sure, I can arrange that.

Customer: Oh, and before I forget—are there any discounts for professional arts organizations?

Agent: I’ll check for you. Anything else?

Customer: Yeah, there’s also a pending injury claim—I think it’s DNC-22587. Could you check its status?

Now, the conversation sounds human. It includes hesitations, clarifications, and incremental intent discovery. This single example now tests turn management, memory, context tracking, and intent sequencing, key factors in evaluating how agents handle real calls.

Developing a rubric for dataset quality.

How do we formalize “good” conversational data? Below is a practical rubric you can use when designing or reviewing datasets for AI agent evaluation:

Each score helps identify weaknesses in the evaluation data. For instance, a dataset with high realism but low diversity might train agents that perform well for one demographic but fail for others. The overall dataset quality can be computed as an aggregation of scores from individual datapoints and will range from 0-5, just like the individual dimensions in the rubric.

Applying the rubric to the original example.

Evaluating the original dance company insurance snippet: