Introducing Evals

Every few years the data teams’ job seems to get upended (at least we’re used to it). The latest shift in our role: maintaining and managing analytics agents. As people demand conversational analytics, the data teams’ new job is keeping agents accurate and consistent. This starts with context.

Agents are non-deterministic and you can’t control how users interact with them, the way you could with a dashboard. This makes it hard to know how agents are actually going to use context, and whether answers get better as it changes.

Today, we're introducing Evals in Hex. Hex’s Evals are inspired by the same testing workflow we use internally to ship our own agent features, now yours to build on.

Grading the work, not just the answers

Data analytics is a uniquely hard domain for an agent to work in: easy questions can look hard, hard questions can look easy, and a correct number can still mislead you.

With Evals, you can build tests for the whole range of cases.

In the simplest version, you can specify a numeric result as a specific number or reference SQL query, alongside your tolerance level.

But Hex’s Evals are much more powerful: they use an LLM-as-judge that reads the entire conversation, and sees the specific tools and resources the agent uses. This lets you grade the agent’s work

• Did the agent use the correct tools, like a specific semantic model or guide?
• For an open-ended question did the agent propose multiple approaches back to the user or ask for clarification?
• Did the agent find and call out the data discrepancy that you know exists in the data?
• Did the agent appropriately refuse to answer a question where there is insufficient context?

This lets you measure not only whether it got the “right” answer, but the reasoning and behavior how you got there.

Built for your existing workflow

Your eval test cases are defined in files that you can version control, and you run them from the Hex CLI. Eval suites can live alongside the rest of your context, so you can use the same code review workflow you already use for guides and semantic models.

You can also fork your context and run evals against the fork, called a Context Preview. These are sandboxed version of your agent with an alternate version of your workspace context.

Because each run is scoped to a specific fork and configuration, you can compare results side by side. Agents aren't deterministic, so you can run each test case up to three times and see how much the answer moves before you commit to a default.

What you can do with Evals

Evals turn agent quality and cost into something that you can measure and improve, so you're not just going on vibes.

Specifically, they help you:

• Test context changes before publishing: Start by staging a guide or semantic model change from the CLI. You can run your eval suite against a preview of this change, and publish only if it passes.
• Catch drift before your users do: Data changes over time, and context can too. External context, data and tables can change upstream of the agent. You can run your eval suite on a schedule and catch unexpected regressions to the agent’s performance.
• Run configuration sweeps: Run a suite of evals against different language models and effort levels: you can evaluate a cheaper open-source model like Kimi with frontier models. Compare the cost and quality of responses in the agent. This lets you choose the default LLM that fits your organization’s needs.

How Chime is using Evals

Let’s take a look at how Chime® has integrated Evals into their workflows. Chime®, a financial technology company serving 10M+ members, had already built their own eval harness to keep pace with 50+ data domains. They ran questions against the agent one by one and used an LLM-as-judge for evaluation. But this required manual tuning and maintenance. When we introduced to native Evals in Hex, the team quickly adopted.

See the docs for more details on how to define and run evals.

The full context lifecycle

Evaluating agent performance isn’t a one-off exercise, building maintaining, and improving context is ongoing work. Here’s how you manage the full context lifecycle to give your organization confidence in the answer you get from the agent:

Unify context

Bring in context and build on what you already have in dbt, git repos, docs, or external apps. We never lock you in with proprietary specifications and all context defined in Hex is portable and can be taken out of Hex.

Observe the agent

Get one view of every question and answer, wherever it happened: Hex, Slack, MCP, or the CLI. The Review Agent flags warnings where context is missing and classifies what each answer actually used.

Improve with use

The Review Agent clusters context gaps into Suggestions, so you can quickly review and apply. Once you make context changes, use Evals to verify if there change worked. Want to apply a suggestion? Add the suggestion with an eval to make sure your change doesn’t regress in the future.

Chime® runs this loop at the scale. Jordan, Director of Data Science, segmented the company's data into domains (marketing, lending, and dozens more), each with its own guide, semantic model, and eval set, gated before it's endorsed company wide.

Getting started

Here are some tips for getting started with Evals:

• Install the Hex CLI by pasting the following in your terminal
  • curl installer: curl -fsSL https://hex.tech/install.sh | bash
  • homebrew installer: brew install hex-inc/hex-cli/hex
• Check Suggestions, the Review Agent already has some changes you should probably make.
• Start with simple questions your team already asks, the ones you'd want the agent to get right every single time.
• Run it again next week, even ad hoc runs every week will start surfacing interesting opportunities.

As part of onboarding, we’ve included a bootstrap prompt for Claude Code, Cursor, or your coding agent of choice. It scans your recent Threads, clusters them into your team's most common question topics, and drafts a starter suite from real usage instead of a blank file.

Give Evals a try, we'd love to hear what you find!