Evaluation checklist
How to evaluate a GTM agent or automation
Test four things before anything an agent writes reaches a buyer: whether it does the same right thing every run, whether it uses current data, whether the send layer blocks mistakes, and whether it creates pipeline. Use this checklist on a vendor trial or on something you built.
1. Reliability across repeated runs
One good run is a demo. The τ-bench paper introduced pass^k, the chance an agent gets the same task right on every one of k attempts, because success on a single try overstates how dependable an agent is. PipelineBench saw the same in GTM work: 1 in 10 model and task pairs passed one run and failed the next.
- Pick 10 to 20 real tasks from your own work (research this account, qualify this lead, update this deal).
- Run each task at least five times without changing anything.
- Count the tasks that passed every run, not the ones that passed once. That is your number.
- Write down what a pass means before you run, so you are not grading on feel.
- Re-run the same set after every change to prompts, tools or models.
2. Data freshness
PipelineBench found with a year-old prospect database, models went with the stale number about half the time, even when the company's own site said otherwise. Test whether the agent or workflow checks records before it uses them.
- Seed a test account whose contact left the company months ago. Does it notice before drafting?
- Give it a record with an old title that the company's own website contradicts. Which one does it use?
- Check whether emails are verified at send time or only at import.
- Ask the vendor or builder: how old can a record be before it is re-checked, and where is that set?
3. Send-layer safety controls
Guardrails that live in a prompt fail when the prompt does. The ones that hold are enforced where messages are sent, whatever the agent decided.
- Try to make it email an existing customer as a prospect. It should be blocked.
- Try to make it contact someone on an open deal owned by another rep. It should be blocked.
- Add an address to your do-not-contact list and send to it. It should be blocked.
- Check that suppression applies across every tool that sends, not only one sequencer.
- Confirm there is a daily send limit and a way to stop everything at once.
4. Human review
Review is where wrong facts get caught. Check it is the default, not an option someone has to remember.
- Are drafts, list additions and CRM changes held for approval before they go out?
- Can a reviewer see the evidence behind each draft (the signal, the source, the date)?
- Is it quick enough that people actually review rather than approve everything?
- Are edits recorded, so the system learns what reviewers change?
5. Stamina over long work
GTM work runs for months: follow-ups, re-checks, nurture. In PipelineBench's year-long runs, Gemini 4 Argon stopped working before December in 8 of 30 runs; Claude Sonnet 5.5 in 13 of 30.
- Does work resume where it left off after a failure or a restart?
- Does follow-up happen on schedule weeks later without someone prompting it?
- Is there a daily or weekly record of what was done, so a stall is noticed?
6. Pipeline outcomes
Judge on meetings and qualified pipeline, not on emails sent or tasks completed. Compare against simple baselines: PipelineBench graded every model against an oracle with perfect information and against shortcut agents, so a result means something.
- Agree the outcome metric before the trial: meetings held, qualified pipeline, reply rate from ICP accounts.
- Run a baseline beside it: your current process, or a plain sequence to the same list.
- Track wrong sends and bounces as failures, not noise.
- Give it long enough to reach the outcomes your sales cycle produces, not just the first week's replies.
7. Cost
Model cost varies widely. PipelineBench lists token costs for a simulated year of GTM work from $9 to $411 for a simulated year, depending on the model, and the most expensive model was not the only good one.
- Ask what you pay per month at your expected volume, in writing.
- Ask which model runs which task, and whether you can change it.
- Count the people time for review and maintenance, not only the subscription.
The benchmark work cited
PipelineBench
Frontrunner · Carl Bager · 29 October 2026
Runs AI models as the first GTM hire of a simulated company for a year, with the same tools, data and budget, and measures the qualified pipeline each creates. 23 models, 30 year-long runs each, graded against an oracle ceiling and shortcut baselines.
- Best result: Claude Fable 5.1 at $18.3M of simulated pipeline in a year, 66% of the $27.6M ceiling set by an oracle run. No model was near the ceiling; most of the gap was stale data and stamina rather than writing.
- Reliability: 1 in 10 model and task pairs passed one run and failed the next.
- Freshness: with a year-old prospect database, models went with the stale number about half the time, even when the company's own site said otherwise.
- Baselines: Oracle (perfect information, never stops) $27.6M; No follow-up (right people, reached once) $9.4M; Database as-is (emails everyone a year-old database lists) $4.7M; Do nothing $0.
- The simulated environment was built from anonymized observations across real company environments: 7,437 agent runs, 492,000 rows of accounts and people, 23,700 outreach actions and 5,577 people reached.
PipelineBench is published by Frontrunner, which is also one of the options in this guide's ranking. Its method and full leaderboard are public; read them at the link above.
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Sierra (Yao, Shinn, Razavi, Narasimhan) · June 2024
Introduces the pass^k metric: the chance an agent succeeds on all k repeated trials of the same task, not just one. The standard way to measure reliability across repeated runs.