Can AI agents actually do real, meaningful SaaS work?

Zapier created a benchmark called AutomationBench, specifically designed to test how AI agents perform in real-world business scenarios. Instead of having them write poems or solve math problems, agents are tasked with operating real software tools including Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot to complete cross-application business workflows.

Recently, Artificial Analysis partnered with Zapier to launch an independent iteration called AutomationBench-AA. Unlike Zapier's official ranking, the AA version calculates the proportion of task objectives completed by models, while also checking for violations of business rules (guardrails). If any rule is triggered, the entire task gets a score of zero directly.

**The results are pretty interesting**

Claude Fable 5 (with fallback to Opus 4.8) scored 48.6%, while Claude Opus 4.8 scored 48.5%. The 0.1 percentage point difference is basically negligible. A netizen put it bluntly:

> fable only gets 0.1pp better lol

Gemini 3.5 Flash ranks third with 42.6%, but only costs $0.49 per task. GPT-5.5 (xhigh) scored 42.1% at a cost of $1.32. Gemini achieved a comparable score with less than 40% of the cost.


**All models make mistakes**

A key metric is "how many objectives can be completed per rule violation". Gemini 3.5 Flash achieves 15.0 objectives per violation, compared to 13.5 for Claude Opus 4.8. Fable 5 actually has no advantage on this metric because it frequently falls back to Opus.


Common rule violations include sending emails to unintended recipients, updating incorrect records, and failing to convert currencies at the latest exchange rate. Zapier gives an example in the documentation: the model needs to mark a €120,000 deal as "won" and send notifications according to the company hierarchy. The model might end up updating the wrong record because there are three similar entries for "Meridian" in the database, or use an outdated exchange rate table.

**Finance is the hardest domain**

When broken down by business domain, models complete only around one-third of Finance tasks on average, while Support and Operations reach roughly 60%.


**Cost varies wildly**

DeepSeek V4, Gemini 3.1 Flash-Lite, and Qwen3.7 Plus cost less than 5 cents per task, while Claude Opus 4.8 (max) costs $1.46. Gemini 3.5 Flash sits in the middle at $0.49, yet ranks third for overall score.


**Working patterns also differ**

GPT-5.5 (xhigh) calls tools an average of 49 times per task, spread across 25 conversation turns. Claude Opus 4.8 is more restrained: 35 tool calls in just 14 turns, with fewer rule violations. Grok 4.3 (high) has the fewest turns (13), but also a lower score—it often declares the task finished early when it's actually incomplete.


On Zapier's official AutomationBench leaderboard, even the top model (Claude Fable 5 Max) only gets 17.4% of tasks fully correct. Note that the AA version measures proportion of objectives completed, while the Zapier version measures full task pass rate—these two metrics cannot be directly compared.


**Key observation**

These models can score over 90% on coding and math problems, but when put into real-world SaaS workflows, even the best complete less than half of all objectives. What's more, they often "confidently make mistakes"—they report the task as completed, but the data in the database is wrong. Zapier's analysis found that 72% of Opus's failures and 91% of Gemini's failures are this type of "false success".

So don't be fooled by all those benchmark leaderboards. We still need to wait before AI agents can take over real work for you.

---

More information:

- [Full results from Artificial Analysis](https://artificialanalysis.ai/evaluations/automationbench-aa)

- [Official Zapier Benchmark](https://zapier.com/benchmarks)

- [AutomationBench Research Paper](https://arxiv.org/abs/2604.18934)

- [GitHub Repository](https://github.com/zapier/AutomationBench)