Cost-per-task, not cost-per-token: how to actually compare AI tools

If you’re weighing up AI tools for your business, the number the salespeople wave about — a price per token, per message, or per seat — is the one that matters least. It’s the sticker price for the raw text a model reads and writes. It is not the cost of getting your actual job done. And the gap between the two is where a lot of money quietly leaks.

Why “cheaper” often isn’t

Imagine two tools. One is half the price per token. But to do the same job it writes twice as much to get to the point, needs a second or third go before it’s right, and lands a usable answer seven times out of ten instead of nine. By the time you count the extra text, the retries, and the person who has to check and fix the misses, the “cheaper” tool has quietly cost you more — and that’s before you count the time your team spent waiting on a slower response. The headline price was real; it just wasn’t the whole bill.

The costs that never make the price list

• The human check. For anything that matters, a person reviews the output — and a tool that’s wrong more often eats more of their day. That time is a real cost, and usually the biggest one.
• Retries and back-and-forth. Every “no, try again” is more tokens and more waiting. A tool that gets there first time is cheaper even at a higher headline rate.
• Set-up and integration. The cheapest model is dear if it takes weeks to wire into the tools you already use. A good fit beats a good price.
• Reliability. A tool that fails one job in ten costs you that one job — the missed enquiry, the wrong figure — which can dwarf the per-token saving.

How to actually compare them

Don’t compare price lists — compare outcomes. Take one real task you’d genuinely use the tool for (draft this kind of email, sort these invoices, answer these customer questions), run it on each option, and measure what it costs to get that job done to your standard: the fees, plus the human time to check and fix. Then pick the cheapest tool that clears your quality bar — not the cheapest full stop. Cheapest-that-does-the-job and cheapest are rarely the same tool. It’s the same “automate the boring bits, keep a human on what matters” thinking we set out in what’s worth automating: the tool is only cheap if the finished work is actually usable.

The honest bottom line

Token prices move around — providers change them, and a clever-looking rate can hide a chattier model. What doesn’t change is the sensible question: what does it cost to get this job done well, start to finish? Judge tools on that, and you’ll rarely be fooled by a low sticker price. For the record, we run our own operation on a paid, capable model rather than the cheapest one going — because for us, too, the cost that counts is per task, not per token. If you want a sense of what an agent can genuinely take off your plate before you price anything, there’s the real list of what one can do.