Stop measuring token costs. Start measuring outcomes.
Your AI agents are a unit economics problem, not an AI problem.
One silent error can wipe out the savings from a dozen clean runs. This is why the question worth asking is not whether agents work, but whether anyone is measuring both sides of the arithmetic.
The technology changes, but the arithmetic does not.
01 · The arithmetic that does not change
If the value per task is lower than the cost per task, you stop running the task, and that is true of every automation decision ever made. It was true of the first spreadsheet macro, and it is true of the most advanced agentic system running in production today.
Most businesses deploying agentic tools can tell you the cost per task to the penny, because the token cost arrives on an invoice at the end of the month. The value per task is harder to see because it sits in hours that were never billed, in work that did not need a second pass, and in errors that were never made. No invoice arrives for errors that never happened, which is why they so often go unmeasured.
Until a business measures both sides (the value and the full cost), the comparison that decides whether automation pays cannot happen. We believe you need three numbers on the table before any agent goes live: what a successful task is worth, what consistency threshold the agent has proven against representative data, and what the system actually costs to deliver that success, including the human hand-off. The rest of this piece explains how to get them.
02 · Where the value hides
Accounts payable is a useful place to see this clearly, because the value side is unusually easy to price. Invoice processing is a clear example: a finance team receives an invoice, extracts the data, matches it to a purchase order, routes it for approval, and posts it to the ERP. Done by hand, the fully loaded cost of that work sits somewhere between $12 and $22 per invoice once you count labour, approval routing, and the time spent correcting the inevitable errors. Industry benchmarks have held in that range for years. That number is your value per task, because it is the cost you no longer pay when a successful autonomous run replaces it.
03 · The distinction most teams miss
Most teams miss the distinction between tool performance and work performance. Tool performance asks whether the model produces accurate results, whereas work performance asks whether the process itself has become faster, with fewer handoffs, less rework, secure governance, and outputs flowing into systems the business depends on. Many AI initiatives score well on the first and poorly on the second, impressing in demos but disappointing in daily operations.
04 · What this looks like in practice
The difference between tool performance and work performance is not abstract, and we saw it clearly in a compliance traceability audit for a major Australian infrastructure project. A team of engineers needed to review over 8,000 design specifications, linking evidence to each design spec and then back to contractual obligations. By hand, this would have taken weeks, so we deployed AI agents to perform the initial review, allowing them to sift through thousands of pages, find evidence, and link it back to the appropriate specifications. The agents assigned a reliability score to each match, indicating to the engineers exactly where they needed to step in for manual checks.
The result was that AI matched 8,000 design specifications to 500 contract specifications in under one hour. Compliance verification dropped from ten weeks to one hour, which represented a 99 per cent time saving and freed the equivalent of ten staff from the audit window. Engineers were redirected from checking and ticking boxes to fixing the genuine issues the agents surfaced. The agents did not replace the engineers; they redesigned the work so human time was spent where it actually mattered, and that is the practical difference between tool performance and work performance.
05 · How Gysho closes the gap
Knowing the value per task is only half the equation, because the other half is proving, with evidence, that the agent will improve the work itself before it ever touches a live workflow. We start with the bottleneck, because the places where teams are slowed down (manual data entry, approval delays, rework loops, reconciliation backlogs) are the clearest indicators of where AI can change the economics. If you cannot measure the baseline, you cannot prove the agent changed anything. We establish that baseline in metrics the business already understands: processing time per task, end-to-end lead time, variable and fixed costs, failure and error rates, and volume throughput.
With the baseline set, we run a measured convergence pattern that separates what an agent can reliably own from what still requires human judgement. First, we build a gold-standard test stack drawn from real documents: different vendors, formats, currencies, edge cases, and PO match scenarios. This benchmark defines success and underpins every test cycle that follows.
Next, we run parallel automated test cycles at scale. Each pipeline variation, prompt strategy, and validation rule is tested against that stack autonomously. We are measuring whether the agent improves performance against the baseline metrics: not just whether it produces accurate extractions, but whether the process becomes faster, cheaper, and cleaner under load.
Then we converge on a threshold that must be met before production deployment. The configuration does not go live until it hits a target consistency level, typically ninety per cent or higher, against the test stack. Below that threshold, the economics do not hold, because one failure in ten invoices erodes the margin too fast. Consistency is a condition for sustained results, not the result itself, so the testing process runs in the background, often for extended periods, until the numbers prove the agent is ready.
Finally, we define the hand-off with a reliability score so that anything outside the threshold, whether ambiguous vendor details, non-standard line items, or PO mismatches that fall outside validated rules, is not left to the agent's best guess. It is flagged automatically and routed to a human operator. The agent handles production volume while the human handles judgement at the edge. That boundary is engineered into the workflow from the start, giving the finance team both control and an audit trail for every decision the system makes.
Every new solution is then tested for its real-world performance against the baseline. If the investment in tokens, infrastructure, and human oversight does not offset the measured outcomes (faster processing, lower costs, fewer errors, higher volume), the deployment does not graduate from test to production.
We require the arithmetic to close in practice, not just in theory.
This changes the question from 'Will the agent work?' to 'What, exactly, has the agent proven it can do, and what is explicitly reserved for human review?' When you can answer that, you have control, and when you cannot, you are experimenting with your general ledger.
06 · Hidden failure costs
When a converged agent runs the same invoice task, a clean run (where the agent extracts the fields, matches the PO, and posts without a human touching it) costs well under a dollar in tokens. Against $12 to $22 of avoided cost, the return is not a close call, and it is likely the easiest decision a finance leader will make this year. But these figures only hold when the agent succeeds.
Without the test stack and the threshold, the agent pulls the wrong total, or the PO match does not resolve, and the error lands in the ERP unnoticed. Now you are not comparing a dollar of tokens against $16 of labour. You are comparing the tokens, plus the retries, plus the roughly $53 it costs to find and fix a single posted error, plus the downstream cost of a payment that went out wrong. One silent failure can wipe out the savings from a dozen clean runs. The unit economics did not change, but your measurement of them did the moment you started counting outcomes instead of attempts.
| Scenario | Cost | Value |
|---|---|---|
| Clean run | <$1 tokens | $12 to $22 saved |
| Failed run (unmeasured) | <$1 + $53 fix + downstream | Negative |
| Failed run (with hand-off) | <$1 + human review at $8 to $12 | Controlled, visible |
The third row is the one most deployments miss. When the agent knows its own boundary and escalates, the failure becomes a known, bounded cost, not a silent liability.
07 · What most deployments miss
Token tracking tells you what you spent, but it tells you nothing about whether the task succeeded. A system that completes 95 per cent of runs cleanly and a system that completes 70 per cent can show the same token bill and have completely different economics. The difference only appears when you track outcomes with the same discipline you track cost.
Most teams skip the test-stack phase, prompting and eyeballing a few results before switching on the tap. That is how you end up with a production system that looks automated but is actually generating a hidden queue of corrections, rework, and quietly compounding errors. The token bill is low, but the true cost is invisible until reconciliation.
There are three failure patterns we see repeatedly:
- Treating governance as an afterthought. AI carries real risk depending on what data you give it and where it is sent. If governance is not designed into the workflow early, it becomes a liability or a deployment blocker later.
- Confusing activity with outcomes. Usage metrics are seductive but meaningless if processes are not actually faster or cheaper. Expense must tie to tangible outcomes.
- Running pilots without production paths. If experiments are not designed to survive beyond proof-of-concept, they were never real value tests. They were theatre.
08 · The standard we hold ourselves to
This is the standard we apply to every production deployment at Gysho today. We do not claim every project we have touched met it from day one, and we do not claim every project in the market meets it. But it is the bar we hold for live systems, because anything less is an experiment.
We have seen what happens when teams skip this step, and we have seen what happens when they follow it. In a major infrastructure compliance audit, measuring outcomes revealed a 99 per cent time saving, freeing the equivalent of ten staff from the audit window. In a recent deployment, the three-number discipline protected the economics from day one.
So before we put an agent into production, we want three numbers on the table: what a successful task is worth, what consistency threshold the agent has proven against a representative test set, and what the system actually costs to deliver that success, including the human hand-off for everything that sits outside the threshold.
If we cannot state all three, we are not deploying an automation; we are running an experiment.
Most businesses are still only watching the cost side, but the ones who win the next few years will be the ones who learnt to measure the value, prove the accuracy, and design the boundary where automation ends and human judgement begins.
09 · Your next step
If you are deploying agents today, start by auditing one workflow and write down the value per task, the cost per successful completion, and the consistency threshold your agent has actually proven against real data. If you cannot fill in all three numbers, you are not ready for production.