A company introduces an AI assistant and discovers that customer-support agents can answer substantially more questions in an hour. The slide is persuasive: labor productivity has improved. A chief financial officer, however, should ask a different question. Did the organization increase profitable output, lower its total cost of serving customers, or improve retention? If not, the productivity improvement may be real without appearing in earnings. The distinction matters because business value is produced by a system, not an isolated task. A faster draft still needs approval, a suggested answer can be wrong, and an employee's free hour is only valuable economically when the organization can put it to useful work. We should celebrate speed but resist turning it into a financial claim before the evidence exists.
What the empirical evidence actually establishes
The 2025 Quarterly Journal of Economics article “Generative AI at Work,” by Brynjolfsson, Li and Raymond, studied 5,172 support agents. The introduction of AI assistance increased issues resolved per hour by about 15 percent on average, with larger benefits for less experienced workers. That is credible evidence of task-level productivity in a specific setting, not evidence that the participating firms enjoyed an equivalent margin expansion. A separate NBER field experiment covering 7,137 knowledge workers across 66 firms found that users spent roughly two fewer hours a week on email, while researchers did not detect a change in the quantity or composition of work from individual access alone. Both studies are important precisely because they describe what was measured and what was not. Time saved can coexist with unchanged production, reallocated attention or intangible gains such as reduced frustration. We cannot infer a company's net profit from those outcomes.
The six places where the value can disappear
First, demand may be constrained: doubling the ability to produce marketing copy creates little additional revenue if nobody wants another campaign. Second, the saved minutes may be fragmented across the day and impossible to convert into capacity that can be redeployed. Third, human review, correction, monitoring and rework can consume a meaningful share of the nominal gain. Fourth, software access, inference, integration, security and governance impose recurring costs that are usually excluded from a demo. Fifth, adoption can demand organizational redesign, training and changes in incentives before the faster work becomes the normal way of operating. Sixth, competitors may gain access to the same cheaper capabilities and pass the benefit to customers through lower prices. A productivity improvement is therefore an input to a business case, not the business case itself.
A numerical example—and why it must remain hypothetical
Imagine a service team that costs 100 monetary units per month to run. An assistant improves throughput by 20 percent, but the organization does not immediately reduce payroll; workers use part of the released capacity to check quality and handle difficult cases. AI licenses and governance add 8 units. If the company cannot sell more work or demonstrably reduce churn, its short-term operating profit may fall rather than rise, even though agents feel more productive. This is an invented illustration rather than a claim about a real company. Change the assumptions—suppose demand is waiting, or improved response times demonstrably improve renewals—and the economics can reverse. The lesson is not that AI is unprofitable. It is that capacity, revenue, pricing and cost do not move together by definition. Those relationships have to be observed and tested.
Who captures the gain?
Suppose all firms in an industry acquire comparable AI capabilities. Costs may fall across the industry, but competition can push prices down as well. Customers gain convenience and affordability; suppliers may earn higher revenue from model usage; employees might benefit from lighter workloads—or face new performance expectations. The adopting company does not automatically keep the whole surplus. Even inside a company, the beneficiaries may differ: finance sees a cost, employees see a simpler task, customers see faster service, and engineering sees ongoing platform obligations. A sophisticated analysis accounts for these distributional effects rather than assuming that an efficiency chart can tell us where the economic value ultimately lands. Durable advantage often comes from integration, quality and trusted customer relationships, not merely from obtaining the same assistant that every competitor can buy.
Move the measurement from activity to outcomes
The wrong scorecard focuses on generated words, completed prompts or nominal hours saved. A better one connects operational metrics to business consequences: cost per successfully resolved case, corrected error rate, contribution margin per customer segment, customer retention, cycle time for a genuinely completed process, and time spent on supervision. It also records baseline performance before adoption, the experimental comparison group when feasible, and variance between experienced and new employees. Before an agent is allowed to act autonomously, define what it can change and what needs human approval. When the measured outcome is consequential, maintain an audit trail from original input through model recommendation to final action. This is where product instrumentation and sound management become more valuable than adding another dashboard of AI usage.
A more useful executive decision
Rather than ask whether the organization is “using enough AI,” select one recurring process with measurable cost, frequent decisions and an obvious owner. Document the starting quality and total unit economics. Introduce the tool in bounded conditions, monitor rework and failure, and compare actual outcomes to a credible baseline. If the outcome improves, decide whether to expand, renegotiate a vendor contract or redesign the workflow. If it does not, retain the evidence and change course. Stanford's AI Index documents that inference costs have fallen markedly for comparable benchmark performance, but cheaper underlying technology does not eliminate the work of implementation or validate a company's value proposition. The enduring question is not how quickly the model responds. It is whether the organization can show that a better result occurred at an acceptable, fully measured cost.
The cost of errors must be priced before scale
A business case that lists the price of the model but ignores the consequences of mistakes is incomplete. A low-cost draft that causes a contract to be sent to the wrong client may erase months of efficiency gains. Even where errors are relatively harmless, someone must design evaluations, inspect samples and resolve edge cases. The relevant unit is not cost per generated answer but cost per answer accepted after the appropriate level of verification. This requires tracking the proportion of outputs corrected, the severity of mistakes, the cost of rework and the proportion routed to a human. As the organization scales deployment, the workload mix may also change: simple requests disappear into automation while human teams inherit a harder and more expensive set of cases. Comparing “before” and “after” agent throughput without adjusting for case difficulty can therefore produce an illusion of deterioration or improvement. A sound measurement system watches the full journey from customer intent to verified outcome.
An honest conclusion leaves room for uncertainty
There are benefits that conventional accounting captures poorly. Reducing repetitive administrative work may lower frustration, improve access to expertise and make a new employee more confident. Those outcomes can be strategically valuable even when immediate operating profit does not change. Their presence, however, is an argument for better measurement, not permission to invent a financial return. Managers should state which effects have been observed, which are expectations, and which remain hypotheses. The strongest pilot is not one that produces the largest productivity percentage. It is one that teaches the organization what would have to change—pricing, demand generation, service design, quality assurance or staffing—to convert a verified operational gain into sustainable economic value. Sometimes the outcome will justify expansion; sometimes the correct decision will be to stop. Both are legitimate results of serious evaluation.