| Decision leadership actually faces |
Evidence needed before deciding |
Reasonable outcomes |
| **Is this activity worth supporting at all?** |
Baseline pain, frequency, economic relevance, current quality/cycle time |
Support; tolerate experimentation; postpone; explicitly stop |
| **What problem are we solving?** |
Outcome metric independent of AI usage |
Faster completion, better quality, capacity, customer outcome, learning, or no compelling problem |
| **Is GenAI the right intervention?** |
Comparison with no change, process simplification and deterministic automation |
GenAI; conventional software/automation; workflow change; nothing |
| **At what level should intervention occur?** |
Where the bottleneck actually sits |
Individual assistant; team tool; workflow redesign; process automation |
| **Which uses can remain decentralized?** |
Reversibility, information sensitivity, consequences of error |
Unmanaged/ lightly governed experimentation; approved sandbox; central control |
| **Should the company provide an approved general tool?** |
Breadth of genuinely useful low-risk tasks, security requirements, utilization economics |
Enterprise tool; limited seats; reimbursement; no general license |
| **Does a role-specific copilot make sense?** |
Repeatable role-specific work and stable context requirements |
Deploy; prototype; retain general assistant |
| **Should AI act or merely advise?** |
Error recoverability, verification ability, authorization risk |
Draft only; recommend; execute with approval; bounded autonomy; no automation |
| **Does the workflow itself need redesign?** |
Evidence that local time savings fail to change end-to-end performance |
Keep assistant; redistribute work; remove steps; redesign process |
| **Is a pilot enough?** |
Objective of pilot |
Continue as local experiment; gather more evidence; productionize; terminate |
| **Is evidence strong enough to scale?** |
Representative users/tasks, objective outcomes, guardrails, economics |
Expand; restrict to subgroup; redesign; stop |
| **What governance is proportionate?** |
Data class, decision consequence, regulation, external exposure, autonomy |
Minimal rules through formal high-risk controls |
| **Who owns the use case?** |
Location of business outcome and technical/control dependencies |
Business owner, product/process owner, centralized platform, shared accountability |
| **What capability must users acquire?** |
Observed failure modes rather than generic “AI readiness” |
No training; contextual guidance; role-specific training; formal competence requirement |
| **How will saved capacity be converted into value?** |
Downstream bottlenecks and management response |
More throughput; shorter queues; quality improvement; capacity reduction; no realized value |
| **When do we stop?** |
Pre-agreed failure conditions |
Abandon; return to human process; substitute deterministic automation; re-scope |
This separation is important because the empirical literature largely measures **specific tasks under specific human–AI configurations**, rather than an abstract organizational state called “AI adoption.” Noy and Zhang tested occupational writing; Brynjolfsson, Li and Raymond studied customer-support agents; Cui and colleagues studied professional developers; Dell’Acqua and colleagues tested consultants and later P&G product-development professionals. Their different results are evidence against treating GenAI as a homogeneous intervention. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence)
**Decision implication:** an “AI transformation program” may be useful as an umbrella for common infrastructure and controls, but it is a poor unit for deciding whether individual uses create value.
### When organizational adoption is justified
Organizational support becomes increasingly defensible when six conditions hold simultaneously:
**A meaningful bottleneck exists.** There is repeated work whose time, quality, capacity or service level matters economically. A cool demo is not a bottleneck.
**The intervention improves an outcome that matters.** AI changes quality, cycle time, throughput, rework, customer experience, learning or another endpoint—not merely prompt volume or perceived productivity. Controlled studies demonstrate that measured gains can be real, but METR's randomized developer experiment demonstrates that users can believe themselves substantially faster while objectively becoming slower. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence)
**Task fit is demonstrated rather than presumed.** Dell’Acqua et al.'s BCG experiment found a “jagged technological frontier”: AI improved performance on tasks inside the model's capabilities but over-reliance could reduce performance outside them. This means superficial task similarity is insufficient evidence of reliability. [2](https://pubsonline.informs.org/doi/10.1287/orsc.2025.21838)
**Verification is economically possible.** A system whose outputs require more expert checking than the work it saves can have negative net value even when generation itself is fast. The importance of including verification is reinforced by experiments in which speed and quality move differently and by METR's finding that workflow and task-selection behavior can materially complicate productivity measurement. [3](https://arxiv.org/abs/2304.11771)
**Risk can be bounded.** Data exposure, consequential decisions, customer harm, security privileges and regulatory requirements are sufficiently controlled. In the EU, some governance is not discretionary: Article 4 of the AI Act requires providers and deployers to take measures supporting an appropriate level of AI literacy, while high-risk deployers face additional obligations including competent human oversight, monitoring and logging. [4](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-4)
**There is a credible mechanism for converting local gains into organizational value.** Cutting a worker's drafting time from 50 to 30 minutes has no automatic P&L value. It matters only when released capacity results in more valuable output, faster service, reduced backlog, avoidance of hiring, reduced external expenditure, improved quality, or some other organizational endpoint. The major causal studies establish task and worker effects; they do **not** establish a universal conversion from saved minutes to firm profit. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence)
Conversely, organizational adoption is **not yet justified** merely because employees like a tool, colleagues use it, competitors announce deployments, vendor benchmarks improve, a demonstration looks impressive, or a pilot reports “hours saved.”
Doing nothing—or retaining decentralized experimentation—is rational when the potential benefit is small, the task is infrequent, workflow change costs dominate, verification is expensive, errors are difficult to detect, simpler software solves the problem better, or current evidence is too weak to justify irreversible investment.
## Evidence on value and heterogeneity
The evidence base is now materially better than it was in 2023, but it remains much stronger for **task productivity** than for **firm-level economic performance**.
### What has actually been demonstrated
| Claimed benefit |
Best evidence found |
Evidence class |
What can reasonably be concluded |
| **Time savings** |
In Noy & Zhang's preregistered experiment with 453 professionals, ChatGPT reduced average task time by about 40% on professional writing tasks while increasing evaluated quality. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence) |
Controlled experiment |
Strong evidence for selected writing tasks; not a company-wide productivity estimate |
| **Higher throughput** |
Randomized field experiments involving 4,867 developers across Microsoft, Accenture and another Fortune 100 company found a 26.08% increase in completed tasks among developers given an AI coding assistant. [5](https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535) |
Controlled field experiments |
Strong evidence that coding assistants can increase output in these settings |
| **Higher throughput** |
A staggered deployment covering 5,172 customer-support agents was associated with about 15% more issues resolved per hour. [3](https://arxiv.org/abs/2304.11771) |
Quasi-experimental/observational field evidence |
Strong real-work evidence, though weaker causally than random assignment |
| **Quality improvement** |
Noy & Zhang found approximately 18% higher evaluated writing quality alongside faster work. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence) |
Controlled experiment |
Speed and quality need not trade off |
| **Quality improvement—and deterioration** |
BCG consultants performed better on tasks inside AI's frontier but could perform worse on tasks outside it because of over-reliance on AI. [2](https://pubsonline.informs.org/doi/10.1287/orsc.2025.21838) |
Preregistered field experiment |
Quality effects are task-dependent; productivity measurement without quality is unsafe |
| **Reduced skill bottlenecks** |
Less experienced/lower-skill customer-support agents obtained the largest gains; experienced/high-skill agents gained less and saw small quality declines in some measures. [3](https://arxiv.org/abs/2304.11771) |
Field evidence |
AI may diffuse elements of top-worker knowledge, but benefits are heterogeneous |
| **Reduced skill bottlenecks in innovation** |
In P&G's randomized experiment, individuals using AI produced ideas comparable in evaluated quality to two-person human teams and became less constrained by their own commercial versus R&D specialization. [6](https://pubsonline.informs.org/doi/10.1287/orsc.2025.20702) |
Preregistered field experiment |
AI can partly substitute for missing complementary expertise in some ideation work |
| **Learning** |
Customer-service evidence is consistent with less-experienced workers learning useful patterns from AI suggestions, and gains were particularly strong on relatively uncommon problems. [3](https://arxiv.org/abs/2304.11771) |
Field evidence, mechanism analysis |
Credible evidence of knowledge diffusion; not proof that every copilot builds durable skill |
| **Innovation** |
P&G participants using GPT-4 produced higher-quality product ideas, but AI did not similarly improve people's ability to select the best ideas. [6](https://pubsonline.informs.org/doi/10.1287/orsc.2025.20702) |
Controlled field experiment |
GenAI can improve generation without improving evaluation |
| **Employee experience** |
P&G participants receiving AI reported more positive and fewer negative emotions during the task. [6](https://pubsonline.informs.org/doi/10.1287/orsc.2025.20702) |
Controlled treatment, self-reported outcome |
Evidence of improved short-run subjective task experience; not evidence of long-run engagement |
| **Customer interaction** |
In the customer-support deployment, customers became more polite and less likely to ask for a manager. [3](https://arxiv.org/abs/2304.11771) |
Field observational/quasi-experimental evidence |
Some downstream customer interaction benefits are plausible and observed |
| **Cost reduction** |
Klarna reported substantial marketing-production savings from GenAI, including an estimated $10 million annual reduction in marketing costs; these were company-reported figures, not a controlled evaluation. [7](https://www.reuters.com/technology/klarna-using-genai-cut-marketing-costs-by-10-mln-annually-2024-05-28/) |
Company case claim reported externally |
Useful hypothesis and case evidence, not causal proof transferable to another company |
| **Firm revenue/profit** |
The strongest studies reviewed above generally stop at task, worker, quality, service or innovation outcomes rather than randomized firm P&L. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence) |
Evidence gap |
Broad claims that enterprise GenAI “drives revenue” remain much less established than task-level productivity effects |
The strongest counterexample to relying on perception comes from METR. Sixteen experienced open-source developers completed 246 real issues in repositories they knew well. With early-2025 AI tools, AI access made them **19% slower**, even though they believed AI had accelerated them by roughly 20% and had expected even larger gains beforehand. The authors explicitly caution against generalizing this result to all developers or all AI systems. [8](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)
METR's follow-up is equally important for management science. By 2026 the researchers concluded that changes in developer willingness to work without AI, task selection and parallel agent usage made later causal estimates substantially less reliable. In other words, as AI becomes embedded in work, **measurement itself becomes endogenous**: workers may choose different tasks because AI exists. [9](https://metr.org/blog/2026-02-24-uplift-update/)
### Who benefits most?
There is no universal answer.
Lower-skilled or less-experienced people benefited disproportionately in the customer-support field study, and less-experienced developers obtained greater gains in the three large coding experiments. [3](https://arxiv.org/abs/2304.11771)
But that does **not** imply a general “AI helps novices most” law. The BCG study found gains were contingent on whether the task sat inside the AI capability frontier, while METR found highly experienced developers working in mature codebases were slowed by the tested tools. [2](https://pubsonline.informs.org/doi/10.1287/orsc.2025.21838)
A more defensible hypothesis is:
> AI produces larger gains where it possesses useful task knowledge that the worker does not already have **and** where its output can be cheaply incorporated and verified.
That proposition fits the customer-service, coding and P&G findings, but should still be treated as a mechanism to test rather than a universal rule. [3](https://arxiv.org/abs/2304.11771)
### Can productivity rise while quality falls?
Yes.
The customer-support evidence shows different speed and quality effects across skill groups, and the BCG study shows that AI can produce impressive-looking but wrong answers that cause humans to underperform when they trust it outside its capability frontier. [3](https://arxiv.org/abs/2304.11771)
Therefore **throughput alone is an unsafe success metric**.
### Does individual productivity imply company productivity?
Not automatically.
The experimental evidence principally demonstrates **partial equilibrium effects**: one worker or team can perform a task differently when given AI. Organizational value depends on what happens next. An employee who drafts twice as fast may simply wait longer for approval; a developer who produces more code may create more review burden; a support agent who resolves tickets faster may reduce queues—or merely create excess capacity. The causal studies demonstrate the local effects but do not prove that each local improvement survives downstream constraints. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence)
Leadership should therefore distinguish:
**potential capacity created → capacity actually released → capacity redeployed → economically valuable outcome.**
Only the last step constitutes organizational value.
## Adoption options, portfolio logic and prerequisites
There is no empirically defensible global ranking of organizational approaches.
The relevant question is **which organizational commitment matches the evidence and risk of the use case**.
| Approach |
Becomes sensible when… |
Main upside |
Main trade-off / failure mode |
Reversibility and learning |
| **Laissez-faire experimentation** |
Uses are low-consequence, data-safe and reversible; objective is discovery |
Very low cost; discovers bottom-up use cases |
Shadow AI, inconsistent handling of sensitive information, no systematic evidence |
Very high reversibility; fast but noisy learning |
| **Approved general enterprise assistant** |
Useful tasks are numerous but shallow; main need is safer access |
Reduces friction and potentially shadow use |
Seat cost can become detached from value; broad access may institutionalize novelty |
High reversibility unless integrations proliferate |
| **Policy plus contextual training** |
Employees are already using AI and need clear boundaries |
Low implementation burden |
Policies without useful alternatives may simply drive use underground |
Highly reversible |
| **Role-specific copilots** |
A family of recurring tasks shares context, tools and quality criteria |
Better fit than generic chat; easier measurement |
Configuration, integration and maintenance cost |
Moderate reversibility |
| **Team-level pilots** |
Benefits are plausible but heterogeneous or uncertain |
Strong learning before irreversible investment |
Volunteer teams and enthusiasts may not represent later users |
Very high if designed with controls |
| **Targeted workflow redesign** |
Local task gains are proven but do not improve the end-to-end process |
Can turn local productivity into organizational performance |
Substantial process/change-management burden |
Moderate or low reversibility |
| **Central AI enablement** |
Common procurement, identity, evaluation, security and model access are repeated across teams |
Economies of scale in common capabilities |
Central team can become a use-case approval bottleneck |
Platform choices create some lock-in |
| **Distributed champions** |
Use cases are local and contextual while common controls already exist |
Faster contextual learning |
Quality and governance can diverge between teams |
High |
| **Custom AI application** |
Workflow-specific context/integration creates defensible value unavailable from commodity tools |
Higher task fit and control |
Engineering, evaluation and maintenance burden; model/vendor churn |
Lower |
| **Agentic/automated workflow** |
Actions are frequent, bounded, observable and recoverable; interfaces are stable |
Removes work rather than merely accelerating it |
Errors acquire operational side effects; permissions and monitoring become critical |
Potentially low once embedded |
| **Delay / do nothing** |
Evidence is weak, verification expensive, risk high or simpler alternatives exist |
Avoids premature cost and lock-in |
Opportunity cost if the capability becomes genuinely useful |
Maximum reversibility |
The empirical basis for avoiding a one-size-fits-all answer is unusually strong: writing, support, consulting and software-development studies produce materially different effects, sometimes in opposite directions. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence)
### Treat adoption as a portfolio
A more useful portfolio taxonomy is not “low/medium/high AI maturity,” but seven observable classes of work:
| Exposure |
Characteristic examples |
Minimum sensible controls |
| **Low-consequence, reversible assistance** |
Brainstorming, rewriting non-sensitive text, personal organization |
Basic acceptable-use rules, AI literacy, approved data boundaries |
| **Internal work product** |
Analysis, code, internal documents, knowledge retrieval |
Access controls, quality expectations, source/verification practices where appropriate, logging sufficient for incident investigation |
| **Externally consequential output** |
Customer responses, contracts, published claims, production code |
Named accountable owner, systematic evaluation, human review or equivalent control, incident path, change monitoring |
| **Decisions affecting people or material rights** |
Some hiring, evaluation, access-to-service or other consequential decisions |
Legal classification, documented oversight, testing for relevant harms, stronger logging and governance; EU high-risk rules may apply depending on use |
| **Agentic execution** |
Systems able to send, modify, approve, purchase, deploy or alter records |
Least privilege, bounded action space, approval gates for irreversible actions, comprehensive auditability, rollback/kill path, operational monitoring |
NIST's Generative AI Profile explicitly provides a risk-management extension to the voluntary AI Risk Management Framework for organizations deploying GenAI; it should be treated as an optional governance framework rather than law. [10](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence)
### What is compliance versus optional good practice in the EU?
As of the October 3, 2026 snapshot, the EU AI Act is in force and has been amended by Regulation (EU) 2026/1744; current interpretation should therefore use the consolidated text rather than the original 2024 act alone. [11](https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng)
**AI literacy:** Article 4 requires AI providers and deployers to take measures supporting sufficient AI literacy among staff and others operating AI systems on their behalf, taking their knowledge, experience and context into account. This requirement is broader than high-risk AI. [4](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-4)
**High-risk AI:** where a company is a deployer of a high-risk system, Article 26 requires appropriate technical and organizational measures, competent and authorized human oversight, monitoring of system operation and, where logs are under the deployer's control, retention for an appropriate period of at least six months unless other law provides otherwise. [12](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-26)
For high-risk workplace systems, Article 26 additionally requires employers to inform workers' representatives and affected workers before putting the system into service or use. This is a specific notification requirement; organizations should not assume that satisfying it exhausts separate labor-law duties that may apply in their jurisdiction. [12](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-26)
The Act's Annex III covers categories of potentially high-risk systems, including employment-related applications; classification nevertheless depends on the actual system and intended use rather than the fact that something uses generative AI. [13](https://ai-act-service-desk.ec.europa.eu/en/ai-act/annex-3)
**Transparency:** Article 50 transparency requirements for certain AI systems began applying on August 2, 2026, and the Commission issued implementation guidance in July/August 2026. [14](https://digital-strategy.ec.europa.eu/en/library/guidelines-transparency-obligations-providers-and-deployers-ai-systems)
**Personal data:** the GDPR remains independently applicable where personal data are processed. The EDPB's Opinion 28/2024 specifically addresses legal basis, anonymization/pseudonymization and personal-data processing in development and deployment of AI models. [15](https://www.edpb.europa.eu/documents/opinion-of-the-board-art-64/opinion-282024-on-certain-data-protection-aspects-related-to_en)
Automated individual decision-making may additionally engage GDPR rules and established EDPB guidance; “there is a human somewhere in the process” should not be assumed to resolve those questions without examining what decision the system actually makes and the meaningfulness of human involvement. [16](https://www.edpb.europa.eu/documents/guideline/automated-decision-making-and-profiling_en)
### Acceptable-use policy should answer operational questions
A useful policy should not merely say “use AI responsibly.” It should make it possible for an employee to determine:
**What information may I provide? What systems may I use? What outputs require verification? What decisions may AI influence? What may it execute? When must I disclose AI involvement? Who remains accountable? Where do I report an incident or surprising behavior?**
An organization with strong rules but no usable sanctioned alternative may still generate shadow use. Conversely, issuing an approved assistant without specifying data and decision boundaries converts a procurement decision into a de facto governance decision.
### Human review is not automatically a sufficient control
The BCG evidence is particularly relevant here: humans can over-rely on plausible AI output and become worse at tasks beyond AI's frontier. [2](https://pubsonline.informs.org/doi/10.1287/orsc.2025.21838)
“Human in the loop” therefore has value only if the reviewer:
- has the competence to detect important mistakes;
- receives enough information and time to review;
- has actual authority to override the system; and
- is not asked to validate such a volume of output that review becomes ceremonial.
For EU high-risk AI, competence, training, authority and support are explicitly part of the human-oversight obligation. [12](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-26)
## Pilot-to-scale and measurement framework
A pilot should answer a **decision question**, not demonstrate that the model can produce output.
“People used it and liked it” is enough evidence for neither workflow redesign nor a large license purchase.
### What a serious pilot should test
Before starting, specify:
**The counterfactual.** What would happen without the AI intervention?
**The unit of analysis.** Task, employee, customer case, team, workflow or process.
**The primary outcome.** For example, customer resolution quality—not “AI usage.”
**One paired productivity/quality measure.** Faster output must be examined alongside errors, rework, customer outcomes or another quality endpoint. The experimental literature shows why both dimensions matter. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence)
**Guardrails.** Security events, privacy breaches, inappropriate outputs, compliance failures, customer harm or unacceptable employee effects.
**Representative tasks and users.** Do not test only enthusiast volunteers doing tasks selected because AI is known to perform them well. Heterogeneous worker effects and METR's later task-selection problem make representativeness a first-order methodological concern. [3](https://arxiv.org/abs/2304.11771)
**Verification effort.** Count checking and correction, not just generation time.
**Total operating cost.** Licenses/inference, integration, training, evaluation, human review, support, incidents and ongoing maintenance.
**Pre-agreed decision thresholds.** Define the smallest benefit worth paying for and the guardrail deterioration that is unacceptable *before* seeing the result.
Where feasible, random assignment or phased rollout with a valid comparison group is substantially more informative than before/after surveys. The strongest organizational studies reviewed used randomization, preregistration or staggered deployment precisely to separate AI effects from ordinary differences among workers and tasks. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence)
### Evidence that justifies scale
A positive pilot should clear four separate gates.
**Outcome gate:** economically meaningful improvement appears in the intended endpoint, not merely usage or satisfaction.
**Safety/quality gate:** relevant quality, customer, security, privacy and compliance guardrails remain acceptable.
**Economics gate:** benefit survives verification, integration, model, support and maintenance costs.
**Transferability gate:** the result persists across representative tasks/users and is not dependent on a handful of champions or a deliberately easy task subset.
Only then should the next question be scale.
Even then, scale should generally proceed in **stages with continued measurement**, because models, product interfaces, usage patterns and task selection can change after initial rollout. METR's later measurement difficulties are a concrete example of the work system adapting around AI. [9](https://metr.org/blog/2026-02-24-uplift-update/)
### Evidence that should trigger redesign
Redesign rather than abandonment when:
- workers become faster but the end-to-end cycle does not improve;
- gains occur primarily in one skill group, suggesting differentiated deployment;
- verification or rework absorbs the generation savings;
- AI shifts a bottleneck from creation to review;
- workers benefit on common cases but fail on unusual high-consequence cases;
- a useful assistant requires so much repetitive human transfer that workflow integration is the actual bottleneck.
The customer-support and BCG evidence supports precisely this kind of differentiated treatment: effects vary by worker experience and task capability frontier. [3](https://arxiv.org/abs/2304.11771)
### Evidence that should trigger stopping
Stop or materially narrow a use when:
**net value remains negative after a reasonable learning period; quality deterioration persists; critical errors are too difficult to detect; verification cost overwhelms savings; users abandon it for real work; simpler deterministic automation performs better; required controls destroy the economics; or the organization cannot establish a lawful/acceptable operating model.**
A stop decision is not an AI failure. It is portfolio discipline.
### Observable measurement framework
| Indicator |
What it tells leadership |
Main limitation / gaming risk |
| **Task time** |
Local efficiency |
Workers may select easier tasks; may omit verification time. METR demonstrates both perception and task-selection problems. [8](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) |
| **End-to-end cycle time** |
Whether local gains survive downstream bottlenecks |
Can be affected by unrelated queue/process changes |
| **Throughput** |
Capacity actually realized |
Easy to inflate by lowering task complexity or quality |
| **Task quality** |
Whether output remains useful/correct |
Requires credible rubric, human evaluation or objective endpoint |
| **Error rate** |
Reliability |
Only measures detectable/defined errors |
| **Rework** |
Hidden downstream cost |
Rework may occur in another team and be poorly attributed |
| **Verification effort** |
Cost of making probabilistic output trustworthy |
Often invisible in product telemetry |
| **Customer outcome** |
Whether internal gains create external value |
May lag and have many confounders |
| **Sustained active use** |
Whether employees continue finding the tool useful |
Usage is a mediator, not an outcome; mandates can game it |
| **Workflow penetration** |
Whether AI is actually part of targeted work |
High penetration can mean forced adoption rather than benefit |
| **Employee experience** |
Friction, perceived control, stress and usefulness |
Self-reports may diverge sharply from actual productivity, as METR showed. [8](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) |
| **Skill/learning retention** |
Whether AI complements or substitutes capability |
Requires testing performance without assistance over time |
| **Incidents / near misses** |
Safety and governance performance |
Under-reporting and rare-event statistics |
| **Abandonment** |
Reveals use cases that fail in practice |
Does not explain whether cause is poor tool, workflow or training |
| **Net operating cost** |
Whether economics work |
Hidden review, integration and support costs are easy to omit |
| **Business endpoint** |
Revenue, cost, backlog, retention, conversion, quality or risk outcome the project was intended to affect |
Attribution becomes harder as distance from the AI intervention increases |
**Do not aggregate these into a single AI score.** A system that scores well on speed and badly on consequential errors should not be allowed to average those facts into a reassuring 82/100.
## Leadership questions and decision reversals
Before spending materially more, redesigning roles, committing to long-term platforms or forcing broad adoption, leadership should be able to give concrete answers to the following.
| Area |
Question |
| **Value** |
What specific business or work-system problem are we trying to change? |
| **Value** |
What happens if we make no AI investment for the next twelve months? |
| **Value** |
What measured outcome would constitute enough improvement to justify the full operating cost? |
| **Alternative** |
Have we compared GenAI with process simplification, search, templates, rules or conventional automation? |
| **Task fit** |
On which exact task classes does AI outperform the current process, and where does it fail? |
| **Heterogeneity** |
Which employees benefit, which do not, and are we hiding those differences inside an average? |
| **Quality** |
How would we know when AI produces a fluent but substantively wrong answer? |
| **Verification** |
Who can reliably verify it, how long does that take, and is verification included in the economics? |
| **Workflow** |
If a worker saves an hour, what organizational outcome actually changes? |
| **Evidence** |
Is the claim based on measured behavior, self-report, a vendor benchmark or a company anecdote? |
| **Pilot validity** |
Were pilot users and tasks representative of the eventual rollout population? |
| **Governance** |
What information may enter the system and what actions may leave it? |
| **Accountability** |
Who owns the result when the AI-assisted process fails? |
| **Legal** |
Does this use process personal data, affect people materially, fall into a regulated sector, or potentially meet an AI Act high-risk category? |
| **Autonomy** |
What is the maximum damage the system can cause before a human can intervene? |
| **Economics** |
What does the system cost after licenses/inference, integration, checking, support, monitoring and model change? |
| **Lock-in** |
What must be rebuilt if the chosen model or vendor becomes unsuitable next year? |
| **Workforce** |
Are we improving work, moving work to reviewers, removing learning opportunities, or changing staffing assumptions? |
| **Scale** |
What new evidence would justify the next increment of investment? |
| **Stop** |
What observable result would cause us to terminate the initiative? |
A leadership team unable to answer those questions does not yet have an adoption problem. It has an **evidence problem**.
### Decision reversals
Many sensible-sounding AI recommendations become wrong when underlying conditions change.
| Apparently sensible advice |
It is sensible when… |
Reverse the advice when… |
| **“Give everyone an approved copilot.”** |
Valuable low-risk tasks are broad and recurring |
Benefits are concentrated in a few roles and unused licenses dominate economics |
| **“Let teams experiment independently.”** |
Mistakes are reversible and information is non-sensitive |
Data exposure, external actions or consequential decisions appear |
| **“Centralize AI adoption.”** |
Common security, procurement, evaluation and integration dominate |
Central approval becomes the bottleneck for highly contextual low-risk uses |
| **“Train everyone.”** |
There are common behaviors everyone genuinely needs, including applicable literacy obligations |
Uses remain undefined and training becomes generic theater; prefer contextual just-in-time learning |
| **“Keep a human in the loop.”** |
Human review can actually detect errors and reviewers have authority/capacity |
Review becomes rubber-stamping, cannot scale, or experts cannot reliably verify outputs. BCG's over-reliance result is the warning case. [2](https://pubsonline.informs.org/doi/10.1287/orsc.2025.21838) |
| **“Scale the successful pilot.”** |
Tasks/users were representative and effects survive verification and cost |
Enthusiasts self-selected, task mix was cherry-picked, or behavior changes after rollout. [9](https://metr.org/blog/2026-02-24-uplift-update/) |
| **“AI helps junior workers most.”** |
AI contains useful expertise they lack and output is verifiable, as in support and some coding contexts. [3](https://arxiv.org/abs/2304.11771) |
The model's knowledge is unreliable for the task or junior staff lack the ability to detect failures |
| **“AI will make experts even faster.”** |
The tool removes genuine expert bottlenecks |
Interaction overhead exceeds assistance value; METR found exactly this for its experienced-developer sample. [8](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) |
| **“Automate after the copilot proves useful.”** |
Actions are bounded, repeatable and recoverable |
A copilot is useful precisely because human judgment resolves ambiguity |
| **“Build a custom solution.”** |
Proprietary workflow/context/integration creates durable value |
Commodity products rapidly cover the need and maintenance becomes the dominant cost |
| **“Standardize on one vendor.”** |
Integration and governance economies outweigh optionality |
Model capability changes quickly or the use portfolio requires materially different providers |
| **“Use adoption rate as the KPI.”** |
Only as a diagnostic for discoverability/friction |
The organization begins forcing usage; adoption then measures compliance rather than value |
| **“Scale because workers report large time savings.”** |
Self-report is corroborated by objective outcomes |
Objective evidence disagrees; METR found workers perceived sizable acceleration while measured completion became slower. [8](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) |
| **“Redesign the whole workflow immediately.”** |
Local evidence is strong and the workflow bottleneck is understood |
Task fit is still uncertain; preserve reversibility through a limited assistant/pilot first |
| **“Wait until AI is more mature.”** |
Error cost and integration burden exceed current benefit |
Experiments are cheap, reversible and create decision-relevant learning now |
The deeper principle is that **governance intensity, organizational commitment and evidence burden should rise together**. The mistake is not decentralization or centralization; it is applying either indiscriminately.
## Claim-to-primary-source table
The table below separates the central decision claims from weaker market narratives and traces them to primary research or official sources.
| Decision-relevant claim |
Primary source |
Evidence type |
What the source actually establishes |
Important boundary |
| GenAI can materially accelerate professional writing while increasing quality |
Noy & Zhang, *Science* / Stanford research record. [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence) |
Randomized, preregistered experiment |
453 professionals; average completion time fell roughly 40%, evaluated output quality rose roughly 18% |
Selected occupational writing tasks, not whole jobs or firms |
| GenAI can raise real-world customer-support productivity |
Brynjolfsson, Li & Raymond, *QJE*, “Generative AI at Work.” [3](https://arxiv.org/abs/2304.11771) |
Staggered field deployment / quasi-experimental evidence |
5,172 agents; roughly 15% more issues resolved per hour on average |
One organizational/work setting |
| GenAI's effects differ substantially by worker skill/experience |
Brynjolfsson, Li & Raymond. [3](https://arxiv.org/abs/2304.11771) |
Field evidence |
Larger gains for less-experienced/lower-skill agents; small gains and some quality decline among top workers |
Does not prove novices always gain most |
| Coding assistants can increase completed development work |
Cui et al., *Management Science*, 2026. [5](https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535) |
Three randomized field experiments |
Combined n=4,867; treatment increased completed tasks by 26.08% |
Specific assistant/work environments |
| AI can also slow experienced developers |
METR randomized productivity study. [8](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) |
Randomized field experiment |
16 experienced open-source developers, 246 real issues; early-2025 AI made completion roughly 19% slower |
Narrow sample and tool era; authors explicitly warn against broad generalization |
| Self-reported productivity can disagree sharply with actual productivity |
METR. [8](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) |
Experimental measurement |
Developers believed AI made them about 20% faster despite measured slowdown |
Particularly strong warning against survey-only ROI |
| AI has a “jagged” task capability frontier |
Dell’Acqua et al., *Organization Science*, 2026. [2](https://pubsonline.informs.org/doi/10.1287/orsc.2025.21838) |
Preregistered randomized field experiment at BCG |
AI improved performance on some realistic consulting tasks but could cause over-reliance and worse performance on tasks outside its frontier |
Frontier changes as models improve |
| A productivity intervention can simultaneously create quality risk |
Dell’Acqua et al.; Brynjolfsson et al. [2](https://pubsonline.informs.org/doi/10.1287/orsc.2025.21838) |
Controlled/field evidence |
Speed/output and correctness/quality need not move together |
Requires task-specific quality definitions |
| GenAI can partly substitute for complementary expertise during ideation |
Dell’Acqua et al., P&G “Cybernetic Teammate.” [6](https://pubsonline.informs.org/doi/10.1287/orsc.2025.20702) |
Preregistered randomized field experiment, 791 professionals |
Individual+AI performance could match human teams; AI reduced functional specialization differences |
Product ideation, not universal team substitution |
| Better generation does not imply better judgment |
P&G experiment. [6](https://pubsonline.informs.org/doi/10.1287/orsc.2025.20702) |
Randomized field evidence |
AI improved idea quality but did not comparably improve selection of the best ideas |
Particularly relevant to decision-support claims |
| AI can improve short-run subjective work experience |
P&G experiment. [6](https://pubsonline.informs.org/doi/10.1287/orsc.2025.20702) |
Experimental treatment with self-reported affect |
Positive emotions increased and negative emotions declined in AI conditions |
Does not establish retention, engagement or long-run well-being |
| One-hour AI training was part of a successful intervention, but its causal necessity is unknown |
P&G study design. [6](https://pubsonline.informs.org/doi/10.1287/orsc.2025.20702) |
Experimental protocol |
AI participants received training/prompt guidance |
Training itself was not independently randomized |
| Later AI-productivity measurement can be distorted by worker/task selection |
METR 2026 update. [9](https://metr.org/blog/2026-02-24-uplift-update/) |
Methodological update from continuing field study |
Willingness to work without AI, task selection and parallel agent use complicated causal estimation |
Highlights a measurement issue rather than estimating universal impact |
| EU providers/deployers have an AI-literacy obligation |
EU AI Act Article 4, official AI Act Service Desk. [4](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-4) |
Binding regulation / official text |
Organizations must take measures supporting sufficient contextual AI literacy |
“Sufficient” is context-dependent; not a mandate for one standardized course |
| EU high-risk AI deployers have stronger operational obligations |
AI Act Article 26. [12](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-26) |
Binding regulation |
Includes competent human oversight, monitoring, certain logging and workplace information obligations |
Applies when the system/use is classified as high-risk |
| High-risk workplace AI requires workers and representatives to be informed before use |
AI Act Article 26(7). [12](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-26) |
Binding regulation |
Explicit workplace information obligation for high-risk AI |
Other national/EU labor duties may be separate |
| AI-related personal-data processing remains subject to GDPR analysis |
EDPB Opinion 28/2024. [15](https://www.edpb.europa.eu/documents/opinion-of-the-board-art-64/opinion-282024-on-certain-data-protection-aspects-related-to_en) |
Official supervisory-board opinion |
Addresses legal basis, anonymity and processing of personal data in AI development/deployment |
Exact obligations depend on processing roles and facts |
| NIST's GenAI Profile is a risk-management framework, not a legal mandate |
NIST AI RMF Generative AI Profile. [10](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence) |
Official voluntary framework |
Provides GenAI-specific risk-management guidance |
Voluntary unless incorporated into contracts/rules elsewhere |
### Overall evidentiary judgment
The evidence now supports a proposition narrower—and more useful—than “GenAI makes companies productive”:
**Generative AI can generate substantial causal improvements in particular cognitive tasks and work settings. Those effects are highly heterogeneous across tasks, workers and human–AI configurations; objectively negative effects are possible; and evidence that these task gains systematically become firm-level revenue, profit or broad organizational performance remains substantially weaker.** [1](https://scale.stanford.edu/publications/experimental-evidence-productivity-effects-generative-artificial-intelligence)
Accordingly, the strongest leadership posture is neither “AI-first” nor “AI-resistant.” It is **evidence-first and portfolio-based**:
**decentralize discovery where consequences are low; centralize common controls and infrastructure where economies or risks justify it; test outcomes rather than enthusiasm; redesign workflows only after locating the real bottleneck; increase governance as consequences and autonomy increase; and treat stopping a weak use case as successful capital allocation rather than failed transformation.**