Research Looks At What It Takes To Review AI’s Work
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Research Looks At What It Takes To Review AI’s Work on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

Georgia Tech cognitive engineer Karen Feigh argues that AI’s productivity gains can be offset by the time and mental effort people spend checking its work. Research she cites reports mixed results: AI slowed experienced developers in one trial, sped up consultants on tasks within its capabilities, and led to more errors on a task outside them.

Georgia Tech cognitive engineer Karen Feigh argued in a November 16 essay that workers’ productivity gains from AI can be offset by the time and mental effort needed to check whether its output is accurate and appropriate. The issue matters as organizations use AI for coding, writing and knowledge work, but the studies she cites show that the effects vary by task and do not establish a single productivity result for all users.

Feigh calls the work of checking AI output the “verification burden.” In The Conversation, she describes an executive assistant who uses AI for itineraries and briefings but has encountered enough problems to set aside time to check for inaccuracies and bias. Feigh compares the new responsibility to supervising a fast, polished intern: the system can produce useful work, but also errors in the same confident tone, and it does not bear responsibility for the consequences.

One study she cites, a randomized trial by the research group METR, involved 16 experienced open-source developers completing 246 tasks in large software projects. The trial found that developers took 19% longer on average when AI tools were allowed. Before the study, participants expected AI to reduce completion time by 24%; afterwards, they still estimated that it had saved them 20%. The trial used tools including Cursor Pro and Claude 3.5 and 3.7 Sonnet during February to June 2025.

METR also reported that developers accepted fewer than 44% of AI code generations, while 75% said they read every line of generated code. On a subset of 44 tasks with usable screen recordings, about 9% of AI-allowed working time went to reviewing and cleaning up AI output. These measures describe this particular software trial, not all AI use. Other studies Feigh cites found gains in different settings, including faster professional writing and consulting tasks that fell within the AI’s abilities.

At a glance
reportWhen: Feigh’s essay was published November 16…
The developmentA November 16 essay by Georgia Tech’s Karen Feigh draws on workplace and experimental research to examine the time and attention required to verify AI-generated work.

AI and work · explainer

The verification loop: where AI’s time savings can go

The draft is fast. Knowing whether it is right is the work that follows.

Time for one task (schematic)

AI draft Checking Fixing Re-checking What a speed metric sees What the work costs

Draft: secondsgood and bad parts in the same confident voice About 9% of the timewent to reviewing and cleaning AI output in METR’s trial with AI allowed Speed is not productivitycount the whole loop, errors included

How the hidden cost builds up

  1. DraftThe AI answers in seconds, fluent and confident“few clues about where scrutiny is needed”
  2. CheckDecide what to check and how, without knowing if or where an error iscognitive load
  3. FixFinding an error does not end the work, it starts a correction cycletime
  4. Re-checkEvery fix needs checking again; when nothing is wrong, the effort feels wastedmotivation
  5. Count it allProductivity is the whole human-AI workflow, not the first draftnet, not gross
The real measure: “whether the entire human-AI workflow produces trustworthy work at a lower total cost”

Source: Karen Feigh, Georgia Tech, The Conversation, Nov 16, 2026 (the three burdens, the quotes); METR, arXiv 2507.09089, p. 20 (time spent reviewing and cleaning AI output). The bars are a schematic, not data.

Graphic: Powered by Thorsten Meyer AI

Why Verification Changes Productivity

The findings show why the speed of an AI-generated first draft is not, on its own, a measure of time saved. Workers may need to inspect, test, correct or discard output before they can use it. That follow-up work can take enough time to reduce or erase gains, as the METR trial found for its participants and tasks.

The risk is not limited to time. In the consulting experiment Feigh cites, AI users performed better on tasks within the system’s capabilities but were 19% less likely to produce a correct solution on a task designed to fall outside them. That result makes task fit and review quality relevant to organizations deciding where AI can be used and who remains accountable for its output. The studies do not establish that AI is generally harmful or unproductive; they show that results depend on the work and on whether people can detect mistakes.

Products Worth Considering

Bose QuietComfort Wireless Noise-Cancelling Headphones, Sandstone

Bose QuietComfort Wireless Noise-Cancelling Headphones, Sandstone

As an affiliate, we earn on qualifying purchases.

Different Tasks, Different Results

The research Feigh discusses does not point to one uniform effect. In a study of mid-level professional writing tasks, researchers found that ChatGPT users completed work 40% faster and produced work rated 18% higher in quality. In a separate experiment involving Boston Consulting Group and 758 knowledge workers, participants using AI completed 12.2% more tasks and worked 25.1% faster on tasks within the AI’s capabilities. On a complex task outside those capabilities, they were 19% less likely to give a correct answer.

A separate experiment with 2,784 people examined how well reviewers detected errors in AI-style suggestions. Participants usually caught typos, but their accuracy was 31% for errors requiring an understanding of the task’s rules. That finding suggests that surface proofreading may not be enough to identify substantive problems. It does not mean that every reviewer or workplace will have the same accuracy; the result belongs to the specific experiment.

Feigh also connects the issue to a longstanding concern in automation: a system may exercise practical influence over decisions while a human retains responsibility. Her essay is an expert perspective drawing together existing research, rather than a new experiment. The studies differ in participants, tasks, tools and measures, so their results should not be treated as directly interchangeable.

“AI’s productivity gains “are often countered by time and mental energy spent determining whether the output is accurate and appropriate.””

— Karen Feigh, cognitive engineer at Georgia Tech, in The Conversation

Where the Evidence Has Limits

The evidence does not show that AI always slows work or that review costs outweigh productivity gains in every setting. METR’s result came from a small group of experienced developers working on real tasks in mature repositories with particular tools available in 2025. The authors cautioned that their findings do not show that current AI systems are useless in other economically relevant settings, or rule out gains from future systems.

The studies also do not establish a standard amount of time or training needed to review AI output safely. Results from writing, consulting and coding tasks involve different measures and cannot be combined into a single estimate of workplace productivity. It remains unclear how well the findings apply to other occupations, newer AI models, less experienced users or organizations with different review procedures.

Measure the Full Work Cycle

Feigh’s practical conclusion is that employers should measure the whole workflow, rather than only how quickly AI produces an initial draft. That includes the time spent checking, testing, correcting and integrating output, as well as whether the final work is accurate and appropriate.

She also argues that workers who remain accountable for AI-assisted work need time and training to review it. The research cited does not set out a single review standard or predict when AI will produce net time savings across an organization. Further evidence will need to show how those outcomes vary by task, tool and review process.

Key Questions

What does Karen Feigh mean by the “verification burden”?

She means the time and mental effort workers spend checking AI output for accuracy and appropriateness, which can offset some of the time saved generating it.

Did AI make developers slower in the METR trial?

In the randomized trial cited by Feigh, 16 experienced open-source developers took 19% longer on average when AI tools were allowed across 246 tasks. That finding applies to the study’s participants, tools and tasks, not every software project.

Did the research find that AI always reduces productivity?

No. Other studies cited in Feigh’s essay found faster completion on writing and consulting tasks. The results varied by task, including a consulting task outside the AI’s capabilities where AI users were less likely to answer correctly.

Why can AI errors be hard to spot?

In one experiment with 2,784 people, reviewers were generally better at catching typos than errors that required understanding task rules. Accuracy for the latter errors was 31% in that experiment.

What should employers measure when evaluating AI tools?

Feigh argues they should track the full work cycle: generation, review, correction and integration, along with the quality of the final work. The research does not prescribe one universal measurement method.

Source: ThorstenMeyerAI.com

You May Also Like

How Lifestyle Trends Are Making Gym Shorts Tying More Convenient

Emerging lifestyle shifts are making it easier to tie gym shorts, driven by consumer preferences for convenience and quick solutions, experts say.

How The Research Framed Microsoft’s AI Comparison With Doctors

Microsoft’s peer-reviewed study reported high diagnostic scores for MAI-DxO, but tested it on historical cases against individual generalists, not specialist teams.

Open ASR Leaderboard Expands With First Language From The Global South

The Hugging Face Open ASR Leaderboard now includes Hindi and Indian English, marking the first Indic and Global South languages on the platform, with new diverse datasets.

AI Is Removing The Middle Class Of Software Engineering?

Experts warn AI automation could displace mid-level software engineers, raising concerns about job security and industry shifts.