🔍 Read the full analysis: Did ChatGPT’s Data Science Help Translate Into Lasting Skills? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A randomized trial involving Boston Consulting Group consultants found that ChatGPT improved performance on coding, statistics and prediction tasks outside many participants’ skill sets. After the chatbot was removed, researchers found no evidence that its users performed better than a control group on technical questions, suggesting the measured gains did not translate into demonstrated lasting skills.
A randomized trial at Boston Consulting Group found that consultants using ChatGPT performed better on data science tasks but showed no evidence of greater technical knowledge when tested without the chatbot afterward. Published October 26 in Nature Human Behaviour, the study suggests that AI can help workers complete unfamiliar tasks without necessarily teaching them to perform those tasks independently.
The study assigned consultants to use either GPT-4 with 15 to 20 minutes of prompting training or conventional online resources, including Google and Stack Overflow. Participants completed two of three 90-minute tasks: Python coding, statistical analysis or prediction. The researchers designed the tasks so ChatGPT could help but could not correctly answer the questions on its own.
Compared with the control group, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The gains brought participants closer to the average scores of 44 BCG data scientists who completed the tasks without AI, but the consultants using ChatGPT still scored below that expert benchmark on average.
After the work tasks, participants answered technical questions about coding, probabilities and machine learning without ChatGPT. The authors reported “no evidence” of better performance by the group that had used the chatbot. They describe the task gains as temporary upskilling. The study also found that ChatGPT users became more confident that the tool could solve problems it could not; the authors characterize that result as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Gains Are Not Skill Gains
The findings separate doing a task with AI from learning how to do it without AI. For employers, that distinction matters when evaluating whether a chatbot has improved workforce capability, rather than simply helping people produce better work while the tool is available.
If businesses treat assisted performance as evidence of independent expertise, they could misjudge training outcomes, hiring needs and quality controls. The experiment does not show that AI can never teach durable skills; it shows that, in this trial, a brief training session and access to ChatGPT did not produce a detectable advantage on the researchers’ subsequent unaided test.
The results also point to a practical trade-off. ChatGPT users completed coding and prediction tasks faster on average—by about nine minutes and 14 minutes, respectively—while time spent on statistics did not differ significantly. That may make AI useful for immediate workplace output, even if the study does not support counting those gains as lasting individual skill.
Products Worth Considering
Samsung Odyssey OLED G5 27-Inch QHD Gaming Monitor
LG UltraGear evo 27-inch 5K Mini-LED Gaming Monitor
Apple MacBook Air 13-inch with M5 Chip, 16GB Memory, 512GB SSD, Silver
As an affiliate, we earn on qualifying purchases.
Inside the BCG Experiment
The trial took place in late March and early April 2024 and was preregistered before data collection. Of 986 consultants allocated to the groups, 573 began the survey and 487 submitted work on both assigned tasks, forming the main analysis sample. Participants had at least a bachelor’s degree; the authors said most reported no coding experience.
Researchers compared participant results with work from 44 BCG data scientists who completed the same tasks without AI. The coding task involved cleaning, merging and filtering datasets; the statistics task asked participants to identify modeling errors and interpret data; and the prediction task involved modeling the predictability of international football matches. The authors say tasks were designed with BCG data scientists and OpenAI’s Economic Impacts Research team so that copying ChatGPT’s output would not be sufficient.
The paper’s title, “Generative AI and the temporary upskilling of knowledge workers,” reflects its central interpretation. The authors compare AI assistance to an exoskeleton: it can extend what a person can do while in use, without necessarily changing the person’s unaided ability. The result is based on a specific workplace experiment, not a test of every AI tool, training approach or job.
““We find no evidence” that the ChatGPT group performed better on the follow-up technical questions without the tool.”
— The study authors
Limits of the Follow-Up Test
The follow-up consisted of five technical questions administered after the task work, not a long-term assessment of participants’ skills. The reported result therefore does not establish whether repeated practice, a longer course of instruction or different ways of using AI could lead to durable learning. Nor does it show how participants would perform on every real-world data science assignment.
The experiment involved consultants at one company, tasks developed for the study and GPT-4, rather than a broad sample of workers or all current AI systems. While the authors report no evidence of a group advantage after tool removal, that finding should not be read as proof that no individual learned anything. The authors also label the finding about increased confidence in ChatGPT’s abilities exploratory, so it is not a definitive measure of how that confidence affects work.
Testing Durable Workplace Learning
The published study establishes its results for this trial; the source material does not identify a scheduled follow-up experiment or a next research milestone. A key next step for researchers would be to test whether longer or more structured training, repeated practice and feedback produce better performance after AI access ends.
For employers applying the findings, the distinction can be tested directly: assess workers’ output with AI, then evaluate relevant skills independently when those skills matter to the role. The study does not prescribe a specific workplace policy, but it supports reporting AI-assisted output and unaided capability separately rather than treating one as a substitute for the other.
Key Questions
Did ChatGPT improve the consultants’ task performance?
Yes. The ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group.
Did the consultants retain more data science knowledge?
The researchers found no evidence of better performance by the ChatGPT group on technical questions answered without the tool. The follow-up was brief, so it does not settle whether other forms of training could produce lasting learning.
Did ChatGPT users match the BCG data scientists?
No, not on average. Participants using ChatGPT moved closer to the benchmark set by 44 BCG data scientists, but the group remained below the experts across the three tasks.
How was the comparison group equipped?
The control group received training in using Google and resources such as Stack Overflow and Khan Academy, rather than ChatGPT, to work on the tasks.
Does this prove AI cannot teach lasting skills?
No. The study tests a particular short training session, tool, workplace and set of tasks. Its finding is narrower: in this trial, the ChatGPT group showed no detectable advantage on the later unaided test.
Source: ThorstenMeyerAI.com
