TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT scored higher on three data science tasks than a control group using Google and reference sites. After ChatGPT was removed, researchers found no evidence that the users had gained more technical knowledge than the control group.
A randomized trial involving Boston Consulting Group consultants found that access to ChatGPT raised scores on coding, statistics and prediction tasks, but a follow-up test found no evidence of a measurable knowledge gain once the chatbot was unavailable. The study, published October 26 in Nature Human Behaviour, bears on whether workplace AI assistance builds lasting skills or mainly helps people perform while the tool is at hand.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
The trial was conducted at BCG in late March and early April 2024. Researchers allocated 986 consultants equally between a ChatGPT group and a control group. Of those, 487 submitted work on both tasks and made up the main analysis sample. Most participants reported no coding experience, according to the authors.
The ChatGPT group received 15 to 20 minutes of prompting instruction and used GPT-4. The control group received instruction on searching with Google and using sites including Stack Overflow and Khan Academy. Participants each completed two 90-minute tasks selected from coding, statistics and prediction. The tasks were designed by BCG data scientists and OpenAI’s Economic Impacts Research team so that ChatGPT could offer guidance but could not answer every question correctly on its own.
Compared with the control group, ChatGPT users scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers also compared participants’ task scores with the average of 44 BCG data scientists who completed the tasks without AI. ChatGPT users closed much of the gap on some tasks but, on average, still scored below the data scientists. After the work, participants answered five technical questions without ChatGPT. The authors reported no evidence that the ChatGPT group performed better on that test; the joint comparison had a p-value of 0.531.
Performance Gains Depend on Access
The results separate doing a task with AI from learning to do it independently. In this trial, brief training and ChatGPT access helped consultants produce stronger work in areas outside their expertise. The follow-up test, however, did not show that those gains carried over to an unaided technical assessment.
For employers, that distinction affects how they evaluate AI in training, staffing and quality control. A worker who can complete a task with a chatbot may still need expert review, especially when the work involves code, statistical reasoning or forecasts. The study suggests that companies should measure both assisted output and unaided understanding before describing AI use as skills development.
The finding does not show that AI can never teach technical skills. It describes outcomes in one workplace trial, with specific tasks, a short training session and a particular model. Still, it gives companies evidence that short-term productivity and lasting learning are different outcomes, and that one should not be assumed from the other.
Products Worth Considering
Bose QuietComfort Wireless Noise-Cancelling Headphones, White Smoke
Samsung Odyssey G3 G30D 27-Inch FHD Gaming Monitor
Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite
As an affiliate, we earn on qualifying purchases.
How BCG Tested ChatGPT
The study, titled “Generative AI and the temporary upskilling of knowledge workers,” was published in Nature Human Behaviour on October 26, 2026. The trial was preregistered on March 13, 2024, before data collection began. Its comparison group used conventional online search and reference resources rather than working without digital assistance.
Each participant completed two of three tasks: cleaning, merging and filtering datasets in Python; assessing a statistical model and interpreting residual plots; or building a model related to the predictability of international football matches. The study also included the data scientist benchmark, giving researchers a reference point for how consultants performed relative to specialists.
The authors describe generative AI as an “exoskeleton”: a tool that can extend a user’s capabilities while it is being used. They report that participants who had ChatGPT were also more confident that the tool could solve problems it could not actually solve. The authors characterize this confidence result as exploratory, so it should be treated as a secondary observation rather than a central finding.
““Generative AI and the temporary upskilling of knowledge workers””
— Wiles, Krayer and co-authors
What the Follow-Up Can Show
The follow-up consisted of five technical questions, and the study does not establish whether different instruction, longer practice or repeated use would lead to stronger unaided skills. The reported result is no evidence of a difference between groups on this test; it does not prove that no participant learned anything.
The findings also come from a single consulting firm, a particular set of tasks and GPT-4 as used in 2024. They do not establish how other occupations, newer models or longer training programs would compare. The source material reports the main analysis sample and key score differences, but does not provide enough detail here to assess every possible limitation or subgroup result.
The higher confidence in ChatGPT’s ability to solve problems it could not solve was labeled exploratory by the authors. Its size and practical effects are not specified in the material available for this report. It remains unclear how often that confidence would affect real workplace decisions.
Testing Lasting Workplace Skills
The paper provides a basis for further studies that track workers over longer periods and test whether learning persists after repeated practice with AI. Useful follow-up work could compare short prompting sessions with more extensive instruction, and measure both task quality with AI and technical performance without it.
For employers applying the findings now, the immediate question is how to evaluate AI-assisted work alongside workers’ independent understanding. The BCG trial does not prescribe a training or hiring policy. It does show why assessments that only measure output while a chatbot is available may not answer whether employees have acquired skills they can use on their own.
Key Questions
What did the BCG trial find?
Consultants with ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. They still scored below BCG data scientists on average.
Did ChatGPT users learn more technical skills?
After ChatGPT was removed, participants answered five technical questions. The authors found no evidence that the ChatGPT group did better than the control group on that test.
When and where was the trial conducted?
The trial took place at Boston Consulting Group in late March and early April 2024. The study was published in Nature Human Behaviour on October 26, 2026.
What did the control group use?
The control group received instruction on using Google and online reference sites, including Stack Overflow and Khan Academy. It did not receive ChatGPT for the tasks.
Does this prove AI cannot teach people?
No. The trial tested a particular model, a brief training session and specific tasks. Its results show no measurable advantage on the study’s unaided follow-up test, not that longer or different forms of AI-supported learning cannot work.
Source: Thorsten Meyer AI
