🔍 Read the full analysis: How Consultants’ Data Science Performance Changed With ChatGPT on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. But in a follow-up test without ChatGPT, researchers found no evidence that the users had gained more technical knowledge.
A randomized trial involving Boston Consulting Group consultants found that using ChatGPT improved performance on coding, statistics and prediction tasks, but the boost did not carry over to a later technical test taken without the chatbot. The study, published October 26 in Nature Human Behaviour, suggests AI can help workers perform tasks outside their usual expertise without necessarily teaching them to do those tasks independently.
Researchers assigned consultants to use either GPT-4 or conventional online resources, including Google and Stack Overflow, after a short training session. The trial was conducted at BCG in late March and early April 2024. Participants completed two of three 90-minute tasks: a Python coding exercise, a statistics exercise or a task predicting the outcomes of international football matches.
Compared with the control group, consultants using ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers also reported that the ChatGPT group completed coding and prediction faster on average, by about nine and 14 minutes, respectively. They found no significant difference in time spent on the statistics task.
The comparison with specialists was a separate benchmark: 44 BCG data scientists completed the tasks without AI. The ChatGPT group as a whole remained below their average on all three. After the tasks, participants answered technical questions without ChatGPT. The authors said they found no evidence of a learning advantage for the group that had used the tool.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Gains Did Not Equal Skill Gains
The findings distinguish between doing a task with AI and acquiring the knowledge to do it without AI. For employers, that difference affects how they interpret productivity gains, staff training and workers’ readiness to take on technical assignments.
The results indicate that a chatbot can help non-specialists produce stronger work in the moment, at least on these structured tasks. But the follow-up test does not show that the short AI-assisted experience made participants more capable when the support was removed. Companies that treat assisted performance as proof of independent skill could overestimate what employees have learned or what they can reliably do without the tool.
The study does not establish that AI cannot teach technical skills. It tests a particular short training, set of tasks and follow-up assessment. Its evidence is narrower: in this trial, the immediate performance gains were not matched by a detectable improvement on the subsequent unaided questions.
Products Worth Considering
Canon SELPHY CP1500 Wireless Dye-Sublimation Photo Printer Bundle with 108 Sheets of Paper and 3 Ink Cartridges
HP OfficeJet 250 Wireless Mobile All-in-One Inkjet Printer
Dell XPS Tower 8960 Desktop, Intel Core i7-13700, NVIDIA GeForce RTX 4060, 16GB DDR5, 512GB SSD
As an affiliate, we earn on qualifying purchases.
Inside the BCG Randomized Trial
The researchers allocated 986 consultants to treatment and control groups. Of those, 573 began the survey and 487 submitted work on both assigned tasks, forming the main analysis sample. Participants had at least a bachelor’s degree; the authors reported that most said they had no coding experience.
The ChatGPT group received 15 to 20 minutes of instruction on prompting the model for technical problems. The control group was trained to use Google and sites such as Stack Overflow and Khan Academy. Researchers designed the tasks with BCG data scientists and OpenAI’s Economic Impacts Research team so that ChatGPT could assist but could not independently provide correct answers to every question.
The trial was preregistered on March 13, 2024, before data collection. The data scientists’ scores provided an expert benchmark, rather than a comparison between AI and experts using the same tools. The researchers also tested participants after the work tasks, asking five questions about probability, machine learning and coding without permitting ChatGPT.
“The authors report that they “find no evidence” that the ChatGPT group did better on technical questions answered without the tool.”
— The study authors
Limits of the Learning Test
The follow-up consisted of five technical questions asked after the work tasks. The reported finding of no evidence of a difference does not prove that no participant learned anything, nor does it show how performance might change after longer training or repeated use. The source material does not provide enough detail to establish whether learning might emerge over a longer period.
The results also come from one company, a particular group of consultants and three deliberately constructed tasks. The tasks were developed so ChatGPT could be useful without reliably solving them on its own. That design helps assess how people work with an imperfect assistant, but it does not represent every workplace, AI system or technical assignment.
The authors also reported that ChatGPT users were more confident that the tool could solve problems it actually could not. They characterized that analysis as exploratory, so it should not be treated as a definitive measure of how often users overestimate AI. The study does not settle whether longer or different forms of training can produce durable skills.
Testing Longer-Term Skill Retention
The published results establish what happened in this trial, not whether the same pattern holds across other occupations or longer training programs. Further research would need to test whether repeated practice, feedback or instruction changes unaided performance, and whether any skills persist over time.
For employers applying the findings, a practical next step is to distinguish AI-assisted output from independently demonstrated ability. The study does not prescribe a hiring or quality-control policy, but its results give organizations reason to test both: what employees can produce with a tool, and what they can explain or complete without it.
Key Questions
What did the BCG study find about ChatGPT and data science tasks?
Consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. The reported gaps were 49, 20 and 17 percentage points, respectively.
Did the ChatGPT users outperform BCG data scientists?
No. The ChatGPT group, on average, remained below the 44 BCG data scientists who completed the tasks without AI.
Did using ChatGPT improve consultants’ technical knowledge?
On a later test taken without ChatGPT, the authors said they found no evidence that the ChatGPT group did better than the control group. That result does not establish that no individual learned anything.
How long was the ChatGPT training?
The treatment group received 15 to 20 minutes of training on prompting ChatGPT for technical problems before completing the tasks with GPT-4.
Can the study show whether longer AI training works?
No. The trial examined a short training and a particular set of tasks and follow-up questions. It does not determine whether longer instruction or repeated practice could lead to lasting skills.
Source: ThorstenMeyerAI.com
