In the study, titled "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality," professors from Harvard Business School, engineers from MIT, and consultants from Boston Consulting Group (BCG) explored how AI can be integrated into the workflows of knowledge workers, focusing on how AI affects performance on complex tasks.
The study involved 758 consultants at BCG, and the findings were striking: the AI-augmented consultants completed 12.2% more tasks on average, finished them 25.1% faster. They also delivered work that was 40% higher in quality than that of their counterparts who did not use AI.
These results held true across 18 different tasks, ranging from creative idea generation to analytical problem-solving. Whether the task involved drafting a press release or segmenting a market, AI proved to be a powerful ally.
AI is a skill leveler with uneven capabilities
One of the key findings in the study is AI's ability to level the playing field. Consultants who initially performed poorly saw the biggest improvement when using AI – up to 43%. While top performers also benefited, their gains were smaller. This "skill leveling" effect could have profound implications for the workplace, potentially reducing the gap between average and high performers.
Image credit: Study paper, via Ethan Mollick's article
AI isn't perfect, and understanding its limitations is key. The study also introduces the concept of the "jagged frontier" – a metaphor for the uneven capabilities of AI. Imagine a fortress wall with towers jutting out unpredictably. Inside the wall, AI excels at tasks. Outside, it struggles and is prone to errors. The challenge is that this wall is invisible, and tasks that seem equally difficult can fall on different sides of the frontier.
Image credit: Study paper, via Ethan Mollick's article
We’ve seen this happen over and over: AI might breeze through generating creative ideas but occasionally, yet hilariously, stumbles over basic math problems or randomization. The study found that while AI-augmented consultants consistently outperformed their peers on tasks within the frontier, they sometimes struggled with tasks just outside of AI's capabilities. These tasks are beyond the so-called jagged frontier. It's important to know where the frontier goes, and play to your strengths wherever you excel over AI.
Are you a centaur or a cyborg?
The scientists analyzed how people involved in the study interacted with AI and identified three distinct styles: centaur, cyborg, and sleeping driver. Understanding these styles can help individuals and organizations optimize their use of AI and avoid common pitfalls.
Centaurs: Strategic division of labor
The centaur style draws inspiration from the mythical creature that is half-human and half-horse, symbolizing a clear division between human and machine tasks. In this approach, the human user strategically decides which tasks to handle and which to delegate to AI. This style reflects an awareness of both the strengths and weaknesses of AI, allowing for a balanced and efficient collaboration.
For example, a centaur-style worker might use AI to generate ideas for a new product but take on the task of refining and finalizing those ideas themselves. In the BCG study, participants who adopted this style were able to maintain control over their work while leveraging AI's capabilities for specific tasks. This approach proved particularly effective for tasks within AI's technological frontier, where AI's capabilities are well-suited to the task at hand.
The paper offers a detailed case in point: during a task that involved conceptualizing a new footwear product, participants acting as centaurs delegated certain aspects – such as generating a list of product features – to AI, while they focused on refining the market strategy and messaging. This strategic division led to a 43% improvement in work quality for those who initially performed at a lower level and a 17% improvement for high performers.
Cyborgs: Seamless integration
While the centaur approach emphasizes a division of labor, the cyborg style represents a more seamlessly integrated collaboration between human and AI. In this approach, the boundary between human and machine becomes blurred, much like the cybernetic organisms of science fiction. Cyborg-style workers use AI not just as a tool but as an extension of their own cognitive processes.
In the BCG study, cyborgs worked closely with AI, engaging in a back-and-forth interaction where tasks were not simply handed off to the machine but co-created. For instance, a consultant working on a creative writing task might begin by drafting a paragraph, then ask AI for suggestions, and subsequently refine the text based on the AI's input. This iterative process can lead to higher-quality outputs and a more efficient workflow.
An example from the study illustrates this approach. When tasked with creating a marketing strategy for a niche footwear product, cyborg-style participants collaborated with AI to brainstorm ideas, write drafts, and edit the final product. This close collaboration led to significant improvements in both task completion rates and overall quality.
Interestingly, while both centaur and cyborg approaches were effective, the study found that those who received additional training in prompt engineering – a key skill for maximizing AI effectiveness – outperformed those who used AI without such guidance. These participants were able to achieve a 46.6% improvement in their work quality, highlighting the importance of understanding how to communicate effectively with AI.
Blind reliance on AI is like sleeping at the wheel
But there's a catch. Whether you’re a cyborg or a centaur, over-reliance on AI can backfire, leading to what one of the study's authors, Fabrizio Dell'Acqua, calls "falling asleep at the wheel" in his study titled Falling Asleep at the Wheel: Human/AI Collaboration in a Field Experiment on HR Recruiters. Self-driving cars are a great off-the-cuff example of this problem: AI performs so well that humans stop paying attention, essentially "falling asleep at the wheel" and letting the AI take over. However, when human input is suddenly needed, relying too much on AI can lead to serious consequences.
Similar risks from over-relying on AI can be found in many business environments as well. Dell’Acqua found that recruiters who used high-quality AI became lazy and made worse decisions than those who relied on their own judgment. This is an example of people ignoring the jagged frontier of AI.
Similarly to the car example above, in the BCG experiment, consultants who relied too heavily on AI made more mistakes on a task specifically designed to be outside the AI's frontier. While human consultants got the task right 84% of the time, those using AI dropped to 60-70%. The AI provided convincing but incorrect answers, and consultants who trusted it blindly paid the price.
In contrast to the centaur and cyborg styles, the sleeping driver approach represents a passive and overly reliant use of AI. Like a driver who falls asleep at the wheel, these individuals allow AI to take full control of the task, often without critical oversight. This approach can lead to significant errors and lower-quality work.
For example, in a business case analysis task, sleeping drivers who relied entirely on AI to interpret data and make recommendations were 19 percentage points less likely to arrive at the correct solution compared to those who maintained a more active role in the process. This highlights the dangers of over-reliance on AI, particularly in tasks that fall outside the technological frontier where AI's capabilities are limited.
The takeaway
The tools that elite consultants used in this study aren't exclusive. They're the same tools available to anyone with access to GPT-4. And as AI continues to evolve, these tools will only get better. But with great power comes great responsibility. AI can make work more productive, interesting, and meaningful – but only if we make conscious choices about how to use it.
As the jagged frontier of AI capabilities expands, the question isn't whether AI will reshape work, but how we want that transformation to happen. Will we become cyborgs and centaurs, working alongside AI to produce the best possible outcomes? Or will we fall asleep at the wheel, letting AI take over without realizing its limitations?
For knowledge workers and organizations, the key takeaway is that AI should be seen as a powerful tool, but not a replacement for human expertise. By adopting a centaur or cyborg approach, workers can maximize the benefits of AI while avoiding the pitfalls of over-reliance.
For instance, when working on tasks that involve creative problem-solving, such as developing a new product or writing persuasive content, a cyborg approach can help workers leverage AI's strengths without losing control of the final output. Meanwhile, for more routine tasks – such as data analysis – a centaur approach can allow workers to delegate effectively while focusing on higher-level strategic decisions.