The Jagged Frontier
AI is excellent at some tasks and unreliable at others of similar difficulty — and the boundary is invisible until you test it.
To an executive
AI is excellent at some tasks and unreliable at others that look similar, and mistakes look convincing. Staff need guidance on where to trust it and where to check.
To an engineer
Test AI on your own tasks rather than assuming difficulty matches human difficulty. Keep human checks on work near the boundary, where fluent but wrong output is most likely.
The idea
A 2023 field experiment with 758 Boston Consulting Group consultants, run by researchers from Harvard, Wharton, MIT and BCG, tested GPT-4 on realistic consulting work. On tasks inside the AI's capability, consultants with AI finished more tasks, worked about a quarter faster and produced noticeably higher-quality work. On a task designed to sit just outside that capability, consultants with AI were less likely to get the right answer than those without it, because the AI produced convincing but wrong results.
Ethan Mollick, one of the authors, described the boundary as a jagged frontier: tasks that look equally hard to us may be easy or impossible for the AI, and the edge is not visible in advance.
Why it works
AI difficulty is not human difficulty. Models can write a polished strategy memo yet stumble on a simple-looking calculation or a subtle factual check.
Wrong answers look like right ones. Fluent output hides errors, so people stop checking — the study's authors compared it to falling asleep at the wheel.
Where it applies
- Deciding which tasks to automate or assist with AI.
- Designing review and verification steps in AI-assisted processes.
- Setting expectations with executives and teams adopting AI.
How to apply
- Map tasks by testing, not by intuition. Try the AI on real examples of each task and record where it succeeds and fails.
- Rank by cost of error. Where a wrong answer is costly, keep a person or a deterministic check in the loop.
- Design for verification. Make it easy to check sources, see reasoning and compare against known answers.
- Re-map regularly. The frontier moves with every model release.
Example scenario
An insurer rolled out an AI assistant to claims handlers. It drafted customer letters well, but on a test set it occasionally misread policy exclusions with total confidence.
The team kept letter drafting fully assisted, required a second check on any exclusion decision, and published a short guide to where the assistant was and was not reliable.
Questions to ask
- Which of our tasks have we actually tested inside the frontier, and which are we assuming?
- Where would a confident wrong answer cause real harm?
- How do people know when to trust the AI, and when to check its work?
When not to use it
- Low-stakes, easily checked work, where occasional errors are cheap and mapping every task is not worth the effort.
How it misleads you
Treating a map of the frontier as permanent. Each model upgrade moves it, sometimes in surprising directions, so yesterday's no-go task may now be safe and yesterday's safe task may behave differently.
In the wild
Further reading
Last reviewed · Suggest an improvement