Skip to content
Latticework
Esc
Try
No. 050Risk & Resilience

Ironies of Automation

Automating the routine work leaves people with the hardest tasks, monitoring they do poorly, and fading skills, just when a failure needs them most.

Automation takes the routine and leaves people the hardest cases, just as their practice and knowledge fade.

To an executive

Automation does not remove the need for skilled people; it makes their job rarer and harder. Plan for how staff keep the skill to catch and fix AI mistakes.

To an engineer

A human in the loop is not a safety net by default. Design review, alerts and hand-back so people can spot failures and act, and keep them practised.

The idea

In a short 1983 paper in Automatica, the cognitive psychologist Lisanne Bainbridge described how automating industrial process control could make the human operator's problems worse rather than better. The more advanced the automation, she argued, the more crucial the human's contribution becomes.

The ironies follow from how automation is designed. Designers automate what they can and leave the operator what they cannot: the rare, unusual and hardest situations. The operator's remaining job is to monitor the system and take over when it fails. But people are poor at sustained monitoring of something that rarely goes wrong, and the manual skill and process knowledge needed to take over fade without practice. So the operator is least prepared at the moment they are most needed.

The same pattern appears when AI agents draft code, decisions or customer replies and a human "reviews" them. The more reliable the AI, the rarer and harder the errors left for the reviewer.

Why it works

The leftover tasks are the hardest ones. What cannot be automated is what designers could not specify: novel faults and edge cases. Those need the most skill, not the least.

Monitoring is a task humans do badly. Vigilance studies show attention fades quickly when little happens. Reviewing a stream of mostly correct AI output invites rubber-stamping.

Skill and understanding decay without use. Operators build their knowledge of a system by working with it. Take away the routine work and you take away the practice that would let them diagnose a failure.

Automation can hide problems. A system that compensates for a fault quietly may do so until the situation is beyond recovery, and then hand it over abruptly.

Where it applies

  • Human review of AI output, such as approving agent-written code, claims decisions or contract summaries.
  • Agentic workflows across many applications, where an agent acts in several systems and a person must step in when it goes wrong.
  • Operations and incident response, where automated remediation and AIOps tools take over routine handling.
  • Knowledge retention, where the people who understood a process move on once it is automated.

How to apply

  1. Design the failure path first. Decide how the AI signals uncertainty or failure, who takes over, and what context they get.
  2. Make review a real task. Give reviewers small batches, the evidence behind each output and a reason to look closely; measure whether they catch seeded errors.
  3. Keep people practised. Rotate staff through doing the work by hand, run drills and simulations, and let them take control for short periods.
  4. Prefer obvious failure. Where you can, make the system stop or fail visibly rather than degrade silently.
  5. Set autonomy by risk. Match how much the AI does alone to how well people can still catch its mistakes.

Example scenario

A bank used an AI agent to prepare payment-exception cases and a team of analysts to approve them. Within months the agent's drafts were right almost every time, approval rates rose close to one hundred per cent, and the team was cut. Then a change in a partner bank's message format led the agent to misread a field, and dozens of wrong payments were approved before anyone noticed. The remaining analysts had not resolved a case by hand for months.

The architects redesigned the loop: the agent flagged low-confidence cases, analysts handled a share of cases end to end every week, the team ran monthly drills with seeded faults, and format changes from partners triggered a pause and a manual check.

Questions to ask

  • When the AI gets it wrong, who notices, how, and what do they need to know to take over?
  • How will reviewers keep the skill and context to judge the output if they rarely do the work themselves?
  • Are we asking people to watch for rare errors in a stream of mostly correct output, a task humans are known to do badly?

When not to use it

  • Fully automated tasks with safe failure, where the system can simply stop and nobody needs to take over.
  • Low-stakes outputs that are cheap to correct after the fact.

How it misleads you

The ironies can be read as an argument against automation, which they are not; Bainbridge proposed ways to design around them. The opposite mistake is common too: adding a "human in the loop" and treating it as a control without checking whether that person can actually catch errors. Test the human part of the system as you would the software.

In the wild

Further reading

Last reviewed · Suggest an improvement