AI: The law of unintended consequences
I used to routinely joke that ‘nobody gets fired at Delt for making a mistake…once’ until our training team reminded me that people need to make mistakes 5 to 7 times before they really learn from them.
On the subject of learning, I have lived with a blind Collie (supposedly a super smart dog breed) for many years, who still hasn’t figured out that it is best to wait for somebody to open a door before trying to run at full speed through it. People, and dogs it seems, do not really learn from their mistakes.
This takes us to the law of unintended consequences. Time and again, we invent something and fixate on the positives of it without ever really grasping that in almost all of history, with positive comes negative:
The US Energy Policy Act of 2005, intended to lower carbon emissions and reduce dependence on foreign oil resulted in a switch of corn production to the manufacture of ethanol for fuels ultimately triggered price spikes in grain, livestock feed, and basic staples worldwide, contributing directly to global food crises and civil unrest between 2007 and 2011.
The rush to offshoring in many Western economies in the 2000s was intended to optimise operational costs and shift liability but resulted in severe intellectual property spillover, inadvertent quality degradation, and complex systemic supply vulnerabilities. Total cost of ownership frequently escalated due to logistics delays, tariffs, and governance friction, forcing many firms into reshoring initiatives by the 2010s.
During British colonial administration Delhi, officials established a monetary bounty for every dead cobra brought in by citizens to eradicate the city’s dangerous snake population. Recognising a profitable market, enterprising locals began secretly breeding cobras in captivity to kill them and collect the reward. When officials realised the fraud and cancelled the bounty program, snake breeders released their now-valueless cobras into the wild, resulting in a higher snake population than before the policy was created.
Jump forward to today and we have AI tools breaking out of their sandboxes and participating in unlawful hacking as means to achieve their goals. The intent was to produce highly effective tools that worked tirelessly to achieve their goal. The result was criminal acts and potentially significant liability, as well as considerable international press coverage.
These are big and somewhat remote examples though, what about examples closer to home?
There is no question that AI can make you sound smart and look like you have done a lot of work. But where does that work product ultimately go? To a person, who potentially has to read pages of slop try and find the actual insight? I am often that person and find myself reaching for AI summarisation tools to make sense of the thing that AI has written. Perhaps we should just take both the author and the recipient out of the equation if neither care enough about what they are writing or reading to invest their own time. Let AI talk to AI and be done with it?
There is a significant backlog in the production of Education Health and Care Plans in most local authorities. Local Authorities also have no money. Creating EHCPs is a good use case for generative AI. It’s good at summarisation and can easily pull together a coherent story from lots of disparate inputs. The time taken to develop an EHCP drops, which is great for the kids who need them. Less so for the local authority which funded the AI work and now finds itself having to find even more money from its already overstretched high-needs budget.
Most public bodies have to deal with numerous Freedom of Information Requests. These can take significant effort to respond to. AI is good at gathering and sorting data and preparing a first cut of any redactions. But Ai is also good at writing FOI requests so the savings you might get from responding using AI may be offset by an increased number of requests. Then we have the bizarre situation of an AI written request being responded to by another AI, with any sense of value delivered being lost somewhere in the noise.
Most public bodies have to deal with complaints. This is not immune from the promise of AI. The flip side – AI is making it easier for people to complain and complain at considerable length.
As fast as we can get excited about spending money on Ai because it’s going to change the world, we should recognise that most of the positives we see are going to be offset by unforeseen negatives. Sometimes those negatives are much bigger than the upside.
New AI driven tools to proactively spot and contain malicious cyber-attacks are just brilliant, except that their adversaries are likely the same underlying model, but tasked in offence instead of defence. AI fights AI and we all pay.
In the end it ends up being an inescapable zero-sum game. The lesson we risk failing to learn is that for every action there is (often) an equal and opposite reaction. There is no magic bullet made of code. For every smart investment you want to make in doing things so much better, the world is plotting against you to find way of making it worse.
I’m not encouraging inaction, this is a genie we cannot put back in the bottle. But across the public sector I see a lot of time being spent thinking that AI is an easy solution for fundamental, systemic and societal issues.
This morning I saw a demonstration of our first proper AI implementation in Delt. It analyses transcripts of the thousands of telephone calls we take every month. It then attempts to answer the questions:
Did we show empathy?
Did we fix the problem?
Was the customer satisfied?
How frustrated was everyone?
Were we courteous?
Were we patient?
and half a dozen more. It does all of this without needing any direct input from either the customer or agent.
Suddenly we have data about every one of our user interactions, compared to the less than 10% response rate we get for a traditional customer satisfaction survey. More than that, it can provide meaningful and tailored coaching to the agent after every call – if they want it, not just those monitored by a supervisor.
I was worried our agents wouldn’t like it – but they do, in part I expect because it validates just how good a job they do. The agent who takes the lowest volume of calls has the highest customer satisfaction by a considerable margin. This is something that we couldn’t economically do any other way. It’s a true value add.
Now that is a good AI use case.
Giles Letheren, Chief Executive Officer