Computer says no: The need for explainable AI

I’ve always been a bit of anAI cynic. For all the talk of robots taking over the world, the reality is thatmost artificial intelligence has been fundamentally crap.

Half of the AI efforts we’veseen have either turned out to be thinly veiled If-This-Then-That algorithms,or chatbots that are no more sophisticated than Clippy the paperclip.

Despite this personalcynicism, even I have to admit that in 2019, “genuine” AI has seen an explosionin real-world applications.

Artificial intelligence iseverywhere. It’s being used to make investment decisions, to decide whetheremployees should be hired or fired, and even to pre-emptively diagnosediseases. In fact, AI-poweredtrading algorithms are already responsible for 50% of all stockmarket transactions, while 14 of the UK’s 43 police forces are adopting AI-powered‘crime-prediction’ software to profile potential criminals.

For the Luddites among us, allof this may sound terrifying. But the reality is that there’s nothing wrongwith AI being used to make these decisions — as long as it can justify itsreasoning.

This, however, is where theproblem lies. Currently, the vast majority of AI systems operate as a ‘blackbox’, with data being input and seemingly unrelated decisions being output.Unlike a traditional computer model, AI-based systems don’t follow a clearlogic path, their decisions are — for want of a better phrase — their own.

This problem is only set toget worse with the rise of geneticprogramming. Here, computer algorithms ‘design themselves’through a process of Darwinian natural selection. Computer code is initiallygenerated randomly and then repeatedly shuffled to emulate reproduction. Everyso often, a random mutation is thrown in to liven things up.

This natural selection ofcode, combined with the opaque decision making of AI, is making it harder thanever to understand what is going on under the hood of our machines.

To address this concern,Google has recently thrown its weight behind the push for ‘explainable AI’.

Along with a series of new visual AI development tools,Google’s push also includes the adoption of counterfactuals within AI. Inshort, this means that AI systems are asked to justify their decisions, testingwhether they would come to the same conclusions if certain data sets wereremoved. By running this process multiple times, a clearer picture can emergeof how AI is reaching its decisions.

Sadly, even Google admits that this system isn’t perfect. Commenting on a recent Tech Tent podcast, Google’s Dr Andrew Moore suggested that counterfactuals cannot truly explain AI's decisions. Instead, they simply provide a better “diagnosis” for how the decision came about.

While this sounds concerning,in reality, such a broad diagnosis may be more than enough to go on.

Black box thinking has always been an element of decision making — regardless of whether it’s a decision by a human or an AI. For most professionals, whether a lawyer, doctor, stockbroker or judge, there is an element of decision making that simply can’t be explained, if only because, as a species, we’re terrible at knowing our own minds.

The economist Daniel Susskind hasexplained this idea with the example of Tiger Woods:

“If you asked Tiger Woods toexplain how he hits a golf ball so far, he might be able to offer you insightinto a few of the thoughts that pass through his mind as he swings the club. Hemight also perhaps pass on a few hints. But he would struggle to articulate thecomplex network of accumulated heuristics, intuitions, and hand-to-eyeinteractions that have contributed to his supremacy as a golfer. Many of thesewill be unconscious, inculcated through repeated practice and use, and some sodeeply embedded that even Tiger himself would be unaware of them. Yet none ofthis precludes us from building a mechanical swinging arm that could hit thegolf ball further and straighter than Tiger.”

In short, most of us cannotexplain every aspect of our own thinking. Instead, we can only give a roughdiagnosis of how we arrived at our conclusions.

The question is, if such broaddiagnosis is considered good enough for humans, should it also be considered goodenough for AI?