
Last week I bought a refill gas bottle and discovered when I got home that I couldn’t open it. The screw cap was extraordinarily tight.
So, naturally, I asked Gemini.
It gave me a long list of things I could try, accompanied by warnings about gas, pressure and safety. It was all very thorough. It was also not particularly useful.
There was still enough time to go back to the gas shop, so I put the bottle in the car and drove back. The woman in charge called one of the young men out to help me. He put on a pair of gloves, took the bottle out of the car and opened the cap.
It still needed a little adjustment because it was too tight for me to open by hand. He made the adjustment and handed it back to me. Problem solved.
The sophisticated AI solution had been a long list of instructions. The human solution was a pair of work gloves, a bit of practical knowledge and some muscle.
It was a useful reminder that having access to more information does not necessarily mean having the right answer.
Which brings us back to an old computing maxim: garbage in, garbage out.
With generative AI, however, there is a slightly different problem. Sometimes the answer isn’t garbage at all. It is perfectly plausible, beautifully written and completely beside the point.
There was a time when computers came with a useful warning: garbage in, garbage out.
Generative AI has given us a new problem. Sometimes the garbage comes out looking remarkably well researched.
That is one of the peculiarities of AI. It can produce an answer in seconds, write it with complete fluency and present it with the confidence of an expert who has spent years studying the subject. And yet the answer may be incomplete, misleading or simply wrong.
That doesn’t make AI useless. Far from it. It makes it something we need to use intelligently.
The sensible use of AI is to ask it for information that helps us make a decision. The less sensible use is to ask it to make the decision for us. There is already a growing collection of cases showing what happens when humans quietly move from the driver’s seat to the passenger seat.
When the lawyer believed the machine
One of the classic examples is Mata v. Avianca, a 2023 case in a New York federal court.
A lawyer used ChatGPT to help prepare a court filing. The chatbot obligingly produced a string of impressive-looking legal precedents, complete with case names, quotations and citations.
There was one minor inconvenience: the cases did not exist.
When the court couldn’t find them, the lawyer went back to ChatGPT to check. The machine assured him that the cases were genuine and could be found in reputable legal databases.
They couldn’t.
The lawyers were sanctioned and fined $5,000, and the episode became an early exhibit in the legal profession’s education in generative AI.
The interesting part is that the lawyer wasn’t punished because he used AI. He was punished because he failed to check what AI had produced.
The machine supplied the fiction. The human supplied the signature.
When the chatbot plays doctor
The stakes become rather higher when the subject is health.
The parents of 19-year-old Sam Nelson are suing OpenAI after their son died following what they allege was dangerous advice from ChatGPT about combining drugs. According to the lawsuit, the chatbot discussed the use of Xanax and kratom without adequately recognising the danger. Nelson subsequently died after taking alcohol, Xanax and kratom.
The parents’ account is part of a lawsuit and has yet to be tested in court. But the case raises an uncomfortable question.
A chatbot can explain what a drug does. It cannot examine you.
It doesn’t know whether the pain in your chest is indigestion, anxiety or something considerably more urgent. It doesn’t see that you are pale, confused or struggling to breathe. And it cannot take responsibility for getting the diagnosis wrong.
That last bit is important.
The chatbot that took off without the airline
Air Canada provides a more entertaining example of what happens when companies put AI into customer service and then discover that it has developed an enthusiasm for improvisation.
A passenger, Jake Moffatt, used the airline’s website chatbot after his grandmother died. The chatbot told him he could buy his ticket and apply retrospectively for a bereavement fare.
Moffatt did exactly that.
Air Canada subsequently told him that the fare had to be arranged before travel. It also argued that it should not be held responsible for information supplied by the chatbot.
A Canadian tribunal was not persuaded.
It found Air Canada liable for negligent misrepresentation and ordered it to pay Moffatt about C$650 in damages, interest and fees.
The lesson for companies was rather neatly delivered: if you put the chatbot on your website, customers may reasonably assume that it speaks for you.
Air Canada had, in effect, discovered that artificial intelligence does not come with an “only joking” disclaimer.
The human still gets the bill
These examples point to a more useful way of thinking about AI.
The problem isn’t that AI makes mistakes. Humans make plenty of those without any technological assistance.
The problem is that AI can make mistakes persuasively.
It can produce a legal precedent that looks authentic, medical advice that sounds reassuring or an answer from a customer-service department that appears authoritative. The better the prose, the easier it is to forget that the machine is generating an answer rather than exercising human judgment.
That is why AI works particularly well as a research assistant, sounding board and first-draft machine.
Ask it to explain something. Ask it to find holes in an argument. Ask it what questions you should be asking. Ask it to give you several possible interpretations.
Then think.
That last step remains stubbornly resistant to automation.
The lawyer still has to check the case. The patient still needs a doctor when the circumstances demand one. The company still owns what its chatbot tells customers.
And if an investment goes spectacularly wrong because you followed an AI-generated tip, the machine will not be calling your financial adviser afterwards to explain itself.
AI can provide the map. It can even suggest a few alternative routes. But you are still driving the car.









