Strengths and Limits of Generative AI

Generative AI is genuinely useful and genuinely unreliable in the same breath, and the skill that matters is telling the two apart. Because these systems produce fluent, confident text on any subject, it is easy to mistake fluency for accuracy — but they are not the same thing. A model predicts likely wording; it does not check whether that wording is true, and it has no live connection to the world unless a tool gives it one. Knowing which tasks play to its strengths, and which walk into its weaknesses, is the difference between a fast assistant and a confident source of plausible errors. This lesson maps that boundary so you can lean on the tool for what it does well and verify the rest.

What it is good at: shaping language you supply or can check

Generative models excel at transforming and generating text: drafting a first version, summarising a document you paste in, rephrasing for a different audience, brainstorming, translating, and explaining a concept in plainer words. Notice the pattern — these are tasks about the form of language, where many good answers exist and you can judge the result yourself.

Where it fails: anything needing ground truth

The same system is weak wherever there is a single correct answer it must retrieve rather than generate: specific facts, real citations, events after or outside its training data, exact arithmetic, and tight logical chains. It was built to sound right, not to be right, and here sounding right is not enough.

Hallucination

When a model does not "know" something, it does not stop — it generates the most plausible-looking continuation, which can be a confident fabrication: an invented statistic, a fake citation, a quote no one said. This is called hallucination. It is not the model lying; it is the normal behaviour of a next-token predictor asked for facts it never reliably stored. The danger is that a fabrication reads exactly as fluently as the truth.

Not a search engine, not an oracle

A language model has no built-in live access to the world and no real sense of its own certainty. Some products bolt on search or other tools, but the underlying model still generates rather than looks up. Treat its output as a capable draft to check, not an authority to trust.

You ask a model to summarise a five-page report you paste in and suggest three headline options. This plays to its strengths: the source is right there, and you can check the summary against the original in a minute. Now you ask the same model, "What was our company's Q3 revenue?" with no data supplied. It has no way to know — so it may return a specific, confident, entirely invented figure. Same tool, opposite reliability, because one task provides ground truth and the other demands it.

Strengths and weaknesses are not fixed to a topic — they shift with whether you supply the ground truth. "List the key arguments in this article" (pasted) is a strength; "What does the law say about X?" (from memory) is a weakness, though both look like reading tasks. The dividing line is not the subject but whether the correct answer must be retrieved from the world or generated from the text in front of the model.

A widely reported 2023 case in a United States federal court showed the risk concretely. Lawyers submitted a brief containing case citations that a generative AI tool had produced — and several of the cited cases did not exist. They were realistic-looking but pure fabrication, and the lawyers faced sanctions for filing them unchecked. The exact penalty is specific to that case, but the lesson is general and has recurred since: a model asked for citations will happily invent ones that look authoritative. It connects to Critical Thinking's core habit — verify, do not trust. The fluency of an answer is not evidence of its truth; the check is on you.

• "If it sounds confident and detailed, it's probably right." Fluency and confidence are generated, not earned; they say nothing about accuracy.
• "It's basically a search engine." It generates likely text; by default it does not look anything up.
• "Hallucination is a rare glitch." It is the expected behaviour when a generator is asked for facts it never reliably stored.
• "It can do the maths." It predicts the look of an answer; precise arithmetic and strict logic are exactly where it slips.

Sort a set of AI tasks into "safe to use directly" and "must verify," then justify each call by naming whether the task needs ground truth the model cannot supply.

Think Like a Maester: Ask a generator to shape language and it shines; ask it for the truth and it may hand you something that merely sounds true. Match the task to the tool, and verify anything that has a right answer.

1. Name three tasks generative AI is genuinely good at, and say what they have in common.
2. Why does the model fail on facts, citations, and current events?
3. What is hallucination, and why does it read as convincingly as the truth?
4. Why can supplying the source text turn a risky question into a safe one?
5. How does the "verify, don't trust" habit apply to AI output?

Generative AI is strong wherever the task is about the shape of language — drafting, summarising, brainstorming, translating, reformatting, explaining — especially when you supply the source and can judge the result. It is weak wherever a single correct answer must be retrieved from the world: facts, citations, current events, arithmetic, and precise logic. Because it predicts plausible wording rather than checking truth, it can hallucinate, fabricating confident and realistic-looking errors, and its fluency is no signal of accuracy. Used well, it is a fast drafting-and-thinking aid whose factual claims you verify — not an oracle, and not a search engine.

• Bias, Data, and the Limits of Machine Learning
• Using Generative AI Responsibly