Gen AI systems are not substitutes for Google or even a damn good book – please stop treating like they are, then complaining when they aren’t!
So I’ve been hearing a lot about what AI can’t do as the resistance to AI in education mounts, just as the pressure to engage increases (Newtonian physics playing out in human behaviour). The core argument I hear most frequently is about how AI can ‘make things up’ when you ask it for information – well maybe, just maybe don’t do that and you might see your experience of AI improve!
I’m going to make a deliberately provocative statement: generative AI is not a source of information. Now, before the technically minded among you start typing furious corrections, let me clarify what I actually mean.
A raw large language model is a pattern matching system, designed specifically for creating plausible-sounding text. That’s its job. That’s what the transformer architecture was built to do. Can it produce information? Yes, absolutely. Can it produce accurate information? Often, yes. But here’s the critical issue: this accuracy is neither guaranteed nor verifiable without external checking – and that’s the fundamental problem.
Getting information from an LLM is like getting directions from someone with a photographic memory but no sense of which photos they’re remembering. They might be perfectly accurate, describing the route in vivid, confident detail. Or they might be just as confidently describing a completely different city. The confidence level doesn’t change, the plausibility doesn’t waver, but the reliability is fundamentally uncertain without verification.
What Pattern Matching Actually Means
A large language model isn’t just a very clever internet search engine. That’s not what it’s doing. In fact, it works quite differently. It’s a probabilistic pattern matching engine. That’s a simplification, and I’m sure people who know far more about LLMs than I do will come and tell me just how wrong that is. But essentially, it is stochastic. It’s making predictions based on patterns in what you’ve asked it, patterns in its training data, the weights it’s been given, the reinforcement learning it’s undergone, and the temperature setting it’s running at.
Is it meant to be very deterministic, sticking to the highest probability responses? Or is it going to get a little bit creative, throwing some randomness in there depending on where its temperature is set? Different models are designed to be different and these are technical parameters that fundamentally affect output, but they’re all about pattern generation, not fact retrieval.
Now, ontologically speaking, if you ask “What’s the capital of France?” and it responds “Paris,” yes, that IS information. You’ve learned something. But here’s the crucial point: the LLM didn’t retrieve that fact from a database. It didn’t look it up. It pattern-matched your question structure to patterns in its training data where “capital” + “France” appeared near “Paris” with sufficient frequency and weighting.
Yes, these models ARE explicitly trained to reproduce information accurately – through massive datasets, optimisation algorithms, and reinforcement learning from human feedback. But the process is fundamentally different from retrieval. The model reconstructs patterns rather than accessing stored facts. And that’s where reliability becomes unpredictable.
Consistent Process, Inconsistent Output
Here’s where the reliability question gets interesting. LLMs have a consistent process but produce inconsistent outputs. You can absolutely trust a large language model to generate patterns that match what it’s been programmed to match based on your input and its training data. That’s 100% reliable. What you cannot trust is that those patterns will be factually accurate, complete, or even the same from one query to the next.
The probabilistic and predictive nature of transformer architecture makes outputs inherently unstable. A subtle variation in how the model processes data, a tiny change in your input, a different random seed – any of these can shift the output pattern. Sometimes these patterns are deliberately built to lack stability. Sometimes a change in one tiny component echoes out to create a bigger change in output. This is not a bug, it’s a feature. But it means treating an LLM as a reliable source of information is fundamentally misunderstanding what the technology does.
The Reliability Gradient (Or: When Can You Actually Trust This Thing?)
Here’s the nuance that matters for practical use: reliability varies wildly depending on what you’re asking.
Very reliable (well-established, frequently reinforced patterns):
- Basic factual information (capitals, dates of major historical events)
- Common knowledge (Hamlet’s themes, basic scientific principles)
- Standard explanations of mainstream concepts
Moderately reliable (more complex synthesis, less common patterns):
- Explanations of specialised topics
- Historical interpretations
- Process descriptions
Highly unreliable (requiring current data, precise accuracy, or rare information):
- Current events or data beyond training cutoff
- Precise numbers, statistics, or quotes
- Specialised technical information
- Anything involving reasoning about novel situations
- Questions requiring current verification (exchange rates, stock prices, recent legislation)
This gradient matters enormously for educators and parents. Asking an LLM to explain photosynthesis to a Year 7 student? Probably fine, but verify key details. Asking it for the exact wording of a recent policy document? Absolutely not reliable without checking the source.
The problem is that the LLM presents all these answers with the same confident tone, the same apparent authority. It doesn’t signal “I’m less certain about this one.” And most users – including our students – haven’t learned to recognise which questions fall into which category.
The Boundaries Are Blurring (And That Matters)
Now, I need to acknowledge that the landscape is changing rapidly. The distinction I’m drawing between pattern matching and information retrieval is deliberately being collapsed by newer systems:
- Perplexity, SearchGPT, and Gemini with search capabilities are combining traditional search with LLM processing
- RAG (Retrieval Augmented Generation) increasingly grounds LLM outputs in retrieved documents
- Citation features in newer models show sources alongside generated text
- Tools like Notebook LM ground themselves entirely in user-provided materials
These developments are significant. Notebook LM, for instance, is extremely useful and powerful precisely because it will ground itself only in the material you give it. Its language model interprets the data you provide, the information you supply. It’s a knowledge extraction tool, and this is where treating LLMs differently becomes crucial.
When we use large language models as direct sources of information, they become inherently dangerous – that’s where hallucinations are genuinely problematic. But when we use them to extract knowledge from verified sources, when we can check where that knowledge came from and verify its accuracy, they become far more powerful and useful. The issue is that most people – including most of our students and their parents – are using them in the former manner, not the latter.
But Even Search-Enabled AI Has Serious Limitations
Here’s something crucial that often gets overlooked: even when AI systems have search capabilities, they’re operating with one hand tied behind their back compared to human researchers.
What search-enabled AI cannot access:
- PDFs embedded in websites or linked as downloads
- Content behind paywalls (news sites, academic journals, specialist databases)
- Academic databases requiring institutional login (JSTOR, ProQuest, PubMed full texts)
- Supplementary materials and attachments
- Content requiring registration or login
- Many government documents in non-web formats
- Specialist archives and collections
This means that whilst a university student can access comprehensive academic databases through their institution, the AI assisting them cannot. Whilst a teacher can download and read that curriculum document PDF, the AI cannot. Whilst a researcher can access paywalled journal articles, the AI is limited to abstracts and summaries.
The practical implication: Search-enabled AI is essentially limited to the free, open, crawlable internet. That’s valuable, but it’s a fraction of what human researchers with proper access can reach. It’s like having a research assistant who can only look at books visible through shop windows but can’t actually go inside the library.
This matters enormously for:
- Academic research: Where the best sources are often behind institutional paywalls
- Policy work: Where official documents are frequently PDFs
- Professional contexts: Where industry reports and specialist publications require subscriptions
- Verification: Where you need to check primary sources that aren’t freely available
So when we talk about AI’s limitations as an information source, this isn’t just about hallucinations or pattern matching. Even with search capabilities, these systems have structural access limitations that mean they’re working with a fundamentally incomplete information landscape compared to what humans with appropriate access can reach.
For educators, this means: Teaching students that AI can “search the internet” isn’t enough. They need to understand that proper research often requires accessing sources the AI simply cannot reach, and that human researchers with library access, institutional subscriptions, or the ability to request documents still have capabilities AI lacks.
The Screwdriver and the Chisel
Let me try an analogy. Using an LLM as an information source is like using a screwdriver as a chisel. Both are tools. A flat-head screwdriver might actually work as a chisel for a little while. You might even get some results. But you’re likely to get a poorer outcome than if you’d used the correct tool. More importantly, you’re likely to injure yourself and break the tool in the process.
The problem isn’t that the screwdriver is useless – it’s brilliant at what it’s designed for. The problem is using it for something it wasn’t built to do, and then complaining when it doesn’t work as well as a proper chisel. This isn’t an argument against screwdrivers. It’s an argument for understanding which tool to use when.
Now, here’s the caveat: large language models ARE being marketed as chisels when they’re really screwdrivers. They phrase their output in ways that seem to suggest they are authoritative information sources. The companies running these systems aren’t in a hurry to counter that idea, because it’s a very useful way to get people engaged.
So this isn’t purely user error – it’s a combination of marketing manipulation and user misunderstanding, and we need to address both. The companies need to be clearer about limitations. But we, as educators, can’t wait for that to happen. We need to teach the distinction now, because our students are using these tools today.
What This Means for Educators (And Parents)
This matters enormously in educational contexts because:
Students ARE using ChatGPT as a Wikipedia replacement. They’re asking it direct questions and accepting the answers uncritically. This creates immediate assessment integrity issues, but more worryingly, it’s teaching them fundamentally flawed research practices.
Parents are using it to “help” with homework. “Just ask the AI” becomes the default response, training the next generation to outsource thinking rather than develop it.
We’re teaching the wrong skills. If we don’t clarify the distinction between pattern matching and information retrieval, we’re setting students up to be manipulated by confident-sounding but unreliable outputs for the rest of their lives.
But here’s what we should be worried about even more: we’re designing assessments that can be easily gamed by pattern matching systems. If an LLM can produce a plausible essay answer, what does that tell us about our essay questions? If it can explain a concept well enough to get marks, what does that say about what we’re actually assessing?
What LLMs Actually Excel At
Let me be absolutely clear: this doesn’t mean LLMs aren’t useful tools or powerful technologies. They are both. It’s just that what they do well is different from what most people think they do.
LLMs are brilliant pattern framework devices for reflecting back your own thinking, for refining things, for working in different modes. That becomes a metacognitive tool, and that’s genuinely powerful. You’re not looking at it for information. You’re looking at it to reflect your thought process, your query, getting it to ask questions, getting it to test you.
What LLMs ARE good for in education:
- Brainstorming and ideation
- Reflection and metacognition
- Drafting and iteration
- Explaining concepts in different ways
- Generating practice questions
- Providing feedback on written work (with appropriate guardrails)
- Creating structured outlines and frameworks
What LLMs are NOT reliably good for:
- Fact-checking and primary research
- Citation and referencing
- Current events (unless specifically designed with search capabilities)
- Specialised technical information
- Anything requiring absolute accuracy
Using them for clarification, for conceptualising, for ideation – that’s powerful. But looking at a large language model to produce facts without verification is an inherent error, because that’s not what they’re primarily designed for.
The “Good Enough” Context
Now, here’s something important: sometimes LLM outputs ARE good enough even with reliability uncertainties. Context matters.
Low-stakes contexts where accuracy is less critical:
- Brainstorming sessions exploring possibilities
- Rough drafts that will be edited and verified
- Creative writing prompts where accuracy is irrelevant
- Generating example scenarios for discussion
- Explaining concepts that you’ll verify against other sources
High-stakes contexts requiring verification:
- Assessment submissions
- Research papers and citations
- Medical, legal, or financial information
- Anything influencing important decisions
- Information being shared with others as fact
Teaching students to recognise this distinction – when is “good enough” acceptable, and when do you need verified accuracy? – is a crucial digital literacy skill.
Practical Verification Strategies
Saying “check sources diligently” sounds good, but what does that actually mean for a busy teacher or a parent helping with homework? Here are practical strategies:
Cross-reference with authoritative sources: If the LLM tells you something, check it against established, reliable sources in that field. Academic databases, official government sites, established educational resources. Your own knowledge for pity’s sake! – If it feels off, it probably is.
Use multiple LLMs and compare outputs: Ask the same question to ChatGPT, Claude, and Gemini. Do they agree? If there’s divergence, that’s your signal to dig deeper.
Look for citations and check them: If the LLM provides sources, actually follow those links. Check they say what the LLM claims they say. Check they’re legitimate sources – remember the AI may authoritatively quote an academic paper and equally authoritatively quote a conspiracy theory website!
Be especially sceptical of specific details: Numbers, dates, quotes, statistics – these are exactly where LLMs are most likely to hallucinate whilst sounding completely confident.
Use specialised tools when you need sourced information: Perplexity, SearchGPT, or other search-enabled systems are better choices when you need verifiable information with sources – most LLMs now have a search feature, but again – check the source!
Teach students the verification process: Don’t just do it for them. Show them how to check, why to check, and what to look for. This is the digital literacy skill that will serve them for life.
The Source-Checking Imperative
One of the most valuable developments in recent LLM versions is that they now show their working – not actual thinking, but the different patterns emerging as they feed their own inputs back in and refine outputs accordingly. More importantly, many now show you sources: “This is where I got this information from.”
If you’re using an LLM for research – particularly with deep research models or RAG-enabled systems – you’re not actually getting facts from the model. The model is collecting sources, synthesising information for you. But here’s the crucial bit: you need to be diligent about checking those sources. The sources of information are the sources, not the LLM.
This is a skill we have to hone and develop in ourselves and, more importantly, teach to our students. In a world where AI can generate plausible-sounding text about anything, the ability to verify information, to trace claims back to legitimate sources, to distinguish between confident assertion and actual evidence – these become fundamental literacy skills, not nice-to-haves.
Stop Blaming the Tool (But Hold Everyone Accountable)
So much discussion still frames this as “Can we trust AI?” or “AI isn’t a good source of information.” But this misses the point entirely. It’s like saying screwdrivers are bad because they make terrible chisels.
The tool does what it was designed to do. The problem is the gap between what it was designed for (pattern matching and language generation) and how it’s being marketed (as an authoritative information source) combined with how users are treating it (as a substitute for research and verification).
We need to hold multiple parties accountable:
- Companies need to be clearer about limitations and capabilities
- Educators need to teach proper use and verification skills
- Users (students, parents, all of us) need to learn to engage critically with AI outputs
But ultimately, if you can’t deal with uncertainty in your output, then a raw generative AI tool probably isn’t the right choice for your use case. And that’s fine. Different tools for different jobs.
Building a Culture of Verification
Here’s what good AI literacy actually looks like in practice:
Students who:
- Understand that LLMs are processing tools, not knowledge databases
- Automatically verify specific claims, especially numbers and quotes
- Know when AI is appropriate to use and when it isn’t
- Can evaluate the reliability gradient of different question types
- Use AI to enhance their thinking, not replace it
Educators who:
- Model critical engagement with AI outputs
- Design assessments that require human verification and synthesis
- Teach verification strategies as core digital literacy
- Understand the capabilities and limitations well enough to guide students
- Use AI appropriately in their own practice
Parents who:
- Understand the difference between using AI as a thinking partner vs answer machine
- Can help children verify information rather than just accepting it
- Model good digital literacy in their own use
- Support learning rather than short-cutting it
This isn’t about rejecting AI. It’s about using it intelligently, with eyes wide open to both its capabilities and its limitations.
The Path Forward
We need to fundamentally rethink how we discuss and teach about AI. The technology creates patterns based on input. In terms of language generation, in terms of coding, in terms of ideation and reflection – that’s exactly what you’re looking for. But it is not a source of truth or a direct source of information, because that’s not how the model works. That’s not how the output is created.
The real question isn’t “Can we trust AI?” but rather “What can we trust AI to do, and how do we verify what it produces?” The answer: we can trust it to match patterns, to process language, to generate plausible text, to reflect our thinking back at us in useful ways, to accelerate certain types of work. We cannot trust it to be our primary information source without verification, our fact-checker, or our research librarian.
For educators and parents, the imperative is clear: teach students to understand what these tools actually do, how to use them effectively, and crucially, how to verify anything they produce. Pattern matching systems are powerful. They’re transformative. They’re here to stay. They will shape the informational landscape our students navigate for the rest of their lives.
The future belongs not to those who can avoid AI, but to those who can use it wisely whilst maintaining critical thinking and verification skills. These tools should augment human intelligence, not replace it. They should accelerate research, not substitute for it. They should support learning, not short-circuit it.
Maybe it’s time we stopped arguing about whether AI is trustworthy and started teaching the skills needed to engage with it critically and effectively. The technology isn’t going away. The question is: will we shape how it’s used, or will we let marketing and misconceptions do that for us?
Pattern-matching systems are here to stay. The question is no longer if our students will use them, but how. Will we teach them to be masters of these powerful tools, or leave them to be passive consumers of their plausible fictions? The choice is ours.
And, as ever, if this was in any way helpful, do sign up for alerts and don’t forget to follow me on LinkedIn, Facebook, Instagram, Bluesky and X (Twitter).