When you type a question into ChatGPT, Claude, Gemini, or one of the many other AI tools on the market, the answer can look so good that you think: 'Wow, this is smart.'
It looks smart, but it is a guessing engine. Although the engine is sophisticated and fast, and although its makers train it on an enormous amount of text, it is still a guessing engine. The difference between the people who use AI well and the people who use it badly is whether they understand that.
I use AI heavily, on the best models on the market and on the worst, both on my own hardware and in the cloud. I also models and start pre-training jobs. Because I see what happens inside these systems, I want to show you what happens when you type a prompt and receive a response.
What happens when you ask AI a question
Suppose that you type 'Write me a marketing plan for my new SaaS' into an AI tool. A human marketer considers your target market and crafts a strategy, and the AI appears to do the same, although it does something else.
The AI takes your words and predicts which word is most likely to come next. It then predicts the word after that, repeating the step thousands of times until it produces something that looks like a marketing plan.
That is the entire mechanism.
Here's an analogy. Imagine that you read every book, article, and blog post that anyone has ever written about marketing without understanding a word of it, so that you still don't know what marketing is. You do, however, know every pattern of words by heart: which words tend to follow which other words, which sentences tend to appear in which contexts, and which paragraphs tend to come after which headings.
Now suppose that someone asks you to write a marketing plan. You can't think about marketing, because you don't understand it. But you can produce something that looks exactly like a marketing plan, because you know the pattern from every book that you read.
That is what AI does, which is to match patterns at an enormous scale.
The maths behind the magic
The technical details
A large language model is a neural network that its makers train on massive text corpora. During training, the model's weights come to encode probability distributions over sequences of . When you give it a prompt, it performs a forward pass through the network and produces a probability distribution over the vocabulary for the next token. It samples from that distribution (with ) and appends the token that it samples, after which it repeats the whole pass for the next token.
The key phrase is probability distribution. The model assigns a likelihood to every possible next word, and then it samples one. Sometimes the sampled word is a good one and sometimes it is a bad one. Either way, the model never 'knows' what it outputs, because the network has no comprehension layer, no reasoning engine, and no understanding. Some researchers find that models build internal representations that look like more than surface-level pattern matching, so the debate on that point is still open. However you read that debate, what you interact with when you type a prompt is, at its core, a prediction engine.
It's statistics and probability from start to finish.
An 80-billion parameter model has 80 billion numerical weights, and the training process nudges each one to make the model's predictions slightly more accurate. Those weights encode patterns about language, but they encode no understanding of the world. The model can output 'the capital of Ghana is Accra' without any concept of what a capital city is, because 'Accra' overwhelmingly follows 'capital of Ghana' in its training data.
The makers of today's models put them through further training stages such as reinforcement learning from human feedback (RLHF), and they also build chain-of-thought prompting, tool use, and retrieval systems on top of them. With those additions the outputs improve, but the mechanism underneath stays the same, which means that token prediction is still at the base of every response.
Why your results look so good
This is the point at which people mistake the output for intelligence.
During training, a model takes in more text than any human can read in a thousand lifetimes, because its makers compress Wikipedia, textbooks, research papers, code repositories, news articles, and forum discussions into those billions of parameters. When the model produces output, it draws on that enormous volume of text.
The results are impressive because of the breadth of the data behind them, which has nothing to do with intelligence.
Think of it this way: if you give a person access to every document that anyone has ever written and ask them to synthesise an answer, they can produce something excellent too. AI does a version of that, except that in place of synthesis it predicts which words should come next from the patterns in its training data.
This is why AI tends to default to the most common, most predictable response. That is the one with the highest probability: the safe answer, the obvious answer, the 'you are right' answer, and the answer that looks right because it matches the patterns that appear most frequently in the training data.
And to a reader in a hurry, that's enough. Because the output looks polished, sounds confident, and covers the main points, the reader accepts it and moves on, which is the trap.
The most probable answer is the average answer
From my years of working with these systems, I know this: whatever output you get from AI is the most probable answer, which can differ from the complete answer.
The most probable answer is the one that fits the patterns best, so it is the average of everything in the model's training data: the consensus view, the safe take, and the essay that gets a B+ on an exam. It is rarely the insight that changes how someone sees a problem.
If you ask AI to write a business proposal, you get the proposal structure that appears most frequently in its training data. If you ask it to debug your code, you get the fix that most commonly appears with the error pattern. If you ask it to draft an email, it produces the phrasing that most writers in its training data use in that context.
You get the median outcome, which falls short of the best one.
The gap between the median outcome and the best one is where you add your value as a human. You hold the context about your specific situation that the model lacks. You can make the creative leap that it can't, because creativity goes beyond recombining patterns. You can also make the judgement call about what matters most to the person who reads your work. And you have the lived experience from which you know that an approach will fail in your market, your team, or your culture.
AI can do none of that, but you can.
The ownership illusion
When you use AI to produce work and then present that work, you say 'I did this', 'here's my proposal', or 'I wrote this report'. Look closely at why you claim ownership.
The work is the reviewing, the thinking through, the challenging, and the improving. If you skip all of that, then you operate a tool and deliver its raw output, which is a different act from doing the work.
And a hiring manager, a client, or a colleague can tell, because output straight from a model has a flatness to it, a sameness that comes from its convergence towards the statistical middle. A model makes the same most-probable choices in every piece that it generates, which is why one email from a model sounds much like another and why a strategy that a model generates looks like a textbook summary.
How to use AI and keep your own judgement
My advice is simple.
Every time that you get output from AI, treat it as a first draft from someone who reads everything and understands nothing, because at a mechanical level that is exactly what it is.
Use AI as a thinking partner and keep the writing yours
Pull information out of it, challenge its output, and build your own solution on top. The people who are most likely to thrive in a world full of AI are the ones who recognise when the model is wrong, when its output is generic, and when the obvious answer that it returns is the wrong one.
- 1
Review the response with fresh eyes
Read the output slowly. Ask yourself: what does the output leave out? What context is missing from it? What does someone with deep experience in this area add? Where is the output generic, and where should it be specific?
- 2
Think with your own brain
Get up, walk around, and think about the problem with your own mind, which has judgement, experience, taste, and the ability to reason about a situation that is new to you. A prediction engine has none of those, but you do.
- 3
Push back on the AI
Give feedback and challenge the output: ask it to consider the angles that its output leaves out, tell it where its output is generic, and force it into specifics. The harder you push, the more useful the output tends to become.
- 4
Keep your judgement sharp
Use AI as a research partner to speed up your thinking while you keep your own judgement and creativity in the work, because if you accept whatever the model produces, you only get faster.
AI is a tool. Your brain is the product.
AI can find information faster than any human. It can scan enormous datasets, identify patterns, and produce structured output in seconds. Use it, because it is powerful.
But a fast answer can still be a poor one, a long answer can still lack insight, and a probable answer can still be false.
When you think through a problem, notice what is missing, know your audience, and apply judgement from years of lived experience, you always produce better solutions than raw AI output.
I'm a proponent of AI, so use it, lean on it, and let it speed up your work. The question that matters is whether you let AI do your thinking for you or use it to think better.
On the first path you become dependent on a prediction engine, while on the second you become someone who produces work that matters.
Choose carefully.
