How AI Chatbots Actually Work
ChatGPT, Claude and Gemini can feel like magic, or like a person. They're neither. Learn what's really going on, in plain English, and play with a tiny word predictor that works on the same basic idea.
The big idea: a very good guesser
An AI chatbot is built on a large language model, or LLM. Strip away the chat window and an LLM does one job: given some text, it guesses what comes next, one small piece at a time.
You already know a tiny version of this. Your phone keyboard suggests the next word as you type. An LLM is the same idea scaled up enormously: it has learned from a huge amount of text, so its guesses take in grammar, facts, style, and the whole conversation so far.
- “Peanut butter and …” → almost certainly “jelly”
- “The capital of France is …” → “Paris”
- A whole answer is just this step repeated hundreds of times, each new word added to the text before guessing the next.
Why it feels smart: to guess the next word well across billions of examples, the model has to pick up patterns about how the world works and how people explain things. That's why it can summarize, translate and reason through problems. But underneath, it's still predicting text, not looking up facts in a database.
How a chatbot is trained
- Collect text. Companies gather a massive amount of writing: public web pages, books, articles, code and more.
- Pre-training. The model reads that text and plays “guess the next word” billions of times. Each wrong guess slightly adjusts its inner settings, called parameters. Big models have billions of them. This step takes huge computer clusters, weeks or months, and a lot of electricity.
- Fine-tuning. A raw model just continues text. To turn it into a helpful assistant, it's trained further on example conversations and on feedback from people who rate which answers are better. This is where it learns to follow instructions, be polite, and refuse harmful requests.
- Release. After training, the model is mostly frozen. Using it (“inference”) doesn't teach it new things in real time.
It stops learning at a date
The model only knows what was in its training text, up to a cutoff date. Ask about something newer and it may not know, or may guess.
Some chatbots can look things up
Many apps add tools like web search. Then the chatbot reads current pages and writes an answer from them, often with links you can check.
Memory is a feature, not the model. Within one chat, the model sees the earlier messages because the app sends the whole conversation each time. Features that “remember” you between chats work by saving notes and adding them to the conversation, not by retraining the model.
Tokens: how AI reads text
Models don't read letters or whole words. They split text into tokens: common words, pieces of words, spaces and punctuation. Each token becomes a number the model can do math on.
OpenAI's rule of thumb for English: one token is about 4 characters, or about ¾ of a word. So 100 tokens is roughly 75 words. Other languages can use more tokens for the same idea.
This is a simplified splitter that breaks off common word parts like “un-”, “-ly” and “-ing”. Real tokenizers learn their pieces from data, so their exact splits differ, but the idea is the same.
Why chatbots fumble letter-counting: a word like “strawberry” may reach the model as one or two tokens, not ten letters. It never “sees” the individual r's, so questions about spelling, letter counts or rhymes can trip it up.
Context window. A model can only pay attention to a limited number of tokens at once: your messages, its replies and any files you add. In a very long chat, the earliest parts may fall out of view or get less attention. Starting a fresh chat often helps.
Predicting the next word
For every step, the model gives each possible next token a probability. “Peanut butter and” might give “jelly” 85%, “chocolate” 6%, “bananas” 4%, and tiny chances to thousands of others.
Then the app picks one. Always taking the top choice makes text dull and repetitive, so chatbots usually add a little randomness. A setting called temperature controls how much:
- Low temperature: sticks to the most likely words. Predictable, focused, sometimes repetitive.
- High temperature: gives unlikely words a better chance. More surprising and creative, but more likely to go off the rails.
Drag left for safe, predictable picks. Drag right to give rare words more of a chance.
This toy looks only at the last word and knows a small table of word pairs. A real model weighs the whole conversation using billions of parameters, but it still ends with a list of probabilities like these.
Same question, different answer? That's the randomness at work. It's great for brainstorming. For facts, if two tries disagree, that's a sign to check a trusted source.
Why chatbots make mistakes
A chatbot is trained to produce text that sounds right. Most of the time, sounding right and being right line up. When they don't, you get a confident, fluent, wrong answer. This is called a hallucination.
| Mistake | Why it happens | What to do |
|---|---|---|
| Made-up facts, quotes or links | It predicts what a citation or quote would look like, and may invent one that fits | Click every link. Search for quotes yourself. |
| Outdated info | Training stopped at a cutoff date | Ask it to search the web, or check an official site |
| Math slips | It predicts digits as text instead of calculating | Use a calculator, or ask it to use a code or calculator tool if it has one |
| Agrees with you too easily | Trained to be helpful, it can follow a wrong premise in your question | Ask neutrally: “Is this true?” not “Why is this true?” |
| Bias or one-sided answers | It learned from human writing, which has biases | Ask for other viewpoints and check sources |
Don't rely on a chatbot alone for medical, legal, tax or financial decisions. It's useful for understanding a topic and preparing questions, then confirm with a professional or the official source.
What it's great at
- Drafting and rewriting: emails, letters, summaries, outlines.
- Explaining things in simpler words, again and again, without getting tired of you.
- Brainstorming lists of ideas, names or options.
- Working with text you give it: summarizing a long document, pulling out key points, translating.
The common thread: tasks where you can easily check the result. Want to get more out of it? Read How to Write Good AI Prompts.
FAQCommon questions
Does the chatbot understand what it says?
Researchers still debate what “understand” means here. What's clear is that it doesn't experience anything or have beliefs the way people do. It produces text based on patterns it learned. Treat it like a capable tool, not a person or an oracle.
Does it search the internet every time?
No. On its own, a model only knows its training data. Many chatbots can also run a web search when needed, and they usually show links when they do. If you see no sources, assume it's answering from memory.
Is it learning from my chats right now?
Not in real time. The model doesn't change while you talk to it. But companies may save conversations and use them to train future models, depending on your settings. ChatGPT, Claude and Gemini all have a setting to opt out.
Why does it say “I can't help with that”?
During fine-tuning, models are trained to decline some requests, like ones that could cause harm. Sometimes they're overly cautious. Rephrasing with more context about what you're actually trying to do often helps.
What's the difference between AI, machine learning and an LLM?
AI is the broad goal of computers doing tasks that seem to need intelligence. Machine learning is one way to get there: learning patterns from data instead of following hand-written rules. An LLM is a kind of machine learning model trained on text, which powers today's chatbots.
Can AI detectors tell if text was written by AI?
Not reliably. Detectors can flag human writing as AI and miss AI writing, so their results shouldn't be treated as proof.