What Is a Large Language Model? How LLMs Work (2026)
What is a large language model? How LLMs turn text into tokens and predict the next one, real examples, LoRA and chain-of-thought, and how they read websites.
Kazi Khalid Hasan Tamim · BugViso Team
17 min read
A large language model (LLM) is a neural network trained on huge amounts of text to predict the next token, a word or word fragment, given everything before it. Repeat that prediction thousands of times and you get an answer, a summary or code. ChatGPT, Claude, Gemini and Llama are all built on large language models.
By the end you will know how an LLM turns text into tokens, why the transformer made today's models possible, what LoRA fine-tuning and chain-of-thought prompting actually do, and how these models read a website, with a script that counts how many tokens your own pages cost.
Data used: all 274 blog pages of bugviso.com from the static export built on 2026-11-15, tokenised with OpenAI's
o200k_baseencoding (tiktoken 0.12.0) on 2026-11-16. We counted tokens in the raw HTML, in all visible text, and in the<main>/<article>text only. The script is at the end of the post.
How a Large Language Model Works, in Four Steps
An LLM works in four steps: it splits text into tokens, turns each token into a vector, mixes those vectors with attention layers, and outputs a probability for every possible next token. It then picks one, appends it, and runs again. Everything an LLM "knows" lives in the numbers, called parameters, that control those steps.
- Tokenise. The input is cut into tokens from a fixed vocabulary. Common words are one token; rare words split into pieces.
- Embed. Each token ID maps to a long list of numbers (a vector) that encodes its meaning and position in the sequence.
- Attend. A stack of transformer layers lets every token look at every earlier token and decide which ones matter. This is how "it" in a sentence gets linked to the right noun.
- Predict. The last layer scores every token in the vocabulary. The model samples one, adds it to the text, and repeats until it stops.
The architecture behind step 3 is the transformer, introduced by Google researchers in the 2017 paper Attention Is All You Need. It replaced recurrent networks with attention alone, which made training far more parallel. That is the reason models could grow from millions to hundreds of billions of parameters: GPT-3, described in Language Models are Few-Shot Learners, has 175 billion.
"Large" refers to both sides of that equation. The parameter count is large, and so is the training text, typically a big slice of the public web plus books, code and licensed data.
Tokens: The Unit Every LLM Reads, Bills and Forgets
A token is the unit an LLM reads and writes, and it is usually smaller than a word. Models are priced per token, their memory (the context window) is measured in tokens, and anything that falls outside that window is invisible to them.
Here is real output from OpenAI's open-source tokenizer, run on 2026-11-16:
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # GPT-4o-era encoding
for text in ["What is a large language model?", "Low-rank adaptation",
"bugviso.com/blog", "Großes Sprachmodell"]:
ids = enc.encode(text)
print(len(ids), [enc.decode([i]) for i in ids])7 ['What', ' is', ' a', ' large', ' language', ' model', '?']
4 ['Low', '-r', 'ank', ' adaptation']
4 ['bug', 'viso', '.com', '/blog']
5 ['Gro', 'ß', 'es', ' Sprach', 'modell']| Text | o200k_base tokens | cl100k_base tokens (GPT-4 era) |
|---|---|---|
| What is a large language model? | 7 | 7 |
| Low-rank adaptation | 4 | 4 |
| bugviso.com/blog | 4 | 4 |
| Großes Sprachmodell (German for "large language model") | 5 | 7 |
Three things show up straight away. Common English words cost one token each, including the leading space. A brand name the tokenizer has rarely seen, like "bugviso", splits into fragments. And the newer, larger vocabulary handles German more cheaply: 5 tokens instead of 7 for the same phrase, so the same budget holds more non-English text.
Context windows on current frontier models range from tens of thousands of tokens to around a million, depending on the model and plan. That sounds like a lot until you see how many tokens a real web page costs, which is the data section below.
Large Language Model Examples You Already Use
The best-known large language model examples are OpenAI's GPT models (behind ChatGPT), Anthropic's Claude, Google's Gemini, Meta's Llama, and open-weight families from Mistral and DeepSeek. Some are only available as hosted products; others publish their weights so anyone can run them.
| Model family | Company | How you get it | Product most people know |
|---|---|---|---|
| GPT | OpenAI | Hosted (app and API) | ChatGPT |
| Claude | Anthropic | Hosted (app and API) | Claude |
| Gemini | Hosted (app, API, inside Search) | Gemini, Google AI Overviews | |
| Llama | Meta | Open weights | Meta AI, self-hosted apps |
| Mistral | Mistral AI | Open weights and hosted | Le Chat |
| DeepSeek | DeepSeek | Open weights and hosted | DeepSeek app |
Keep two ideas apart. A model is the trained network. A product like ChatGPT or Perplexity wraps one or more models with a chat interface, web search, memory and safety filters. Perplexity, for example, is an answer engine that runs language models over live search results rather than a single model of its own brand.
That distinction matters for anyone who publishes on the web. The model's weights decide what it remembers from training; the product's search layer decides which pages it fetches today. Our breakdown of how AI search engines choose sources covers that second layer engine by engine.
How LLMs Are Trained: Pre-Training, Fine-Tuning and LoRA
LLMs are trained in two stages: pre-training on raw text to learn language, then fine-tuning on curated examples to learn a task or a style of answering. Pre-training is where almost all the compute goes. Fine-tuning is where a general text predictor becomes an assistant that follows instructions.
Pre-training is self-supervised. The model sees a stretch of text, predicts the next token, gets scored against the real one, and its parameters shift slightly. Repeat that across trillions of tokens and grammar, facts and reasoning patterns emerge as a side effect of getting predictions right.
Fine-tuning continues training on a much smaller, targeted dataset: question-answer pairs, conversations, code reviews, or human preference ratings. The catch is cost. Full fine-tuning updates every parameter, so a 175-billion-parameter model produces a 175-billion-parameter copy per task.
What LoRA (Low-Rank Adaptation) Changes
LoRA, short for low-rank adaptation of large language models, fine-tunes a model by freezing its original weights and training small add-on matrices instead. The 2021 paper LoRA: Low-Rank Adaptation of Large Language Models reports that, compared with fully fine-tuning GPT-3 175B, LoRA cut the trainable parameters by 10,000 times and the GPU memory needed by 3 times, with quality on par or better.
The trick is that the change a task needs is "low rank": it can be written as the product of two thin matrices. Instead of editing a huge weight matrix directly, LoRA learns those two thin ones and adds their product on top.
# Plain-English shape of a LoRA layer (not a training script)
# W: frozen pre-trained weight, d x k (e.g. 4096 x 4096 = 16.8M numbers)
# A: trainable, r x k B: trainable, d x r with rank r = 8
# Trainable numbers: 8*4096 + 4096*8 = 65,536 (about 0.4 % of W)
output = x @ W.T + x @ (B @ A).T * scaleBecause the base model stays untouched, one copy can serve many tasks, each with its own few-megabyte adapter. That is why LoRA became the default way to customise open-weight models like Llama.
Chain-of-Thought Prompting: Why Showing the Steps Works
Chain-of-thought prompting asks a large language model to write out intermediate reasoning steps before its final answer, and on multi-step problems that improves accuracy. The 2022 paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models showed that giving a 540-billion-parameter model just eight worked examples with reasoning reached state-of-the-art accuracy on GSM8K, a benchmark of grade-school math word problems.
The reason is mechanical. An LLM spends a fixed amount of computation per token. A question that needs four steps of arithmetic gets more computation, and a written record of partial results, when the model generates those steps as tokens instead of jumping to the answer.
Standard few-shot prompt (the example shows only the answer):
Q: A page is 50,000 tokens and 10 % is article text. How many article tokens?
A: 5,000.
Q: A page is 55,900 tokens and 8 % is article text. How many article tokens?
A:
Chain-of-thought prompt (the example shows the steps):
Q: A page is 50,000 tokens and 10 % is article text. How many article tokens?
A: 10 % is 0.10. 0.10 × 50,000 = 5,000. The answer is 5,000.
Q: A page is 55,900 tokens and 8 % is article text. How many article tokens?
A:The only difference is the worked example. In the second prompt the model copies the pattern, writes "8 % is 0.08. 0.08 × 55,900 = …", and does the arithmetic in visible steps.
The paper also found the effect appears mainly in large models; small ones produced fluent but wrong reasoning chains. Today's "reasoning" models build the same idea in: they generate reasoning tokens before the visible answer, which is why they are slower and bill more tokens per reply.
What Generative AI With Large Language Models Can and Can't Do
Generative AI with large language models is good at producing fluent text from patterns it has seen, and unreliable on facts it has not seen or cannot look up. The model has no database of truth. It has probabilities.
What they do well:
- Drafting, rewriting and summarising text in a requested tone or format.
- Translating, including between programming languages.
- Extracting structured data (names, prices, dates) from messy text.
- Answering questions that are well covered in their training data.
Where they fail:
- Hallucination. When the likeliest next token is wrong, the model states it just as confidently. Invented citations are the classic case.
- Knowledge cutoff. Weights freeze at the end of training, so anything newer is unknown unless the product fetches it.
- Arithmetic and counting over long inputs, unless the product hands the work to a tool.
- Brand facts. A small company's pricing or features may appear a handful of times on the web, or never, so the model guesses.
The fix the industry settled on for the last two is retrieval: search the web or a document store at question time, put the results into the context window, and let the model answer from them. That fix is exactly where your website comes in.
How Large Language Models Read Your Website
LLMs read websites through two separate paths: training crawlers that collect pages for future models, and search or user-triggered fetchers that pull pages live to answer a question today. Each major vendor documents separate user agents for these, so you can allow one and block the other in robots.txt.
| Company | Training crawler | Search / live fetch | Source |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User | OpenAI crawler docs |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User | Anthropic crawler help article |
| Google-Extended (a robots.txt token, not a separate crawler) | Googlebot | Google common crawlers | |
| Perplexity | none listed in its docs | PerplexityBot, Perplexity-User | Perplexity bot docs |
Google is explicit that blocking Google-Extended does not affect inclusion in Google Search or ranking, and its AI features guidance says AI Overviews draw on the normal Search index. OpenAI's documentation describes the same split from the other side: allow OAI-SearchBot to appear in ChatGPT search, disallow GPTBot to keep content out of training.
# robots.txt: stay quotable in AI search, opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /If you want the full template set and the trade-offs, the guide on how to block AI training but allow AI search goes through every vendor's tokens.
Only 8 % of a Page's Tokens Are the Article
On bugviso.com's own blog, the article text is a median 8.2 % of the page's raw HTML tokens. The median page is about 55,900 tokens as HTML but only about 4,550 tokens of actual article, measured across all 274 posts.
| Measure (274 pages) | 10th percentile | Median | 90th percentile |
|---|---|---|---|
| Raw HTML tokens | 47,758 | 55,869 | 63,412 |
| All visible text tokens | 3,474 | 4,678 | 5,881 |
Article (<main>) text tokens | 3,345 | 4,549 | 5,752 |
| Article share of raw HTML | 6.7 % | 8.2 % | 9.6 % |
Where do the other 92 % go? In a median page, inline SVG icons account for 28.1 % of raw tokens and <script> tags, including the page data Next.js embeds for hydration, for 16.3 %. Almost everything else is HTML tags and their attributes. Inline <style> is zero, because the CSS ships as a separate file.
This is a static, server-rendered site, so the article text is in the first HTML response. That is the good case. Three practical conclusions follow:
- Text extraction is mandatory. Every page in the sample was over 32,000 raw tokens. A pipeline that fed raw HTML to a model would spend more than 90 % of its budget on markup, so retrieval systems strip it first, and they can only keep text that is actually in the HTML they fetched.
- Content rendered only by JavaScript is at risk. If your article appears after client-side rendering, a fetcher that does not run JavaScript sees the shell and none of the text.
- Your article becomes chunks. At a median 4,549 tokens, a typical post splits into about 9 chunks of 512 tokens. Each chunk is retrieved and judged on its own, so a section that only makes sense after the previous one loses when it is quoted alone.
That last point is the core of generative engine optimization: write sections that answer one question completely, so any chunk an LLM retrieves can stand on its own.
Run It Yourself: Count Your Page's LLM Tokens
This script fetches any URL (or reads a saved HTML file) and prints its token count as raw HTML, as visible text, and as <main>/<article> text, with the article's share. It uses only the Python standard library plus tiktoken.
#!/usr/bin/env python3
"""Count how many LLM tokens a web page costs: raw HTML vs the text a reader sees.
Usage: python3 page_tokens.py <url-or-file> [more ...]
Needs: pip install tiktoken (o200k_base is the GPT-4o-era tokenizer)
"""
import sys
import urllib.request
from html.parser import HTMLParser
import tiktoken
SKIP = {"script", "style", "noscript", "svg", "template", "head"}
class Text(HTMLParser):
def __init__(self):
super().__init__()
self.depth, self.main_depth, self.all, self.main = 0, 0, [], []
def handle_starttag(self, tag, attrs):
if tag in SKIP:
self.depth += 1
if tag in ("main", "article") or self.main_depth:
self.main_depth += 1
def handle_endtag(self, tag):
if tag in SKIP and self.depth:
self.depth -= 1
if self.main_depth:
self.main_depth -= 1
def handle_data(self, data):
if self.depth or not data.strip():
return
self.all.append(data.strip())
if self.main_depth:
self.main.append(data.strip())
def load(src):
if src.startswith("http"):
req = urllib.request.Request(src, headers={"User-Agent": "page-tokens/1.0"})
with urllib.request.urlopen(req, timeout=20) as r:
return r.read().decode("utf-8", "replace")
return open(src, encoding="utf-8").read()
def measure(html, enc):
p = Text()
p.feed(html)
return (len(enc.encode(html, disallowed_special=())),
len(enc.encode(" ".join(p.all), disallowed_special=())),
len(enc.encode(" ".join(p.main), disallowed_special=())))
if __name__ == "__main__":
enc = tiktoken.get_encoding("o200k_base")
print(f"{'raw_html':>9} {'visible':>8} {'main':>7} {'main%':>6} page")
for src in sys.argv[1:]:
raw, vis, main = measure(load(src), enc)
print(f"{raw:>9,} {vis:>8,} {main:>7,} {main / raw:>6.1%} {src}")Sample output from two of the pages in our dataset:
raw_html visible main main% page
47,488 3,544 3,415 7.2% blog/accessibility-audit-services-vs-tools.html
62,191 5,564 5,435 8.7% blog/accessibility-website-audit-tool-wcag.htmlTwo readings to watch for. A main count of 0 means the page has no <main> or <article> element, so extractors have to guess where the content is. A visible count far below what you see in the browser means the text is injected by JavaScript after load, and the raw HTML a simple fetcher receives does not contain it.
How BugViso Checks Whether LLMs Can Read and Quote Your Pages
BugViso's AI Search Readiness engine runs the checks above on every crawled page, in a real browser, as part of a normal audit. It matches each AI crawler against your robots.txt with RFC 9309 rules, so a stray Disallow for OAI-SearchBot or Claude-SearchBot shows up as a finding rather than a silent loss of visibility. It validates llms.txt if you have one, scores extractability and E-E-A-T signals, and rolls them into a 0–100 GEO citability score.
The RAG chunk readiness simulator does the chunking step from the data section for you: it splits each page's main content into roughly 512-token chunks along sentence boundaries and scores every chunk for fact density and answerability. You see it chunk by chunk in the UI and as a section in the PDF, next to AI-written fixes. Details are on the AI search readiness feature page.
Frequently Asked Questions
What is a large language model in simple terms?
A large language model is software that has read a huge amount of text and learned to guess the next word. Ask it a question and it writes an answer one small piece at a time, each piece chosen because it is the most likely continuation. It feels like understanding, but underneath it is very well-trained prediction.
What are some examples of large language models?
Well-known examples are OpenAI's GPT models (used in ChatGPT), Anthropic's Claude, Google's Gemini, Meta's Llama, and models from Mistral and DeepSeek. Llama, Mistral and DeepSeek publish open weights that anyone can download and run. GPT, Claude and Gemini are used through apps and paid APIs.
Is ChatGPT a large language model?
Not exactly. ChatGPT is a product built on OpenAI's GPT family of large language models. The product adds a chat interface, web search, memory, file handling and safety layers around the model. The same distinction applies to Claude, Gemini and Perplexity: the app and the underlying model are different things.
What is the difference between generative AI and large language models?
Generative AI is the broad category of models that create new content: text, images, audio, video or code. Large language models are the text branch of it. Image generators are generative AI but not LLMs, and many modern LLMs are multimodal, so they also accept images or audio as input.
What is LoRA in large language models?
LoRA (low-rank adaptation) is a cheap way to fine-tune a large language model. It freezes the original weights and trains small extra matrices added to each layer. The original paper reports 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning of GPT-3 175B, with similar quality.
What is chain-of-thought prompting?
Chain-of-thought prompting means asking a model to write out its reasoning steps before the final answer, often by showing it a few worked examples. The 2022 paper that named it found large gains on math word problems for very large models. Modern reasoning models now do this internally before replying.
The One Thing Website Owners Should Take From How LLMs Work
An LLM can only quote what reaches its context window, and on a typical page that is the 8 % of tokens that are real text, cut into chunks and judged one at a time. Make sure the right bots can fetch your pages, keep the text in the HTML, and write sections that stand alone, then use the BugViso AI Visibility Tracker to see whether ChatGPT, Gemini, Perplexity, Claude and AI Overviews actually mention you.
Written by
Kazi Khalid Hasan Tamim · BugViso Team
Kazi writes BugViso’s guides on website audits, Core Web Vitals, technical SEO, accessibility and AI search. The data in them comes from audits BugViso runs in a real Chromium browser, including our 219-site study of website health.
Summarise with AI
Open this guide in your assistant.
Follow BugViso
New guides and audit data as we publish them.
AI visibility tracker
Is your brand in AI answers?
See how often AI assistants mention you, cite your site and rank you against your competitors.
- ChatGPT
- Gemini
- Perplexity
- Claude
- AI Overviews
See where your site stands
Run a free BugViso audit for SEO, speed, accessibility and AI search readiness — with fixes you can ship today.