Token
A unit of text AI processes (roughly 4 characters). LLM.txt file size and content length are measured in tokens.
What is Token?
A token is the unit into which a language model breaks text before processing it. Tokens are produced by a tokeniser that splits text into common chunks: a frequent short word is usually one token, a longer or rarer word may be several, and punctuation, spaces and parts of words get their own tokens. In English, one token is on average about four characters or three-quarters of a word, so 1,000 tokens is roughly 750 words. Other languages, code and unusual formatting typically use more tokens per word.
Everything a model does is counted in tokens. The context window is a token limit, API pricing is per token, and the length of a prompt, a retrieved passage or a generated answer is measured in tokens. Different model families use different tokenisers, so the same text can count differently across vendors, and vendors publish tokeniser tools to estimate counts.
For content, tokens are a practical way to think about budget. A retrieval system that supplies a few thousand tokens of source material to the model can include only a handful of passages; a page whose key answer is a 100-token paragraph is easy to include, while one whose answer requires 2,000 tokens of surrounding context is not. Files intended for AI consumption, such as llms.txt, are best kept compact for the same reason.
Why it matters for AI search
Tokens are the currency of AI attention, and every answer has a limited budget. Content that delivers its answer in few tokens, without preamble, is cheaper for an engine to include and more likely to be quoted intact. Thinking in tokens also helps with technical files: an llms.txt that runs to tens of thousands of tokens defeats its purpose, while a compact one that states the site's key facts and links can be consumed in a single request.
Related terms
Context Window
The amount of text an AI can consider at once. Affects how much of your content AI can process.
LLM (Large Language Model)
A type of AI trained on vast amounts of text data to understand and generate human-like language (e.g., GPT-4, Claude 3).
LLM.txt
A proposed standard file (similar to robots.txt) that provides a Markdown-formatted map of a website for AI crawlers to consume efficiently.
Embedding
A numerical representation of text that AI uses to understand meaning and find similar content.
Frequently asked questions
How many tokens is a typical web page?+
A 1,500-word article is roughly 2,000 tokens of body text in English, before counting navigation, boilerplate and markup that a crawler may or may not strip. Answer engines usually extract and trim passages rather than using the full count.
Do all AI models count tokens the same way?+
No. Each model family uses its own tokeniser, so the same text can produce different counts on OpenAI, Anthropic and Google models. The four-characters-per-token rule is an English-language average, and vendors offer tools for exact counts.