An AI token is the granularity at which Gemini and other generative AI models process input and output. Tokens can be single characters like z or whole words like cat. Long words are broken up into several tokens, according to Google’s Gemini API documentation.
The language-model sense applies throughout this entry: The subword unit a model reads and writes, rather than the blockchain asset that shares the name. The context window defines the combined limit of input and output tokens. When billing is enabled, the cost of a call to the Gemini API is determined in part by the number of input and output tokens.
Key Takeaways
- For Gemini models, a token is equivalent to about 4 characters, per Google’s Gemini API documentation. 100 tokens is equal to about 60-80 English words.
- For Claude, a token approximately represents 3.5 English characters, per Anthropic’s documentation glossary. The exact number can vary depending on the language used.
- Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text, according to Anthropic’s pricing documentation.
- Knowing how many tokens are in a text string shows whether the string is too long for a text model to process. It also shows how much an OpenAI API call costs, since usage is priced by token.
- Claude Opus 5 is priced at $5 per million base input tokens and $25 per million output tokens. A cache hit costs 10% of the standard input price.
- Video is counted at 263 tokens per second and audio at 32 tokens per second. Images with both dimensions less than or equal to 384 pixels count as 258 tokens.
How Does AI Tokenization Work?
The set of all tokens used by the model is called the vocabulary. The process of splitting text into tokens is called tokenization, per Google’s Gemini API documentation. Tokens are the smallest individual units of a language model. They can correspond to words, subwords, characters, or even bytes in the case of Unicode, according to Anthropic’s documentation glossary.
1. Text arrives as a string of characters
Claude is provided with text consisting of a series of characters. That text is encoded into a series of tokens for the model to process. Tokens are typically hidden when interacting with language models at the text level. They become relevant when examining the exact inputs and outputs of a language model.
The step never shows up in a chat window, so the unit surfaces only on an invoice or in a context-limit error.
2. The tokenizer splits it into subword units
Byte pair encoding is a way of converting text into tokens, and it attempts to let the model see common subwords. BPE encodings will often split “encoding” into tokens like “encod” and “ing”, per OpenAI’s tiktoken repository. Peer-reviewed work on neural machine translation introduced the approach, encoding rare and unknown words as sequences of subword units. That segmentation is based on the byte pair encoding compression algorithm.
A tokenizer works like a phrase book of the syllables a language repeats most. A common word costs one entry, and a rare surname costs several. It also works like a postal line that bundles frequent street names in a single pass and spells unfamiliar ones letter by letter.
3. Each unit maps to an entry in the vocabulary
Long words are broken up into several tokens. For Gemini models, a token is equivalent to about 4 characters. Byte pair encoding also works on arbitrary text, even text that is not in the tokenizer’s training data.
4. The model reads and generates in those units
Given a text string and an encoding such as cl100k_base, a tokenizer can split the text string into a list of tokens. Each Gemini model has a maximum number of tokens it can handle, and the context window defines the combined limit of input and output tokens.
| Step | What happens | What the model sees |
|---|---|---|
| Input | Text arrives as a series of characters | Nothing yet |
| Tokenize | Byte pair encoding splits the text into subword units | Common subwords such as “encod” and “ing” |
| Look up | Each unit maps to an entry in the vocabulary | Vocabulary entries rather than letters |
| Generate | The model reads and writes in those units | Output tokens counted alongside input tokens |
Source: Google Gemini API documentation; OpenAI tiktoken repository; Anthropic Claude documentation glossary.
Tokenization opens the pipeline our large language model explainer walks through end to end.
How Many Tokens Is a Word?
For Gemini models, a token is equivalent to about 4 characters, according to Google’s Gemini API documentation. 100 tokens is equal to about 60-80 English words. Anthropic publishes a rough estimate of its own: 1 token is approximately 4 characters or 0.75 words in English. The exact count varies by language and content type.
Anthropic’s documentation glossary publishes a lower figure again. A token approximately represents 3.5 English characters for Claude, and the exact number can vary depending on the language used. Those ratios come from the vendors themselves, and they disagree. Treating one of them as a constant is where token-cost arithmetic goes wrong.
The spread widens inside a single vendor. Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text. Claude Sonnet 4.6 and earlier models use the previous tokenizer, per Anthropic’s pricing documentation. The exact increase depends on the content and workload shape.
| Vendor documentation | Documented characters per token | Documented word equivalence | Stated qualifier |
|---|---|---|---|
| Google, Gemini API tokens page | About 4 | 100 tokens is about 60 to 80 English words | “equivalent to about” |
| Anthropic, Claude documentation glossary | Approximately 3.5 | Not published | “can vary depending on the language used” |
| Anthropic, pricing documentation | Approximately 4 | Approximately 0.75 words per token | “as a rough estimate” |
Source: Google Gemini API documentation; Anthropic Claude documentation glossary and pricing documentation.
A price quoted per million tokens is a rate rather than a total. Two sticker prices become comparable only once you know which tokenizer produced the count underneath them.
The two assistants readers ask about most sit on different tokenizers, one gap among several in our Claude and ChatGPT comparison data.
Why Do AI Models Charge Per Token?
Usage is priced by token, according to OpenAI’s Cookbook. Knowing how many tokens are in a text string shows how much an OpenAI API call costs. When billing is enabled, the cost of a call to the Gemini API is determined in part by the number of input and output tokens.
Anthropic denominates its published prices in MTok, which its pricing documentation defines as million tokens.
| Model | Base input, per million tokens | Output, per million tokens |
|---|---|---|
| Claude Opus 5 | $5 | $25 |
| Claude Sonnet 4.6 | $3 | $15 |
| Claude Haiku 4.5 | $1 | $5 |
Source: Anthropic Claude pricing documentation, accessed July 2026.
Output carries the heavier rate across the whole line. Claude Opus 5 is priced at $5 per million base input tokens and $25 per million output tokens. Claude Sonnet 4.6 sits at $3 and $15, and Claude Haiku 4.5 at $1 and $5.
Two documented multipliers move the arithmetic the other way. A cache hit costs 10% of the standard input price. The Batch API allows asynchronous processing of large volumes of requests with a 50% discount on both input and output tokens.
Published rates move often, and the count underneath them moves too. Current per-token rates and context sizes across the major models sit in our AI model tracker.
The same unit sets the ceiling as well as the bill. Anthropic’s glossary defines the context window as the amount of text a language model can look back on and reference when generating new text.
Metered per-token billing is the business model underneath the meter, and its revenue side sits in our OpenAI revenue and usage data.
Tokens Beyond Text: Images, Video, and Audio
Images with both dimensions less than or equal to 384 pixels count as 258 tokens. Larger images are tiled into 768×768 pixel tiles, each counting as 258 tokens, according to Google’s Gemini API documentation. Video is counted at 263 tokens per second, and audio is counted at 32 tokens per second.
| Input type | Documented token cost | Scope |
|---|---|---|
| Image, both dimensions 384 pixels or less | 258 tokens | Gemini API |
| Image tile, 768×768 pixels | 258 tokens per tile | Gemini API |
| Video | 263 tokens per second | Gemini API |
| Audio | 32 tokens per second | Gemini API |
Source: Google Gemini API documentation, multimodal token counts.
Those are Google’s published rates for the Gemini API, and they do not transfer to other vendors. Non-text inputs also draw on the same budget as text. Gemini and other generative AI models process input and output at a granularity called a token. The context window defines the combined limit of input and output tokens.
Advantages and Trade-offs of Token-Based Accounting
Advantages
- Byte pair encoding is reversible and lossless, so tokens convert back into the original text.
- It works on arbitrary text, even text that is not in the tokenizer’s training data.
- It compresses the text, so the token sequence is shorter than the bytes corresponding to the original text.
- Larger tokens enable data efficiency during inference and pretraining, and are used when possible.
- Smaller tokens allow a model to handle uncommon or never-before-seen words.
Trade-offs and Risks
- The choice of tokenization method can impact the model’s performance and vocabulary size. It also affects the ability to handle out-of-vocabulary words.
- For Claude, a token approximately represents 3.5 English characters. The exact number can vary depending on the language used. Writers working in some languages pay more units for the same meaning.
- Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text. Cost baselines built against an older model generation do not carry forward.
Real-World Token Accounting
Anthropic changed the tokenizer between Claude generations
Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer that contributes to their improved performance on a wide range of tasks. That tokenizer produces approximately 30% more tokens for the same text. Claude Sonnet 4.6 and earlier models use the previous tokenizer.
A team that budgeted against Sonnet 4.6 and then moved up a generation counts different units for an identical prompt.
OpenAI ships a different encoding per model family
Encoding name o200k_base is used by the gpt-4o and gpt-4o-mini models, and cl100k_base by gpt-4-turbo, gpt-4 and gpt-3.5-turbo. Encoding name p50k_base is used by Codex models, and r50k_base by GPT-3 models like davinci.
Counting with the wrong encoding returns a number that looks plausible and bills incorrectly. Adoption across the assistants built on those model families sits in our AI assistant usage and pricing data.
Google publishes a per-second rate for audio and video
Google counts video at 263 tokens per second and audio at 32 tokens per second for the Gemini API. An hour of recorded audio therefore carries a published token cost before anyone writes a prompt around it.
Is an AI Token the Same as a Crypto Token?
No. The AI sense names the granularity at which Gemini and other generative AI models process input and output. That unit counts against the context window and against the cost of a call to the Gemini API.
The blockchain sense describes a digital asset recorded on a ledger. The two senses share a word and nothing else, and no ratio on this page applies to a traded asset.
How Do I Count Tokens Before Sending a Prompt?
tiktoken is a fast open-source tokenizer by OpenAI. Given a text string and an encoding, a tokenizer can split the text string into a list of tokens, according to OpenAI’s Cookbook. Matching the encoding to the model matters. The encoding o200k_base is used by the gpt-4o and gpt-4o-mini models, while cl100k_base is used by gpt-4-turbo, gpt-4, and gpt-3.5-turbo.
A rough sanity check helps before a tokenizer runs. As a rough estimate, 1 token is approximately 4 characters or 0.75 words in English. The exact count varies by language and content type.
Conclusion
An AI token is the unit models read and write in, equivalent to about 4 characters for Gemini models. 100 tokens is equal to about 60-80 English words. For Claude, a token approximately represents 3.5 English characters. Two vendors, two published ratios, and no single conversion that holds across both.
Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text. A per-token price is a rate whose multiplier is set by the model. Budget the tokenizer before budgeting the token.
