--- title: "Token Counter" date: 2026-10-12 author: "Barry Elad" featured_image: "https://sqmagazine.co.uk/wp-content/uploads/2026/10/token-counter.jpg" --- # Token Counter The token counter splits any text into OpenAI tokens using the o200k\_base or cl100k\_base encoding and reports characters, words, lines and characters per token. Its default 72-character prompt comes to **16 tokens**, about 4.5 characters per token. Developers and prompt writers can size a prompt before sending it to a model. Token Counter Examples Reset Short prompt Code snippet Same text, GPT-4 tokenizer Text Paste a prompt, a system message or a document here to count its tokens. Counted locally. Your text never reaches our server and is never saved. Tokenizer GPT-5, GPT-4oo200k\_base: GPT-5 family, GPT-4.1, GPT-4o, o-series GPT-6 (estimate)GPT-6 family: an estimate with o200k\_base GPT-4, GPT-3.5cl100k\_base: GPT-4, GPT-3.5 Turbo Tokens 16 Copy This text is 16 tokens in o200k\_base (GPT-5 family, GPT-4.1, GPT-4o, o-series), about 4.5 characters per token. Tokens16 Characters72 Words14 Lines1 Characters per token4.5 Tokenizer usedo200k\_base (GPT-5 family, GPT-4.1, GPT-4o, o-series) Show token boundaries A dashed outline marks one character that takes more than one token. ↵ marks a line break, and · a joining character you cannot see. Copy all details How this works How it worksCounts use the o200k\_base and cl100k\_base encodings published in [OpenAI's tiktoken (opens in new window)](https://github.com/openai/tiktoken), through the open-source gpt-tokenizer library. tiktoken maps the GPT-5 family to o200k\_base and lists no encoding for GPT-6, so the GPT-6 choice counts with o200k\_base, the newest one published, and is an estimate; [OpenAI's input-token-count API endpoint (opens in new window)](https://developers.openai.com/api/docs/guides/token-counting) gives the exact figure for a GPT-6 request. Other providers use their own tokenizers, so the same text can count differently on Claude or Gemini. Chat APIs add a few tokens of message overhead per request. Not coveredOpenAI tokenizers only; GPT-6 is an estimate; other vendors count differently; chat overhead excluded. Data checkedOct 9, 2026: the model-to-encoding map in the tiktoken repository (tiktoken/model.py): gpt-5 maps to o200k\_base, and no GPT-6 entry exists Privacy**What you enter never reaches our server.** It is processed on your device, inside this page. CorrectionsReport a wrong result to [media@sqmagazine.co.uk](mailto:media@sqmagazine.co.uk?subject=Correction%3A%20Token%20Counter); every report is reviewed and confirmed errors are corrected. Include the inputs and the answer you expected. With your permission, we thank you by name on this page. Updates- Oct 13, 2026 "Show token boundaries" shades each token of your text in turn, so you can see where the tokenizer splits words, code and emoji. - Oct 9, 2026 The tokenizer menu names the GPT-5 family and adds GPT-6, counted with o200k\_base and labeled as an estimate. - Oct 9, 2026 A "Price this text" link opens the LLM cost calculator with this count as the input tokens. FeedbackHow useful was this tool? The figures shown are the worked example. Enable JavaScript to calculate with your own numbers. ## How to Use the Token Counter Two inputs drive **6 outputs**, and every figure refreshes as you type. The counter runs entirely in your browser; the “Text” field notes that your text never leaves the page and is never saved. 1. Paste a prompt, a system message or a document into “Text”. 2. Choose a “Tokenizer”: o200k\_base, the default, or cl100k\_base for older models. The menu names the models each one serves. 3. Read the “Tokens” figure, then the other counts below it. The last line, “Tokenizer used”, confirms which encoding produced the count. 4. Tap an “Examples” chip (“Short prompt”, “Code snippet” or “Same text, GPT-4 tokenizer”) to load a preset. “Copy all details” copies every figure; “Share result” keeps your tokenizer choice but never your text. The token figure comes from OpenAI’s own encodings, run on your device. ## How the Token Counter Counts Tokens Byte pair encoding (BPE) does the splitting, with the encodings OpenAI publishes. OpenAI describes its open-source tiktoken library as a fast BPE tokeniser for use with OpenAI’s models. Its [help center](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them) adds that a token can represent a character, part of a word, a whole word, or punctuation. According to the [tiktoken README](https://github.com/openai/tiktoken), BPE is reversible and lossless, so you can convert tokens back into the original text. It also works on arbitrary text, even text that is not in the tokeniser’s training data. On average, each token corresponds to about **4 bytes** in practice. The method traces to a paper that segmented words based on the byte pair encoding compression algorithm, accepted at ACL 2016. This openai token counter runs the gpt-tokenizer library in the page, which its maintainers call a port of OpenAI’s tiktoken. A primer on [how large language models work](/what-are-large-language-models/) explains why every model family needs a tokenizer. Four more outputs are simple counts: - Characters: Unicode code points, so a simple emoji counts once while a flag or a joined emoji counts as several. - Words: runs of text separated by spaces or line breaks. - Lines: line breaks plus one; a final line break does not add a line. - Characters per token: characters divided by tokens. Special strings such as “<|endoftext|>” also count as ordinary text. The default prompt shows every step. ## Token Counter Example: A 72-Character Prompt The default text, *Paste a prompt, a system message or a document here to count its tokens.*, returns **16 tokens** in o200k\_base. Four figures in the token calculator’s panel explain it: 1. **72 characters**, counting spaces and punctuation. 2. **14 words** and 1 line. 3. **16 tokens**: every word is one token, plus one for the comma and one for the period. Each later word keeps its leading space, which matters: OpenAI notes that red, Red, and red do not contain identical text: the last example includes a leading space. 4. **4.5 characters per token**: 72 divided by 16. That 16 covers the text alone. OpenAI’s help center warns that a plain-text token count does not necessarily include all tokens in an API request. For GPT-4o, the [OpenAI Cookbook’s counting function](https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken) adds tokens\_per\_message = 3 and encodes each field of the message. A comment in the same code reads every reply is primed with assistant, worth 3 more tokens. Sent as one user message, the prompt picks up 3 tokens for the message and 1 for the role string “user”. Add 16 for the text and 3 for the reply: **23 input tokens**. ## What Your Token Count Means In other words, the count measures your text, and it is a floor for any request built from it. OpenAI lists message structure, tools, schemas, images, and files as inputs that can affect the full count. The Cookbook calls its own function an estimate, not a timeless guarantee. Output, by contrast, follows its own rules. OpenAI says reasoning tokens count toward output usage and are billed as output tokens, so no input count predicts what a reply will use. The limit that matters most is the context window. Per OpenAI, the context window limits the tokens a model can work with in a request. Compare the counter’s figure with the limit in your model’s documentation and leave room for the reply. Read the number as the size of the text: the right starting point for trimming a prompt, never a quote for a whole request. ## o200k\_base vs cl100k\_base: Which Tokenizer Does Your Model Use? Start by matching the encoding to the model. OpenAI’s Cookbook explains that encodings specify how text is converted into tokens. Different models use different encodings. The [model map in tiktoken’s source](https://github.com/openai/tiktoken/blob/main/tiktoken/model.py) sends each model-name prefix to one encoding: | Model name prefix in tiktoken | Encoding | |---|---| | gpt-5, gpt-4.5, gpt-4.1, gpt-4o | o200k\_base | | o1, o3, o4-mini | o200k\_base | | gpt-4, gpt-3.5-turbo | cl100k\_base | *Source: OpenAI tiktoken* GPT-5 is absent from the tool’s “Tokenizer” menu. Its entry in that map reads “gpt-5”: “o200k\_base”, so the default setting fits it. The gpt-tokenizer library makes the same default choice: importing from gpt-tokenizer uses o200k\_base. The “Same text, GPT-4 tokenizer” chip reruns the default prompt through cl100k\_base. It returns **16 tokens** there too, because every word in it is common English. ## Why Code Uses More Tokens per Character Code packs fewer characters into each token than prose. OpenAI’s help center gives the English rule of thumb, 1 token is approximately 4 characters, and the counter’s own presets bracket it: - Short prompt: 75 characters and 15 tokens, or **5.00 characters per token**. - Default prompt: 72 characters and 16 tokens, or **4.5 characters per token**. - Code snippet: 62 characters and 23 tokens, or **2.70 characters per token**. The four-line JavaScript function spends tokens on indentation, brackets and symbols that prose rarely uses. The 4-character rule would put it at 15.5 tokens instead of 23. Similarly, language moves the ratio. OpenAI adds that other languages can have different relationships between characters, words, and tokens. A [study](https://arxiv.org/abs/2305.15425) published at NeurIPS 2023 put numbers on the gap. The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. Paste a non-English sentence and switch the “Tokenizer” to see how much the two encodings differ on it. ## Do Claude and Gemini Count Tokens the Same Way? No. Each provider uses its own tokenizer, so the same text can count differently on Claude or Gemini. This counter’s figure applies to OpenAI models only. For a side-by-side of the two assistants, see [Claude vs ChatGPT statistics](/claude-vs-chatgpt-statistics/). [Anthropic says](https://platform.claude.com/docs/en/build-with-claude/token-counting) Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer. On those models, the same input text produces approximately **30%** more tokens than on earlier models. The exact increase depends on the content and workload shape. Its own counting endpoint returns an estimate. The per-million-token Claude prices those counts feed into are tracked in [Claude AI statistics](/claude-ai-statistics/). Meanwhile, [Google puts Gemini](https://ai.google.dev/gemini-api/docs/tokens) close to OpenAI’s rule of thumb: for Gemini models, a token is equivalent to about 4 characters. 100 tokens is equal to about 60-80 English words. As a result, any model outside OpenAI’s needs that provider’s own counting tool. ## Is One Token Always Four Characters? No. OpenAI offers 1 token is approximately 4 characters and 100 tokens are approximately 75 words as guides, then adds: “These are estimates, not exact counts.” The counter’s own presets run from **2.70 to 5.00** characters per token. ## Sources Used on This Page - [What are tokens and how to count them](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them). - [tiktoken](https://github.com/openai/tiktoken). - [tiktoken model.py](https://github.com/openai/tiktoken/blob/main/tiktoken/model.py). - [gpt-tokenizer](https://github.com/niieani/gpt-tokenizer). - [How to count tokens with tiktoken](https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken). - [Token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting). - [Understand and count tokens](https://ai.google.dev/gemini-api/docs/tokens). - [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909). - [Language Model Tokenizers Introduce Unfairness Between Languages](https://arxiv.org/abs/2305.15425). ## Is there anything else my agent or I should know about this page? Yes. The counter runs in your browser and nothing you paste reaches our server. Tokens come from OpenAI’s published o200k\_base or cl100k\_base encoding, through the open-source gpt-tokenizer library. Characters, words and lines come from the same text. The count matches what the chosen encoding produces for the raw text, and three things sit outside it. A chat API adds message overhead to each request. Other providers’ tokenizers, such as Anthropic’s or Google’s, can count the same text differently. The counter also shows no price for the tokens. SQ Magazine publishes the page for information. A token count helps you plan a prompt against a context window or a budget; it promises no cost or output from any model. ## Conclusion The default 72-character prompt is **16 tokens** in o200k\_base and 23 once it travels as a chat message. For developers, that gap is the margin to leave below a context limit; for writers pasting long documents, the count shows how much of a model’s context window the text takes up. Try the “Code snippet” chip to watch the ratio drop, then paste your own system prompt and switch tokenizers to match your model. When a new model ships, check which encoding tiktoken maps it to before trusting any count. The per-million-token API prices in [OpenAI statistics](/openai-statistics/) turn the figure into a cost estimate. Definition of Context Window. Link to full glossary entry follows the description.**Context Window**A context window is all the text an AI model can reference when generating a response, measured in tokens and shared with the model's own output. [Read more about Context Window](https://sqmagazine.co.uk/glossary/context-window/) ## More AI Tools [All ai tools (1) →](https://sqmagazine.co.uk/tools/)