The token counter splits any text into OpenAI tokens using the o200k_base or cl100k_base encoding and reports characters, words, lines and characters per token. Its default 72-character prompt comes to 16 tokens, about 4.5 characters per token. Developers and prompt writers can size a prompt before sending it to a model.
Tokens
16
This text is 16 tokens in o200k_base (GPT-5 family, GPT-4.1, GPT-4o, o-series), about 4.5 characters per token.
- Tokens
- 16
- Characters
- 72
- Words
- 14
- Lines
- 1
- Characters per token
- 4.5
- Tokenizer used
- o200k_base (GPT-5 family, GPT-4.1, GPT-4o, o-series)
Show token boundaries
A dashed outline marks one character that takes more than one token. ↵ marks a line break, and · a joining character you cannot see.
How this works
- How it works
- Counts use the o200k_base and cl100k_base encodings published in OpenAI's tiktoken (opens in new window), through the open-source gpt-tokenizer library. tiktoken maps the GPT-5 family to o200k_base and lists no encoding for GPT-6, so the GPT-6 choice counts with o200k_base, the newest one published, and is an estimate; OpenAI's input-token-count API endpoint (opens in new window) gives the exact figure for a GPT-6 request. Other providers use their own tokenizers, so the same text can count differently on Claude or Gemini. Chat APIs add a few tokens of message overhead per request.
- Not covered
- OpenAI tokenizers only; GPT-6 is an estimate; other vendors count differently; chat overhead excluded.
- Data checked
- : the model-to-encoding map in the tiktoken repository (tiktoken/model.py): gpt-5 maps to o200k_base, and no GPT-6 entry exists
- Privacy
- What you enter never reaches our server. It is processed on your device, inside this page.
- Corrections
- Report a wrong result to media@sqmagazine.co.uk; every report is reviewed and confirmed errors are corrected. Include the inputs and the answer you expected. With your permission, we thank you by name on this page.
- Updates
- "Show token boundaries" shades each token of your text in turn, so you can see where the tokenizer splits words, code and emoji.
- The tokenizer menu names the GPT-5 family and adds GPT-6, counted with o200k_base and labeled as an estimate.
- A "Price this text" link opens the LLM cost calculator with this count as the input tokens.
- Feedback
- How useful was this tool?
Table of Contents
How to Use the Token Counter
Two inputs drive 6 outputs, and every figure refreshes as you type. The counter runs entirely in your browser; the “Text” field notes that your text never leaves the page and is never saved.
- Paste a prompt, a system message or a document into “Text”.
- Choose a “Tokenizer”: o200k_base, the default, or cl100k_base for older models. The menu names the models each one serves.
- Read the “Tokens” figure, then the other counts below it. The last line, “Tokenizer used”, confirms which encoding produced the count.
- Tap an “Examples” chip (“Short prompt”, “Code snippet” or “Same text, GPT-4 tokenizer”) to load a preset. “Copy all details” copies every figure; “Share result” keeps your tokenizer choice but never your text.
The token figure comes from OpenAI’s own encodings, run on your device.
How the Token Counter Counts Tokens
Byte pair encoding (BPE) does the splitting, with the encodings OpenAI publishes. OpenAI describes its open-source tiktoken library as a fast BPE tokeniser for use with OpenAI’s models. Its help center adds that a token can represent a character, part of a word, a whole word, or punctuation.
According to the tiktoken README, BPE is reversible and lossless, so you can convert tokens back into the original text. It also works on arbitrary text, even text that is not in the tokeniser’s training data. On average, each token corresponds to about 4 bytes in practice. The method traces to a paper that segmented words based on the byte pair encoding compression algorithm, accepted at ACL 2016.
This openai token counter runs the gpt-tokenizer library in the page, which its maintainers call a port of OpenAI’s tiktoken. A primer on how large language models work explains why every model family needs a tokenizer. Four more outputs are simple counts:
- Characters: Unicode code points, so a simple emoji counts once while a flag or a joined emoji counts as several.
- Words: runs of text separated by spaces or line breaks.
- Lines: line breaks plus one; a final line break does not add a line.
- Characters per token: characters divided by tokens.
Special strings such as “<|endoftext|>” also count as ordinary text.
The default prompt shows every step.
Token Counter Example: A 72-Character Prompt
The default text, Paste a prompt, a system message or a document here to count its tokens., returns 16 tokens in o200k_base. Four figures in the token calculator’s panel explain it:
- 72 characters, counting spaces and punctuation.
- 14 words and 1 line.
- 16 tokens: every word is one token, plus one for the comma and one for the period. Each later word keeps its leading space, which matters: OpenAI notes that red, Red, and red do not contain identical text: the last example includes a leading space.
- 4.5 characters per token: 72 divided by 16.
That 16 covers the text alone. OpenAI’s help center warns that a plain-text token count does not necessarily include all tokens in an API request.
For GPT-4o, the OpenAI Cookbook’s counting function adds tokens_per_message = 3 and encodes each field of the message. A comment in the same code reads every reply is primed with assistant, worth 3 more tokens. Sent as one user message, the prompt picks up 3 tokens for the message and 1 for the role string “user”. Add 16 for the text and 3 for the reply: 23 input tokens.
What Your Token Count Means
In other words, the count measures your text, and it is a floor for any request built from it. OpenAI lists message structure, tools, schemas, images, and files as inputs that can affect the full count. The Cookbook calls its own function an estimate, not a timeless guarantee.
Output, by contrast, follows its own rules. OpenAI says reasoning tokens count toward output usage and are billed as output tokens, so no input count predicts what a reply will use.
The limit that matters most is the context window. Per OpenAI, the context window limits the tokens a model can work with in a request. Compare the counter’s figure with the limit in your model’s documentation and leave room for the reply.
Read the number as the size of the text: the right starting point for trimming a prompt, never a quote for a whole request.
o200k_base vs cl100k_base: Which Tokenizer Does Your Model Use?
Start by matching the encoding to the model. OpenAI’s Cookbook explains that encodings specify how text is converted into tokens. Different models use different encodings. The model map in tiktoken’s source sends each model-name prefix to one encoding:
| Model name prefix in tiktoken | Encoding |
|---|---|
| gpt-5, gpt-4.5, gpt-4.1, gpt-4o | o200k_base |
| o1, o3, o4-mini | o200k_base |
| gpt-4, gpt-3.5-turbo | cl100k_base |
Source: OpenAI tiktoken
GPT-5 is absent from the tool’s “Tokenizer” menu. Its entry in that map reads “gpt-5”: “o200k_base”, so the default setting fits it. The gpt-tokenizer library makes the same default choice: importing from gpt-tokenizer uses o200k_base.
The “Same text, GPT-4 tokenizer” chip reruns the default prompt through cl100k_base. It returns 16 tokens there too, because every word in it is common English.
Why Code Uses More Tokens per Character
Code packs fewer characters into each token than prose. OpenAI’s help center gives the English rule of thumb, 1 token is approximately 4 characters, and the counter’s own presets bracket it:
- Short prompt: 75 characters and 15 tokens, or 5.00 characters per token.
- Default prompt: 72 characters and 16 tokens, or 4.5 characters per token.
- Code snippet: 62 characters and 23 tokens, or 2.70 characters per token.
The four-line JavaScript function spends tokens on indentation, brackets and symbols that prose rarely uses. The 4-character rule would put it at 15.5 tokens instead of 23.
Similarly, language moves the ratio. OpenAI adds that other languages can have different relationships between characters, words, and tokens.
A study published at NeurIPS 2023 put numbers on the gap. The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. Paste a non-English sentence and switch the “Tokenizer” to see how much the two encodings differ on it.
Do Claude and Gemini Count Tokens the Same Way?
No. Each provider uses its own tokenizer, so the same text can count differently on Claude or Gemini. This counter’s figure applies to OpenAI models only. For a side-by-side of the two assistants, see Claude vs ChatGPT statistics.
Anthropic says Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer. On those models, the same input text produces approximately 30% more tokens than on earlier models. The exact increase depends on the content and workload shape. Its own counting endpoint returns an estimate. The per-million-token Claude prices those counts feed into are tracked in Claude AI statistics.
Meanwhile, Google puts Gemini close to OpenAI’s rule of thumb: for Gemini models, a token is equivalent to about 4 characters. 100 tokens is equal to about 60-80 English words. As a result, any model outside OpenAI’s needs that provider’s own counting tool.
Is One Token Always Four Characters?
No. OpenAI offers 1 token is approximately 4 characters and 100 tokens are approximately 75 words as guides, then adds: “These are estimates, not exact counts.” The counter’s own presets run from 2.70 to 5.00 characters per token.
Sources Used on This Page
- What are tokens and how to count them.
- tiktoken.
- tiktoken model.py.
- gpt-tokenizer.
- How to count tokens with tiktoken.
- Token counting.
- Understand and count tokens.
- Neural Machine Translation of Rare Words with Subword Units.
- Language Model Tokenizers Introduce Unfairness Between Languages.
Is there anything else my agent or I should know about this page?
Yes. The counter runs in your browser and nothing you paste reaches our server. Tokens come from OpenAI’s published o200k_base or cl100k_base encoding, through the open-source gpt-tokenizer library. Characters, words and lines come from the same text.
The count matches what the chosen encoding produces for the raw text, and three things sit outside it. A chat API adds message overhead to each request. Other providers’ tokenizers, such as Anthropic’s or Google’s, can count the same text differently. The counter also shows no price for the tokens.
SQ Magazine publishes the page for information. A token count helps you plan a prompt against a context window or a budget; it promises no cost or output from any model.
Conclusion
The default 72-character prompt is 16 tokens in o200k_base and 23 once it travels as a chat message. For developers, that gap is the margin to leave below a context limit; for writers pasting long documents, the count shows how much of a model’s context window the text takes up.
Try the “Code snippet” chip to watch the ratio drop, then paste your own system prompt and switch tokenizers to match your model. When a new model ships, check which encoding tiktoken maps it to before trusting any count. The per-million-token API prices in OpenAI statistics turn the figure into a cost estimate.
