What Is a Token in AI? Why Turkish Text Uses More Tokens

In short: A token is the smallest piece of text an AI model works with when it reads and writes, and it can be a word, part of a word or a single character. In RuneLab's measurement of 2 November 2026, the Turkish version of the same text took 1.45 times as many tokens as the English version on OpenAI's o200k_base tokenizer, and up to 1.96 times as many on the other general-purpose tokenizers we measured. API costs, context windows and usage limits are all counted in tokens.

The usage limits, context windows and API bills you run into in ChatGPT, Claude or Gemini are all counted in the same unit: the token. How much text a token covers depends on the language, though, and the rules of thumb written for English do not hold for Turkish.

A token is the smallest unit of text a language model reads and writes. According to the LLM module of Google's Machine Learning Crash Course, a token can be a word, part of a word or a single character, and most modern language models split text into subwords. The model does not see text letter by letter but as these pieces. When it writes an answer, it moves forward by predicting the next token.

We explain how a large language model (LLM) works in general in What is an LLM. The token is the unit that measures both what goes into that model and what comes out.

As a rough guide, Google writes that in English one token is about 4 characters and 400 tokens are about 300 words. The same source says that tokenization is language specific and that the number of characters per token differs from language to language. We measured that ratio for Turkish ourselves.

The software that splits text into tokens is called a tokenizer. Every model has a tokenizer and a vocabulary. The Understand and count tokens page of Google's Gemini API documentation explains that the set of all tokens a model uses is called its vocabulary, and the process of splitting text into tokens is called tokenization.

One common method is BPE (byte pair encoding). The paper Neural Machine Translation of Rare Words with Subword Units, published by Sennrich, Haddow and Birch on 31 August 2015, brought the idea of encoding rare and unknown words as smaller subword units to machine translation. OpenAI's tiktoken library also uses BPE. According to the library's description, BPE is reversible and lossless, so you can get the original text back from the tokens. It works on any text and tries to let the model see common subwords as single pieces.

A common word becomes a single token, while a rarer word is split into several pieces. We tried this ourselves with OpenAI's o200k_base tokenizer on 3 November 2026. The Turkish sentence in the table means "AI models spend more tokens on Turkish text", and "evlerimizden" means "from our houses".

In the English sentence almost every word is one token. In the Turkish sentence even "yapay zeka" (artificial intelligence) is split into four pieces. These two sentences are only an example. The real comparison is the full-text measurement.

On general-purpose tokenizers, Turkish text uses more tokens than the English version of the same content, because Turkish words are long, carry many suffixes and get split into more pieces. To measure this, we ran the Turkish and English versions of the same content through ten tokenizers. The text is the Universal Declaration of Human Rights and its Turkish translation, İnsan Hakları Evrensel Beyannamesi, both published by the UN Office of the High Commissioner for Human Rights (OHCHR). The Turkish text has 10,383 characters and 1,396 words, the English text 10,742 characters and 1,779 words. We measured on 2 November 2026.

A ratio above 1 means the Turkish version takes more tokens for the same content, below 1 means fewer. The table shows four things.

• General-purpose tokenizers use 1.41 to 1.96 times as many tokens for Turkish. OpenAI's newer o200k_base encoding shows a gap of 1.45 times and the older cl100k_base encoding 1.88 times, so OpenAI's newer encoding is clearly more efficient for Turkish than the older one.
• In English, every general-purpose tokenizer uses about 1.25 tokens per word. For Turkish, the same tokenizers use 2.2 to 3.1 tokens per word.
• On tokenizers trained on Turkish text, the picture reverses. Kumru-2B, turkish-gpt2-large and TURNA use 0.51 to 0.62 times as many tokens for the Turkish version as for the English one.
• Turkish-Llama-8b-v0.1 stays close to the general-purpose tokenizers (1.41 times). The Turkish-Llama-8b-v0.1 model card on Hugging Face describes the model as a version of Llama 3 8B trained on Turkish data. Its vocabulary of 128,256 tokens is also the same size as Meta's Llama 3 tokenizer. So the model was trained on Turkish data, but its tokenizer most likely comes from Llama 3.

The gap comes mostly from the text a tokenizer was trained on. Turkish is an agglutinative language, and a single word such as "evlerimizden" carries what English says in three words ("from our houses"). In our measurement the Turkish text has fewer but longer words than the English one, and general-purpose tokenizers split those long words into more pieces. The fact that the gap reverses on tokenizers trained on Turkish shows that the problem lies less in the language itself than in the tokenizer's training data.

The study Language Model Tokenizers Introduce Unfairness Between Languages, published by Petrov, La Malfa, Torr and Bibi on 17 May 2023, likewise finds that the same text translated into different languages can differ in token length by up to 15 times in some cases. According to the authors, the gap shows up in cost, latency and how much context can be given to the model.

We counted the two language versions of the same text with each tokenizer's own library, without adding special tokens.

• Text: The Turkish and English PDFs of OHCHR's Universal Declaration of Human Rights, with the title, the preamble and the 30 articles. The text was extracted from the PDFs with pdftotext from Poppler 24.04, without layout mode. Form feed characters were removed, both languages were normalized to Unicode NFC, and no words were changed. Text extracted with a different pdftotext build changes the token counts by about 5 percent, while the Turkish / English ratios stay close (1.47 instead of 1.45 on o200k_base).
• Counting: Tokens were counted without special tokens. No start or end tokens and no chat template were added, so the numbers in the table are content tokens only. API calls add a few formatting tokens to each message.
• Words: Words were counted by splitting on whitespace, with the same rule in both languages. Because Turkish words are longer, the fair comparison is the Turkish / English ratio for the whole text.
• Tools: The OpenAI encodings were measured with tiktoken 0.14.0 and the others with Hugging Face transformers 5.22.1 and tokenizers 0.23.3. Each Hugging Face tokenizer was loaded at a pinned revision (commit), and we checked that the text decoded from the tokens matched the input. TURNA's tokenizer normalizes line breaks and spaces, so its check was run on a version of the text with whitespace collapsed. TURNA's token count is the same on that version (1,915).
• Cross-check: The Mistral NeMo result was measured again with Mistral's own mistral-common 1.13.0 library and came out the same (3,460 and 2,264).
• Rerun: The whole measurement was rerun from scratch on 3 November 2026 and gave the same numbers.
• Model mapping: The OpenAI model names in the table are based on which encoding tiktoken 0.14.0 assigns to which model. That version maps GPT-4o, GPT-4.1, o3 and GPT-5 to o200k_base.

This measurement gives the result for one text on the tokenizers we measured, not a fixed multiplier that holds for every text and every model.

• One text: This is a formal legal text in an old-fashioned style. The Turkish translation uses older words such as hürriyet (liberty) and haysiyet (dignity), and the OHCHR PDF has at least one typo ("oralak" instead of "olarak"). We left the text as it is. Everyday Turkish, email, chat messages, code or technical documents will give different ratios.
• Tokenizers measured: The table covers OpenAI's two encodings, the Qwen, Mistral NeMo and DeepSeek-V3 tokenizers, and the tokenizers of four models built for Turkish. For Claude and Gemini, the most accurate token count comes from the provider's own token counting tool.
• Tokens are not prices: Each provider prices tokens differently, charges input and output separately, and prices vary from model to model. There are no prices in this table. If you pay per input token on the same model, 1.45 times as many tokens means roughly 1.45 times the input cost for the same text.
• Fewer tokens do not mean a better model: A tokenizer trained on Turkish splitting Turkish text into fewer tokens does not show that the model is better or cheaper overall.

The token count matters because API costs, context windows and usage limits are all measured in tokens.

1. API cost: OpenAI, Anthropic and Google price their APIs per token. The Gemini API documentation says that the cost of a call depends in part on the number of input and output tokens. If Turkish text takes more tokens, the same work in Turkish is billed for more tokens. Turning a per-token price into a cost per task is a separate calculation. RuneLab's AI FinOps: Token Economics, Budgeting and Unit Cost Management course works through it with token economics, model routing, caching and unit economics.
2. Context window: The amount of text a model can see at once is also measured in tokens. In the same context window, less Turkish content fits than English. We explain what a context window is in our post on large language models.
3. Limits: Rate and usage limits are often defined in tokens too. Anthropic's Token counting documentation says that counting tokens helps you manage rate limits and costs ahead of time. Subscription chat products are moving to tokens as well. OpenAI has announced that from December 2026 it will measure ChatGPT and Codex usage in tokens on the Plus, Pro and Business plans. We cover the details in ChatGPT usage limits.

Changing models changes the token count too. According to Anthropic's token counting documentation, Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer, and the same text produces about 30 percent more tokens on them than on earlier models. The documentation adds that the exact increase depends on the content. So even with the same provider, switching models can change the token count. Do not use a count measured on an older model to estimate costs on a newer one.

Tool definitions use tokens as well. OpenAI's Counting tokens guide says that tool definitions such as function schemas and MCP servers add tokens to the context, and that these tokens are hard to count locally. When you connect many tools to an AI assistant through MCP (Model Context Protocol), those definitions take up room in the context window on every request.

The most accurate token count comes from the provider's own token counting tool.

• OpenAI: You can count locally with the tiktoken library. OpenAI's guide says tiktoken works for plain text, while for requests with images, files, tools and schemas the input token count endpoint returns the exact count the model will receive. That count includes formatting tokens such as message roles.
• Anthropic (Claude): The count_tokens endpoint of the Messages API returns the token count before you send the message. According to the documentation it is free to use but subject to a requests-per-minute limit, and the result is an estimate that can differ from the actual count by a small amount.
• Google (Gemini): The count_tokens (countTokens) call returns the token count of the input only. The usage field in the response breaks down input, output, thinking, cached and tool use tokens separately.

If you are planning a Turkish application, take a sample of your own real texts, count it with the tool of the model you will use, and calculate costs from that count. The "1 token is about 4 characters" rule is a guide for English. In our measurement, Turkish text averaged 3.23 characters per token on o200k_base.

There are four ways to reduce token usage and token costs for Turkish.

• Compare tokenizers: In our measurement, the same Turkish text differed by 1,087 tokens between OpenAI o200k_base and DeepSeek-V3 (3,213 and 4,300). When choosing a model, count your own text on several models.
• Cache long repeated instructions: Anthropic's Pricing page says that input tokens read from the cache (prompt caching) cost a small fraction of the standard input price. Reading the same long system prompt or document from the cache is therefore cheaper than processing it again on every request. This lowers the price of repeated tokens, not the token count.
• Leave unneeded text out of the context: Instead of sending a long document archive with every request, a RAG setup that passes only the relevant passages can cut token use. We explain how it works in How RAG finds answers.
• Consider local models for Turkish-heavy work: In our measurement, tokenizers trained on Turkish text split Turkish into fewer tokens. If you want to run a model on your own server, our guide to running an LLM locally and offline is a place to start. Test the model's quality separately.

Choosing a model does not have to be a one-time decision. It can also happen per request. RuneLab's RuneGate gives applications a single OpenAI-compatible endpoint, routes each request by difficulty and sends simple requests to an economical model. It logs which request went to which model along with its cost, sets a monthly budget for each team and stops requests before the budget is exceeded.

In AI, a token is the smallest piece of text a model works with. It can be a word, part of a word or a single character. The model reads text in these pieces and also produces its answer token by token.

In large language models (LLMs), the token is the unit that measures input and output. The amount of text a model can see at once, usage limits and API prices are all calculated from the token count.

It depends on the language. Google writes that in English 1 token is about 4 characters and 400 tokens are about 300 words. In RuneLab's measurement, the English text took about 1.25 tokens per word on general-purpose tokenizers, while the Turkish version of the same text took 2.2 to 3.1 tokens per word.

Turkish is an agglutinative language and its words are long. General-purpose tokenizers split these words into more pieces. In RuneLab's measurement of 2 November 2026, the Turkish version of the same text took 1.45 times as many tokens on OpenAI o200k_base and 1.96 times as many on DeepSeek-V3.

If you pay per token on the API, the same content is billed for more tokens in Turkish because it takes more tokens. In our measurement the gap on OpenAI's o200k_base encoding was 1.45 times. ChatGPT subscription plans are paid per plan, but OpenAI has announced that from December 2026 it will measure usage on the Plus, Pro and Business plans in tokens.

A tokenizer is the software that splits text into the tokens a model understands and turns tokens back into text. Every model has its own tokenizer and vocabulary, so the same text takes a different number of tokens on different models.

For the most accurate count, use the tool of the model you plan to use. On OpenAI, tiktoken or the input token count endpoint gives the count, on Claude the count_tokens endpoint, and on Gemini the countTokens call. Estimating with average character rules can be misleading for Turkish.

Among the general-purpose tokenizers RuneLab measured, Turkish-Llama-8b-v0.1 (1.41 times) and OpenAI o200k_base (1.45 times) used the fewest tokens for Turkish. Kumru-2B, turkish-gpt2-large and TURNA, which were trained on Turkish, split the Turkish text into fewer tokens than the English one. That does not mean the model is better.

• RuneLab, Turkish and English token measurement (2 November 2026, rerun on 3 November 2026): the token table, word and character counts, tokens per word and characters per token
• OHCHR, İnsan Hakları Evrensel Beyannamesi (Turkish PDF): the Turkish text of the measurement
• OHCHR, Universal Declaration of Human Rights (English PDF): the English text of the measurement
• Petrov, La Malfa, Torr and Bibi, Language Model Tokenizers Introduce Unfairness Between Languages (arXiv): token length differences between languages and their effects
• Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (arXiv): subword units and BPE
• Google for Developers, Machine Learning Crash Course, LLM module: the definition of a token, the character and word ratio for English, and tokenization being language specific
• Google, Gemini API documentation, Understand and count tokens: the definitions of vocabulary and tokenization, how token counts affect cost, count_tokens and the usage field
• Anthropic, Token counting: the count_tokens endpoint, free but rate-limited use, estimated counts, the newer tokenizer and the increase of about 30 percent
• Anthropic, Pricing: prompt caching and the price of cache reads
• OpenAI, Counting tokens: the limits of tiktoken, the input token count endpoint, formatting tokens and tool definitions
• OpenAI, tiktoken GitHub repository: the properties of BPE, version 0.14.0 used in the measurement and its model-to-encoding mapping
• Hugging Face model cards: Kumru-2B, turkish-gpt2-large, Turkish-Llama-8b-v0.1 model card and TURNA: the Turkish tokenizers measured and their developers

• What is an LLM? How large language models work
• ChatGPT usage limits and what changes in December
• Claude pricing 2026: Pro, Max, Ultra and Team
• ChatGPT Business and Pro API credits guide
• What is an AI agent?
• RAG explained: how answers are found in documents
• Course: AI FinOps: Token Economics, Budgeting and Unit Cost Management