mothertoken
know the mother tongue of your LLMs
Last verified:
What is mothertoken?
Mothertoken is a CLI toolkit for comparing tokenizer efficiency across languages, model families, and user-supplied Hugging Face references. The tool helps users discover which models make their language cheaper, longer, or harder to fit in context windows by revealing how different models spend tokens differently across languages.
Key features include: the rank command to find token-efficient models for a specific language (e.g., mothertoken rank spanish), the tokenize command to count tokens for exact text with model support (e.g., mothertoken tokenize "Hola Mundo" --model gpt-4o), the compare command to benchmark multiple models on the same text, and the benchmark run command to generate custom benchmark data across languages and models. It also includes mothertoken list to see available tokenizer IDs and which models use them, plus support for file input with --file and direct Hugging Face references.
Mothertoken is designed for AI developers, prompt engineers, multilingual content creators, researchers studying tokenizer fairness, and anyone working with LLMs in non-English languages. The tool addresses
mothertoken pricing
Pricing model: Freemium
Mothertoken is free and open-source under MIT license. The package is available on PyPI with pip install mothertoken at no cost. No paid tiers, subscription plans, or enterprise features exist. All features including rank, tokenize, compare, benchmark, and list commands are included in the free package. Local development requires free tools (uv, git) but no licensing fees.
mothertoken pros
- Open-source tool with MIT license
- Available as pip package for easy installation
- CLI interface with intuitive commands
- Supports 15+ languages including non-English
- Compares multiple models on same text
- Rank command finds best model for your language
- Tokenize command counts exact tokens for text
- Compare command benchmarks model alternatives
- Benchmark run generates custom benchmark data
- Direct Hugging Face model reference support
- File input support with --file flag
- Uses FLORES+ parallel corpus data
- Precomputed benchmark data included
- Shows tokenizer IDs and associated models
- Addresses tokenizer unfairness problem
- Works with GPT, Qwen, Mistral, DeepSeek models
- Local tokenization without API calls
- Language-specific fertility metrics
- Efficiency metrics including rtc (runtime)
- Chars per token measurements
- Vs English comparison baseline
- User-owned benchmark file support
- Configurable benchmark output location
- Cross-platform (Linux/macOS/Windows)
mothertoken cons
- No web GUI - CLI only
- Requires Python and uv for development
- Bundled benchmark not exhaustive
- Limited to precomputed tokenizer set
- No real-time API token counting
- English-equivalent estimates approximate
- No cloud-based interface
- Setup complexity for local development
- Documentation primarily in code/comments
- No paid enterprise features
- Limited visual output options
- No mobile app support
- No automated report generation
- Benchmark regeneration manual process
- No multi-user collaboration
- Limited language coverage beyond 15
- No integration with CI/CD pipelines
- No token usage tracking over time
Frequently asked questions about mothertoken
I speak a language that is not English. Which tokenizer is most efficient for it?
Use the rank command: mothertoken rank spanish (or your language name). This shows which tokenizers are most efficient for your language using precomputed benchmark data from FLORES+. You can also use raw FLORES+ codes like mothertoken rank spa_Latn.
I have this model. Which languages does it tokenize best, which ones worst?
Use the benchmark run command: mothertoken benchmark run --models gpt-oss,YOUR_MODEL1,YOUR_MODEL2. This generates benchmark data showing token efficiency across multiple languages for your specified models.
What languages does Mothertoken support?
Mothertoken supports 15+ languages: English (eng_Latn), French (fra_Latn), Spanish (spa_Latn), Portuguese (por_Latn), German (deu_Latn), Arabic (arb_Arab), Chinese (cmn_Hans), Japanese (jpn_Jpan), Thai (tha_Thai), Hindi (hin_Deva), Korean (kor_Hang), Turkish (tur_Latn), Ukrainian (ukr_Cyrl), Vietnamese (vie_Latn), and Swahili (swh_Latn).
How do I install Mothertoken?
For the published package: pip install mothertoken. For local development: git clone https://github.com/inimaz/mothertoken, then cd mothertoken, uv sync, and uv pip install -e .
What is the difference between local tokenization and English-equivalent count?
Local tokenization counts exact tokens using the model's tokenizer. English-equivalent count (with --language) estimates the token count using a benchmark multiplier derived from FLORES+ data, showing how many tokens your language would cost relative to English.
How do I create and manage custom benchmark data?
Use mothertoken benchmark run to create data (writes to user-owned file and makes it active), mothertoken benchmark use benchmark.json to make an existing benchmark active, mothertoken benchmark status to inspect what commands will use, and mothertoken benchmark use --default to return to the bundled default benchmark.
Is Mothertoken free or does it have paid plans?
Mothertoken is completely free and open-source under MIT license. There are no paid tiers, subscriptions, or enterprise plans. All features are included in the free PyPI package.