sentencepiece
Category: ml-training · License: MIT · Version: 1.0.0
Tags: Tokenization, SentencePiece, Language-Independent, BPE, Unigram, Multilingual, CJK Languages, Unicode, Deterministic, Google
When to use it
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.Example requests
Ask your agent in plain language, for example:- “How do I train a 32k Unigram SentencePiece model on my raw text file?”
- “I’m building a Japanese-Chinese-Korean translation model from corpus_cjk.txt (about 2GB, no word segmentation). I need a subword vocab of ~64k, and I want the same vocab every time I rerun the pipeline. Tokenization currently chokes on missing characters and I see lots of <unk>. What should I do?”
- “I want to fine-tune a T5-style model on multilingual text, and I need to build a tokenizer from scratch that uses the same approach T5 uses, without any language-specific preprocessing. How should I set that up?”
Install
/plugin marketplace add KalarisLabs/research-agent-skills and install the whole ml-training category with /plugin install ml-training@research-agent-skills.