HuggingFace Tokenizers Skill for Hermes Agent | Fast NLP Tokenization

This skill brings the HuggingFace Tokenizers library into Hermes Agent, giving you Rust-speed tokenization and the ability to train custom BPE, WordPiece, or Unigram tokenizers from your own data. You would reach for this when you need to process large corpora in seconds, build a production NLP pipeline, or train a tokenizer that matches your domain vocabulary exactly.

What it does

The skill wraps the tokenizers Python library (Rust core, Python bindings) so you can load pretrained tokenizers from the HuggingFace Hub, train new ones from scratch, and encode or decode text at 10-100x the speed of pure Python implementations. It also exposes the full tokenization pipeline (normalization, pre-tokenization, model, post-processing) and alignment tracking, which maps every token back to its position in the original text. You can use it standalone or integrate the resulting tokenizer with the transformers library for model training.

Before you start

Skill path: optional-skills/mlops/huggingface-tokenizers
Tier: optional, installed on demand
Version: 1.0.0
Author: Orchestra Research
License: MIT
Platforms: linux, macos, windows
Upstream tags: Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production

Install the Python package inside your Hermes Agent environment:

# Install tokenizers
pip install tokenizers

# With transformers integration
pip install tokenizers transformers

No special permissions or API keys are required. The skill works on any supported platform.

Quick start

Load a pretrained tokenizer

from tokenizers import Tokenizer

# Load from HuggingFace Hub
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

# Encode text
output = tokenizer.encode("Hello, how are you?")
print(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']
print(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]

# Decode back
text = tokenizer.decode(output.ids)
print(text)  # "hello, how are you?"

Train a custom BPE tokenizer

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

# Initialize tokenizer with BPE model
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

# Configure trainer
trainer = BpeTrainer(
    vocab_size=30000,
    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
    min_frequency=2
)

# Train on files
files = ["train.txt", "validation.txt"]
tokenizer.train(files, trainer)

# Save
tokenizer.save("my-tokenizer.json")

Training time is roughly 1-2 minutes for a 100MB corpus and 10-20 minutes for 1GB.

Batch encoding with padding

# Enable padding
tokenizer.enable_padding(pad_id=3, pad_token="[PAD]")

# Encode batch
texts = ["Hello world", "This is a longer sentence"]
encodings = tokenizer.encode_batch(texts)

for encoding in encodings:
    print(encoding.ids)
# [101, 7592, 2088, 102, 3, 3, 3]
# [101, 2023, 2003, 1037, 2936, 6251, 102]

Tokenization algorithms

BPE (Byte-Pair Encoding)

BPE starts with a character-level vocabulary, finds the most frequent character pair, merges it into a new token, and repeats until the vocabulary reaches the target size. It is used by GPT-2, GPT-3, RoBERTa, BART, and DeBERTa.

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel

tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>"))
tokenizer.pre_tokenizer = ByteLevel()

trainer = BpeTrainer(
    vocab_size=50257,
    special_tokens=["<|endoftext|>"],
    min_frequency=2
)

tokenizer.train(files=["data.txt"], trainer=trainer)

Advantages: Handles out-of-vocabulary words well by breaking them into subwords, flexible vocabulary size, good for morphologically rich languages.

Trade-offs: Tokenization depends on merge order, may split common words unexpectedly.

WordPiece

WordPiece starts with a character vocabulary, scores each potential merge pair by frequency(pair) / (frequency(first) × frequency(second)), merges the highest scoring pair, and repeats. It is used by BERT, DistilBERT, and MobileBERT.

from tokenizers import Tokenizer
from tokenizers.models import WordPiece
from tokenizers.trainers import WordPieceTrainer
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.normalizers import BertNormalizer

tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))
tokenizer.normalizer = BertNormalizer(lowercase=True)
tokenizer.pre_tokenizer = Whitespace()

trainer = WordPieceTrainer(
    vocab_size=30522,
    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
    continuing_subword_prefix="##"
)

tokenizer.train(files=["corpus.txt"], trainer=trainer)

Advantages: Prioritizes semantically meaningful merges, proven in state-of-the-art BERT models.

Trade-offs: Unknown words become [UNK] if no subword match is found; saves vocabulary rather than merge rules, leading to larger file sizes.

Unigram

Unigram starts with a large vocabulary of all substrings, computes the loss for the corpus, removes tokens with minimal impact on loss, and repeats until the target vocabulary size is reached. It is used by ALBERT, T5, mBART, and XLNet (via SentencePiece).

from tokenizers import Tokenizer
from tokenizers.models import Unigram
from tokenizers.trainers import UnigramTrainer

tokenizer = Tokenizer(Unigram())

trainer = UnigramTrainer(
    vocab_size=8000,
    special_tokens=["<unk>", "<s>", "</s>"],
    unk_token="<unk>"
)

tokenizer.train(files=["data.txt"], trainer=trainer)

Advantages: Probabilistic (finds the most likely tokenization), works well for languages without word boundaries, handles diverse linguistic contexts.

Trade-offs: Computationally expensive to train, more hyperparameters to tune.

Tokenization pipeline

The complete pipeline runs in four stages: Normalization → Pre-tokenization → Model → Post-processing.

Normalization

Clean and standardize text before tokenization:

from tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence

tokenizer.normalizer = Sequence([
    NFD(),           # Unicode normalization (decompose)
    Lowercase(),     # Convert to lowercase
    StripAccents()   # Remove accents
])

# Input: "Héllo WORLD"
# After normalization: "hello world"

Common normalizers include NFD, NFC, NFKD, NFKC (Unicode normalization forms), Lowercase(), StripAccents(), Strip(), and Replace(pattern, content) for regex replacement.

Pre-tokenization

Split text into word-like units:

from tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel

# Split on whitespace and punctuation
tokenizer.pre_tokenizer = Sequence([
    Whitespace(),
    Punctuation()
])

# Input: "Hello, world!"
# After pre-tokenization: ["Hello", ",", "world", "!"]

Common pre-tokenizers include Whitespace(), ByteLevel() (GPT-2 style), Punctuation(), Digits(individual_digits=True), and Metaspace() (SentencePiece style, replaces spaces with ▁).

Post-processing

Add special tokens for model input:

from tokenizers.processors import TemplateProcessing

# BERT-style: [CLS] sentence [SEP]
tokenizer.post_processor = TemplateProcessing(
    single="[CLS] $A [SEP]",
    pair="[CLS] $A [SEP] $B [SEP]",
    special_tokens=[
        ("[CLS]", 1),
        ("[SEP]", 2),
    ],
)

Common patterns:

# GPT-2: sentence <|endoftext|>
TemplateProcessing(
    single="$A <|endoftext|>",
    special_tokens=[("<|endoftext|>", 50256)]
)

# RoBERTa: <s> sentence </s>
TemplateProcessing(
    single="<s> $A </s>",
    pair="<s> $A </s> </s> $B </s>",
    special_tokens=[("<s>", 0), ("</s>", 2)]
)

Alignment tracking

Track token positions in the original text:

output = tokenizer.encode("Hello, world!")

# Get token offsets
for token, offset in zip(output.tokens, output.offsets):
    start, end = offset
    print(f"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}")

# Output:
# hello      → [ 0,  5): 'Hello'
# ,          → [ 5,  6): ','
# world      → [ 7, 12): 'world'
# !          → [12, 13): '!'

This is essential for named entity recognition (mapping predictions back to text), question answering (extracting answer spans), and token classification (aligning labels to original positions).

Integration with transformers

Load with AutoTokenizer

from transformers import AutoTokenizer

# AutoTokenizer automatically uses fast tokenizers
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Check if using fast tokenizer
print(tokenizer.is_fast)  # True

# Access underlying tokenizers.Tokenizer
fast_tokenizer = tokenizer.backend_tokenizer
print(type(fast_tokenizer))  # <class 'tokenizers.Tokenizer'>

Convert custom tokenizer to transformers

from tokenizers import Tokenizer
from transformers import PreTrainedTokenizerFast

# Train custom tokenizer
tokenizer = Tokenizer(BPE())
# ... train tokenizer ...
tokenizer.save("my-tokenizer.json")

# Wrap for transformers
transformers_tokenizer = PreTrainedTokenizerFast(
    tokenizer_file="my-tokenizer.json",
    unk_token="[UNK]",
    pad_token="[PAD]",
    cls_token="[CLS]",
    sep_token="[SEP]",
    mask_token="[MASK]"
)

# Use like any transformers tokenizer
outputs = transformers_tokenizer(
    "Hello world",
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="pt"
)

Common patterns

Train from iterator (large datasets)

from datasets import load_dataset

# Load dataset
dataset = load_dataset("wikitext", "wikitext-103-raw-v1", split="train")

# Create batch iterator
def batch_iterator(batch_size=1000):
    for i in range(0, len(dataset), batch_size):
        yield dataset[i:i + batch_size]["text"]

# Train tokenizer
tokenizer.train_from_iterator(
    batch_iterator(),
    trainer=trainer,
    length=len(dataset)  # For progress bar
)

Performance: processes 1GB in roughly 10-20 minutes.

Enable truncation and padding

# Enable truncation
tokenizer.enable_truncation(max_length=512)

# Enable padding
tokenizer.enable_padding(
    pad_id=tokenizer.token_to_id("[PAD]"),
    pad_token="[PAD]",
    length=512  # Fixed length, or None for batch max
)

# Encode with both
output = tokenizer.encode("This is a long sentence that will be truncated...")
print(len(output.ids))  # 512

Multi-processing

from tokenizers import Tokenizer
from multiprocessing import Pool

# Load tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")

def encode_batch(texts):
    return tokenizer.encode_batch(texts)

# Process large corpus in parallel
with Pool(8) as pool:
    # Split corpus into chunks
    chunk_size = 1000
    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]

    # Encode in parallel
    results = pool.map(encode_batch, chunks)

Speedup: 5-8x with 8 cores.

Performance benchmarks

Training speed

Corpus Size	BPE (30k vocab)	WordPiece (30k)	Unigram (8k)
10 MB	15 sec	18 sec	25 sec
100 MB	1.5 min	2 min	4 min
1 GB	15 min	20 min	40 min

Hardware: 16-core CPU, tested on English Wikipedia.

Tokenization speed

Implementation	1 GB corpus	Throughput
Pure Python	~20 minutes	~50 MB/min
HF Tokenizers	~15 seconds	~4 GB/min
Speedup	80x	80x

Test: English text, average sentence length 20 words.

Memory usage

Task	Memory
Load tokenizer	~10 MB
Train BPE (30k vocab)	~200 MB
Encode 1M sentences	~500 MB

Supported models

Pre-trained tokenizers available via from_pretrained():

BERT family:

bert-base-uncased, bert-large-cased
distilbert-base-uncased
roberta-base, roberta-large

GPT family:

gpt2, gpt2-medium, gpt2-large
distilgpt2

T5 family:

t5-small, t5-base, t5-large
google/flan-t5-xxl

Other:

facebook/bart-base, facebook/mbart-large-cc25
albert-base-v2, albert-xlarge-v2
xlm-roberta-base, xlm-roberta-large

Browse all: https://huggingface.co/models?library=tokenizers

When not to use it

If you need language-independent tokenization (used by T5/ALBERT), use SentencePiece instead. If you are working specifically with OpenAI's GPT models, tiktoken is the native BPE tokenizer. If you only need to load a pretrained tokenizer without training or customizing, the transformers AutoTokenizer already uses this library internally, so you may not need this skill directly.

Limits and gotchas

Training a Unigram tokenizer is computationally expensive and has more hyperparameters to tune than BPE or WordPiece.
WordPiece saves vocabulary but not merge rules, resulting in larger file sizes compared to BPE.
BPE tokenization depends on merge order, which can split common words unexpectedly.
WordPiece maps unknown words to [UNK] if no subword match is found.
The performance benchmarks were run on a 16-core CPU with English Wikipedia; your results will vary with different hardware and languages.

Related references

These companion guides live in the same skill directory and go deeper into specific areas:

Training Guide - Train custom tokenizers, configure trainers, handle large datasets
Algorithms Deep Dive - BPE, WordPiece, Unigram explained in detail
Pipeline Components - Normalizers, pre-tokenizers, post-processors, decoders
Transformers Integration - AutoTokenizer, PreTrainedTokenizerFast, special tokens

Resources

Docs: https://huggingface.co/docs/tokenizers
GitHub: https://github.com/huggingface/tokenizers ⭐ 9,000+
Version: 0.20.0+
Course: https://huggingface.co/learn/nlp-course/chapter6/1
Paper: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)

HuggingFace Tokenizers Skill: Fast BPE, WordPiece & Unigram for Hermes Agent