Multilingual Tokenization Engine
A fairness-audited BPE tokenizer for English, Hindi, and Hinglish, with an explainability dashboard.
Tech stack
- Python
- FastAPI
- Next.js
- Streamlit
- Scikit-learn
- NLTK
The problem
Sub-word tokenizers often treat scripts unequally, fragmenting Hindi and Hinglish more aggressively than English. That imbalance quietly penalizes multilingual models — and it's rarely measured.
The solution
A BPE sub-word tokenizer with script-aware pre-tokenization for English, Hindi, and Hinglish, paired with a fairness-audit pipeline and an interactive explainability dashboard that surfaces tokenization parity across scripts.
Architecture
Script-aware pre-tokenization → BPE training → fairness-audit metrics → interactive dashboard (Streamlit/Next.js) over a FastAPI service. Robustness validated against typographic errors and OCR noise.
Highlights
- Script-aware tokenization across English, Hindi, and Hinglish
- Fairness-audit pipeline + interactive explainability dashboard
- Stress-tested against typos and OCR noise for reliable inference