My model once mistook Tamil for Telugu, so I built an open-source language identification tool for Hinglish, Tanglish, and Kanglish.
Bhasha is an Apache-2.0 word-level language identifier for code-mixed Indian text, including Hinglish, Tanglish, and Kanglish, and runs on a CPU.
Key Takeaways
- Bhasha is a free, Apache-2.0, 278M-parameter CPU-friendly model that assigns language labels to each word in code-mixed Indian text, including Hinglish, Tanglish, and Kanglish, covering major Indian languages and English.
- Word-level language tags matter for code-mixed messages because they help route words to appropriate transliterators, tokenizers, spell-checkers, or translation models, and measure language mixing, unlike message-level identification.
- The model returns per-word labels plus message-level summaries such as main language, script, mixing status, and language shares; when Hindi and Urdu cannot be distinguished, it outputs the combined label hi/ur.
- Training data was built entirely with code from public corpora and transliteration pipelines, avoiding hand labels and LLM-generated text to keep licensing clean, but this can bias results toward news/web-style language.
- On LLM-labelled YouTube comments, Bhasha performed strongly, especially for Malayalam and Kannada, while human-labelled Tamil comments exposed a weakness that LLM-based tests had partially hidden.
Arun Prabhakaran (@LatentAILabs) has published Bhasha (arunp0/bhasha-lid-indic), a free, open-source model that assigns a language label to each word in code-mixed Indian text. It targets Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Odia and English, plus name and other, and reads Roman letters, native scripts and messages that mix both. The model is Apache-2.0, has 278M parameters and runs on a CPU. Its strongest results came on Malayalam and Kannada, while human-labelled Tamil comments revealed a gap that LLM-labelled tests had hidden.
Why word-level language tags matter
Most language identification tools return one label for an entire message. That is useful for routing a message, but it is not enough when a system needs to know which words belong to which language. Word-level labels can help route each word to the right transliterator or spell-checker, select the correct tokenizer or translation model, measure how much code-mixing is present, and filter a corpus when each sentence contains a mix of English and an Indian language.
The pattern is common in messages typed on phones, especially in Hinglish, Manglish (Malayalam) or Kanglish (Kannada). Prabhakaran cites a made-up comment of the kind seen under a Tamil film trailer: “padam semma mass bro, second half konjam slow but climax vera level”. Half the words are Tamil and half are English, but the whole comment is written in Roman letters.
For Hinglish, there are good word-level models. For Tamil, Malayalam and Kannada mixed with English, Prabhakaran found no licensed, released word-level model for romanised text. He found only two downloadable checkpoints without a licence: the 2022 shared-task winner for Kannada, a DistilBERT model that judges one word at a time, and a Malayalam model with no model card. Newer MIT models such as MixLID cover these languages only in native scripts. Bhasha, which simply means language, was built to fill that gap. It is the fourth model in a small open set for Indian languages, after Yama for zero-shot classification, Chitragupta for PII detection and Nandi for complaint routing.
What Bhasha returns
Bhasha returns a label for each word, plus a message summary. The summary includes the main language, the script used (native, Roman or mixed), whether the message is mixed, and the share of each language. Romanised Hindi and Urdu can contain the same words, so when the model cannot tell them apart it returns hi/ur instead of guessing.
The langs argument matters. If the expected language pair is known, such as Tamil plus English, passing it limits the output to the expected language plus English, name and other. Prabhakaran says most users already know their language pair: a Tamil film channel, for example, can reasonably expect Tamil and English.
Training data was built entirely by code
No openly licensed, word-labelled code-mixed corpus covered these languages, so every training row was built by code from public text. Prabhakaran says there were no hand labels and no LLM-written text in the training data. That keeps the licence clean, but it also means all Indian-language text started as news and web writing, a limitation that later shows up in the Tamil results.
The pipeline began with real native sentences from IndicCorp v2 (CC0), which contains news and web text in all 11 languages and English. Code then romanised native words using Aksharantar (CC-BY), the IndicXlit transliteration model (MIT) and simple rules, weighted by how often each spelling appears in real romanised web text using FineWeb-2 word counts. A Hindi word, for example, becomes “nahi” most of the time and “nahin” or “nhi” some of the time.
English spans of 1-6 words were inserted into sentences, and the language of every word was recorded by code. Name and English-loanword tags used simple rules, so they are less exact. The pipeline also added names from Wikidata, emoji, @mentions, numbers, stretched words such as “superrr”, dropped vowels such as “nhi”, random capitals and native-script words mixed into Roman text.
The result was 1.65 million rows across 11 languages: romanised, mixed, native, native mixed, mixed script and plain English. Prabhakaran says no training row overlapped an evaluation set, except that two native-script test sentences that matched training text were dropped from that test. A later check found no overlap with the human-labelled sets either. He fine-tuned XLM-RoBERTa-base for one epoch, which took about an hour on one L4 GPU.
The author’s LLM-labelled test showed strong results
Before training, Prabhakaran wrote an evaluation plan and a bar for each language pair: beat the better of two GlotLID baselines, with a margin of 0.10 over GlotLID applied word by word.
He then built a real test set from 425 YouTube comments in DravidianCodeMix (CC-BY): 150 Tamil, 150 Malayalam and 125 Kannada comments, each mixed with English. Each word was labelled by two Claude passes and by Gemini, and a 2-of-3 vote decided the label. The three labellers agreed on 95-99% of words. These labels were LLM-reviewed, not human-verified. The frozen model was scored once.
The released Hinglish and Indic word-level models were built for Hindi, Bengali and Gujarati. They have no Tamil, Malayalam or Kannada label, so they fail on these tests. In this context, macro-F1 averages two F1 scores, one for the Indian language and one for English. A perfect score is 1.0, and a model that never finds the Indian language tops out near 0.5.
On the LLM-labelled YouTube comments, Bhasha scored 0.87-0.89, while the released Hindi/Bengali word-level models scored 0.28-0.44 and the other released models tested scored 0.64-0.79. Malayalam scored 0.888 and Kannada scored 0.888, clearing the bars comfortably. Tamil scored 0.873, beating the strongest baseline, GlotLID on the whole sentence at 0.812, but missing Prabhakaran’s own bar of 0.878 by 0.005. That bar was set by the author, not an external benchmark.
For Hindi-English, Bhasha scored 0.896 on the COMI-LINGUA test set, ahead of muril-hinglish-lid at 0.857. On LinCE it scored 0.922 against muril-hinglish-lid’s 0.952. Prabhakaran notes that muril-hinglish-lid was trained on real Hinglish and that he used LinCE as a development check during training, so the LinCE number is not independent.
At that stage, he says he would have shipped the model with the Tamil miss stated. Then he found human-labelled data.
Human labels changed the Tamil picture
Prabhakaran found three sets of Dravidian code-mixed text with word-level language labels written by people. CoLI-Dravidian 2024, part of the FIRE shared task, contains Tamil, Malayalam and Kannada YouTube comments in Roman letters, labelled by native speakers. CoLI-Kanglish, from ICON 2022, contains Kannada-English YouTube comments and is independent of CoLI-Dravidian, with no shared text. WoLLaI, from Pondicherry University and CC BY, contains 12,401 Malayalam-English sentences.
He used these sets for evaluation only, with the model frozen, and scored each set once. He reports totals only and does not redistribute them, noting that the CoLI files state no licence. The CoLI files have one word per row and no sentence breaks, so they were scored in 12-word windows.
On human-labelled sets, Bhasha scores 0.84-0.90 on Kannada, 0.91-0.95 on Malayalam and 0.90 on Hindi-English, measured by macro-F1. Tamil, however, scored 0.72 on human-labelled comments, compared with 0.87 on his own test set. About a third of the Tamil words were tagged Telugu. Malayalam and Kannada held up, but Tamil did not.
Prabhakaran says a drop that large is a finding, not noise. One caveat is that, of the three CoLI files, the Tamil one kept its original word order least well, and context helped it least. Part of the drop may come from scoring in windows rather than from the model itself. The error pattern appears either way.
Bhasha is not the best Kannada model everywhere
The released models tested show a more mixed picture. distilka, the 2022 Kanglish winner, beats Bhasha on CoLI Kannada: 0.954 against 0.903, or 0.936 with the language hint. It was trained on data from the same shared-task series. On independent Kannada comments, Bhasha leads, 0.888 against 0.724. Prabhakaran gives two caveats: those comments have LLM labels, and distilka’s labels are unnamed, so he mapped them using its own training data.
The Malayalam XLM-R checkpoint ties Bhasha on WoLLaI at 0.952 each, but scores 0.682 on CoLI Malayalam and 0.638 on the author’s comments, against Bhasha’s 0.911 and 0.888. MixLID gets almost no romanised Indian words right, with Indian-language F1 near 0, as expected for a native-script model. On native-script sentences it is close to Bhasha: 0.9916 against 0.9956. Prabhakaran ran it the way its own helper code does, using plain Transformers and no remote code, taking each word’s first sub-token as the word’s label.
So Bhasha is not the best Kannada model on every test. But Prabhakaran says it is the only model he found that scores 0.84 or higher on every Kannada and Malayalam set, and the only licensed word-level model he found for romanised Tamil, Malayalam or Kannada.
Where the Tamil words went
To investigate the Tamil drop, Prabhakaran counted, for every Tamil word in the human-labelled set, which label the model gave it. He counted labels only and did not read the comments, so the evaluation stayed blind.
Telugu is as common an answer as Tamil itself. This is not a labelling mismatch: names, numbers and English barely move. The model calls Tamil words Telugu or another South Indian language. The same pull appears more weakly elsewhere: 9% of Kannada words and 5.5% of Malayalam words are also tagged Telugu.
The author’s own 425-comment set did not show this, probably because its Tamil comments were easier. The comment at the top of the article is an example: without a language hint, Bhasha tags padam, semma, konjam and vera as Telugu.
Prabhakaran’s best explanation is the kind of Tamil the model learned. Written Tamil, the kind found in news, differs a lot from spoken Tamil: a newspaper writes irukkiradhu where a commenter writes irukku. Bhasha only ever saw news Tamil converted to Roman letters. When it meets spoken forms it has never seen, it falls back on the nearest South Indian language it has.
A quick fix: tell it the language pair
Because most users already know their language pair, the wrapper has a langs option that limits the output to the expected language plus English, name and other. Prabhakaran describes this as a fixed rule with nothing tuned on the test sets. He scored it once on every set.
The hint helps every human-labelled Dravidian set. For Tamil, it mostly moves the errors to English. With Telugu ruled out, the Tamil score itself rises from 0.53 to 0.78. But 2,188 Tamil words then come out as English, and the English score falls from 0.92 to 0.73. Among the allowed labels, the model ranks English above Tamil for those words, and another 653 become name, so it does not recognise them as Tamil at all. On the author’s own 425-comment set, the hint does not help Tamil either: the score moves from 0.873 to 0.865.
That result supports the spoken-Tamil explanation. The language-pair hint reduces Telugu confusion, but it does not make Bhasha recognise spoken Tamil forms it was never trained on.
What this means for developers
Bhasha is a practical option for developers who need word-level language tagging in code-mixed Indian text, especially for Malayalam, Kannada and Hindi-English, and for anyone who needs a licensed word-level model for romanised Tamil, Malayalam or Kannada. It is Apache-2.0, has 278M parameters and runs on a CPU.
For Tamil, the model card and evaluation logs suggest caution. The LLM-labelled test gave 0.87, but human-labelled comments gave 0.72, with many Tamil words labelled Telugu. Passing a known language pair helps, but the underlying weakness appears to be the absence of spoken Tamil forms in the training data.
Prabhakaran says parts of the article were drafted with AI assistants and that he checked every number against the model card and the evaluation logs. The broader lesson is that code-mixed Indian language models need evaluation sets that match real spoken typing, not only news-style text converted into Roman letters.
Meta description
Bhasha is an open-source word-level language ID model for code-mixed Indian text; it excels on Malayalam and Kannada but struggles with Tamil.
Tags
Bhasha, language identification, code-mixed Indian text, open-source model, Tamil
Featured image
Featured image: abstract, friendly illustration of multilingual word tokens flowing into language tags, with no real people, no logos and no text.

Comments (0)