NVIDIA and Hugging Face Open 2.8 Trillion Token Indic Pretraining Corpus With Full Pipeline

NVIDIA has released a 2.8-trillion-token Indic language pretraining corpus alongside a fully open data generation pipeline, published on Hugging Face. The dataset covers a wide range of Indic languages and is accompanied by the tooling needed to reproduce, extend, or adapt the pipeline for other low-resource language families. This is one of the largest publicly available non-English pretraining datasets, addressing a significant gap in multilingual AI development resources. For developers building applications targeting South Asian language markets, this corpus provides a foundation for training or fine-tuning models without constructing datasets from scratch. The open pipeline is particularly valuable, as it enables teams to apply the same methodology to other underrepresented languages.