How I Built a 150 GB Multilingual & Code Dataset for Central Asian AI (And Fought Out-of-Memory Errors for 10 Hours)

Описание и применение

Создан 150 ГБ многоязычный и кодовый датасет для центральноазиатских языков, сжатый до ~27,7 ГБ и размещённый на Hugging Face, чтобы поддержать обучение LLM и технический перевод.

Задача

Central Asian languages lack high‑quality open‑source datasets, especially for code, hindering LLM training and technical translation.

Результат

Successfully created a 150 GB multilingual and code dataset, compressed to ~27.7 GB, enabling efficient download and use for LLM fine‑tuning and translation tasks.

Рабочий процесс

• Collect raw text and source code from public repositories and corpora
• Parse and clean data, separating languages and code files
• Store cleaned data in .txt and .jsonl files per language and language code
• Compress the entire dataset using 7z with maximum compression profile
• Handle out‑of‑memory errors by tuning parameters and restarting the process
• Upload the compressed archive to Hugging Face
• Publish dataset under CC BY 4.0 license

Доказательства

The process took 10 agonizing hours. The compression tool kept crashing, my OS froze constantly, and the console kept throwing the dreaded Failed message due to Out-of-Memory (OOM) errors.

Источник