How I Built a 150 GB Multilingual & Code Dataset for Central Asian AI (And Fought Out-of-Memory Errors for 10 Hours)
说明与用途
为中亚语言构建了150 GB多语言和代码数据集,压缩至约27.7 GB,并发布在 Hugging Face 上,以支持 LLM 微调和技术翻译。
问题
Central Asian languages lack high‑quality open‑source datasets, especially for code, hindering LLM training and technical translation.
结果
Successfully created a 150 GB multilingual and code dataset, compressed to ~27.7 GB, enabling efficient download and use for LLM fine‑tuning and translation tasks.
工作流
- Collect raw text and source code from public repositories and corpora
- Parse and clean data, separating languages and code files
- Store cleaned data in .txt and .jsonl files per language and language code
- Compress the entire dataset using 7z with maximum compression profile
- Handle out‑of‑memory errors by tuning parameters and restarting the process
- Upload the compressed archive to Hugging Face
- Publish dataset under CC BY 4.0 license
证据
The process took 10 agonizing hours. The compression tool kept crashing, my OS froze constantly, and the console kept throwing the dreaded Failed message due to Out-of-Memory (OOM) errors.
- 记录日期: 2026-09-22T01:45:16.592Z