Building a Synthetic Data Generator

Описание и применение

Создана гибкая система генерации синтетических данных на основе Red Hat SDG Hub, поддерживающая множество форматов, фильтрацию и экспорт.

Задача

Need versatile synthetic data to support multiple downstream workflows, but existing tools did not fully fit specific pipeline requirements.

Результат

The custom SDG App delivers a flexible, extensible synthetic data pipeline that supports diverse input formats, advanced generation strategies, rigorous quality filtering, and multiple export options, all operable locally or via API.

Рабочий процесс

• Install SDG Hub via pip or plugin marketplace
• Adapt SDG Hub to project needs by adding custom YAML flows and modules
• Build SDG App with core components (seed_parser, prompt_builder, providers, generator, validator, exporter, job_manager, flow_runner)
• Expose services via FastAPI, Gradio UI, and Typer CLI
• Configure LLM backend (Ollama/llama.cpp) and quality filters (toxicity, PII, dedup)
• Run generation flows (e.g., RAG evaluation, Knowledge QA, MCP distillation)
• Export datasets to JSONL, Parquet, or Hugging Face Dataset
• Integrate with Bob and Claude AI assistants via plugins

Доказательства

SDG Hub is an open‑source library providing composable blocks and flows for synthetic data generation.

Источник

The resulting application leverages SDG Hub under the hood while extending its capabilities to match my project’s specific requirements, context, and environment.

Источник