Building a Synthetic Data Generator

说明与用途

基于 Red Hat SDG Hub 构建的灵活合成数据生成系统，支持多种格式、过滤和导出。

问题

Need versatile synthetic data to support multiple downstream workflows, but existing tools did not fully fit specific pipeline requirements.

结果

The custom SDG App delivers a flexible, extensible synthetic data pipeline that supports diverse input formats, advanced generation strategies, rigorous quality filtering, and multiple export options, all operable locally or via API.

工作流

• Install SDG Hub via pip or plugin marketplace
• Adapt SDG Hub to project needs by adding custom YAML flows and modules
• Build SDG App with core components (seed_parser, prompt_builder, providers, generator, validator, exporter, job_manager, flow_runner)
• Expose services via FastAPI, Gradio UI, and Typer CLI
• Configure LLM backend (Ollama/llama.cpp) and quality filters (toxicity, PII, dedup)
• Run generation flows (e.g., RAG evaluation, Knowledge QA, MCP distillation)
• Export datasets to JSONL, Parquet, or Hugging Face Dataset
• Integrate with Bob and Claude AI assistants via plugins

证据

SDG Hub is an open‑source library providing composable blocks and flows for synthetic data generation.

来源

The resulting application leverages SDG Hub under the hood while extending its capabilities to match my project’s specific requirements, context, and environment.

来源