Reproducible CSV catalog QA with Python: validation, traceability, and safe exports

Описание и применение

Рецепт автоматизации QA CSV‑каталогов на Python обеспечивает воспроизводимость, трассируемость и безопасные экспорты, устраняя типичные ошибки в заголовках, ценах и дубликатах SKU.

Задача

Small catalog CSV files often fail in production due to header changes, text prices, duplicate SKUs, or encoding issues, causing import errors into e‑commerce or ERP systems.

Результат

The recipe delivers a repeatable, traceable CSV QA process that catches common data issues before they reach production, reducing import failures and enabling clear audit trails.

Рабочий процесс

• 1. Preserve the original file and compute a SHA‑256 hash before any transformation.
• 2. Define a small internal contract (sku, title, optional description, category, price) and create a mapping YAML file to translate source columns to contract fields.
• 3. Normalize deterministic details: strip whitespace, enforce UTF‑8‑sig encoding, and keep operations idempotent.
• 4. Detect and report ambiguous duplicates by returning them in an errors file with row numbers and conflicting values.
• 5. Validate keys and values: check for missing SKUs, duplicate SKUs, and warn on negative prices; record row counts, preserved key counts, mapping version, and hashes.
• 6. Produce a handoff package: normalized CSV, HTML/Markdown report, machine‑readable errors file, and a change log linking output rows to source rows.
• 7. Run the pipeline in CI with fail‑closed checks: reject exports if required columns disappear, duplicate keys increase, or output count mismatches accepted input; support dry‑run mode and bind approvals to input hash, mapping version, and destination.

Доказательства

Start by treating the source file as evidence. Keep the original column names and compute a hash before transforming anything.

Источник

A mapping file makes the transformation reviewable instead of hiding assumptions in code.

Источник

Do not silently merge ambiguous duplicates. Return them in an errors file with the row numbers and the conflicting values so a client can decide.

Источник

Run the same checks in CI and reject the export when required columns disappear, duplicate keys increase, or the output count does not match the accepted input set.

Источник