Reproducible CSV catalog QA with Python: validation, traceability, and safe exports

说明与用途

使用Python的可复现CSV目录QA自动化配方，规范化、验证并导出产品数据，生成详细报告并实现CI fail‑closed 检查。

问题

Small catalog CSV files often fail in production due to header changes, text prices, duplicate SKUs, or encoding issues, causing import errors into e‑commerce or ERP systems.

结果

The recipe delivers a repeatable, traceable CSV QA process that catches common data issues before they reach production, reducing import failures and enabling clear audit trails.

工作流

• 1. Preserve the original file and compute a SHA‑256 hash before any transformation.
• 2. Define a small internal contract (sku, title, optional description, category, price) and create a mapping YAML file to translate source columns to contract fields.
• 3. Normalize deterministic details: strip whitespace, enforce UTF‑8‑sig encoding, and keep operations idempotent.
• 4. Detect and report ambiguous duplicates by returning them in an errors file with row numbers and conflicting values.
• 5. Validate keys and values: check for missing SKUs, duplicate SKUs, and warn on negative prices; record row counts, preserved key counts, mapping version, and hashes.
• 6. Produce a handoff package: normalized CSV, HTML/Markdown report, machine‑readable errors file, and a change log linking output rows to source rows.
• 7. Run the pipeline in CI with fail‑closed checks: reject exports if required columns disappear, duplicate keys increase, or output count mismatches accepted input; support dry‑run mode and bind approvals to input hash, mapping version, and destination.

证据

Start by treating the source file as evidence. Keep the original column names and compute a hash before transforming anything.

来源

A mapping file makes the transformation reviewable instead of hiding assumptions in code.

来源

Do not silently merge ambiguous duplicates. Return them in an errors file with the row numbers and the conflicting values so a client can decide.

来源

Run the same checks in CI and reject the export when required columns disappear, duplicate keys increase, or the output count does not match the accepted input set.

来源