Your Demo Dashboard Is Not a Funnel: Auditing Synthetic Events in ClickHouse

Description and use

SensorFlow’s demo dashboard displays a funnel that appears to show conversion, but it is built from synthetic data, so the metrics do not reflect real user behavior. Inspecting raw ClickHouse rows and user overlap reveals no real conversions. Proper funnel measurement requires identity rules, event ordering, time windows, and filtering out synthetic traffic.

Problem

A demo analytics stack can display a funnel chart that appears to show conversion, but the data is synthetic and does not represent real user behavior, leading to misleading conversion metrics.

Outcome

The audit demonstrates that synthetic demo dashboards can mislead stakeholders into thinking conversion rates are real. By inspecting raw data, checking user overlap, and applying proper funnel definitions, teams can avoid false metrics and ensure dashboards reflect actual user behavior.

Workflow

• 1. Read the demo or seed‑data generator to understand its synthetic nature.
• 2. Query raw rows and distinct IDs for each event stage using ClickHouse SQL.
• 3. Verify overlap between stages before calculating any conversion percentage.
• 4. Add ordered‑event, time‑window, identity, and deduplication rules to the real‑event test.
• 5. Match Superset dashboard filters to the SQL query and exclude synthetic traffic from production reports.

Evidence

A newly installed analytics stack can show a convincing sequence of "app open → product view → add to cart → purchase" before it has ingested a single real user event.

Source

The current demo SQL inserts rows from numbers(240). It selects the event name with number % 12, and the demo user with number % 48. Because 48 is divisible by 12, each demo-user-* stays in one event category throughout this generated batch.

Source

Recreating the published generator's 240 rows with ClickHouse Local produced 16 / 4 / 0: sixteen viewers, four buyers, and zero IDs in both sets.

Source

First, choose a stable identity rule. Decide whether distinct_id represents a logged-in account, an anonymous browser ID, or a merged identity.

Source