I deleted my own optimization because the benchmark said it was 2.1x slower

Description and use

Optimizing pandas with a hand‑written NumPy kernel was slower and used ~200× more memory, so it was removed.

Problem

Optimizing pandas column statistics with a hand‑written NumPy kernel increased runtime and memory usage.

Outcome

The hand‑written NumPy kernel was slower and used far more memory than the original pandas code; the optimization was removed, and the library now relies on pandas for speed and memory efficiency.

Workflow

• 1. Implemented a NumPy kernel that converted numeric columns to a contiguous 2‑D array and computed moments in one pass.
• 2. Benchmarked the kernel against the original pandas implementation using realistic frame shapes and multiple runs.
• 3. Observed that the kernel was slower (799 ms vs 557 ms) and used far more memory (298 MiB vs 1.6 MiB).
• 4. Identified that the copy operation in .to_numpy() was the bottleneck.
• 5. Removed the branch that chose between per‑column and frame‑level strategies.
• 6. Reverted to the pandas implementation, which was fastest and smallest.
• 7. Documented the benchmark results and the decision to delete the optimization in docs/performance.md.
• 8. Updated the contributing guide to require benchmarks and willingness to delete regressions.

Evidence

The hand‑written NumPy block kernel was slower (799 ms) and used 298 MiB versus 557 ms and 1.6 MiB for the pandas code.

Source

pandas never pays that. It computes each column's moments against the block that already exists, in place, and allocates almost nothing.

Source