Thirty years of confident wrong answers

Description and use

A database on a private cloud slowed every Monday due to storage tiering moving blocks to SATA. Pinning VMs to Fibre Channel resolved the issue.

Problem

A production database slowed to a crawl every Monday morning after a two‑and‑a‑half‑year deployment on a new private cloud, with no clear root cause from logs or instrumentation.

Outcome

Pinning database VMs to a Fibre Channel storage tier eliminated the Monday morning slowdown, improving performance by two orders of magnitude.

Workflow

• Reproduce the error in a clean test harness
• Load‑test the database in a controlled environment
• Observe that the slowdown still occurs
• Investigate infrastructure layers: hypervisor, virtual storage drivers, tiered storage platform, storage network
• Identify increased access latencies in the storage layer
• Determine that the smart storage policy is moving blocks to SATA
• Pin database VMs to Fibre Channel tier
• Confirm performance improvement

Evidence

Our newly deployed production database would slow to a crawl every Monday morning.

Source

We kept testing all week long. Friday came, I stopped my test, and on Monday I flew back to the customer. Sitting at my desk at the customer, I started the load test and hit the same problem again.

Source

We noticed the access latencies. On Monday they started super slow. By Tuesday performance had improved by an order of magnitude, and by Friday it had jumped two orders of magnitude from the first measurement.

Source

We fixed it by simply pinning every database VM to a Fibre Channel storage tier.

Source