Thirty years of confident wrong answers

说明与用途

在私有云上部署的数据库每周一都会变慢，原因是存储分层将块移动到 SATA。将所有 VM 固定到 Fibre Channel 解决了问题。

问题

A production database slowed to a crawl every Monday morning after a two‑and‑a‑half‑year deployment on a new private cloud, with no clear root cause from logs or instrumentation.

结果

Pinning database VMs to a Fibre Channel storage tier eliminated the Monday morning slowdown, improving performance by two orders of magnitude.

工作流

• Reproduce the error in a clean test harness
• Load‑test the database in a controlled environment
• Observe that the slowdown still occurs
• Investigate infrastructure layers: hypervisor, virtual storage drivers, tiered storage platform, storage network
• Identify increased access latencies in the storage layer
• Determine that the smart storage policy is moving blocks to SATA
• Pin database VMs to Fibre Channel tier
• Confirm performance improvement

证据

Our newly deployed production database would slow to a crawl every Monday morning.

来源

We kept testing all week long. Friday came, I stopped my test, and on Monday I flew back to the customer. Sitting at my desk at the customer, I started the load test and hit the same problem again.

来源

We noticed the access latencies. On Monday they started super slow. By Tuesday performance had improved by an order of magnitude, and by Friday it had jumped two orders of magnitude from the first measurement.

来源

We fixed it by simply pinning every database VM to a Fibre Channel storage tier.

来源