Thirty years of confident wrong answers

说明与用途

在私有云上部署的数据库每周一都会因存储分层策略将块移动到 SATA 而变慢。将 VM 固定到 Fibre Channel 解决了问题。

问题

A production database on a private cloud slowed to a crawl every Monday, with no clear root cause from logs or instrumentation.

结果

By pinning database VMs to Fibre Channel storage, the Monday slowdowns were eliminated, restoring normal performance and allowing the production rollout to proceed.

工作流

• Deploy database on private cloud
• Observe Monday slowdowns
• Read database and system logs
• Instrument database VMs
• Reproduce slowdown in clean test harness
• Test load on clean environment
• Identify no root cause in database or VM layers
• Consult infrastructure team and vendor architect
• Instrument hypervisor, virtual storage drivers, tiered storage platform, storage network
• Measure access latencies
• Observe latency spike on Monday and improvement by Friday
• Determine storage tiering policy as cause
• Pin database VMs to Fibre Channel tier
• Resolve slowdown issue

证据

Our newly deployed production database would slow to a crawl every Monday morning.

来源

We fixed it by simply pinning every database VM to a Fibre Channel storage tier.

来源

The storage tiering policy would relegate lightly used blocks to SATA storage. As blocks got hot, it would move them to Fibre Channel storage, and as usage grew further, to NVMe. Over the weekend, when no tests were running, it moved them back to SATA.

来源