Thirty years of confident wrong answers

Description and use

A database on a private cloud slowed every Monday due to a storage tiering policy moving blocks to SATA. Pinning VMs to Fibre Channel resolved the issue.

Problem

A production database on a private cloud slowed to a crawl every Monday, with no clear root cause from logs or instrumentation.

Outcome

By pinning database VMs to Fibre Channel storage, the Monday slowdowns were eliminated, restoring normal performance and allowing the production rollout to proceed.

Workflow

• Deploy database on private cloud
• Observe Monday slowdowns
• Read database and system logs
• Instrument database VMs
• Reproduce slowdown in clean test harness
• Test load on clean environment
• Identify no root cause in database or VM layers
• Consult infrastructure team and vendor architect
• Instrument hypervisor, virtual storage drivers, tiered storage platform, storage network
• Measure access latencies
• Observe latency spike on Monday and improvement by Friday
• Determine storage tiering policy as cause
• Pin database VMs to Fibre Channel tier
• Resolve slowdown issue

Evidence

Our newly deployed production database would slow to a crawl every Monday morning.

Source

We fixed it by simply pinning every database VM to a Fibre Channel storage tier.

Source

The storage tiering policy would relegate lightly used blocks to SATA storage. As blocks got hot, it would move them to Fibre Channel storage, and as usage grew further, to NVMe. Over the weekend, when no tests were running, it moved them back to SATA.

Source