Thirty years of confident wrong answers

Описание и применение

База данных, развернутая на внутреннем облаке, замедлялась каждое понедельник. Проблема была связана с политикой хранения, которая перемещала блоки на SATA. Переназначение всех VM на Fibre Channel устранило задержки.

Задача

A production database slowed to a crawl every Monday morning after a two‑and‑a‑half‑year deployment on a new private cloud, with no clear root cause from logs or instrumentation.

Результат

Pinning database VMs to a Fibre Channel storage tier eliminated the Monday morning slowdown, improving performance by two orders of magnitude.

Рабочий процесс

• Reproduce the error in a clean test harness
• Load‑test the database in a controlled environment
• Observe that the slowdown still occurs
• Investigate infrastructure layers: hypervisor, virtual storage drivers, tiered storage platform, storage network
• Identify increased access latencies in the storage layer
• Determine that the smart storage policy is moving blocks to SATA
• Pin database VMs to Fibre Channel tier
• Confirm performance improvement

Доказательства

Our newly deployed production database would slow to a crawl every Monday morning.

Источник

We kept testing all week long. Friday came, I stopped my test, and on Monday I flew back to the customer. Sitting at my desk at the customer, I started the load test and hit the same problem again.

Источник

We noticed the access latencies. On Monday they started super slow. By Tuesday performance had improved by an order of magnitude, and by Friday it had jumped two orders of magnitude from the first measurement.

Источник

We fixed it by simply pinning every database VM to a Fibre Channel storage tier.

Источник