Eight months in production and no automation had ever touched it

Описание и применение

Сервис ценообразования работал 8 месяцев без автоматизации: отсутствовали мониторинг, логирование, резервные копии и обновления образов. После внедрения метаданных, автоматических проверок и регулярных задач сервис стал устойчивым.

Задача

A pricing lookup service went down for 14 trading hours with no alerts, due to missing monitoring, logging, backups, and outdated base images.

Результат

The service now has automated deployment, monitoring, logging, backups, and a registry that ensures all operational aspects are tracked and compliant.

Рабочий процесс

• Add a service file with metadata to the repo
• Configure build to reject artifacts without a valid service file
• Update platform rolling jobs to use registry from service files
• Set up a weekly job to compare running services against registry
• Add monitoring and alerting for memory usage
• Ship logs to a central log aggregator
• Schedule regular backups for the database

Доказательства

The outage itself was dull: the process had run out of memory and the container had not restarted because nothing was watching it.

Источник

Its logs were not shipped anywhere, so for the whole incident window we had nothing to read.

Источник