Eight months in production and no automation had ever touched it

说明与用途

一个定价服务在8个月内没有自动化：缺少监控、日志、备份和镜像更新。通过添加元数据、自动检查和定期任务，提升了可靠性。

问题

A pricing lookup service went down for 14 trading hours with no alerts, due to missing monitoring, logging, backups, and outdated base images.

结果

The service now has automated deployment, monitoring, logging, backups, and a registry that ensures all operational aspects are tracked and compliant.

工作流

• Add a service file with metadata to the repo
• Configure build to reject artifacts without a valid service file
• Update platform rolling jobs to use registry from service files
• Set up a weekly job to compare running services against registry
• Add monitoring and alerting for memory usage
• Ship logs to a central log aggregator
• Schedule regular backups for the database

证据

The outage itself was dull: the process had run out of memory and the container had not restarted because nothing was watching it.

来源

Its logs were not shipped anywhere, so for the whole incident window we had nothing to read.

来源