Eight months in production and no automation had ever touched it

Description and use

A pricing service ran 8 months without automation: missing monitoring, logging, backups, and image updates. After adding metadata, automated checks, and regular jobs, it became resilient.

Problem

A pricing lookup service went down for 14 trading hours with no alerts, due to missing monitoring, logging, backups, and outdated base images.

Outcome

The service now has automated deployment, monitoring, logging, backups, and a registry that ensures all operational aspects are tracked and compliant.

Workflow

• Add a service file with metadata to the repo
• Configure build to reject artifacts without a valid service file
• Update platform rolling jobs to use registry from service files
• Set up a weekly job to compare running services against registry
• Add monitoring and alerting for memory usage
• Ship logs to a central log aggregator
• Schedule regular backups for the database

Evidence

The outage itself was dull: the process had run out of memory and the container had not restarted because nothing was watching it.

Source

Its logs were not shipped anywhere, so for the whole incident window we had nothing to read.

Source