I Added Prometheus to My Go Job Scraper and It Changed How I Think About Production

Description and use

Adding Prometheus to a Go scraper provided observability, tracking job counts, duration, and errors, turning the project into a production-ready service.

Problem

Running a Go scraper in production without observability makes it invisible and hard to detect failures or performance issues.

Outcome

Adding Prometheus metrics and a health endpoint transformed the scraper into a production-ready service with observability, enabling early detection of failures and performance monitoring.

Workflow

• Create metrics/metrics.go with Prometheus counters and histograms
• Expose /metrics and /healthz endpoints in main.go using promhttp.Handler()
• Instrument scraping logic to record job counts, status, and duration
• Install Prometheus client library (go get github.com/prometheus/[email protected])
• Run the application and verify metrics via curl http://localhost:2112/metrics
• Configure Prometheus scrape_configs to target the scraper
• Create Grafana dashboards using the exposed metrics

Evidence

Without metrics, you are blind. You don't know: How many jobs you scraped? Is it failing silently? How long does it take?

Source

Step A: Create metrics/metrics.go ... JobsScraped = promauto.NewCounterVec(...

Source

Step B: Expose /metrics and /healthz in main.go ... http.Handle("/metrics", promhttp.Handler())

Source

Now you can create Grafana dashboards with jobs_scraped_total and scrape_duration_seconds.

Source