Alerting on infrastructure you built yourself

Setting up Prometheus and Grafana is a weekend project. Deciding on alert thresholds that don't lie to you is the part that takes longer to get right. Early on, I alerted on everything I could measure — CPU, memory, disk, request latency — and within days I had alert fatigue from noise: spikes that resolved themselves in seconds, disk usage warnings during routine log rotation, nothing actionable. I moved to a tiered model instead. Tier one is nginx and disk space, because those fail loudly and affect everything downstream. Tier two is the NestJS backend — error rate and p95 latency, tracked over a rolling window rather than instant thresholds, so a single slow request doesn't trigger a false alarm. Tier three is PostgreSQL — connection pool saturation and slow query counts, which matter but rarely need an immediate response. Loki and Promtail tie it together by letting me jump from a metric spike straight to the structured logs from that time window, instead of grepping through files manually. The real shift was mental, not technical: an alert should mean 'something needs a human,' not 'something happened.' Most of the tuning work was quietly deleting alerts I had originally been proud of setting up.