Why you shouldn’t notify on warnings

routes:
- match_re:
alertname: QuietHours
receiver: blackhole
- match_re:
severity: critical
receiver: onduty
System health dashboard
sum(sum_over_time(ALERTS{service="policy",alertstate="firing"}[$interval])) or on () vector(0)
- alert: HighGoRoutinesCountAcmeService
expr: sum (go_goroutines{service="acme"}) > 1000
for: 2m
labels:
service: acme
severity: warning
team: core
- alert: TooManyWarnings
expr: count_over_time(sum by(service, alertname, triggered_alertname) (label_replace(ALERTS{alertstate="firing",severity="warning"}, "triggered_alertname", "$1", "alertname", "(.*)"))[1h:5m]) > 8
labels:
onduty: working-hours-only
severity: critical
inhibit_rules:
- source_match:
alertname: QuietHours
target_match:
onduty: working-hours-only
- alert: QuietHours
#hour() is in UTC
expr: day_of_week() == 6 or day_of_week() == 7 or hour() >= 15 or hour() <= 6
for: 1m
labels:
severity: info

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store