author | Matthew Wild <mwild1@gmail.com> |
Wed, 13 Jul 2022 11:15:43 +0100 | |
changeset 5000 | 031e0dd90f4b |
parent 4628 | eba7e68120d8 |
child 5489 | 67190744b1eb |
permissions | -rw-r--r-- |
4626
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
1 |
--- |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
2 |
labels: |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
3 |
- 'Stage-Alpha' |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
4 |
summary: Alertmanager webhook receiver for pubsub |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
5 |
--- |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
6 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
7 |
# Introduction |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
8 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
9 |
This module lets |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
10 |
[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
11 |
publish alerts to [pubsub][doc:pubsub] via |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
12 |
[webhooks](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config). |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
13 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
14 |
# Setup |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
15 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
16 |
The relevant pubsub nodes must be created and configured somehow. |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
17 |
Because the request IP address is used to publish, the `publisher` |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
18 |
affiliation should be given to the IP address Alertmanager sends |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
19 |
webhooks from. |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
20 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
21 |
# Configuration |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
22 |
|
4627
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
23 |
## Prometheus |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
24 |
|
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
25 |
A Prometheus `rule_files` might contain something along these lines: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
26 |
|
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
27 |
``` yaml |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
28 |
groups: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
29 |
- name: Stuff |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
30 |
rules: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
31 |
- alert: Down |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
32 |
expr: up == 0 |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
33 |
for: 5m |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
34 |
annotations: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
35 |
title: 'Stuff is down!' |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
36 |
labels: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
37 |
severity: 'critical' |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
38 |
``` |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
39 |
|
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
40 |
## Alertmanager |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
41 |
On the Alertmanager site the webhook configuration may look something |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
42 |
like this: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
43 |
|
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
44 |
``` yaml |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
45 |
receivers: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
46 |
- name: pubsub |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
47 |
webhook_configs: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
48 |
- url: http://pubsub.localhost:5280/pubsub_alertmanager |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
49 |
``` |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
50 |
|
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
51 |
And then finally some Alertmanager routes would point at that receiver: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
52 |
|
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
53 |
``` yaml |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
54 |
route: |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
55 |
receiver: pubsub |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
56 |
``` |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
57 |
|
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
58 |
## Prosody |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
59 |
|
4626
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
60 |
On the Prosody side, apart from creating and configuring the node(s) |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
61 |
that will be used, configure your pubsub service like this: |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
62 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
63 |
``` lua |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
64 |
Component "pubsub.example.com" "pubsub" |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
65 |
modules_enabled = { |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
66 |
"pubsub_alertmanager", |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
67 |
} |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
68 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
69 |
-- optional extra settings: |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
70 |
alertmanager_body_template = [[ |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
71 |
*ALARM!* {annotations.title?Alert} is {status} |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
72 |
Since {startsAt}{endsAt& until {endsAt}} |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
73 |
Labels: {labels% |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
74 |
{idx}: {item}} |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
75 |
Annotations: {annotations% |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
76 |
{idx}: {item}} |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
77 |
]] |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
78 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
79 |
alertmanager_node_template = "alerts/{alert.labels.severity}" |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
80 |
``` |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
81 |
|
4628
eba7e68120d8
mod_pubsub_alertmanager: Add some words about the node template
Kim Alvefur <zash@zash.se>
parents:
4627
diff
changeset
|
82 |
If no node template is given, either an optional part after |
eba7e68120d8
mod_pubsub_alertmanager: Add some words about the node template
Kim Alvefur <zash@zash.se>
parents:
4627
diff
changeset
|
83 |
"pubsub_alertmanager" in the HTTP path is used as node, or the string |
eba7e68120d8
mod_pubsub_alertmanager: Add some words about the node template
Kim Alvefur <zash@zash.se>
parents:
4627
diff
changeset
|
84 |
"alerts". Here, an alerts would be published to different nodes based on |
eba7e68120d8
mod_pubsub_alertmanager: Add some words about the node template
Kim Alvefur <zash@zash.se>
parents:
4627
diff
changeset
|
85 |
the 'severity' label, so e.g. `alerts/critical` in this example. |
eba7e68120d8
mod_pubsub_alertmanager: Add some words about the node template
Kim Alvefur <zash@zash.se>
parents:
4627
diff
changeset
|
86 |
|
4627
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
87 |
## All Options |
622c6308d7af
mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents:
4626
diff
changeset
|
88 |
|
4626
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
89 |
Available configuration options: |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
90 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
91 |
`alertmanager_body_template` |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
92 |
: Template for the textual representation of alerts. |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
93 |
|
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
94 |
`alertmanager_node_template` |
61ce3394fe8b
mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff
changeset
|
95 |
: Template for the pubsub node name, defaults to `"{path?alerts}"` |