Reusable Actions For Alert
This example shows you how to check both database prod1
, prod2
and determines if any of them hit 90%
threshold of disk space then using an action template to send notification to two telegram channels.
Here is the full configurations:
# Check both databases prod1 and prod2 and alert if any of them has less than 10% disk space left.
alert:
id: db-disk-space
description: Check that the DB has enough disk space
steps:
- name: db-prod1-no-space
provider:
type: mock
config: "{{ providers.db-server-mock }}"
with:
command: df -h | grep /dev/disk3s1s1 | awk '{ print $5}' # Check the disk space
command_output: 91% # Mock
- name: db-prod2-no-space
provider:
type: mock
config: "{{ providers.db-server-mock }}"
with:
command: df -h | grep /dev/disk3s1s1 | awk '{ print $5}' # Check the disk space
command_output: 94.5% # Mock
actions:
- name: trigger-telegram1
use: @trigger-telegrame
provider:
config:
authentication:
bot_token: "{{ env.TELEGRAM_BOT_TOKEN1 }}"
with:
chat_id: "{{ env.TELEGRAM_CHAT_ID1 }}"
- name: trigger-telegram2
use: @trigger-telegrame
provider:
config:
authentication:
bot_token: "{{ env.TELEGRAM_BOT_TOKEN2 }}"
with:
chat_id: "{{ env.TELEGRAM_CHAT_ID2 }}"
actions:
- name: trigger-telegram
use: @trigger-telegram
condition:
- type: threshold
value: "{{ steps.db-prod1-no-space.results }}"
compare_to: 90% # Trigger if more than 90% full
alias: A
- type: threshold
value: "{{ steps.db-prod2-no-space.results }}"
compare_to: 90% # Trigger if more than 90% full
alias: B
# trigger the action if any of the conditions are met:
if: "{{ A }} or {{ B }}"
provider:
type: telegram
with:
message: Keep Alert Test
providers:
db-server-mock:
description: Paper DB Server
authentication:
Breakdown
Steps
In this example we can see two steps:
- db-prod1-no-space - checks db space of db prod1
- db-prod2-no-space - checkd db space of db prod2
Conditions
The action has two threshold conditions:
condition:
- type: threshold
value: "{{ steps.this.results }}"
compare_to: 90% # Trigger if more than 90% full
But now we’ve added an alias
to each condition, so it’ll be easier to check it in the action
itself.
Action
The action template is defined as.
actions:
- name: trigger-telegram
use: @trigger-telegram
condition:
- type: threshold
value: "{{ steps.db-prod1-no-space.results }}"
compare_to: 90% # Trigger if more than 90% full
alias: A
- type: threshold
value: "{{ steps.db-prod2-no-space.results }}"
compare_to: 90% # Trigger if more than 90% full
alias: B
# trigger the action if any of the conditions are met:
if: "{{ A }} or {{ B }}"
provider:
type: telegram
with:
message: Keep Alert Test
The action uses the if
statement to alert if one of the databases has less than 10% disk space left.
Note that we don’t define any telegram chat_id
and bot_token
here because we want to define two separate telegram credentials for the two channels.
The credentials are defined in actions
definitions within the alert
configuration.
Note that we declare use: @trigger-telegram
to use the defined action template.
alert:
...
actions:
- name: trigger-telegram1
use: @trigger-telegram
provider:
config:
authentication:
bot_token: "{{ env.TELEGRAM_BOT_TOKEN1 }}"
with:
chat_id: "{{ env.TELEGRAM_CHAT_ID1 }}"
- name: trigger-telegram2
use: @trigger-telegram
provider:
config:
authentication:
bot_token: "{{ env.TELEGRAM_BOT_TOKEN2 }}"
with:
chat_id: "{{ env.TELEGRAM_CHAT_ID2 }}"