In this short intro, we’ll go through the steps required to set up push notifications, emails and automated phone calls related to validator uptime, voting power and network statistics. The software to be used is Simply Staking’s self-hosted and open-source alerting tool, PANIC.
PANIC ground control
Situations alerted:
- Missing blocks
- Validator voting power changes
- Validator jailing
- P2P peer fluctuations
- Out of sync nodes
- Unreachable nodes
- New Github releases
Notification channels:
- Logging — alerts logged to an alerts log
- Telegram alerts — alerts delivered to a Telegram chat via a Telegram bot
- Automated phone calls — alerts trigger a phone call to grab the node operator’s attention
- Email notifications — alerts sent by email using a personal SMTP server
The source documentation for this article can be found here: Installing and Running P.A.N.I.C.
We’ve previously published an overview of the monitoring options for Cosmos node operators: Monitoring and Alerting for Validators.
Requirements
We’ll be going over the installation steps for the following requirements below:
- Ubuntu 22.04 system
- Python3
- PIP Package Manager
- Pipenv Packaging Tool
- Optional: Redis Database Server
- Optional: Telegram bot
- Optional: Twilio account
Setting up the dependencies
- Python3
Ubuntu 22.04 comes with Python 3.10 by default. You should be able to check Check if Python 3 is installed by running
$ python3 --version
- Installing PIP Package Manager, Pipenv Packaging Tool and Redis Database Server
# sudo apt-get install python3-pip redis-server -y
# sudo pip3 install pipenv
# sudo systemctl enable redis-server.service
- Configuring Redis
- Access the redis.conf file:
# nano /etc/redis/redis.conf
In the GENERAL
section, add the line bind 127.0.0.1 ::1
to bind Redis only to localhost, if it is not yet already added. This assumes that Redis is installed on the same system that will run the alerter.
In the SECURITY
section, add (or uncomment) the line requirepass <PASS>
, where <PASS>
is a complex password of your choosing. The alerter will ask for this password during setup.
requirepass password123
In the SECURITY
section, disable some unused commands to prevent some actions from taking place. For each command, add a line: rename-command <COMMAND> “”
.
rename-command FLUSHALL ""
rename-command PEXPIRE ""
rename-command CONFIG ""
rename-command SHUTDOWN ""
rename-command BGREWRITEAOF ""
rename-command BGSAVE ""
rename-command SAVE ""
rename-command SPOP ""
rename-command SREM ""
rename-command RENAME ""
rename-command DEBUG ""
2. Restart Redis to apply changes:
# systemctl restart redis-server
- Creating a Telegram Bot
Add BotFather on Telegram, press Start and follow the below steps:
1. Send a /newbot
command and fill in the requested details, including a bot name and username.
2. Take a note of the API token, which looks something like 123456:ABC-DEF1234ghIkl-zyx56789g1u163eh41
3. Access the link t.me/<username>
to your new bot given by BotFather and press Start.
4. Send a /start
message to the bot and then any second message.
5. Access the link api.telegram.org/bot<token>/getUpdates
, replacing <token>
with the bot’s API token. This gives a list of the bot’s activity, including messages sent to the bot.
6. The result section should contain at least one message, due to us sending a message. Take a note of the “id”
number in the “chat”
section of this message.
Note: A group chat can also be retrieved by adding the bot to the chat and finding this chat’s ID through the same Telegram API method. Group chat IDs start with a negative sign (-).
- Setting up a Twilio account
- To create a free trial Twilio account, head to the try-twilio page and sign up using your details.
- Visit your account dashboard.
- Click the ‘Get a Trial Number’ button to generate a unique number.
- Take note of the (i) Twilio phone number.
- Take note of the (ii) account SID and (iii) auth token.
- Navigate to the Verified Called IDs page.
- Press the red + to add a new personal number and follow the verification steps.
Installing PANIC
1. Create a new user to run the alerter as:
# adduser panic_alerter
2. Create a directory that will have the PANIC alerter cloned into it:
# mkdir /opt/panic_alerter
3. Since this is now owned by the root user, set it to be owned by the new panic_alerter
user, then switch to that user:
# chown -R panic_alerter:panic_alerter /opt/panic_alerter
# su panic_alerter
4. Now logged in as the panic_alerter
user, clone the PANIC Github repository:
$ cd /opt/panic_alerter/ && git clone https://github.com/SimplyStaking/initia_panic.git && cd panic && git checkout main
5. Update the project’s packages and run the alerter setup:
$ pipenv update
$ pipenv run python run_setup.py
Initialising the PANIC alerter setup
From here onward, follow the installation wizard and input the API tokens, node info, etc. to configure the different notification channels and the nodes which are to be monitored.
Starting PANIC as a Service
Running the alerter as an enabled service means that it starts up automatically on boot and restarts automatically if it runs into some issue and stops running.
The service file is created as follows:
# nano /etc/systemd/system/panic_alerter.service
Entering the service definition:
[Unit]
Description=P.A.N.I.C.
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
User=panic_alerter
TimeoutStopSec=90s
WorkingDirectory=/opt/panic_alerter/panic
ExecStart=/usr/local/bin/pipenv run python /opt/panic_alerter/panic/run_alerter.py
[Install]
WantedBy=multi-user.target
Lastly, we will enable and start the alerter service:
# sudo systemctl enable panic_alerter.service
# sudo systemctl start panic_alerter.service
Check out systemctl status panic_alerter.service
or the logs in /opt/panic_alerter/panic/logs
to confirm that the alerter is running. Alternatively, if you set up Telegram, try interacting with the Telegram bot (using /help
).
Changing configuration
Configuration options are stored in .ini files in the config directory of panic. Here you may find:
- user_config_main.ini for the notification channel setup
- user_config_nodes.ini for nodes to be monitored
- user_config_repos.in for GitHub repos to be monitored (only if this feature is enabled)
- Website: https://simplystaking.com
- Twitter: https://twitter.com/simplystaking
- Telegram: https://t.me/SimplyStaking