Monitoring your validator
Tail your log file
Watch the latest entries in your log file for errors. It moves fast...so may need to stop the tail and read before restarting. Here are several ways to run it.
When the validator is first starting up, it asks the entrypoint in the start script "what IP address do you see me at?" Then it proceeds to ask the entrypoint if the required ports are open for it to use. This can be searched for during the startup sequence using the last tail option below that will grep 10 lines before and 50 after the search term to see if you are reachable. There is a ton of good connection debugging info in this search. See screenshot below.
tail -f ~/xand-validator.logtail -f ~/xand-validator.log | grep "search terms"This will restart your validator then grep the log file for initial connection info:
sudo systemctl restart xand.service &&
tail -f ~/xand-validator.log | grep --color=always -B 10 -A 50 "Checking that tcp ports"
mon.sh
Create a monitor script to easily run the monitor command
Run as xand user
Create a blank file named mon.sh in your home dir with editor.
cd ~vim mon.shCopy the code block into the file, correcting your ledger path if needed. Save and exit.
solana-validator --ledger ~/ledger monitorMake the file executable
chmod +x mon.shRun the monitor from home dir
./mon.shNOTE: Press enter to drop a line to compare old values and press ctrl+c to exit the monitor command.
catchup.sh
Create a catchup script that compares your machine to the RPC that you are connected to.
Create a blank file named catchup.sh in your home dir with editor.
cd ~vim catchup.shCopy the code block into the file. Save and exit.
Note: if using the ALT method for catchup because localhost is not working for you...be sure to grab your validator ID pubkey using solana-keygen pubkey ~/validator-keypair.json
solana catchup -k ~/validator-keypair.json --our-localhost --follow --verboseMake the file executable
chmod +x catchup.shRun the monitor from home dir
./catchup.shNOTE: Press enter to drop a line to compare old values and press ctrl+c to exit the catchup command.
Watchtower
NOTE: solana-watchtower is an optional monitoring system running on a separate computer that will alert you in your own personal discord. Setup is required that is not shown in this guide.
Run as xand user without sudo
Watchtower should be ran from a remote computer that is running 24/7. It works by asking the RPC node if your Validator passes all the sanity checks. It can be added as a service or ran in a tmux window that never closes. You will need the software compiled to the point that solana -V works after a reboot. You will need to create a Discord or Slack channel with a webhook to make this work. Telegram, PagerDudy, and Twilio are also supported. This example script checks every <interval> seconds and alerts to Discord and Slack if <unhealthy-threshold> number of failures show in a row...ie 900 seconds
Multiple scripts can be running with different --validator-identity and pumped into the same alert channel and use the --name-suffix to uniquely identify which machine is failing.
NOTE: If our RPC node goes down or is unreachable from your location...you will get false positives that your machine is down...this can be added if desired:
--ignore-http-bad-gateway
Ignore HTTP 502 Bad Gateway errors from the JSON RPC URL. This flag can help reduce false positives, at the expense of no alerting should a Bad Gateway error be a side effect of the real problem
vim watchtower-alerts.shAdd text to file and modify for your needs:
export DISCORD_WEBHOOK=https://discord.com/api/webhooks/xxxxxxx/yyyyyyyyyy
export SLACK_WEBHOOK=https://hooks.slack.com/services/xxxxxxxxx/yyyyyyyy/zzzzzz
#! /bin/sh
exec solana-watchtower \
--url http://xand-rpc2.devnet.xandeum.com:8899/ \
--validator-identity <Validator ID> \
--name-suffix ::<hostname>:: \
--interval 300 \
--unhealthy-threshold 3 \
--minimum-validator-identity-balance 3 \
chmod +x watchtower-alerts.shConsider next Zabbix Installation