solana-ruby-watchtower 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +14 -0
- data/LICENSE +21 -0
- data/README.md +181 -0
- data/examples/solana-watchtower.service +28 -0
- data/examples/watchtower.yml +56 -0
- data/exe/solana-watchtower +9 -0
- data/lib/solana/ruby/watchtower/check_state.rb +98 -0
- data/lib/solana/ruby/watchtower/cli.rb +133 -0
- data/lib/solana/ruby/watchtower/config.rb +134 -0
- data/lib/solana/ruby/watchtower/heartbeat.rb +27 -0
- data/lib/solana/ruby/watchtower/monitor.rb +189 -0
- data/lib/solana/ruby/watchtower/notifiers.rb +100 -0
- data/lib/solana/ruby/watchtower/observe.rb +46 -0
- data/lib/solana/ruby/watchtower/rpc.rb +111 -0
- data/lib/solana/ruby/watchtower/runner.rb +51 -0
- data/lib/solana/ruby/watchtower/stores/active_record_store.rb +35 -0
- data/lib/solana/ruby/watchtower/stores/file_store.rb +56 -0
- data/lib/solana/ruby/watchtower/stores/memory_store.rb +29 -0
- data/lib/solana/ruby/watchtower/version.rb +9 -0
- data/lib/solana/ruby/watchtower.rb +34 -0
- data/solana-ruby-watchtower.gemspec +43 -0
- metadata +127 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 4753642b3fac09fb773914fdad127effd85850119172f75d4febf17b024d9f65
|
|
4
|
+
data.tar.gz: 127f3df754c1fedaea7f47d526b1ff0307be60a44d76a854ece8214d66a0b8d5
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 14c9d556c637573e16588ff8c0c211ff79a162dd1fc7399c4d24e101648c684f07fc025d50c971e965c1c6f973b7df0e63a600574c735967f34bd2f2bc08d213
|
|
7
|
+
data.tar.gz: 73c7685e7014ba160ee16fe00f840450f98d6216c408407610892e2e1587f2e86887254e55c8f4917c72e9b9ceba422e130ba1dc10cdeeeace6fa62fcbdf88e9
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0 (unreleased)
|
|
4
|
+
|
|
5
|
+
First release, extracted from the monitor built for Idhra's mainnet validator pair.
|
|
6
|
+
|
|
7
|
+
- Checks: vote lag and delinquency, vote-account identity, RPC reachability,
|
|
8
|
+
each node in gossip, staked identity on a known host, staked identity
|
|
9
|
+
advertised by two nodes (possible double-active).
|
|
10
|
+
- Alerts once, reminds while failing, reports recovery; an undelivered alert is
|
|
11
|
+
retried on the next run.
|
|
12
|
+
- Notifiers: Telegram, webhook (Slack/Discord), log. Optional heartbeat URL.
|
|
13
|
+
- State stores: memory, JSON file, ActiveRecord.
|
|
14
|
+
- `solana-watchtower` CLI: `run`, `check`, `status`, `notify-test`.
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Paul Zupan, Idhra Inc.
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
# solana-ruby-watchtower
|
|
2
|
+
|
|
3
|
+
An off-box monitor for a Solana validator, built for **active/standby HA pairs**.
|
|
4
|
+
|
|
5
|
+
`agave-watchtower` tells you when your vote account goes delinquent. With a
|
|
6
|
+
failover pair, that isn't enough. The failover daemon's own alerts fire on a
|
|
7
|
+
role change, so they stay silent in the cases that matter most:
|
|
8
|
+
|
|
9
|
+
- the active node dies and the standby can't take over,
|
|
10
|
+
- both nodes end up running the staked identity,
|
|
11
|
+
- the failover daemon has lost its RPC and is quietly doing nothing.
|
|
12
|
+
|
|
13
|
+
This gem watches the validator from the cluster's side, over RPC, from a machine
|
|
14
|
+
that is not one of your nodes. It checks the vote account and also the gossip
|
|
15
|
+
table, so it can see each node and which identity it's running.
|
|
16
|
+
|
|
17
|
+
It depends only on the Ruby standard library.
|
|
18
|
+
|
|
19
|
+
## What it checks
|
|
20
|
+
|
|
21
|
+
| Check | Every | Alerts when |
|
|
22
|
+
|---|---|---|
|
|
23
|
+
| `voting` | 30 s | Last vote more than `max_vote_lag_slots` (100) behind the tip, or delinquent, for 2 checks in a row |
|
|
24
|
+
| `vote_identity` | 30 s | The vote account points at an identity other than `identity` |
|
|
25
|
+
| `rpc:votes`, `rpc:gossip` | each run | Every RPC endpoint failed (4 in a row for votes). "Couldn't ask" is never reported as "down" |
|
|
26
|
+
| `node:<name>` | 2 min | A node is missing from gossip, or it's running a key that is neither the staked identity nor its own standby key |
|
|
27
|
+
| `staked:gossip` | 2 min | The staked identity is missing from gossip, or it's gossiping from an IP that isn't one of your nodes |
|
|
28
|
+
| `staked:location` | 2 min | The staked identity's gossip address flipped between hosts 2+ times in 15 min, which suggests **two nodes running the staked identity** |
|
|
29
|
+
|
|
30
|
+
A single change of the staked identity's address is a normal role switch. It
|
|
31
|
+
gets an informational message, not an alert.
|
|
32
|
+
|
|
33
|
+
Every alert is sent once when the check goes bad. It's repeated every
|
|
34
|
+
`realert_minutes` (30) while the check stays bad, and followed by ✅ RESOLVED
|
|
35
|
+
when it clears. Each message starts with `[name]`, so one chat can serve
|
|
36
|
+
several validators. If the notifier fails, the alert is offered again on the
|
|
37
|
+
next run.
|
|
38
|
+
|
|
39
|
+
## Install
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
gem install solana-ruby-watchtower
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
Requires Ruby 3.2 or newer.
|
|
46
|
+
|
|
47
|
+
## Configure
|
|
48
|
+
|
|
49
|
+
Copy [`examples/watchtower.yml`](examples/watchtower.yml) and fill it in:
|
|
50
|
+
|
|
51
|
+
```yaml
|
|
52
|
+
name: my-validator
|
|
53
|
+
vote_account: <vote account pubkey>
|
|
54
|
+
identity: <staked identity pubkey>
|
|
55
|
+
nodes:
|
|
56
|
+
node-1: { ip: 203.0.113.10, unstaked_identity: <node-1 standby pubkey> }
|
|
57
|
+
node-2: { ip: 203.0.113.20, unstaked_identity: <node-2 standby pubkey> }
|
|
58
|
+
rpc_urls:
|
|
59
|
+
- https://api.mainnet-beta.solana.com
|
|
60
|
+
notify:
|
|
61
|
+
telegram:
|
|
62
|
+
bot_token: <%= ENV['TELEGRAM_BOT_TOKEN'] %>
|
|
63
|
+
chat_id: <%= ENV['TELEGRAM_CHAT_ID'] %>
|
|
64
|
+
heartbeat_url: https://hc-ping.com/<uuid>
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
- **nodes**: every machine that may run the staked identity, keyed by the IP
|
|
68
|
+
it gossips from. `unstaked_identity` is the key the node runs while it is the
|
|
69
|
+
standby. Leave it out for a single node with no standby.
|
|
70
|
+
- **rpc_urls**: tried in order, first answer wins. **Use providers your
|
|
71
|
+
failover tooling does not use.** If both rely on the same provider, one
|
|
72
|
+
outage or rate limit blinds them together.
|
|
73
|
+
- **notify**: `telegram` and/or `webhook` (`url:`, works with Slack and
|
|
74
|
+
Discord incoming webhooks). With neither, alerts are only logged.
|
|
75
|
+
- **heartbeat_url**: a dead-man's switch such as healthchecks.io, Uptime Kuma or
|
|
76
|
+
Better Stack. It's pinged after every vote check. Without it, a monitor that
|
|
77
|
+
dies takes its alerts down with it.
|
|
78
|
+
|
|
79
|
+
The file goes through ERB, so secrets can stay in the environment. Every value
|
|
80
|
+
is validated at startup; a bad config exits with status 2 and a message.
|
|
81
|
+
|
|
82
|
+
## Run
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
solana-watchtower notify-test -c watchtower.yml # check the alert channel first
|
|
86
|
+
solana-watchtower check votes -c watchtower.yml # one check, prints the results
|
|
87
|
+
solana-watchtower run -c watchtower.yml # run until SIGINT/SIGTERM
|
|
88
|
+
solana-watchtower status -c watchtower.yml # stored state of every check
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
The `run` command keeps each check's state in `state_file` (JSON, written
|
|
92
|
+
atomically), so after a restart it neither repeats alerts already sent nor
|
|
93
|
+
forgets one that is firing. `-v` logs every check result.
|
|
94
|
+
|
|
95
|
+
For systemd, see [`examples/solana-watchtower.service`](examples/solana-watchtower.service).
|
|
96
|
+
|
|
97
|
+
**Where to run it:** anywhere except your validator nodes, and ideally not with
|
|
98
|
+
the same hosting provider. A small VPS is plenty. Each gossip check downloads
|
|
99
|
+
the full cluster node list (about 1.5 MB), which comes to roughly 1 GB a day.
|
|
100
|
+
|
|
101
|
+
## Inside a Rails app
|
|
102
|
+
|
|
103
|
+
If you already run a Rails app with a job scheduler, you can run the checks as
|
|
104
|
+
jobs and keep their state in a table.
|
|
105
|
+
|
|
106
|
+
```ruby
|
|
107
|
+
# db/migrate/xxx_create_watchtower_checks.rb
|
|
108
|
+
create_table :watchtower_checks do |t|
|
|
109
|
+
t.string :key, null: false
|
|
110
|
+
t.boolean :alerting, null: false, default: false
|
|
111
|
+
t.integer :consecutive_failures, null: false, default: 0
|
|
112
|
+
t.datetime :failing_since
|
|
113
|
+
t.datetime :last_alerted_at
|
|
114
|
+
t.datetime :last_checked_at
|
|
115
|
+
t.text :last_message
|
|
116
|
+
t.jsonb :data, null: false, default: {}
|
|
117
|
+
t.timestamps
|
|
118
|
+
end
|
|
119
|
+
add_index :watchtower_checks, :key, unique: true
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
```ruby
|
|
123
|
+
# app/models/watchtower_check.rb
|
|
124
|
+
class WatchtowerCheck < ApplicationRecord; end
|
|
125
|
+
|
|
126
|
+
# app/jobs/watchtower_job.rb
|
|
127
|
+
class WatchtowerJob < ApplicationJob
|
|
128
|
+
queue_as :watchtower
|
|
129
|
+
# Solid Queue: drop a run that would overlap the previous one.
|
|
130
|
+
limits_concurrency to: 1, key: ->(check) { "watchtower-#{check}" }, on_conflict: :discard
|
|
131
|
+
|
|
132
|
+
def perform(check)
|
|
133
|
+
Solana::Ruby::Watchtower::Monitor.new(
|
|
134
|
+
Solana::Ruby::Watchtower::Config.load(Rails.root.join('config/watchtower.yml')),
|
|
135
|
+
store: Solana::Ruby::Watchtower::Stores::ActiveRecordStore.new(WatchtowerCheck),
|
|
136
|
+
logger: Rails.logger
|
|
137
|
+
).run(check)
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
```yaml
|
|
143
|
+
# config/recurring.yml (Solid Queue)
|
|
144
|
+
production:
|
|
145
|
+
watchtower_votes:
|
|
146
|
+
class: WatchtowerJob
|
|
147
|
+
args: [ 'votes' ]
|
|
148
|
+
schedule: every 30 seconds
|
|
149
|
+
watchtower_gossip:
|
|
150
|
+
class: WatchtowerJob
|
|
151
|
+
args: [ 'gossip' ]
|
|
152
|
+
schedule: every 2 minutes
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
A failing job never pings the heartbeat, so a crash loop is caught by the
|
|
156
|
+
heartbeat service the same way a dead process is.
|
|
157
|
+
|
|
158
|
+
## Limits
|
|
159
|
+
|
|
160
|
+
- **Double-active detection is a heuristic.** Gossip keeps one address per
|
|
161
|
+
identity, the one pushed most recently. Two nodes holding the staked identity
|
|
162
|
+
make that address flip, and the monitor sees it at its 2-minute sampling. A
|
|
163
|
+
real failover followed by a failback within the window looks the same, and
|
|
164
|
+
is worth hearing about anyway.
|
|
165
|
+
- **Gossip liveness is weak for the staked node.** The cluster keeps a staked
|
|
166
|
+
identity's gossip entry for a long time after the node goes quiet. A dead
|
|
167
|
+
active node is caught by the `voting` check, not by `node:<name>`. Standby
|
|
168
|
+
(unstaked) entries expire quickly, so a dead standby is caught by gossip.
|
|
169
|
+
- **Public RPC rate-limits by IP.** On shared hosting egress you may see
|
|
170
|
+
`rpc:*` alerts. Add a second endpoint to `rpc_urls`.
|
|
171
|
+
|
|
172
|
+
## Development
|
|
173
|
+
|
|
174
|
+
```bash
|
|
175
|
+
bundle install
|
|
176
|
+
bundle exec rspec
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
## License
|
|
180
|
+
|
|
181
|
+
MIT. See [LICENSE](LICENSE).
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# systemd unit for the monitor host (not a validator node).
|
|
2
|
+
#
|
|
3
|
+
# sudo useradd --system --create-home --home-dir /var/lib/solana-watchtower watchtower
|
|
4
|
+
# sudo install -m 0600 -o watchtower watchtower.yml /etc/solana-watchtower/watchtower.yml
|
|
5
|
+
# sudo install -m 0600 -o watchtower watchtower.env /etc/solana-watchtower/watchtower.env
|
|
6
|
+
# sudo systemctl enable --now solana-watchtower
|
|
7
|
+
#
|
|
8
|
+
# watchtower.env holds TELEGRAM_BOT_TOKEN=... and TELEGRAM_CHAT_ID=...
|
|
9
|
+
# Set state_file in the config to /var/lib/solana-watchtower/state.json.
|
|
10
|
+
[Unit]
|
|
11
|
+
Description=solana-watchtower validator monitor
|
|
12
|
+
Wants=network-online.target
|
|
13
|
+
After=network-online.target
|
|
14
|
+
|
|
15
|
+
[Service]
|
|
16
|
+
User=watchtower
|
|
17
|
+
EnvironmentFile=/etc/solana-watchtower/watchtower.env
|
|
18
|
+
ExecStart=/usr/local/bin/solana-watchtower run --config /etc/solana-watchtower/watchtower.yml
|
|
19
|
+
Restart=always
|
|
20
|
+
RestartSec=10
|
|
21
|
+
NoNewPrivileges=true
|
|
22
|
+
ProtectSystem=strict
|
|
23
|
+
ProtectHome=true
|
|
24
|
+
ReadWritePaths=/var/lib/solana-watchtower
|
|
25
|
+
PrivateTmp=true
|
|
26
|
+
|
|
27
|
+
[Install]
|
|
28
|
+
WantedBy=multi-user.target
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# solana-watchtower config. Copy, fill in, and run:
|
|
2
|
+
#
|
|
3
|
+
# solana-watchtower notify-test --config watchtower.yml
|
|
4
|
+
# solana-watchtower run --config watchtower.yml
|
|
5
|
+
#
|
|
6
|
+
# The file goes through ERB first, so secrets can come from the environment.
|
|
7
|
+
# Run the monitor on a machine that is NOT one of your validator nodes: it has to
|
|
8
|
+
# keep working when they don't.
|
|
9
|
+
|
|
10
|
+
# Shown in every alert, so one chat can take alerts for several validators.
|
|
11
|
+
name: my-validator
|
|
12
|
+
|
|
13
|
+
vote_account: YOUR_VOTE_ACCOUNT_PUBKEY
|
|
14
|
+
# The staked (voting) identity.
|
|
15
|
+
identity: YOUR_STAKED_IDENTITY_PUBKEY
|
|
16
|
+
|
|
17
|
+
# Every machine that may run the staked identity, by the IP it gossips from.
|
|
18
|
+
# unstaked_identity is the key a node runs while it is the standby; leave it out
|
|
19
|
+
# for a single node with no standby.
|
|
20
|
+
nodes:
|
|
21
|
+
node-1:
|
|
22
|
+
ip: 203.0.113.10
|
|
23
|
+
unstaked_identity: NODE_1_STANDBY_IDENTITY_PUBKEY
|
|
24
|
+
node-2:
|
|
25
|
+
ip: 203.0.113.20
|
|
26
|
+
unstaked_identity: NODE_2_STANDBY_IDENTITY_PUBKEY
|
|
27
|
+
|
|
28
|
+
# Tried in order; the first to answer is used. Prefer endpoints your failover
|
|
29
|
+
# tooling does NOT use, so one provider's outage cannot blind both at once.
|
|
30
|
+
rpc_urls:
|
|
31
|
+
- https://api.mainnet-beta.solana.com
|
|
32
|
+
|
|
33
|
+
notify:
|
|
34
|
+
telegram:
|
|
35
|
+
bot_token: <%= ENV['TELEGRAM_BOT_TOKEN'] %>
|
|
36
|
+
chat_id: <%= ENV['TELEGRAM_CHAT_ID'] %>
|
|
37
|
+
# webhook: # Slack or Discord incoming webhook
|
|
38
|
+
# url: <%= ENV['WATCHTOWER_WEBHOOK_URL'] %>
|
|
39
|
+
|
|
40
|
+
# Pinged after every vote check; the service alerts you when the pings stop,
|
|
41
|
+
# which is the only way to hear about the monitor itself dying.
|
|
42
|
+
# heartbeat_url: https://hc-ping.com/your-uuid
|
|
43
|
+
|
|
44
|
+
# Relative to this file.
|
|
45
|
+
state_file: watchtower-state.json
|
|
46
|
+
|
|
47
|
+
# Thresholds (these are the defaults).
|
|
48
|
+
# max_vote_lag_slots: 100
|
|
49
|
+
# failures_to_alert: 2
|
|
50
|
+
# rpc_failures_to_alert: 4
|
|
51
|
+
# realert_minutes: 30
|
|
52
|
+
# double_active_moves: 2
|
|
53
|
+
# double_active_window_minutes: 15
|
|
54
|
+
# votes_interval_seconds: 30
|
|
55
|
+
# gossip_interval_seconds: 120
|
|
56
|
+
# rpc_timeout_seconds: 10
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'time'
|
|
4
|
+
|
|
5
|
+
module Solana::Ruby::Watchtower
|
|
6
|
+
# The alert state of one check.
|
|
7
|
+
#
|
|
8
|
+
# #observe records a result and says whether it calls for a message;
|
|
9
|
+
# #acknowledge records that the message went out. They are separate so that an
|
|
10
|
+
# alert the notifier failed to deliver is offered again on the next run
|
|
11
|
+
# instead of being marked as sent.
|
|
12
|
+
class CheckState
|
|
13
|
+
Alert = Data.define(:kind, :key, :text) # kind: :firing, :reminder or :resolved
|
|
14
|
+
|
|
15
|
+
ATTRIBUTES = %i[key alerting consecutive_failures failing_since last_alerted_at last_checked_at last_message data].freeze
|
|
16
|
+
TIMES = %i[failing_since last_alerted_at last_checked_at].freeze
|
|
17
|
+
|
|
18
|
+
attr_accessor(*ATTRIBUTES)
|
|
19
|
+
|
|
20
|
+
def initialize(key:, alerting: false, consecutive_failures: 0, failing_since: nil, last_alerted_at: nil,
|
|
21
|
+
last_checked_at: nil, last_message: nil, data: {})
|
|
22
|
+
@key = key.to_s
|
|
23
|
+
@alerting = alerting ? true : false
|
|
24
|
+
@consecutive_failures = consecutive_failures.to_i
|
|
25
|
+
@failing_since = time(failing_since)
|
|
26
|
+
@last_alerted_at = time(last_alerted_at)
|
|
27
|
+
@last_checked_at = time(last_checked_at)
|
|
28
|
+
@last_message = last_message
|
|
29
|
+
@data = data || {}
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def self.from_h(hash)
|
|
33
|
+
new(**hash.transform_keys(&:to_sym).slice(*ATTRIBUTES))
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
# Times as ISO 8601 strings, ready for JSON.
|
|
37
|
+
def to_h
|
|
38
|
+
ATTRIBUTES.to_h do |attr|
|
|
39
|
+
value = public_send(attr)
|
|
40
|
+
[attr, TIMES.include?(attr) && value ? value.utc.iso8601 : value]
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def alerting? = @alerting
|
|
45
|
+
|
|
46
|
+
def observe(ok:, message:, now:, failures_to_alert:, realert_after:)
|
|
47
|
+
self.last_message = message
|
|
48
|
+
self.last_checked_at = now
|
|
49
|
+
|
|
50
|
+
if ok
|
|
51
|
+
self.consecutive_failures = 0
|
|
52
|
+
# Kept until the recovery message is sent: it is what that message reports.
|
|
53
|
+
self.failing_since = nil unless alerting?
|
|
54
|
+
return unless alerting?
|
|
55
|
+
|
|
56
|
+
return Alert.new(kind: :resolved, key: key, text: "✅ RESOLVED #{key}: #{message} (failing #{since(now)})")
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
self.consecutive_failures += 1
|
|
60
|
+
self.failing_since ||= now
|
|
61
|
+
return if consecutive_failures < failures_to_alert
|
|
62
|
+
|
|
63
|
+
if !alerting?
|
|
64
|
+
Alert.new(kind: :firing, key: key, text: "🚨 #{key}: #{message}")
|
|
65
|
+
elsif last_alerted_at.nil? || now - last_alerted_at >= realert_after
|
|
66
|
+
Alert.new(kind: :reminder, key: key, text: "🚨 STILL FAILING #{key} (#{since(now)}): #{message}")
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def acknowledge(alert, now)
|
|
71
|
+
if alert.kind == :resolved
|
|
72
|
+
self.alerting = false
|
|
73
|
+
self.failing_since = nil
|
|
74
|
+
self.last_alerted_at = nil
|
|
75
|
+
else
|
|
76
|
+
self.alerting = true
|
|
77
|
+
self.last_alerted_at = now
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
private
|
|
82
|
+
|
|
83
|
+
def since(now)
|
|
84
|
+
return 'briefly' if failing_since.nil?
|
|
85
|
+
|
|
86
|
+
minutes = ((now - failing_since) / 60).round
|
|
87
|
+
"since #{failing_since.utc.strftime('%H:%M')} UTC, #{minutes} min"
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def time(value)
|
|
91
|
+
case value
|
|
92
|
+
when nil then nil
|
|
93
|
+
when String then Time.iso8601(value)
|
|
94
|
+
else value.to_time
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
end
|
|
@@ -0,0 +1,133 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'logger'
|
|
4
|
+
require 'optparse'
|
|
5
|
+
|
|
6
|
+
module Solana::Ruby::Watchtower
|
|
7
|
+
# solana-watchtower COMMAND [options]
|
|
8
|
+
#
|
|
9
|
+
# run run the checks on their intervals until SIGINT/SIGTERM
|
|
10
|
+
# check votes|gossip run one check once and exit (for cron or a systemd timer)
|
|
11
|
+
# status print each check's stored state
|
|
12
|
+
# notify-test send a test message through the configured notifiers
|
|
13
|
+
class CLI
|
|
14
|
+
USAGE = <<~TEXT
|
|
15
|
+
Usage: solana-watchtower COMMAND [options]
|
|
16
|
+
|
|
17
|
+
Commands:
|
|
18
|
+
run run the checks on their intervals until SIGINT/SIGTERM
|
|
19
|
+
check votes|gossip run one check once and exit
|
|
20
|
+
status print each check's stored state
|
|
21
|
+
notify-test send a test message through the configured notifiers
|
|
22
|
+
|
|
23
|
+
Options:
|
|
24
|
+
TEXT
|
|
25
|
+
|
|
26
|
+
def initialize(argv, out: $stdout, err: $stderr)
|
|
27
|
+
@argv = argv.dup
|
|
28
|
+
@out = out
|
|
29
|
+
@err = err
|
|
30
|
+
@config_path = ENV.fetch('WATCHTOWER_CONFIG', 'watchtower.yml')
|
|
31
|
+
@log_level = Logger::INFO
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
# Returns the process exit status.
|
|
35
|
+
def run
|
|
36
|
+
parser.parse!(@argv)
|
|
37
|
+
command = @argv.shift
|
|
38
|
+
case command
|
|
39
|
+
when 'run' then run_forever
|
|
40
|
+
when 'check' then check_once(@argv.shift)
|
|
41
|
+
when 'status' then status
|
|
42
|
+
when 'notify-test' then notify_test
|
|
43
|
+
when nil then @err.puts(parser.help) || 2
|
|
44
|
+
else
|
|
45
|
+
@err.puts "unknown command #{command.inspect}\n\n#{parser.help}"
|
|
46
|
+
2
|
|
47
|
+
end
|
|
48
|
+
rescue ConfigError, OptionParser::ParseError => e
|
|
49
|
+
@err.puts "solana-watchtower: #{e.message}"
|
|
50
|
+
2
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
private
|
|
54
|
+
|
|
55
|
+
def parser
|
|
56
|
+
@parser ||= OptionParser.new do |o|
|
|
57
|
+
o.banner = USAGE
|
|
58
|
+
o.on('-c', '--config FILE', "config file (default: $WATCHTOWER_CONFIG or #{@config_path})") { |v| @config_path = v }
|
|
59
|
+
o.on('-v', '--verbose', 'log every check result, not just alerts') { @log_level = Logger::DEBUG }
|
|
60
|
+
o.on('--version', 'print the version') do
|
|
61
|
+
@out.puts VERSION
|
|
62
|
+
exit 0
|
|
63
|
+
end
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def config
|
|
68
|
+
raise ConfigError, "config file #{@config_path} not found (pass --config)" unless File.exist?(@config_path)
|
|
69
|
+
|
|
70
|
+
@config ||= Config.load(@config_path)
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def logger
|
|
74
|
+
@logger ||= Logger.new(@out, progname: 'solana-watchtower', level: @log_level).tap do |l|
|
|
75
|
+
l.formatter = proc { |sev, time, _, msg| "#{time.utc.strftime('%FT%TZ')} #{sev} #{msg}\n" }
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def monitor = @monitor ||= Monitor.new(config, logger: logger)
|
|
80
|
+
|
|
81
|
+
def run_forever
|
|
82
|
+
runner = Runner.new(monitor, logger: logger,
|
|
83
|
+
votes_every: config.votes_interval_seconds,
|
|
84
|
+
gossip_every: config.gossip_interval_seconds)
|
|
85
|
+
%w[INT TERM].each { |signal| trap(signal) { runner.stop! } }
|
|
86
|
+
logger.info("watching #{config.name}: vote account #{config.vote_account}, " \
|
|
87
|
+
"nodes #{config.nodes.map(&:name).join(', ')}, via #{config.rpc_urls.map { |u| URI(u).host }.join(', ')}")
|
|
88
|
+
runner.run
|
|
89
|
+
logger.info('stopped')
|
|
90
|
+
0
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def check_once(check)
|
|
94
|
+
raise ConfigError, "check needs one of: #{Monitor::CHECKS.join(', ')}" unless Monitor::CHECKS.include?(check)
|
|
95
|
+
|
|
96
|
+
monitor.run(check)
|
|
97
|
+
print_states(monitor.store.all.select { |s| belongs_to?(s.key, check) })
|
|
98
|
+
0
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
def status
|
|
102
|
+
print_states(Stores::FileStore.new(config.state_file).all)
|
|
103
|
+
0
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
def notify_test
|
|
107
|
+
notifier = Notifiers.build(config.notify, logger: logger)
|
|
108
|
+
if notifier.deliver("[#{config.name}] ✅ solana-watchtower test message")
|
|
109
|
+
@out.puts 'delivered'
|
|
110
|
+
0
|
|
111
|
+
else
|
|
112
|
+
@err.puts 'delivery failed, see the log above'
|
|
113
|
+
1
|
|
114
|
+
end
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
def belongs_to?(key, check)
|
|
118
|
+
check == 'votes' ? %w[rpc:votes voting vote_identity].include?(key) : !%w[rpc:votes voting vote_identity].include?(key)
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
def print_states(states)
|
|
122
|
+
if states.empty?
|
|
123
|
+
@out.puts "no state yet in #{config.state_file}"
|
|
124
|
+
return
|
|
125
|
+
end
|
|
126
|
+
width = states.map { |s| s.key.size }.max
|
|
127
|
+
states.sort_by(&:key).each do |s|
|
|
128
|
+
flag = s.alerting? ? 'ALERT' : (s.consecutive_failures.positive? ? 'fail ' : 'ok ')
|
|
129
|
+
@out.puts "#{flag} #{s.key.ljust(width)} #{s.last_message}"
|
|
130
|
+
end
|
|
131
|
+
end
|
|
132
|
+
end
|
|
133
|
+
end
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'erb'
|
|
4
|
+
require 'yaml'
|
|
5
|
+
|
|
6
|
+
module Solana::Ruby::Watchtower
|
|
7
|
+
class ConfigError < Error; end
|
|
8
|
+
|
|
9
|
+
# The settings for one validator, from a hash or a YAML file.
|
|
10
|
+
#
|
|
11
|
+
# A file is passed through ERB first, so secrets can stay in the environment:
|
|
12
|
+
#
|
|
13
|
+
# bot_token: <%= ENV['TELEGRAM_BOT_TOKEN'] %>
|
|
14
|
+
#
|
|
15
|
+
# Everything is checked when the config is built, so a typo stops the monitor
|
|
16
|
+
# at startup rather than leaving it running blind.
|
|
17
|
+
class Config
|
|
18
|
+
Node = Data.define(:name, :ip, :unstaked_identity)
|
|
19
|
+
|
|
20
|
+
DEFAULTS = {
|
|
21
|
+
'rpc_urls' => ['https://api.mainnet-beta.solana.com'],
|
|
22
|
+
'rpc_timeout_seconds' => 10,
|
|
23
|
+
# Last vote this many slots behind the tip counts as not voting. Healthy
|
|
24
|
+
# lag is a handful of slots; the cluster's delinquency mark is 128.
|
|
25
|
+
'max_vote_lag_slots' => 100,
|
|
26
|
+
# Consecutive bad vote checks before an alert fires.
|
|
27
|
+
'failures_to_alert' => 2,
|
|
28
|
+
# All RPC endpoints failing, held back longer: public RPC rate-limits in bursts.
|
|
29
|
+
'rpc_failures_to_alert' => 4,
|
|
30
|
+
'realert_minutes' => 30,
|
|
31
|
+
# The staked identity's gossip address changing this many times within the
|
|
32
|
+
# window means two nodes are advertising it.
|
|
33
|
+
'double_active_moves' => 2,
|
|
34
|
+
'double_active_window_minutes' => 15,
|
|
35
|
+
'votes_interval_seconds' => 30,
|
|
36
|
+
# getClusterNodes returns the whole cluster (well over a megabyte), so the
|
|
37
|
+
# gossip checks run less often than the vote check.
|
|
38
|
+
'gossip_interval_seconds' => 120,
|
|
39
|
+
'state_file' => 'watchtower-state.json',
|
|
40
|
+
'notify' => {}
|
|
41
|
+
}.freeze
|
|
42
|
+
|
|
43
|
+
POSITIVE_INTEGERS = %w[
|
|
44
|
+
rpc_timeout_seconds max_vote_lag_slots failures_to_alert rpc_failures_to_alert realert_minutes
|
|
45
|
+
double_active_moves double_active_window_minutes votes_interval_seconds gossip_interval_seconds
|
|
46
|
+
].freeze
|
|
47
|
+
|
|
48
|
+
PUBKEY = /\A[1-9A-HJ-NP-Za-km-z]{32,44}\z/
|
|
49
|
+
|
|
50
|
+
attr_reader :name, :vote_account, :identity, :nodes, :rpc_urls, :notify, :heartbeat_url, :state_file,
|
|
51
|
+
*POSITIVE_INTEGERS.map(&:to_sym)
|
|
52
|
+
|
|
53
|
+
# base_dir resolves a relative state_file; for a file, its directory.
|
|
54
|
+
def self.load(path)
|
|
55
|
+
text = ERB.new(File.read(path)).result
|
|
56
|
+
new(YAML.safe_load(text, aliases: true) || {}, base_dir: File.dirname(File.expand_path(path)))
|
|
57
|
+
rescue Psych::SyntaxError => e
|
|
58
|
+
raise ConfigError, "#{path}: #{e.message}"
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def initialize(hash, base_dir: Dir.pwd)
|
|
62
|
+
settings = DEFAULTS.merge(stringify(hash))
|
|
63
|
+
|
|
64
|
+
@vote_account = pubkey!(settings, 'vote_account')
|
|
65
|
+
@identity = pubkey!(settings, 'identity')
|
|
66
|
+
@name = (settings['name'] || "#{@identity[0, 8]}…").to_s
|
|
67
|
+
@nodes = nodes!(settings['nodes'])
|
|
68
|
+
@rpc_urls = Array(settings['rpc_urls']).map { |u| u.to_s.strip }.reject(&:empty?)
|
|
69
|
+
raise ConfigError, 'rpc_urls is empty' if @rpc_urls.empty?
|
|
70
|
+
|
|
71
|
+
@rpc_urls.each { |u| url!(u, 'rpc_urls') }
|
|
72
|
+
POSITIVE_INTEGERS.each { |k| instance_variable_set(:"@#{k}", positive_integer!(settings, k)) }
|
|
73
|
+
|
|
74
|
+
@notify = settings['notify'] || {}
|
|
75
|
+
raise ConfigError, 'notify must be a mapping' unless @notify.is_a?(Hash)
|
|
76
|
+
|
|
77
|
+
@heartbeat_url = blank?(settings['heartbeat_url']) ? nil : url!(settings['heartbeat_url'], 'heartbeat_url')
|
|
78
|
+
@state_file = File.expand_path(settings['state_file'].to_s, base_dir)
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
private
|
|
82
|
+
|
|
83
|
+
def stringify(value)
|
|
84
|
+
case value
|
|
85
|
+
when Hash then value.to_h { |k, v| [k.to_s, stringify(v)] }
|
|
86
|
+
when Array then value.map { |v| stringify(v) }
|
|
87
|
+
else value
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def nodes!(raw)
|
|
92
|
+
raise ConfigError, 'nodes must map each node name to its ip (and unstaked_identity for a standby)' unless raw.is_a?(Hash) && raw.any?
|
|
93
|
+
|
|
94
|
+
nodes = raw.map do |name, node|
|
|
95
|
+
raise ConfigError, "nodes.#{name} must be a mapping" unless node.is_a?(Hash)
|
|
96
|
+
raise ConfigError, "nodes.#{name}.ip is missing" if blank?(node['ip'])
|
|
97
|
+
|
|
98
|
+
unstaked = node['unstaked_identity']
|
|
99
|
+
unstaked = blank?(unstaked) ? nil : pubkey!(node, 'unstaked_identity', "nodes.#{name}.")
|
|
100
|
+
Node.new(name: name.to_s, ip: node['ip'].to_s, unstaked_identity: unstaked)
|
|
101
|
+
end
|
|
102
|
+
duplicate = nodes.group_by(&:ip).find { |_, same| same.size > 1 }
|
|
103
|
+
raise ConfigError, "nodes #{duplicate[1].map(&:name).join(' and ')} share ip #{duplicate[0]}" if duplicate
|
|
104
|
+
|
|
105
|
+
nodes
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
def pubkey!(settings, key, prefix = '')
|
|
109
|
+
value = settings[key].to_s
|
|
110
|
+
raise ConfigError, "#{prefix}#{key} is missing" if value.empty?
|
|
111
|
+
raise ConfigError, "#{prefix}#{key} is not a base58 public key: #{value.inspect}" unless value.match?(PUBKEY)
|
|
112
|
+
|
|
113
|
+
value
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
def positive_integer!(settings, key)
|
|
117
|
+
value = settings[key]
|
|
118
|
+
raise ConfigError, "#{key} must be a positive integer, got #{value.inspect}" unless value.is_a?(Integer) && value.positive?
|
|
119
|
+
|
|
120
|
+
value
|
|
121
|
+
end
|
|
122
|
+
|
|
123
|
+
def url!(value, key)
|
|
124
|
+
uri = URI.parse(value.to_s)
|
|
125
|
+
raise ConfigError, "#{key}: #{value.to_s.inspect} is not an http(s) URL" unless uri.is_a?(URI::HTTP) && uri.host
|
|
126
|
+
|
|
127
|
+
value.to_s
|
|
128
|
+
rescue URI::InvalidURIError
|
|
129
|
+
raise ConfigError, "#{key}: #{value.to_s.inspect} is not an http(s) URL"
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
def blank?(value) = value.nil? || value.to_s.strip.empty?
|
|
133
|
+
end
|
|
134
|
+
end
|