solana-ruby-watchtower 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 4753642b3fac09fb773914fdad127effd85850119172f75d4febf17b024d9f65
4
+ data.tar.gz: 127f3df754c1fedaea7f47d526b1ff0307be60a44d76a854ece8214d66a0b8d5
5
+ SHA512:
6
+ metadata.gz: 14c9d556c637573e16588ff8c0c211ff79a162dd1fc7399c4d24e101648c684f07fc025d50c971e965c1c6f973b7df0e63a600574c735967f34bd2f2bc08d213
7
+ data.tar.gz: 73c7685e7014ba160ee16fe00f840450f98d6216c408407610892e2e1587f2e86887254e55c8f4917c72e9b9ceba422e130ba1dc10cdeeeace6fa62fcbdf88e9
data/CHANGELOG.md ADDED
@@ -0,0 +1,14 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0 (unreleased)
4
+
5
+ First release, extracted from the monitor built for Idhra's mainnet validator pair.
6
+
7
+ - Checks: vote lag and delinquency, vote-account identity, RPC reachability,
8
+ each node in gossip, staked identity on a known host, staked identity
9
+ advertised by two nodes (possible double-active).
10
+ - Alerts once, reminds while failing, reports recovery; an undelivered alert is
11
+ retried on the next run.
12
+ - Notifiers: Telegram, webhook (Slack/Discord), log. Optional heartbeat URL.
13
+ - State stores: memory, JSON file, ActiveRecord.
14
+ - `solana-watchtower` CLI: `run`, `check`, `status`, `notify-test`.
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Paul Zupan, Idhra Inc.
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,181 @@
1
+ # solana-ruby-watchtower
2
+
3
+ An off-box monitor for a Solana validator, built for **active/standby HA pairs**.
4
+
5
+ `agave-watchtower` tells you when your vote account goes delinquent. With a
6
+ failover pair, that isn't enough. The failover daemon's own alerts fire on a
7
+ role change, so they stay silent in the cases that matter most:
8
+
9
+ - the active node dies and the standby can't take over,
10
+ - both nodes end up running the staked identity,
11
+ - the failover daemon has lost its RPC and is quietly doing nothing.
12
+
13
+ This gem watches the validator from the cluster's side, over RPC, from a machine
14
+ that is not one of your nodes. It checks the vote account and also the gossip
15
+ table, so it can see each node and which identity it's running.
16
+
17
+ It depends only on the Ruby standard library.
18
+
19
+ ## What it checks
20
+
21
+ | Check | Every | Alerts when |
22
+ |---|---|---|
23
+ | `voting` | 30 s | Last vote more than `max_vote_lag_slots` (100) behind the tip, or delinquent, for 2 checks in a row |
24
+ | `vote_identity` | 30 s | The vote account points at an identity other than `identity` |
25
+ | `rpc:votes`, `rpc:gossip` | each run | Every RPC endpoint failed (4 in a row for votes). "Couldn't ask" is never reported as "down" |
26
+ | `node:<name>` | 2 min | A node is missing from gossip, or it's running a key that is neither the staked identity nor its own standby key |
27
+ | `staked:gossip` | 2 min | The staked identity is missing from gossip, or it's gossiping from an IP that isn't one of your nodes |
28
+ | `staked:location` | 2 min | The staked identity's gossip address flipped between hosts 2+ times in 15 min, which suggests **two nodes running the staked identity** |
29
+
30
+ A single change of the staked identity's address is a normal role switch. It
31
+ gets an informational message, not an alert.
32
+
33
+ Every alert is sent once when the check goes bad. It's repeated every
34
+ `realert_minutes` (30) while the check stays bad, and followed by ✅ RESOLVED
35
+ when it clears. Each message starts with `[name]`, so one chat can serve
36
+ several validators. If the notifier fails, the alert is offered again on the
37
+ next run.
38
+
39
+ ## Install
40
+
41
+ ```bash
42
+ gem install solana-ruby-watchtower
43
+ ```
44
+
45
+ Requires Ruby 3.2 or newer.
46
+
47
+ ## Configure
48
+
49
+ Copy [`examples/watchtower.yml`](examples/watchtower.yml) and fill it in:
50
+
51
+ ```yaml
52
+ name: my-validator
53
+ vote_account: <vote account pubkey>
54
+ identity: <staked identity pubkey>
55
+ nodes:
56
+ node-1: { ip: 203.0.113.10, unstaked_identity: <node-1 standby pubkey> }
57
+ node-2: { ip: 203.0.113.20, unstaked_identity: <node-2 standby pubkey> }
58
+ rpc_urls:
59
+ - https://api.mainnet-beta.solana.com
60
+ notify:
61
+ telegram:
62
+ bot_token: <%= ENV['TELEGRAM_BOT_TOKEN'] %>
63
+ chat_id: <%= ENV['TELEGRAM_CHAT_ID'] %>
64
+ heartbeat_url: https://hc-ping.com/<uuid>
65
+ ```
66
+
67
+ - **nodes**: every machine that may run the staked identity, keyed by the IP
68
+ it gossips from. `unstaked_identity` is the key the node runs while it is the
69
+ standby. Leave it out for a single node with no standby.
70
+ - **rpc_urls**: tried in order, first answer wins. **Use providers your
71
+ failover tooling does not use.** If both rely on the same provider, one
72
+ outage or rate limit blinds them together.
73
+ - **notify**: `telegram` and/or `webhook` (`url:`, works with Slack and
74
+ Discord incoming webhooks). With neither, alerts are only logged.
75
+ - **heartbeat_url**: a dead-man's switch such as healthchecks.io, Uptime Kuma or
76
+ Better Stack. It's pinged after every vote check. Without it, a monitor that
77
+ dies takes its alerts down with it.
78
+
79
+ The file goes through ERB, so secrets can stay in the environment. Every value
80
+ is validated at startup; a bad config exits with status 2 and a message.
81
+
82
+ ## Run
83
+
84
+ ```bash
85
+ solana-watchtower notify-test -c watchtower.yml # check the alert channel first
86
+ solana-watchtower check votes -c watchtower.yml # one check, prints the results
87
+ solana-watchtower run -c watchtower.yml # run until SIGINT/SIGTERM
88
+ solana-watchtower status -c watchtower.yml # stored state of every check
89
+ ```
90
+
91
+ The `run` command keeps each check's state in `state_file` (JSON, written
92
+ atomically), so after a restart it neither repeats alerts already sent nor
93
+ forgets one that is firing. `-v` logs every check result.
94
+
95
+ For systemd, see [`examples/solana-watchtower.service`](examples/solana-watchtower.service).
96
+
97
+ **Where to run it:** anywhere except your validator nodes, and ideally not with
98
+ the same hosting provider. A small VPS is plenty. Each gossip check downloads
99
+ the full cluster node list (about 1.5 MB), which comes to roughly 1 GB a day.
100
+
101
+ ## Inside a Rails app
102
+
103
+ If you already run a Rails app with a job scheduler, you can run the checks as
104
+ jobs and keep their state in a table.
105
+
106
+ ```ruby
107
+ # db/migrate/xxx_create_watchtower_checks.rb
108
+ create_table :watchtower_checks do |t|
109
+ t.string :key, null: false
110
+ t.boolean :alerting, null: false, default: false
111
+ t.integer :consecutive_failures, null: false, default: 0
112
+ t.datetime :failing_since
113
+ t.datetime :last_alerted_at
114
+ t.datetime :last_checked_at
115
+ t.text :last_message
116
+ t.jsonb :data, null: false, default: {}
117
+ t.timestamps
118
+ end
119
+ add_index :watchtower_checks, :key, unique: true
120
+ ```
121
+
122
+ ```ruby
123
+ # app/models/watchtower_check.rb
124
+ class WatchtowerCheck < ApplicationRecord; end
125
+
126
+ # app/jobs/watchtower_job.rb
127
+ class WatchtowerJob < ApplicationJob
128
+ queue_as :watchtower
129
+ # Solid Queue: drop a run that would overlap the previous one.
130
+ limits_concurrency to: 1, key: ->(check) { "watchtower-#{check}" }, on_conflict: :discard
131
+
132
+ def perform(check)
133
+ Solana::Ruby::Watchtower::Monitor.new(
134
+ Solana::Ruby::Watchtower::Config.load(Rails.root.join('config/watchtower.yml')),
135
+ store: Solana::Ruby::Watchtower::Stores::ActiveRecordStore.new(WatchtowerCheck),
136
+ logger: Rails.logger
137
+ ).run(check)
138
+ end
139
+ end
140
+ ```
141
+
142
+ ```yaml
143
+ # config/recurring.yml (Solid Queue)
144
+ production:
145
+ watchtower_votes:
146
+ class: WatchtowerJob
147
+ args: [ 'votes' ]
148
+ schedule: every 30 seconds
149
+ watchtower_gossip:
150
+ class: WatchtowerJob
151
+ args: [ 'gossip' ]
152
+ schedule: every 2 minutes
153
+ ```
154
+
155
+ A failing job never pings the heartbeat, so a crash loop is caught by the
156
+ heartbeat service the same way a dead process is.
157
+
158
+ ## Limits
159
+
160
+ - **Double-active detection is a heuristic.** Gossip keeps one address per
161
+ identity, the one pushed most recently. Two nodes holding the staked identity
162
+ make that address flip, and the monitor sees it at its 2-minute sampling. A
163
+ real failover followed by a failback within the window looks the same, and
164
+ is worth hearing about anyway.
165
+ - **Gossip liveness is weak for the staked node.** The cluster keeps a staked
166
+ identity's gossip entry for a long time after the node goes quiet. A dead
167
+ active node is caught by the `voting` check, not by `node:<name>`. Standby
168
+ (unstaked) entries expire quickly, so a dead standby is caught by gossip.
169
+ - **Public RPC rate-limits by IP.** On shared hosting egress you may see
170
+ `rpc:*` alerts. Add a second endpoint to `rpc_urls`.
171
+
172
+ ## Development
173
+
174
+ ```bash
175
+ bundle install
176
+ bundle exec rspec
177
+ ```
178
+
179
+ ## License
180
+
181
+ MIT. See [LICENSE](LICENSE).
@@ -0,0 +1,28 @@
1
+ # systemd unit for the monitor host (not a validator node).
2
+ #
3
+ # sudo useradd --system --create-home --home-dir /var/lib/solana-watchtower watchtower
4
+ # sudo install -m 0600 -o watchtower watchtower.yml /etc/solana-watchtower/watchtower.yml
5
+ # sudo install -m 0600 -o watchtower watchtower.env /etc/solana-watchtower/watchtower.env
6
+ # sudo systemctl enable --now solana-watchtower
7
+ #
8
+ # watchtower.env holds TELEGRAM_BOT_TOKEN=... and TELEGRAM_CHAT_ID=...
9
+ # Set state_file in the config to /var/lib/solana-watchtower/state.json.
10
+ [Unit]
11
+ Description=solana-watchtower validator monitor
12
+ Wants=network-online.target
13
+ After=network-online.target
14
+
15
+ [Service]
16
+ User=watchtower
17
+ EnvironmentFile=/etc/solana-watchtower/watchtower.env
18
+ ExecStart=/usr/local/bin/solana-watchtower run --config /etc/solana-watchtower/watchtower.yml
19
+ Restart=always
20
+ RestartSec=10
21
+ NoNewPrivileges=true
22
+ ProtectSystem=strict
23
+ ProtectHome=true
24
+ ReadWritePaths=/var/lib/solana-watchtower
25
+ PrivateTmp=true
26
+
27
+ [Install]
28
+ WantedBy=multi-user.target
@@ -0,0 +1,56 @@
1
+ # solana-watchtower config. Copy, fill in, and run:
2
+ #
3
+ # solana-watchtower notify-test --config watchtower.yml
4
+ # solana-watchtower run --config watchtower.yml
5
+ #
6
+ # The file goes through ERB first, so secrets can come from the environment.
7
+ # Run the monitor on a machine that is NOT one of your validator nodes: it has to
8
+ # keep working when they don't.
9
+
10
+ # Shown in every alert, so one chat can take alerts for several validators.
11
+ name: my-validator
12
+
13
+ vote_account: YOUR_VOTE_ACCOUNT_PUBKEY
14
+ # The staked (voting) identity.
15
+ identity: YOUR_STAKED_IDENTITY_PUBKEY
16
+
17
+ # Every machine that may run the staked identity, by the IP it gossips from.
18
+ # unstaked_identity is the key a node runs while it is the standby; leave it out
19
+ # for a single node with no standby.
20
+ nodes:
21
+ node-1:
22
+ ip: 203.0.113.10
23
+ unstaked_identity: NODE_1_STANDBY_IDENTITY_PUBKEY
24
+ node-2:
25
+ ip: 203.0.113.20
26
+ unstaked_identity: NODE_2_STANDBY_IDENTITY_PUBKEY
27
+
28
+ # Tried in order; the first to answer is used. Prefer endpoints your failover
29
+ # tooling does NOT use, so one provider's outage cannot blind both at once.
30
+ rpc_urls:
31
+ - https://api.mainnet-beta.solana.com
32
+
33
+ notify:
34
+ telegram:
35
+ bot_token: <%= ENV['TELEGRAM_BOT_TOKEN'] %>
36
+ chat_id: <%= ENV['TELEGRAM_CHAT_ID'] %>
37
+ # webhook: # Slack or Discord incoming webhook
38
+ # url: <%= ENV['WATCHTOWER_WEBHOOK_URL'] %>
39
+
40
+ # Pinged after every vote check; the service alerts you when the pings stop,
41
+ # which is the only way to hear about the monitor itself dying.
42
+ # heartbeat_url: https://hc-ping.com/your-uuid
43
+
44
+ # Relative to this file.
45
+ state_file: watchtower-state.json
46
+
47
+ # Thresholds (these are the defaults).
48
+ # max_vote_lag_slots: 100
49
+ # failures_to_alert: 2
50
+ # rpc_failures_to_alert: 4
51
+ # realert_minutes: 30
52
+ # double_active_moves: 2
53
+ # double_active_window_minutes: 15
54
+ # votes_interval_seconds: 30
55
+ # gossip_interval_seconds: 120
56
+ # rpc_timeout_seconds: 10
@@ -0,0 +1,9 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ require 'solana/ruby/watchtower'
5
+
6
+ # Line by line under systemd/journald, where stdout is a pipe.
7
+ $stdout.sync = true
8
+
9
+ exit Solana::Ruby::Watchtower::CLI.new(ARGV).run
@@ -0,0 +1,98 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'time'
4
+
5
+ module Solana::Ruby::Watchtower
6
+ # The alert state of one check.
7
+ #
8
+ # #observe records a result and says whether it calls for a message;
9
+ # #acknowledge records that the message went out. They are separate so that an
10
+ # alert the notifier failed to deliver is offered again on the next run
11
+ # instead of being marked as sent.
12
+ class CheckState
13
+ Alert = Data.define(:kind, :key, :text) # kind: :firing, :reminder or :resolved
14
+
15
+ ATTRIBUTES = %i[key alerting consecutive_failures failing_since last_alerted_at last_checked_at last_message data].freeze
16
+ TIMES = %i[failing_since last_alerted_at last_checked_at].freeze
17
+
18
+ attr_accessor(*ATTRIBUTES)
19
+
20
+ def initialize(key:, alerting: false, consecutive_failures: 0, failing_since: nil, last_alerted_at: nil,
21
+ last_checked_at: nil, last_message: nil, data: {})
22
+ @key = key.to_s
23
+ @alerting = alerting ? true : false
24
+ @consecutive_failures = consecutive_failures.to_i
25
+ @failing_since = time(failing_since)
26
+ @last_alerted_at = time(last_alerted_at)
27
+ @last_checked_at = time(last_checked_at)
28
+ @last_message = last_message
29
+ @data = data || {}
30
+ end
31
+
32
+ def self.from_h(hash)
33
+ new(**hash.transform_keys(&:to_sym).slice(*ATTRIBUTES))
34
+ end
35
+
36
+ # Times as ISO 8601 strings, ready for JSON.
37
+ def to_h
38
+ ATTRIBUTES.to_h do |attr|
39
+ value = public_send(attr)
40
+ [attr, TIMES.include?(attr) && value ? value.utc.iso8601 : value]
41
+ end
42
+ end
43
+
44
+ def alerting? = @alerting
45
+
46
+ def observe(ok:, message:, now:, failures_to_alert:, realert_after:)
47
+ self.last_message = message
48
+ self.last_checked_at = now
49
+
50
+ if ok
51
+ self.consecutive_failures = 0
52
+ # Kept until the recovery message is sent: it is what that message reports.
53
+ self.failing_since = nil unless alerting?
54
+ return unless alerting?
55
+
56
+ return Alert.new(kind: :resolved, key: key, text: "✅ RESOLVED #{key}: #{message} (failing #{since(now)})")
57
+ end
58
+
59
+ self.consecutive_failures += 1
60
+ self.failing_since ||= now
61
+ return if consecutive_failures < failures_to_alert
62
+
63
+ if !alerting?
64
+ Alert.new(kind: :firing, key: key, text: "🚨 #{key}: #{message}")
65
+ elsif last_alerted_at.nil? || now - last_alerted_at >= realert_after
66
+ Alert.new(kind: :reminder, key: key, text: "🚨 STILL FAILING #{key} (#{since(now)}): #{message}")
67
+ end
68
+ end
69
+
70
+ def acknowledge(alert, now)
71
+ if alert.kind == :resolved
72
+ self.alerting = false
73
+ self.failing_since = nil
74
+ self.last_alerted_at = nil
75
+ else
76
+ self.alerting = true
77
+ self.last_alerted_at = now
78
+ end
79
+ end
80
+
81
+ private
82
+
83
+ def since(now)
84
+ return 'briefly' if failing_since.nil?
85
+
86
+ minutes = ((now - failing_since) / 60).round
87
+ "since #{failing_since.utc.strftime('%H:%M')} UTC, #{minutes} min"
88
+ end
89
+
90
+ def time(value)
91
+ case value
92
+ when nil then nil
93
+ when String then Time.iso8601(value)
94
+ else value.to_time
95
+ end
96
+ end
97
+ end
98
+ end
@@ -0,0 +1,133 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'logger'
4
+ require 'optparse'
5
+
6
+ module Solana::Ruby::Watchtower
7
+ # solana-watchtower COMMAND [options]
8
+ #
9
+ # run run the checks on their intervals until SIGINT/SIGTERM
10
+ # check votes|gossip run one check once and exit (for cron or a systemd timer)
11
+ # status print each check's stored state
12
+ # notify-test send a test message through the configured notifiers
13
+ class CLI
14
+ USAGE = <<~TEXT
15
+ Usage: solana-watchtower COMMAND [options]
16
+
17
+ Commands:
18
+ run run the checks on their intervals until SIGINT/SIGTERM
19
+ check votes|gossip run one check once and exit
20
+ status print each check's stored state
21
+ notify-test send a test message through the configured notifiers
22
+
23
+ Options:
24
+ TEXT
25
+
26
+ def initialize(argv, out: $stdout, err: $stderr)
27
+ @argv = argv.dup
28
+ @out = out
29
+ @err = err
30
+ @config_path = ENV.fetch('WATCHTOWER_CONFIG', 'watchtower.yml')
31
+ @log_level = Logger::INFO
32
+ end
33
+
34
+ # Returns the process exit status.
35
+ def run
36
+ parser.parse!(@argv)
37
+ command = @argv.shift
38
+ case command
39
+ when 'run' then run_forever
40
+ when 'check' then check_once(@argv.shift)
41
+ when 'status' then status
42
+ when 'notify-test' then notify_test
43
+ when nil then @err.puts(parser.help) || 2
44
+ else
45
+ @err.puts "unknown command #{command.inspect}\n\n#{parser.help}"
46
+ 2
47
+ end
48
+ rescue ConfigError, OptionParser::ParseError => e
49
+ @err.puts "solana-watchtower: #{e.message}"
50
+ 2
51
+ end
52
+
53
+ private
54
+
55
+ def parser
56
+ @parser ||= OptionParser.new do |o|
57
+ o.banner = USAGE
58
+ o.on('-c', '--config FILE', "config file (default: $WATCHTOWER_CONFIG or #{@config_path})") { |v| @config_path = v }
59
+ o.on('-v', '--verbose', 'log every check result, not just alerts') { @log_level = Logger::DEBUG }
60
+ o.on('--version', 'print the version') do
61
+ @out.puts VERSION
62
+ exit 0
63
+ end
64
+ end
65
+ end
66
+
67
+ def config
68
+ raise ConfigError, "config file #{@config_path} not found (pass --config)" unless File.exist?(@config_path)
69
+
70
+ @config ||= Config.load(@config_path)
71
+ end
72
+
73
+ def logger
74
+ @logger ||= Logger.new(@out, progname: 'solana-watchtower', level: @log_level).tap do |l|
75
+ l.formatter = proc { |sev, time, _, msg| "#{time.utc.strftime('%FT%TZ')} #{sev} #{msg}\n" }
76
+ end
77
+ end
78
+
79
+ def monitor = @monitor ||= Monitor.new(config, logger: logger)
80
+
81
+ def run_forever
82
+ runner = Runner.new(monitor, logger: logger,
83
+ votes_every: config.votes_interval_seconds,
84
+ gossip_every: config.gossip_interval_seconds)
85
+ %w[INT TERM].each { |signal| trap(signal) { runner.stop! } }
86
+ logger.info("watching #{config.name}: vote account #{config.vote_account}, " \
87
+ "nodes #{config.nodes.map(&:name).join(', ')}, via #{config.rpc_urls.map { |u| URI(u).host }.join(', ')}")
88
+ runner.run
89
+ logger.info('stopped')
90
+ 0
91
+ end
92
+
93
+ def check_once(check)
94
+ raise ConfigError, "check needs one of: #{Monitor::CHECKS.join(', ')}" unless Monitor::CHECKS.include?(check)
95
+
96
+ monitor.run(check)
97
+ print_states(monitor.store.all.select { |s| belongs_to?(s.key, check) })
98
+ 0
99
+ end
100
+
101
+ def status
102
+ print_states(Stores::FileStore.new(config.state_file).all)
103
+ 0
104
+ end
105
+
106
+ def notify_test
107
+ notifier = Notifiers.build(config.notify, logger: logger)
108
+ if notifier.deliver("[#{config.name}] ✅ solana-watchtower test message")
109
+ @out.puts 'delivered'
110
+ 0
111
+ else
112
+ @err.puts 'delivery failed, see the log above'
113
+ 1
114
+ end
115
+ end
116
+
117
+ def belongs_to?(key, check)
118
+ check == 'votes' ? %w[rpc:votes voting vote_identity].include?(key) : !%w[rpc:votes voting vote_identity].include?(key)
119
+ end
120
+
121
+ def print_states(states)
122
+ if states.empty?
123
+ @out.puts "no state yet in #{config.state_file}"
124
+ return
125
+ end
126
+ width = states.map { |s| s.key.size }.max
127
+ states.sort_by(&:key).each do |s|
128
+ flag = s.alerting? ? 'ALERT' : (s.consecutive_failures.positive? ? 'fail ' : 'ok ')
129
+ @out.puts "#{flag} #{s.key.ljust(width)} #{s.last_message}"
130
+ end
131
+ end
132
+ end
133
+ end
@@ -0,0 +1,134 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'erb'
4
+ require 'yaml'
5
+
6
+ module Solana::Ruby::Watchtower
7
+ class ConfigError < Error; end
8
+
9
+ # The settings for one validator, from a hash or a YAML file.
10
+ #
11
+ # A file is passed through ERB first, so secrets can stay in the environment:
12
+ #
13
+ # bot_token: <%= ENV['TELEGRAM_BOT_TOKEN'] %>
14
+ #
15
+ # Everything is checked when the config is built, so a typo stops the monitor
16
+ # at startup rather than leaving it running blind.
17
+ class Config
18
+ Node = Data.define(:name, :ip, :unstaked_identity)
19
+
20
+ DEFAULTS = {
21
+ 'rpc_urls' => ['https://api.mainnet-beta.solana.com'],
22
+ 'rpc_timeout_seconds' => 10,
23
+ # Last vote this many slots behind the tip counts as not voting. Healthy
24
+ # lag is a handful of slots; the cluster's delinquency mark is 128.
25
+ 'max_vote_lag_slots' => 100,
26
+ # Consecutive bad vote checks before an alert fires.
27
+ 'failures_to_alert' => 2,
28
+ # All RPC endpoints failing, held back longer: public RPC rate-limits in bursts.
29
+ 'rpc_failures_to_alert' => 4,
30
+ 'realert_minutes' => 30,
31
+ # The staked identity's gossip address changing this many times within the
32
+ # window means two nodes are advertising it.
33
+ 'double_active_moves' => 2,
34
+ 'double_active_window_minutes' => 15,
35
+ 'votes_interval_seconds' => 30,
36
+ # getClusterNodes returns the whole cluster (well over a megabyte), so the
37
+ # gossip checks run less often than the vote check.
38
+ 'gossip_interval_seconds' => 120,
39
+ 'state_file' => 'watchtower-state.json',
40
+ 'notify' => {}
41
+ }.freeze
42
+
43
+ POSITIVE_INTEGERS = %w[
44
+ rpc_timeout_seconds max_vote_lag_slots failures_to_alert rpc_failures_to_alert realert_minutes
45
+ double_active_moves double_active_window_minutes votes_interval_seconds gossip_interval_seconds
46
+ ].freeze
47
+
48
+ PUBKEY = /\A[1-9A-HJ-NP-Za-km-z]{32,44}\z/
49
+
50
+ attr_reader :name, :vote_account, :identity, :nodes, :rpc_urls, :notify, :heartbeat_url, :state_file,
51
+ *POSITIVE_INTEGERS.map(&:to_sym)
52
+
53
+ # base_dir resolves a relative state_file; for a file, its directory.
54
+ def self.load(path)
55
+ text = ERB.new(File.read(path)).result
56
+ new(YAML.safe_load(text, aliases: true) || {}, base_dir: File.dirname(File.expand_path(path)))
57
+ rescue Psych::SyntaxError => e
58
+ raise ConfigError, "#{path}: #{e.message}"
59
+ end
60
+
61
+ def initialize(hash, base_dir: Dir.pwd)
62
+ settings = DEFAULTS.merge(stringify(hash))
63
+
64
+ @vote_account = pubkey!(settings, 'vote_account')
65
+ @identity = pubkey!(settings, 'identity')
66
+ @name = (settings['name'] || "#{@identity[0, 8]}…").to_s
67
+ @nodes = nodes!(settings['nodes'])
68
+ @rpc_urls = Array(settings['rpc_urls']).map { |u| u.to_s.strip }.reject(&:empty?)
69
+ raise ConfigError, 'rpc_urls is empty' if @rpc_urls.empty?
70
+
71
+ @rpc_urls.each { |u| url!(u, 'rpc_urls') }
72
+ POSITIVE_INTEGERS.each { |k| instance_variable_set(:"@#{k}", positive_integer!(settings, k)) }
73
+
74
+ @notify = settings['notify'] || {}
75
+ raise ConfigError, 'notify must be a mapping' unless @notify.is_a?(Hash)
76
+
77
+ @heartbeat_url = blank?(settings['heartbeat_url']) ? nil : url!(settings['heartbeat_url'], 'heartbeat_url')
78
+ @state_file = File.expand_path(settings['state_file'].to_s, base_dir)
79
+ end
80
+
81
+ private
82
+
83
+ def stringify(value)
84
+ case value
85
+ when Hash then value.to_h { |k, v| [k.to_s, stringify(v)] }
86
+ when Array then value.map { |v| stringify(v) }
87
+ else value
88
+ end
89
+ end
90
+
91
+ def nodes!(raw)
92
+ raise ConfigError, 'nodes must map each node name to its ip (and unstaked_identity for a standby)' unless raw.is_a?(Hash) && raw.any?
93
+
94
+ nodes = raw.map do |name, node|
95
+ raise ConfigError, "nodes.#{name} must be a mapping" unless node.is_a?(Hash)
96
+ raise ConfigError, "nodes.#{name}.ip is missing" if blank?(node['ip'])
97
+
98
+ unstaked = node['unstaked_identity']
99
+ unstaked = blank?(unstaked) ? nil : pubkey!(node, 'unstaked_identity', "nodes.#{name}.")
100
+ Node.new(name: name.to_s, ip: node['ip'].to_s, unstaked_identity: unstaked)
101
+ end
102
+ duplicate = nodes.group_by(&:ip).find { |_, same| same.size > 1 }
103
+ raise ConfigError, "nodes #{duplicate[1].map(&:name).join(' and ')} share ip #{duplicate[0]}" if duplicate
104
+
105
+ nodes
106
+ end
107
+
108
+ def pubkey!(settings, key, prefix = '')
109
+ value = settings[key].to_s
110
+ raise ConfigError, "#{prefix}#{key} is missing" if value.empty?
111
+ raise ConfigError, "#{prefix}#{key} is not a base58 public key: #{value.inspect}" unless value.match?(PUBKEY)
112
+
113
+ value
114
+ end
115
+
116
+ def positive_integer!(settings, key)
117
+ value = settings[key]
118
+ raise ConfigError, "#{key} must be a positive integer, got #{value.inspect}" unless value.is_a?(Integer) && value.positive?
119
+
120
+ value
121
+ end
122
+
123
+ def url!(value, key)
124
+ uri = URI.parse(value.to_s)
125
+ raise ConfigError, "#{key}: #{value.to_s.inspect} is not an http(s) URL" unless uri.is_a?(URI::HTTP) && uri.host
126
+
127
+ value.to_s
128
+ rescue URI::InvalidURIError
129
+ raise ConfigError, "#{key}: #{value.to_s.inspect} is not an http(s) URL"
130
+ end
131
+
132
+ def blank?(value) = value.nil? || value.to_s.strip.empty?
133
+ end
134
+ end