cronwatch 0.10.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +18 -9
- data/lib/cronwatch/alerts/datadog.rb +1 -0
- data/lib/cronwatch/alerts/discord.rb +22 -3
- data/lib/cronwatch/alerts/email.rb +2 -0
- data/lib/cronwatch/alerts/honeybadger.rb +1 -0
- data/lib/cronwatch/alerts/postmark.rb +1 -0
- data/lib/cronwatch/alerts/provider.rb +2 -0
- data/lib/cronwatch/alerts/resend.rb +1 -0
- data/lib/cronwatch/alerts/rollbar.rb +1 -0
- data/lib/cronwatch/alerts/sentry.rb +1 -0
- data/lib/cronwatch/alerts/sigv4.rb +2 -0
- data/lib/cronwatch/alerts/slack.rb +1 -0
- data/lib/cronwatch/alerts/twilio.rb +1 -0
- data/lib/cronwatch/alerts/webhook.rb +18 -5
- data/lib/cronwatch/client.rb +66 -17
- data/lib/cronwatch/cron_pattern.rb +17 -8
- data/lib/cronwatch/deprecation.rb +18 -0
- data/lib/cronwatch/duration.rb +2 -0
- data/lib/cronwatch/environment.rb +93 -17
- data/lib/cronwatch/evaluate.rb +41 -10
- data/lib/cronwatch/flight.rb +2 -0
- data/lib/cronwatch/format.rb +1 -0
- data/lib/cronwatch/http.rb +21 -5
- data/lib/cronwatch/job.rb +2 -0
- data/lib/cronwatch/js.rb +74 -1
- data/lib/cronwatch/monitored.rb +6 -1
- data/lib/cronwatch/output.rb +3 -0
- data/lib/cronwatch/pg_cron.rb +22 -3
- data/lib/cronwatch/rails/active_job.rb +2 -2
- data/lib/cronwatch/schedule.rb +25 -1
- data/lib/cronwatch/scheduler.rb +6 -1
- data/lib/cronwatch/serialize.rb +1 -0
- data/lib/cronwatch/stats.rb +1 -0
- data/lib/cronwatch/stores/active_record.rb +36 -18
- data/lib/cronwatch/ticker.rb +2 -0
- data/lib/cronwatch/triage/anthropic.rb +2 -0
- data/lib/cronwatch/types.rb +148 -31
- data/lib/cronwatch/version.rb +1 -1
- data/lib/cronwatch/walker.rb +2 -0
- data/lib/cronwatch/web/app.rb +108 -33
- data/lib/cronwatch/web/html.rb +6 -3
- data/lib/cronwatch/web/origin.rb +2 -0
- data/lib/cronwatch/web/pwa.rb +2 -0
- data/lib/cronwatch/web/timeline.rb +2 -0
- data/lib/cronwatch/web.rb +1 -2
- data/lib/cronwatch/zone.rb +2 -0
- data/lib/cronwatch.rb +2 -0
- data/lib/generators/cronwatch/install/install_generator.rb +1 -1
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9f5d4d303ae53fd20968b2f416e805a3f840b07b152004878147c3d3c3e4e899
|
|
4
|
+
data.tar.gz: 184ac01a139983fdabc2344586a4bc6c6669b987864a8c84d7f7b8f8bd90909d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 333767b7bcffda2b75c1349644c7fa870c371c9a5c1ee0491083effcdbb1030c54dfc561a36a883b8588411caa59b19ca4c36c06638c61c5652917c29f1252fa
|
|
7
|
+
data.tar.gz: 0ecbafab015195219f3dfb53e03dd202766342fc11195b7d97cdac4610821940a2e8d777898834aa8b015236b5b5a1a11eb0f1ecd424c42f52a4b84eddfb14b1
|
data/README.md
CHANGED
|
@@ -47,10 +47,10 @@ NIGHTLY.run do |job|
|
|
|
47
47
|
job.metric(:cost, 1.2) # watched against budgets and baselines
|
|
48
48
|
end
|
|
49
49
|
|
|
50
|
-
CW.
|
|
50
|
+
CW.start_checking # checks for missed and stuck runs every minute, in a background thread
|
|
51
51
|
```
|
|
52
52
|
|
|
53
|
-
`run` returns what the block returns and raises what it raises, after the run is recorded. A script run from crontab exits when it is done, so instead of `
|
|
53
|
+
`run` returns what the block returns and raises what it raises, after the run is recorded. A script run from crontab exits when it is done, so instead of `start_checking`, add a second crontab line that declares the jobs and calls `CW.check` every five minutes.
|
|
54
54
|
|
|
55
55
|
A run that starts in one call and ends in another (a job that hands work to a queue, a webhook that reports back later) is one run too:
|
|
56
56
|
|
|
@@ -64,7 +64,7 @@ run.finish # or run.fail(error); finish("text") and fin
|
|
|
64
64
|
|
|
65
65
|
Neither raises for the store: failures go to `on_error`, and a finish of a run already finished, not found, or of another job records nothing and returns nil. A run is judged once, however many processes finish it at once, and a store that fails during `finish` leaves the handle active to finish again. A run never finished is marked stuck after the job's `timeout`.
|
|
66
66
|
|
|
67
|
-
Client options: `store`, `alerts`, `triage`, `cron_secret`, `retention` (default `"30d"`), `defaults`, `sources` (where runs the process does not wrap come from, such as pg_cron), `redact` (default: blank values that look like secrets; `false` keeps output as logged, or pass a callable), `deliver` (`:now` by default; `:check` queues alerts for another process's check to send, for a worker that cannot reach Slack), `on_error` and `now`. Methods: `job`, `run`, `resume_run`, `record_run`, `check`, `
|
|
67
|
+
Client options: `store`, `alerts`, `triage`, `cron_secret`, `retention` (default `"30d"`), `defaults`, `sources` (where runs the process does not wrap come from, such as pg_cron), `redact` (default: blank values that look like secrets; `false` keeps output as logged, or pass a callable), `deliver` (`:now` by default; `:check` queues alerts for another process's check to send, for a worker that cannot reach Slack), `on_error` and `now`. Methods: `job`, `run`, `resume_run`, `record_run`, `check`, `start_checking`/`stop`, `silence(name, for: "2h")`/`unsilence`, `forget`, `jobs`, `jobs_with_runs`, `job_summary`, `runs`, `get_run`, `defined_jobs`, `close`. [cronwatch.dev/docs/ruby](https://cronwatch.dev/docs/ruby/#api) has each one.
|
|
68
68
|
|
|
69
69
|
## Rails
|
|
70
70
|
|
|
@@ -100,7 +100,7 @@ class NightlyReportJob < ApplicationJob
|
|
|
100
100
|
end
|
|
101
101
|
```
|
|
102
102
|
|
|
103
|
-
Every `perform` is recorded as a run with the trigger `"active_job"
|
|
103
|
+
Every `perform` is recorded as a run with the trigger `"active-job"` (runs recorded before 1.0 carry `"active_job"`). The name defaults to the class name without `Job`, dasherized, with `::` as `:` (`Reports::NightlyJob` is `reports:nightly`); pass `name:` to choose another. Jobs are declared once the app has booted, so a check knows a job that has never run. A job that raises still raises after the run is recorded, so ActiveJob retries and your error reporter see it as before.
|
|
104
104
|
|
|
105
105
|
### Sidekiq
|
|
106
106
|
|
|
@@ -164,11 +164,11 @@ Or from a crontab: `bin/rails cronwatch:check`.
|
|
|
164
164
|
```ruby
|
|
165
165
|
# config/routes.rb
|
|
166
166
|
Rails.application.routes.draw do
|
|
167
|
-
mount Cronwatch
|
|
167
|
+
mount Cronwatch.client.routes => "/cronwatch"
|
|
168
168
|
end
|
|
169
169
|
```
|
|
170
170
|
|
|
171
|
-
Set `CRONWATCH_TOKEN` to a long random string and open `/cronwatch?token=<it>` once; the browser keeps a cookie. Without a token, while the environment (`Rails.env
|
|
171
|
+
Set `CRONWATCH_TOKEN` to a long random string and open `/cronwatch?token=<it>` once; the browser keeps a cookie. Without a token, while the environment (the first of `CRONWATCH_ENV`, `APP_ENV`, then `Rails.env` (when Rails is loaded), `RAILS_ENV` or `RACK_ENV` that holds more than spaces, trimmed and lowercased) is `development`, `dev`, `local`, `test` or `testing`, it makes a token of its own and prints a sign-in link to standard output on its first request (open it once; unless `origin:` is set or the request's host is loopback, the link leaves the host out, since a client chooses it); anywhere else it answers 503. Puma, Unicorn, Thin and rackup set `RACK_ENV` to `development` when nothing names an environment, production included, so under one of them `RACK_ENV=development` alone does not count: `CRONWATCH_ENV`, `APP_ENV`, `Rails.env` or `RAILS_ENV` must say development (or `RACK_ENV` say `test`, `dev` or `local`). To rely on the app's own sign in, mount it behind that (Devise's `authenticate` block, or a routing constraint) and pass `token: nil`. The URLs, JSON shapes, headers and CSRF rules are the SDK's, so the MCP server reads it unchanged. The request's origin, which the CSRF check compares against, is what Rack reports and so already follows `X-Forwarded-Host` and `X-Forwarded-Proto`; pass `origin: "https://app.example.com"` to pin it instead. `GET /cronwatch/api/check` with a bearer (the token or `CRON_SECRET`) runs the check, for an outside cron.
|
|
172
172
|
|
|
173
173
|
There is no `handler()` as in the TypeScript SDK: for a job triggered over HTTP, wrap the controller action's body in `CW.job(...).run` (declared once, at boot) and check the bearer in the controller.
|
|
174
174
|
|
|
@@ -176,10 +176,19 @@ In any other Rack app:
|
|
|
176
176
|
|
|
177
177
|
```ruby
|
|
178
178
|
# config.ru
|
|
179
|
-
|
|
180
|
-
map("/cronwatch") { run Cronwatch::Web.new(CW) }
|
|
179
|
+
map("/cronwatch") { run CW.routes }
|
|
181
180
|
```
|
|
182
181
|
|
|
182
|
+
`client.routes(**options)` (`token:`, `base_path:`, `origin:`) is the one way to build the dashboard, as `routes()` is in every language.
|
|
183
|
+
|
|
184
|
+
### Public and internal
|
|
185
|
+
|
|
186
|
+
Public means what this README and the docs at cronwatch.dev name. Everything else is internal and may change in any release: anything marked `@api private` in the source (`Evaluate`, `Schedule`, `Format`, `JS`, `HTTP` and the dashboard's helpers among them), and the constants made private with `private_constant`.
|
|
187
|
+
|
|
188
|
+
### Deprecated
|
|
189
|
+
|
|
190
|
+
These still work through every 1.x release and go in 2.0, each with a warning in Ruby's deprecation category (shown under `ruby -w` or `-W:deprecated`): `Cronwatch::Web.new(client, **options)` (use `client.routes(**options)`), `client.start(every)` (use `client.start_checking(every)`), `client.run(id)` without a block (use `client.get_run(id)`) and `client.silence(name, "2h")` (use `client.silence(name, for: "2h")`). The ActiveJob trigger is `active-job` from 1.0; runs recorded before keep `active_job`.
|
|
191
|
+
|
|
183
192
|
## Stores
|
|
184
193
|
|
|
185
194
|
- `Cronwatch::Stores::Memory.new`: the default. Nothing survives a restart.
|
|
@@ -204,7 +213,7 @@ Cronwatch::Alerts::Custom.new("pagerduty") do |alert|
|
|
|
204
213
|
end
|
|
205
214
|
```
|
|
206
215
|
|
|
207
|
-
Every alert goes to every channel. A channel that raises, or takes longer than 15 seconds, goes to `on_error` and never blocks the others. Each condition alerts once when it opens and once more when a clean run closes it; there are no repeat alerts. The webhook
|
|
216
|
+
Every alert goes to every channel. A channel that raises, or takes longer than 15 seconds, goes to `on_error` and never blocks the others. Each condition alerts once when it opens and once more when a clean run closes it; there are no repeat alerts. The webhook's body is the alert's JSON with `"schema": 1` first (its [JSON Schema](https://cronwatch.dev/schemas/webhook/1.json)); parse the fields, not `title` and `message`, whose wording may change. It signs the body with `X-CronWatch-Signature: sha256=<hex>`, the HMAC-SHA256 of the raw body, which a receiver checks with `Rack::Utils.secure_compare("sha256=#{Cronwatch::Alerts::Webhook.signature(secret, raw_body)}", header)`.
|
|
208
217
|
|
|
209
218
|
## Triage
|
|
210
219
|
|
|
@@ -12,6 +12,7 @@ module Cronwatch
|
|
|
12
12
|
# The longest embed description Discord takes. The title (under 256)
|
|
13
13
|
# and it stay well inside the embed's 6000.
|
|
14
14
|
DESCRIPTION_MAX = 4096
|
|
15
|
+
private_constant :COLOR, :DESCRIPTION_MAX
|
|
15
16
|
|
|
16
17
|
attr_reader :name
|
|
17
18
|
|
|
@@ -45,10 +46,28 @@ module Cronwatch
|
|
|
45
46
|
# cap, and escaping can grow both, so the whole is held to
|
|
46
47
|
# DESCRIPTION_MAX (in UTF-16 units) by cutting the message's block,
|
|
47
48
|
# never the triage: Discord refuses a longer one on every retry.
|
|
49
|
+
#
|
|
50
|
+
# Where a cap cuts a surrogate pair the high half is kept, as the SDK's
|
|
51
|
+
# slice keeps it, and written as JSON.stringify writes it (see
|
|
52
|
+
# JS.lone_surrogate). The cut of the block to the room left leaves out
|
|
53
|
+
# a pair or a lone half, as the SDK's cut does.
|
|
48
54
|
def self.embed_description(alert)
|
|
49
|
-
triage =
|
|
50
|
-
|
|
51
|
-
|
|
55
|
+
triage = ""
|
|
56
|
+
triage_units = 0
|
|
57
|
+
if alert.triage && !alert.triage.empty?
|
|
58
|
+
head, lone = JS.slice16(alert.triage, 1000)
|
|
59
|
+
triage = "\n**Triage:** #{escape_markdown(head)}"
|
|
60
|
+
triage_units = JS.length16(triage)
|
|
61
|
+
if lone
|
|
62
|
+
triage += JS.lone_surrogate(lone)
|
|
63
|
+
triage_units += 1
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
room = DESCRIPTION_MAX - "```\n".length - "\n```".length - triage_units
|
|
67
|
+
head, lone = JS.slice16(alert.message, 3800)
|
|
68
|
+
block = code_block_safe(head)
|
|
69
|
+
block = lone && JS.length16(block) + 1 <= room ? block + JS.lone_surrogate(lone) : JS.head16(block, room)
|
|
70
|
+
"```\n" + block + "\n```" + triage
|
|
52
71
|
end
|
|
53
72
|
|
|
54
73
|
# Breaks up ``` so text inside a code block cannot close it.
|
|
@@ -11,6 +11,8 @@ module Cronwatch
|
|
|
11
11
|
# to: one address or several
|
|
12
12
|
# subject_prefix: put in front of the title in the subject, "[prod]" say
|
|
13
13
|
# link: ->(alert) { "https://app.example.com/cronwatch/jobs/#{alert.job}" }
|
|
14
|
+
#
|
|
15
|
+
# @api private
|
|
14
16
|
module Email
|
|
15
17
|
Message = Struct.new(:from, :to, :subject, :text, :html, keyword_init: true)
|
|
16
18
|
|
|
@@ -11,6 +11,7 @@ module Cronwatch
|
|
|
11
11
|
missed: "CronWatch::Missed", failed: "CronWatch::Failed", stuck: "CronWatch::Stuck", slow: "CronWatch::Slow",
|
|
12
12
|
over_budget: "CronWatch::OverBudget", recovered: "CronWatch::Recovered",
|
|
13
13
|
}.freeze
|
|
14
|
+
private_constant :CLASS
|
|
14
15
|
|
|
15
16
|
attr_reader :name
|
|
16
17
|
|
|
@@ -10,6 +10,8 @@ module Cronwatch
|
|
|
10
10
|
# SDK's alerts/shared.ts: the POST that names the provider and the URL's
|
|
11
11
|
# origin on failure with every secret cut out, the stable alert id, and
|
|
12
12
|
# the run summary trackers attach. Standard library only.
|
|
13
|
+
#
|
|
14
|
+
# @api private
|
|
13
15
|
module Provider
|
|
14
16
|
module_function
|
|
15
17
|
|
|
@@ -13,6 +13,7 @@ module Cronwatch
|
|
|
13
13
|
# Envelopes: https://develop.sentry.dev/sdk/data-model/envelopes/
|
|
14
14
|
# Event payload: https://develop.sentry.dev/sdk/data-model/event-payloads/
|
|
15
15
|
class Sentry
|
|
16
|
+
# @api private
|
|
16
17
|
Dsn = Struct.new(:endpoint, :public_key, keyword_init: true)
|
|
17
18
|
|
|
18
19
|
attr_reader :name
|
|
@@ -9,6 +9,8 @@ module Cronwatch
|
|
|
9
9
|
# alerts/sigv4.ts. Spec:
|
|
10
10
|
# https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html
|
|
11
11
|
# Checked against the AWS SigV4 test suite (test/provider_channels_test.rb).
|
|
12
|
+
#
|
|
13
|
+
# @api private
|
|
12
14
|
module SigV4
|
|
13
15
|
SPACES = Regexp.new("[#{JS::WHITESPACE}]+")
|
|
14
16
|
|
|
@@ -4,10 +4,11 @@ require "openssl"
|
|
|
4
4
|
|
|
5
5
|
module Cronwatch
|
|
6
6
|
module Alerts
|
|
7
|
-
# POSTs the alert as JSON to any URL. The body is
|
|
8
|
-
# { type, run, details, job, definition, title,
|
|
7
|
+
# POSTs the alert as JSON to any URL. The body is `"schema": 1` and then
|
|
8
|
+
# the alert's JSON: { schema, type, run, details, job, definition, title,
|
|
9
|
+
# message, at, triage }, described by https://cronwatch.dev/schemas/webhook/1.json.
|
|
9
10
|
# With a secret, each request carries `X-CronWatch-Signature: sha256=<hex>`,
|
|
10
|
-
# the HMAC-SHA256 of the raw body, so the receiver can verify it. Header
|
|
11
|
+
# the HMAC-SHA256 of the raw body (Webhook.signature), so the receiver can verify it. Header
|
|
11
12
|
# values are trimmed. A redirect is an error, not followed (the headers
|
|
12
13
|
# and the signature would go with it): point the url at where the
|
|
13
14
|
# receiver really is.
|
|
@@ -24,13 +25,25 @@ module Cronwatch
|
|
|
24
25
|
@http = http
|
|
25
26
|
end
|
|
26
27
|
|
|
28
|
+
# The version of the body's shape, its first key. It goes up only in a
|
|
29
|
+
# major release; fields are added within one.
|
|
30
|
+
SCHEMA = 1
|
|
31
|
+
private_constant :SCHEMA
|
|
32
|
+
|
|
33
|
+
# The HMAC-SHA256 of `body` with `secret`, as lowercase hex, without the
|
|
34
|
+
# `sha256=` the header puts before it. A receiver compares it with the
|
|
35
|
+
# header in constant time (Rack::Utils.secure_compare).
|
|
36
|
+
def self.signature(secret, body)
|
|
37
|
+
OpenSSL::HMAC.hexdigest("SHA256", secret.to_s, body.to_s)
|
|
38
|
+
end
|
|
39
|
+
|
|
27
40
|
def call(alert)
|
|
28
|
-
body = JS.json(alert.to_h)
|
|
41
|
+
body = JS.json({ "schema" => SCHEMA }.merge(alert.to_h))
|
|
29
42
|
headers = { "content-type" => "application/json", "user-agent" => "cronwatch" }
|
|
30
43
|
# A pasted Authorization value often carries a stray space or newline, which a header would refuse.
|
|
31
44
|
@headers.each { |name, value| headers[name.to_s] = value.is_a?(String) ? JS.trim(value) : value }
|
|
32
45
|
if @secret && !@secret.empty?
|
|
33
|
-
headers["x-cronwatch-signature"] = "sha256=#{
|
|
46
|
+
headers["x-cronwatch-signature"] = "sha256=#{Webhook.signature(@secret, body)}"
|
|
34
47
|
end
|
|
35
48
|
response = @http.post(@url, body, headers)
|
|
36
49
|
# Only the origin: a webhook URL's path or query often is the credential.
|
data/lib/cronwatch/client.rb
CHANGED
|
@@ -42,8 +42,16 @@ module Cronwatch
|
|
|
42
42
|
SILENCE_OPTIONS = %i[for].freeze
|
|
43
43
|
# Run ids that start with this belong to the pg_cron source (Sources::PgCron).
|
|
44
44
|
RESERVED_RUN_ID_PREFIX = "pgcron:"
|
|
45
|
+
# The longest run id, in UTF-16 code units (JavaScript's string length):
|
|
46
|
+
# what start, resume and record_run take, and every store holds.
|
|
47
|
+
MAX_RUN_ID = 200
|
|
48
|
+
private_constant :CHANNEL_TIMEOUT_MS, :DEFAULT_OPTIONS, :FORK_LOCK, :HISTORY_MAX, :HISTORY_PAGE, :MAX_RUN_ID,
|
|
49
|
+
:PRUNE_INTERVAL_MS, :RESERVED_RUN_ID_PREFIX, :RETRY_BUDGET_MS, :SILENCE_OPTIONS, :STATE_ATTEMPTS,
|
|
50
|
+
:TRIAGE_TIMEOUT_MS
|
|
45
51
|
|
|
46
52
|
# What execute returns: the recorded run, and the block's own outcome.
|
|
53
|
+
#
|
|
54
|
+
# @api private
|
|
47
55
|
ExecuteResult = Struct.new(:run, :result, :error, :threw, keyword_init: true)
|
|
48
56
|
# What a channel's call receives with each alert (the SDK's ChannelContext).
|
|
49
57
|
# on_error(error) reports a problem that did not stop the alert going
|
|
@@ -73,7 +81,9 @@ module Cronwatch
|
|
|
73
81
|
# alerts: where alerts go: objects with #call(alert) and #name. Defaults to the console.
|
|
74
82
|
# triage: a callable taking a TriageContext and returning a short diagnosis, added to every alert but recoveries.
|
|
75
83
|
# cron_secret: a second bearer Cronwatch::Web accepts for /api/check, for an outside cron. Defaults to
|
|
76
|
-
# ENV["CRON_SECRET"]; an empty string
|
|
84
|
+
# ENV["CRON_SECRET"]; an empty string or one of only whitespace, given here or in the
|
|
85
|
+
# variable, counts as unset. Pass nil for none. Anything but a String or nil (false, a
|
|
86
|
+
# number, a Symbol) raises TypeError.
|
|
77
87
|
# retention: how long finished runs are kept. Default "30d".
|
|
78
88
|
# defaults: grace, timeout, timezone and failures_before_alert applied to every job unless it sets its own.
|
|
79
89
|
# redact: applied to every run's output and error before it is stored, shown or sent to an alert
|
|
@@ -98,8 +108,11 @@ module Cronwatch
|
|
|
98
108
|
raise ArgumentError, "a source must respond to sync(host)" unless source.respond_to?(:sync)
|
|
99
109
|
end
|
|
100
110
|
@triage = triage
|
|
101
|
-
secret
|
|
102
|
-
@cron_secret =
|
|
111
|
+
# A blank secret, given or read, counts as unset; one given that is not a String raises.
|
|
112
|
+
@cron_secret =
|
|
113
|
+
if cron_secret.equal?(UNSET) then Environment.secret("CRON_SECRET")
|
|
114
|
+
else Environment.secret_option(cron_secret, "cron_secret")
|
|
115
|
+
end
|
|
103
116
|
@retention_ms = Duration.parse(retention || "30d", "retention")
|
|
104
117
|
@defaults = (defaults || {}).transform_keys(&:to_sym)
|
|
105
118
|
unknown = @defaults.keys - DEFAULT_OPTIONS
|
|
@@ -128,7 +141,7 @@ module Cronwatch
|
|
|
128
141
|
@registry = Mutex.new
|
|
129
142
|
@ready = false
|
|
130
143
|
@last_prune_at = 0
|
|
131
|
-
# Seconds before
|
|
144
|
+
# Seconds before start_checking's first check, and how long a channel or triage may take. Tests shorten them.
|
|
132
145
|
@first_tick_s = 1.0
|
|
133
146
|
@channel_timeout_ms = CHANNEL_TIMEOUT_MS
|
|
134
147
|
@triage_timeout_ms = TRIAGE_TIMEOUT_MS
|
|
@@ -158,13 +171,16 @@ module Cronwatch
|
|
|
158
171
|
JobHandle.new(self, definition)
|
|
159
172
|
end
|
|
160
173
|
|
|
161
|
-
#
|
|
162
|
-
#
|
|
163
|
-
#
|
|
174
|
+
# Run a job by name without keeping a handle, declaring it on first use
|
|
175
|
+
# (or again, when options are given).
|
|
176
|
+
#
|
|
177
|
+
# Without a block it reads the run with this id, as get_run does: that
|
|
178
|
+
# form is deprecated (use get_run), still works through 1.x and goes in 2.0.
|
|
164
179
|
def run(name_or_id, **options, &block)
|
|
165
180
|
unless block
|
|
166
181
|
raise ArgumentError, "run(#{name_or_id.inspect}, ...) needs a block; without one, run(id) reads a run" if options.any?
|
|
167
182
|
|
|
183
|
+
Deprecation.warn("run(id) without a block", "get_run(id)")
|
|
168
184
|
return get_run(name_or_id)
|
|
169
185
|
end
|
|
170
186
|
|
|
@@ -327,7 +343,12 @@ module Cronwatch
|
|
|
327
343
|
input = run.is_a?(Run) ? run : Run.from_h(run.is_a?(Hash) ? run.to_h { |k, v| [Naming.camel(k), v] } : run)
|
|
328
344
|
declared = @registry.synchronize { @definitions[input.job] }
|
|
329
345
|
raise ArgumentError, "record_run: job \"#{input.job}\" is not declared; call job first" unless declared
|
|
330
|
-
|
|
346
|
+
# The longest id start takes, and every store holds.
|
|
347
|
+
unless input.id.is_a?(String) && !input.id.empty? && JS.length16(input.id) <= MAX_RUN_ID
|
|
348
|
+
got = input.id.is_a?(String) ? "#{JS.length16(input.id)} characters" : input.id.class.to_s
|
|
349
|
+
raise ArgumentError, "record_run: run ids must be 1 to #{MAX_RUN_ID} characters (got #{got}; job \"#{input.job}\")"
|
|
350
|
+
end
|
|
351
|
+
if input.id.include?("\0")
|
|
331
352
|
raise ArgumentError, "record_run: run ids cannot contain a NUL character (job \"#{input.job}\")"
|
|
332
353
|
end
|
|
333
354
|
# Refused as JobContext#metric refuses them: a store keeps NaN and Infinity as null.
|
|
@@ -388,7 +409,7 @@ module Cronwatch
|
|
|
388
409
|
end
|
|
389
410
|
|
|
390
411
|
# Look for missed and stuck runs across every job, send alerts, retry
|
|
391
|
-
# alerts no channel accepted, and prune old runs. Call it from
|
|
412
|
+
# alerts no channel accepted, and prune old runs. Call it from start_checking, a
|
|
392
413
|
# scheduled job (Cronwatch::CheckJob), or by hand. Concurrent calls share
|
|
393
414
|
# one check.
|
|
394
415
|
def check
|
|
@@ -456,12 +477,15 @@ module Cronwatch
|
|
|
456
477
|
|
|
457
478
|
# Stop alerts for a job for a while. State keeps updating underneath. The
|
|
458
479
|
# end is a whole millisecond, held at 2**53 - 1 (see Evaluate.silence_end).
|
|
459
|
-
# silence("nightly-report", for: "2h")
|
|
480
|
+
# silence("nightly-report", for: "2h")
|
|
481
|
+
# The duration as a second argument, silence("nightly-report", "2h"), is
|
|
482
|
+
# deprecated: it still works through 1.x and goes in 2.0.
|
|
460
483
|
def silence(name, duration = nil, **options)
|
|
461
484
|
unknown = options.keys - SILENCE_OPTIONS
|
|
462
485
|
raise ArgumentError, "silence takes for:, not #{unknown.map(&:inspect).join(", ")}" if unknown.any?
|
|
463
486
|
raise ArgumentError, "silence takes a duration or for:, not both" if !duration.nil? && options.key?(:for)
|
|
464
487
|
|
|
488
|
+
Deprecation.warn("silence(name, duration)", "silence(name, for: duration)") unless duration.nil?
|
|
465
489
|
duration = options[:for] if duration.nil?
|
|
466
490
|
ms = Duration.parse(duration, "silence duration")
|
|
467
491
|
patch_state(name) { |state| state.silenced_until = Evaluate.silence_end(now, ms) }
|
|
@@ -489,8 +513,9 @@ module Cronwatch
|
|
|
489
513
|
# processes. Default every minute; the first check comes after a second.
|
|
490
514
|
# Calling it again while it runs does nothing, and a different interval
|
|
491
515
|
# is reported to on_error and ignored: stop first to change it. A forked
|
|
492
|
-
# child (Puma, Unicorn, Sidekiq) has no thread, so call
|
|
493
|
-
|
|
516
|
+
# child (Puma, Unicorn, Sidekiq) has no thread, so call start_checking
|
|
517
|
+
# there. (A job's start opens a run; this starts the checks.)
|
|
518
|
+
def start_checking(every = "1m")
|
|
494
519
|
after_fork_check
|
|
495
520
|
ms = [5_000, Duration.parse(every, "check interval")].max
|
|
496
521
|
@ticker_lock.synchronize do
|
|
@@ -498,8 +523,8 @@ module Cronwatch
|
|
|
498
523
|
@ticker = nil if @ticker && !@ticker.alive?
|
|
499
524
|
if @ticker
|
|
500
525
|
unless @ticker_ms == ms
|
|
501
|
-
report(ArgumentError.new("
|
|
502
|
-
"call stop first to change it"), "
|
|
526
|
+
report(ArgumentError.new("start_checking(#{every.inspect}) ignored: already checking every #{Duration.format(@ticker_ms)}; " \
|
|
527
|
+
"call stop first to change it"), "start_checking")
|
|
503
528
|
end
|
|
504
529
|
return nil
|
|
505
530
|
end
|
|
@@ -507,7 +532,7 @@ module Cronwatch
|
|
|
507
532
|
@ticker_ms = ms
|
|
508
533
|
if @defer_delivery && !@warned_deferred_start
|
|
509
534
|
@warned_deferred_start = true
|
|
510
|
-
warn '[cronwatch]
|
|
535
|
+
warn '[cronwatch] start_checking was called with deliver: "check", so these checks send no alerts. ' \
|
|
511
536
|
'Another process must run checks with deliver: "now" (the default) to send them.'
|
|
512
537
|
end
|
|
513
538
|
@ticker = Ticker.new(ms / 1000.0, @first_tick_s) do
|
|
@@ -522,6 +547,26 @@ module Cronwatch
|
|
|
522
547
|
nil
|
|
523
548
|
end
|
|
524
549
|
|
|
550
|
+
# start_checking, under its former name. Deprecated, since a job's start
|
|
551
|
+
# opens a run: it still works through 1.x and goes in 2.0.
|
|
552
|
+
def start(every = "1m")
|
|
553
|
+
Deprecation.warn("client.start(every)", "client.start_checking(every)")
|
|
554
|
+
start_checking(every)
|
|
555
|
+
end
|
|
556
|
+
|
|
557
|
+
# The dashboard and JSON API for this client, as a Rack app to mount:
|
|
558
|
+
#
|
|
559
|
+
# mount Cronwatch.client.routes => "/cronwatch" # config/routes.rb
|
|
560
|
+
# run CW.routes # config.ru
|
|
561
|
+
#
|
|
562
|
+
# Options: token:, base_path: and origin: (see Cronwatch::Web). Needs the
|
|
563
|
+
# rack gem, which every Rails app has.
|
|
564
|
+
def routes(**options)
|
|
565
|
+
require_relative "web"
|
|
566
|
+
Web.build(self, **options)
|
|
567
|
+
end
|
|
568
|
+
|
|
569
|
+
# Stop the checks start_checking began.
|
|
525
570
|
def stop
|
|
526
571
|
stop_ticker
|
|
527
572
|
nil
|
|
@@ -1076,9 +1121,9 @@ module Cronwatch
|
|
|
1076
1121
|
|
|
1077
1122
|
# Raises for a run id no store could hold, or one reserved for the pg_cron source.
|
|
1078
1123
|
def check_run_id(job, id, method)
|
|
1079
|
-
unless id.is_a?(String) && !id.empty? && JS.length16(id) <=
|
|
1124
|
+
unless id.is_a?(String) && !id.empty? && JS.length16(id) <= MAX_RUN_ID
|
|
1080
1125
|
got = id.is_a?(String) ? "#{JS.length16(id)} characters" : id.class.to_s
|
|
1081
|
-
raise ArgumentError, "job \"#{job}\": #{method}() needs a run id of 1 to
|
|
1126
|
+
raise ArgumentError, "job \"#{job}\": #{method}() needs a run id of 1 to #{MAX_RUN_ID} characters (got #{got})"
|
|
1082
1127
|
end
|
|
1083
1128
|
# Postgres refuses NUL in text, so no store could hold such an id.
|
|
1084
1129
|
raise ArgumentError, "job \"#{job}\": #{method}() cannot take a run id containing a NUL character" if id.include?("\0")
|
|
@@ -1123,6 +1168,8 @@ module Cronwatch
|
|
|
1123
1168
|
|
|
1124
1169
|
# Alerts written with the state that opened their conditions, and how
|
|
1125
1170
|
# many older ones the queue let go.
|
|
1171
|
+
#
|
|
1172
|
+
# @api private
|
|
1126
1173
|
Outbox = Struct.new(:alerts, :dropped)
|
|
1127
1174
|
|
|
1128
1175
|
# An evaluation as it is written: its drafts composed into alerts and held
|
|
@@ -1309,6 +1356,8 @@ module Cronwatch
|
|
|
1309
1356
|
end
|
|
1310
1357
|
|
|
1311
1358
|
# The wall-clock milliseconds one check has spent retrying, across its jobs.
|
|
1359
|
+
#
|
|
1360
|
+
# @api private
|
|
1312
1361
|
RetryBudget = Struct.new(:spent_ms)
|
|
1313
1362
|
|
|
1314
1363
|
# Send the alerts that no channel accepted last time, once each, oldest
|
|
@@ -7,6 +7,8 @@ require_relative "walker"
|
|
|
7
7
|
module Cronwatch
|
|
8
8
|
# A cron expression's fields, and the next time they match. What is
|
|
9
9
|
# accepted, and which times match, follow croner; Fugit parses the values.
|
|
10
|
+
#
|
|
11
|
+
# @api private
|
|
10
12
|
class CronPattern
|
|
11
13
|
NICKNAMES = {
|
|
12
14
|
"@yearly" => "0 0 1 1 *", "@annually" => "0 0 1 1 *", "@monthly" => "0 0 1 * *", "@weekly" => "0 0 * * 0",
|
|
@@ -27,14 +29,15 @@ module Cronwatch
|
|
|
27
29
|
|
|
28
30
|
def initialize(text)
|
|
29
31
|
fields, dom, dow = croner_fields(text)
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
32
|
+
# Fugit refuses a date no month has ("30 2"), and drops a month the
|
|
33
|
+
# days cannot fall in ("31 2,3" reads as March alone). Croner keeps
|
|
34
|
+
# both, so the days are read with every month and the months with
|
|
35
|
+
# every day, and a date no month has is a cron that never fires.
|
|
36
|
+
cron = fugit(fields.each_with_index.map { |f, i| i == 3 ? dom : (i == 4 ? "*" : (i == 5 ? dow : f)) })
|
|
37
|
+
months = fugit(["0", "0", "0", "*", fields[4], "*"])
|
|
35
38
|
# Croner accepts a few forms Fugit does not read (a range with #, a day of
|
|
36
|
-
# the month with L after it
|
|
37
|
-
raise ArgumentError, "CronPattern: '#{text}' uses a form the Ruby port does not read" unless cron
|
|
39
|
+
# the month with L after it). Those are refused.
|
|
40
|
+
raise ArgumentError, "CronPattern: '#{text}' uses a form the Ruby port does not read" unless cron && months
|
|
38
41
|
|
|
39
42
|
@seconds = flags(cron.seconds, 60, 0)
|
|
40
43
|
@minutes = flags(cron.minutes, 60, 0)
|
|
@@ -45,7 +48,7 @@ module Cronwatch
|
|
|
45
48
|
else
|
|
46
49
|
cron.monthdays.each { |d| @days[d - 1] = true if d.positive? }
|
|
47
50
|
end
|
|
48
|
-
@months = flags(
|
|
51
|
+
@months = flags(months.months, 12, 1)
|
|
49
52
|
@weekdays = Array.new(7, 0)
|
|
50
53
|
if cron.weekdays.nil?
|
|
51
54
|
@weekdays.fill(ANY_NTH)
|
|
@@ -75,6 +78,12 @@ module Cronwatch
|
|
|
75
78
|
|
|
76
79
|
private
|
|
77
80
|
|
|
81
|
+
def fugit(fields)
|
|
82
|
+
Fugit::Cron.parse(fields.join(" "))
|
|
83
|
+
rescue StandardError
|
|
84
|
+
nil
|
|
85
|
+
end
|
|
86
|
+
|
|
78
87
|
def flags(values, size, base)
|
|
79
88
|
return Array.new(size, true) if values.nil?
|
|
80
89
|
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Cronwatch
|
|
4
|
+
# Warns that a name is deprecated: it still works through 1.x and goes in
|
|
5
|
+
# 2.0. The warning is Ruby's own deprecation category, so it shows where
|
|
6
|
+
# Ruby's deprecation warnings show (`ruby -w`, `-W:deprecated`, or
|
|
7
|
+
# `Warning[:deprecated] = true`) and points at the caller's line.
|
|
8
|
+
#
|
|
9
|
+
# @api private
|
|
10
|
+
module Deprecation
|
|
11
|
+
module_function
|
|
12
|
+
|
|
13
|
+
def warn(old, replacement, uplevel: 2)
|
|
14
|
+
Kernel.warn("[cronwatch] #{old} is deprecated and goes in 2.0; use #{replacement}",
|
|
15
|
+
uplevel: uplevel, category: :deprecated)
|
|
16
|
+
end
|
|
17
|
+
end
|
|
18
|
+
end
|
data/lib/cronwatch/duration.rb
CHANGED
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
module Cronwatch
|
|
4
4
|
# "15m", "1h30m", "90s", "2d", or a number of milliseconds.
|
|
5
|
+
#
|
|
6
|
+
# @api private
|
|
5
7
|
module Duration
|
|
6
8
|
UNIT_MS = { "ms" => 1, "s" => 1000, "m" => 60_000, "h" => 3_600_000, "d" => 86_400_000, "w" => 604_800_000 }.freeze
|
|
7
9
|
PART = Regexp.new("(\\d+(?:\\.\\d+)?)[#{JS::WHITESPACE}]*(ms|s|m|h|d|w)")
|