belay 0.1.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 628271c83f016c169bdd9050abf23bbb6932fb27dcf15a74bbfbaae63e1814e4
4
- data.tar.gz: 154ee22a4b15d42c20ae9879bad70b2f55597fd89b8f7a26bd4e96472d9f386e
3
+ metadata.gz: 16138d86493d56b1c9baef13c6b5039519a01a2382679f134dc7db31f721ad39
4
+ data.tar.gz: c044de960b0983f6bbf0d5965983d89bf1b3d6b2923decebc2e2f71907196e91
5
5
  SHA512:
6
- metadata.gz: f75b75dd4980103b44896aac3b8c2b20adbf643975c198cd5127efadc353a86b70c7cd12ad825153ef816696af4066cb58e8c1d8972abbc9a79508ba94837e8b
7
- data.tar.gz: 34595eaa201103123588bb2f08a0111c305ab0579c12b119b15508f55ab0c410ae2bd094feb326f16dc2f4f7063408f6d60823b42d00822613f8d06436d766f6
6
+ metadata.gz: 77143cfb6dd262d9b763a0c7c3166dec5d8fc5366fa1e613575f3ea83a1f289c448f578a279ddaf44c45fccba17d89d1fc4e81894be89450bbb3f071460f7d3e
7
+ data.tar.gz: 806fae91af2d064e8458a94f47aea3ea9562dc3b2dc00f454ee0fad2ad14b4b7d540e76348f65e04a8cf3b98906bf653fecd1a6c4ad9de634567c301ad35db6c
data/README.md CHANGED
@@ -132,7 +132,7 @@ ctx.llm(model:, prompt:, schema: nil, cacheable: false)
132
132
 
133
133
  Calls `Belay.config.provider`, validates the response against `schema:` provider-agnostically, and returns `response.content` directly as the step's own output.
134
134
 
135
- **→ [docs/providers.md](docs/providers.md)** - Belay isn't Anthropic-only; bring your own provider.
135
+ **→ [docs/providers.md](docs/providers.md)** - ships `AnthropicProvider`, `OpenAIProvider`, and `GeminiProvider`; bring your own for anything else.
136
136
  **→ [docs/caching.md](docs/caching.md)** - `cacheable: true` and when it's actually safe to use.
137
137
 
138
138
  ## Dashboard
@@ -146,6 +146,24 @@ A runs index, a run detail page with the full step timeline, a replay diff view,
146
146
 
147
147
  **→ [docs/dashboard.md](docs/dashboard.md)** for mounting, wiring in your own auth (Devise/Pundit/CanCanCan/anything), and what each page shows.
148
148
 
149
+ ## Instrumentation
150
+
151
+ Belay publishes `ActiveSupport::Notifications` events for the run/step/gate lifecycle (`belay.run.started`, `belay.step.completed`, `belay.gate.paused`, and others) - the same mechanism Rails itself uses for `sql.active_record`, so you can pipe what Belay is doing into your own logs/APM without polling the dashboard.
152
+
153
+ **→ [docs/instrumentation.md](docs/instrumentation.md)** for the full event list and payloads.
154
+
155
+ ## Webhooks
156
+
157
+ Built on top of instrumentation: set `config.webhook_url` and a gate pausing, a step getting flagged `needs_review`, or a run completing each POST a JSON body to it - retried automatically, optionally HMAC-signed, no polling or a subscriber block of your own required.
158
+
159
+ **→ [docs/webhooks.md](docs/webhooks.md)** for configuration, signature verification, and retry behavior.
160
+
161
+ ## Cost budgets
162
+
163
+ `config.run_cost_budget_cents` fires `belay.run.budget_exceeded` the first time a single run's total cost crosses it; `config.daily_cost_budget_cents` shows a banner on the dashboard's Stats page once today's spend does. Both nil (off) by default.
164
+
165
+ **→ [docs/cost-budgets.md](docs/cost-budgets.md)** for both thresholds and why they're separate.
166
+
149
167
  ## Roadmap
150
168
 
151
169
  See [ROADMAP.md](ROADMAP.md).
@@ -9,6 +9,8 @@ module Belay
9
9
  class StatsController < ApplicationController
10
10
  def index
11
11
  @daily_stats = Belay::DailyStats.for_range
12
+ @daily_cost_budget_cents = Belay.config.daily_cost_budget_cents
13
+ @today_cost_cents = Belay::DailyStats.today_cost_cents
12
14
  end
13
15
  end
14
16
  end
@@ -0,0 +1,6 @@
1
+ <div class="mb-6 rounded-lg border border-amber-500/25 bg-amber-500/5 p-4" role="alert">
2
+ <p class="text-sm text-amber-200">
3
+ Today's spend (<%= number_to_currency(today_cost_cents / 100.0) %>) has crossed the configured daily budget of
4
+ <%= number_to_currency(daily_cost_budget_cents / 100.0) %>.
5
+ </p>
6
+ </div>
@@ -6,6 +6,7 @@
6
6
  <th class="px-4 py-2 text-left text-xs font-medium uppercase tracking-wide text-slate-500">Runs</th>
7
7
  <th class="px-4 py-2 text-left text-xs font-medium uppercase tracking-wide text-slate-500">Real API calls</th>
8
8
  <th class="px-4 py-2 text-left text-xs font-medium uppercase tracking-wide text-slate-500">Per 100 runs</th>
9
+ <th class="px-4 py-2 text-left text-xs font-medium uppercase tracking-wide text-slate-500">Spend</th>
9
10
  </tr>
10
11
  </thead>
11
12
  <tbody class="divide-y divide-slate-800">
@@ -15,6 +16,7 @@
15
16
  <td class="stat-runs px-4 py-2 font-mono text-xs text-slate-500"><%= day.runs %></td>
16
17
  <td class="stat-calls px-4 py-2 font-mono text-xs text-slate-500"><%= day.real_calls %></td>
17
18
  <td class="stat-rate px-4 py-2 font-mono text-xs font-medium text-slate-100"><%= day.calls_per_100_runs %></td>
19
+ <td class="stat-cost px-4 py-2 font-mono text-xs text-slate-300"><%= number_to_currency(day.cost_cents / 100.0) %></td>
18
20
  </tr>
19
21
  <% end %>
20
22
  </tbody>
@@ -6,6 +6,10 @@
6
6
  </p>
7
7
  </div>
8
8
 
9
+ <% if @daily_cost_budget_cents && @today_cost_cents > @daily_cost_budget_cents %>
10
+ <%= render "budget_banner", today_cost_cents: @today_cost_cents, daily_cost_budget_cents: @daily_cost_budget_cents %>
11
+ <% end %>
12
+
9
13
  <% if @daily_stats.any? %>
10
14
  <%= render "chart", daily_stats: @daily_stats %>
11
15
  <%= render "table", daily_stats: @daily_stats %>
@@ -47,7 +47,57 @@ module Belay
47
47
  # own comment), so the host app owns wherever this string actually comes from.
48
48
  attr_accessor :custom_css
49
49
 
50
+ # Where Belay::WebhookJob POSTs a JSON body ({event:, payload:}) for every event
51
+ # in webhook_events below. nil by default - opt-in, same reasoning as provider/
52
+ # authenticate_with: a host app that never sets this gets no webhook behavior at
53
+ # all, not a silently-failing POST to nowhere. Read at delivery time (inside the
54
+ # subscriber block and inside WebhookJob#perform), not captured when the
55
+ # subscription is installed - so it can be set in config/initializers/belay.rb
56
+ # regardless of initializer load order relative to Belay::Engine's own.
57
+ attr_accessor :webhook_url
58
+
59
+ # Optional HMAC-SHA256 secret. When set, every webhook request carries an
60
+ # X-Belay-Signature header (hex digest of the raw JSON body) so the receiving
61
+ # endpoint can verify the request actually came from this app, the same pattern
62
+ # Stripe/GitHub webhooks use. nil by default - signing is extra integration work
63
+ # for a receiver that may not need it (e.g. a private endpoint on a trusted
64
+ # network); unsigned is still a reasonable default, matching webhook_url's own
65
+ # opt-in-everything posture.
66
+ attr_accessor :webhook_secret
67
+
68
+ # Which belay.* events actually get POSTed. A plain Array a host app can
69
+ # overwrite wholesale (push onto it, or replace it entirely) - not every event
70
+ # Belay fires is something worth waking up a human/on-call for; this starts
71
+ # narrow (gate pause, needs_review, run completion - exactly what
72
+ # FUTURE_ROADMAP.md's webhooks item asks for) rather than defaulting to "every
73
+ # belay.* event," which would surprise anyone who just wants "tell me when a
74
+ # human is needed or a run finishes," not every auto-approved gate too.
75
+ attr_accessor :webhook_events
76
+
77
+ # Cents. nil by default - opt-in, same posture as every other threshold here. Once
78
+ # a single run's total cost_cents (every step's, summed - see Run#cost_cents)
79
+ # crosses this, Executor fires belay.run.budget_exceeded exactly once for that
80
+ # run (not once per step that happens to still be over budget afterward). Not in
81
+ # webhook_events by default even once set - a host app that wants this pushed to
82
+ # its webhook adds "belay.run.budget_exceeded" itself; setting a budget purely
83
+ # for the dashboard banner below shouldn't silently start firing webhooks nobody
84
+ # asked for.
85
+ attr_accessor :run_cost_budget_cents
86
+
87
+ # Cents. nil by default. Purely a dashboard concern (unlike run_cost_budget_cents
88
+ # above, this fires no event) - Belay::StatsController compares it against
89
+ # today's total cost_cents (Belay::DailyStats.for_range) and the Stats page shows
90
+ # a banner once today's total crosses it.
91
+ attr_accessor :daily_cost_budget_cents
92
+
50
93
  def initialize
94
+ assign_execution_defaults!
95
+ assign_webhook_and_budget_defaults!
96
+ end
97
+
98
+ private
99
+
100
+ def assign_execution_defaults!
51
101
  @lease_duration = 300
52
102
  @heartbeat_interval = 30
53
103
  @provider = nil
@@ -56,5 +106,13 @@ module Belay
56
106
  @rule_approval_threshold = 3
57
107
  @custom_css = nil
58
108
  end
109
+
110
+ def assign_webhook_and_budget_defaults!
111
+ @webhook_url = nil
112
+ @webhook_secret = nil
113
+ @webhook_events = %w[belay.gate.paused belay.step.needs_review belay.run.completed]
114
+ @run_cost_budget_cents = nil
115
+ @daily_cost_budget_cents = nil
116
+ end
59
117
  end
60
118
  end
@@ -7,7 +7,7 @@ module Belay
7
7
  # ActiveRecord model of its own: there's no belay_daily_stats table, this is
8
8
  # a read-only view over belay_runs/belay_steps' own created_at timestamps.
9
9
  class DailyStats
10
- Row = Struct.new(:date, :runs, :real_calls, :calls_per_100_runs, keyword_init: true)
10
+ Row = Struct.new(:date, :runs, :real_calls, :calls_per_100_runs, :cost_cents, keyword_init: true)
11
11
 
12
12
  # Calls are grouped by the STEP's own day, not the owning run's - a run
13
13
  # can pause on a gate for a real, unbounded amount of time (nothing times
@@ -25,9 +25,12 @@ module Belay
25
25
  def self.for_range
26
26
  calls_by_day = calls_by_day_counts
27
27
  runs_by_day = runs_by_day_counts
28
+ cost_by_day = cost_cents_by_day
28
29
 
29
- days = (calls_by_day.keys + runs_by_day.keys).uniq.sort
30
- days.map { |date| build_row(date, runs_by_day.fetch(date, 0), calls_by_day.fetch(date, 0)) }
30
+ days = (calls_by_day.keys + runs_by_day.keys + cost_by_day.keys).uniq.sort
31
+ days.map do |date|
32
+ build_row(date, runs_by_day.fetch(date, 0), calls_by_day.fetch(date, 0), cost_by_day.fetch(date, 0))
33
+ end
31
34
  end
32
35
 
33
36
  def self.calls_by_day_counts
@@ -40,6 +43,19 @@ module Belay
40
43
  end
41
44
  private_class_method :runs_by_day_counts
42
45
 
46
+ # Grouped by the step's own day, same reasoning calls_by_day_counts' comment
47
+ # above already gives for calls - a step's cost is incurred when the call it
48
+ # paid for actually happened, not whenever its owning run started. Not scoped to
49
+ # llm_call/cache_hit: false the way calls_by_day_counts is - cost_cents is only
50
+ # ever non-nil on a real, billed llm_call anyway (a cache hit's own cost_cents is
51
+ # explicitly 0, per Context#fetch_cached_llm_response's own comment, not nil), so
52
+ # a plain SUM across every step's cost_cents already excludes everything that
53
+ # shouldn't count without a separate WHERE.
54
+ def self.cost_cents_by_day
55
+ normalized_counts(Step.group("DATE(created_at)").sum(:cost_cents))
56
+ end
57
+ private_class_method :cost_cents_by_day
58
+
43
59
  # .group("DATE(...)").count's hash keys come back as plain Strings on some
44
60
  # adapter/Rails version combinations, real Date objects on others - a raw
45
61
  # SQL group-by fragment (not an actual typed column) gives ActiveRecord
@@ -51,8 +67,19 @@ module Belay
51
67
  end
52
68
  private_class_method :normalized_counts
53
69
 
54
- def self.build_row(date, runs, calls)
55
- Row.new(date: date, runs: runs, real_calls: calls,
70
+ # Deliberately NOT @daily_stats.find { |row| row.date == Date.current } at the
71
+ # call site - every date above comes from Postgres's own DATE(created_at), not
72
+ # Ruby's Date.current, and the two can disagree for part of the day whenever the
73
+ # app's Time.zone differs from the DB session's own timezone setting. Asking
74
+ # Postgres's own CURRENT_DATE here instead of Ruby's Date.current means both
75
+ # sides of "is this today" are computed the same way, in the same session - they
76
+ # can never disagree, regardless of which timezone either happens to be in.
77
+ def self.today_cost_cents
78
+ Step.where("DATE(created_at) = CURRENT_DATE").sum(:cost_cents)
79
+ end
80
+
81
+ def self.build_row(date, runs, calls, cost_cents)
82
+ Row.new(date: date, runs: runs, real_calls: calls, cost_cents: cost_cents,
56
83
  calls_per_100_runs: runs.zero? ? 0.0 : (calls.to_f / runs * 100).round(1))
57
84
  end
58
85
  private_class_method :build_row
data/lib/belay/engine.rb CHANGED
@@ -20,6 +20,15 @@ module Belay
20
20
  class Engine < ::Rails::Engine
21
21
  isolate_namespace Belay
22
22
 
23
+ # Installed once per process, at boot - not per-request. Belay::Webhooks.deliver
24
+ # (the callback this installs) checks Belay.config.webhook_url/webhook_events
25
+ # itself on every call, so it's a safe no-op in any app that never sets
26
+ # webhook_url, and initializer load order relative to a host app's own
27
+ # config/initializers/belay.rb never matters (see Webhooks' own comment).
28
+ initializer "belay.webhooks" do
29
+ Belay::Webhooks.subscribe!
30
+ end
31
+
23
32
  # The dashboard's CSS, inlined into the layout via <style> rather than
24
33
  # served through a <link> and whichever asset pipeline (Sprockets,
25
34
  # Propshaft, none at all) the host app happens to have configured - that's
@@ -186,6 +186,8 @@ module Belay
186
186
  # human. Run#pause! already exists (running -> paused) for exactly this shape
187
187
  # of "stopped, not failed, waiting on something external."
188
188
  @run.pause!
189
+ ActiveSupport::Notifications.instrument("belay.step.needs_review", run_id: @run.id, step_id: existing.id,
190
+ name: existing.name, reason: reason)
189
191
  raise Error, "#{reason} - flagged needs_review; the run is paused until a human resolves it"
190
192
  end
191
193
 
@@ -240,12 +242,85 @@ module Belay
240
242
  output = run_block(step_run, context, step)
241
243
  return [nil, nil] if output.equal?(AUTO_SETTLED) # auto_settle_gate! already wrote this row itself
242
244
 
243
- step.update!(status: settled_status(step_run), output: output, **extra_step_attrs(step_run, context))
244
- [output, nil]
245
+ settle_succeeded!(step_run, step, context, output)
245
246
  rescue StandardError => e
246
- step.update!(status: "failed", error: { "class" => e.class.name, "message" => e.message },
247
+ settle_failed!(step_run, step, context, e)
248
+ ensure
249
+ # One call site, not one in each settle_*! below, covering success, failure,
250
+ # AND (harmlessly) the AUTO_SETTLED early return above - check_run_cost_budget!
251
+ # itself no-ops immediately for anything that isn't a step with a real,
252
+ # just-persisted cost_cents (a gate never has one), so running it unconditionally
253
+ # here is equivalent to calling it explicitly from every exit path, without the
254
+ # duplication (and the risk of a future settle_*! forgetting to call it).
255
+ check_run_cost_budget!(step)
256
+ end
257
+
258
+ def settle_succeeded!(step_run, step, context, output)
259
+ status = settled_status(step_run)
260
+ step.update!(status: status, output: output, **extra_step_attrs(step_run, context))
261
+ instrument_step_completed(step, status: status)
262
+ [output, nil]
263
+ end
264
+
265
+ def settle_failed!(step_run, step, context, error)
266
+ step.update!(status: "failed", error: { "class" => error.class.name, "message" => error.message },
247
267
  **extra_step_attrs(step_run, context))
248
- [nil, e]
268
+ instrument_step_completed(step, status: "failed")
269
+ [nil, error]
270
+ end
271
+
272
+ # One event per settled attempt (success or failure alike), not separate
273
+ # started/completed events - the same consolidation ActionController's own
274
+ # process_action.action_controller notification makes (one event, status: in
275
+ # the payload), rather than one event per outcome. A step's own row already
276
+ # records attempt: - a retried step fires this once per attempt, same as it
277
+ # gets its own Step row per attempt.
278
+ def instrument_step_completed(step, status:)
279
+ ActiveSupport::Notifications.instrument(
280
+ "belay.step.completed",
281
+ run_id: @run.id, step_id: step.id, position: step.position, name: step.name,
282
+ kind: step.kind, status: status, attempt: step.attempt
283
+ )
284
+ end
285
+
286
+ # Fires belay.run.budget_exceeded the FIRST time this run's total cost_cents
287
+ # (every step's, summed - Run#cost_cents) crosses config.run_cost_budget_cents,
288
+ # never again for the same run - a run that's already over budget doesn't need
289
+ # the same alert repeated on every subsequent step that's merely still over it.
290
+ # step.cost_cents.nil? short-circuits before the budget/alert_sent_at checks
291
+ # below even run: a step with no recorded cost (no pricing: configured, or not
292
+ # an llm_call at all) can never be the step that newly crosses a budget, so
293
+ # there's nothing to check.
294
+ def check_run_cost_budget!(step)
295
+ return if step.cost_cents.nil?
296
+
297
+ budget = Belay.config.run_cost_budget_cents
298
+ return if budget.nil? || @run.budget_alert_sent_at
299
+
300
+ total = @run.cost_cents
301
+ return if total <= budget
302
+
303
+ claim_budget_alert!(total, budget)
304
+ end
305
+
306
+ # A conditional UPDATE, not read-then-write - same reasoning as Run#acquire_
307
+ # lease!'s own comment: nothing here takes the same kind of lease
308
+ # run_attempt's own caller assumes holds for the whole run (Workflow.start's
309
+ # inline path takes none at all; flag_needs_review!'s own comment already
310
+ # flags that no lease mechanism exists to tell two workers apart), so two
311
+ # Executor instances racing on the same run could otherwise both read
312
+ # budget_alert_sent_at as nil and both fire this event. update_all's return
313
+ # value (rows actually matched) is the only thing trusted to decide that;
314
+ # @run's in-memory attribute is only set after a successful claim, never used
315
+ # to decide one.
316
+ def claim_budget_alert!(total, budget)
317
+ claimed = Run.where(id: @run.id, budget_alert_sent_at: nil).update_all(budget_alert_sent_at: Time.current)
318
+ return if claimed.zero?
319
+
320
+ @run.budget_alert_sent_at = Time.current
321
+ ActiveSupport::Notifications.instrument(
322
+ "belay.run.budget_exceeded", run_id: @run.id, total_cost_cents: total, budget_cents: budget
323
+ )
249
324
  end
250
325
 
251
326
  # Wrapped in @run.heartbeat_while, not called bare - this is the one call in
@@ -311,6 +386,8 @@ module Belay
311
386
  def auto_settle_gate!(step, context_key, rule)
312
387
  step.update!(status: "succeeded", output: nil, input: { "context_key" => context_key },
313
388
  rule_applied_id: rule.id)
389
+ ActiveSupport::Notifications.instrument("belay.gate.auto_approved", run_id: @run.id, step_id: step.id,
390
+ name: step.name, rule_id: rule.id)
314
391
  AUTO_SETTLED
315
392
  end
316
393
 
@@ -347,6 +424,8 @@ module Belay
347
424
  @run.pause!
348
425
  @run.release_lease!
349
426
  end
427
+ ActiveSupport::Notifications.instrument("belay.gate.paused", run_id: @run.id, step_id: step.id,
428
+ name: step.name)
350
429
  throw :gate_paused
351
430
  end
352
431
 
@@ -0,0 +1,158 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "net/http"
4
+ require "json"
5
+ require "uri"
6
+
7
+ module Belay
8
+ # Calls Google's Gemini API directly over Net::HTTP (stdlib) - same
9
+ # reasoning as AnthropicProvider/OpenAIProvider: no SDK gem, so no second
10
+ # gem's own version matrix for this gem to track for a provider this
11
+ # central.
12
+ #
13
+ # Endpoint and field names verified against Google's own REST reference
14
+ # (ai.google.dev/api/generate-content) as of 2026-10-28.
15
+ #
16
+ # The API key goes in the request URL's own ?key= query param, not a
17
+ # header - that's Gemini's own convention, unlike Anthropic's x-api-key or
18
+ # OpenAI's Authorization: Bearer. URI.encode_www_form escapes it rather
19
+ # than interpolating it directly into the URL string, purely so a key
20
+ # containing a URL-meaningful character (unlikely, but API keys are
21
+ # opaque strings this gem didn't generate) can't corrupt the request line.
22
+ #
23
+ # schema: is passed through to generationConfig.responseSchema completely
24
+ # unvalidated and untranslated - Gemini's schema dialect uses its own,
25
+ # UPPERCASE type names ("OBJECT", "STRING", "INTEGER", ...), not standard
26
+ # JSON Schema's lowercase ones ("object", "string"), and doesn't support
27
+ # every JSON Schema keyword. A schema written for AnthropicProvider or
28
+ # OpenAIProvider will not work here unchanged - same spirit as
29
+ # AnthropicProvider's additionalProperties: false requirement, just a
30
+ # different provider-specific dialect. See docs/providers.md.
31
+ #
32
+ # finishReason is mapped to the same StructuredOutputError reason: values
33
+ # the other two providers use, so a caller can branch on reason: the same
34
+ # way regardless of which provider is configured: "SAFETY"/"RECITATION"
35
+ # (the model declined to produce matching content) -> "refusal";
36
+ # "MAX_TOKENS" -> "max_tokens". Like AnthropicProvider/OpenAIProvider, this
37
+ # check only runs when schema: was given - see OpenAIProvider's own comment
38
+ # for why a refusal/truncation on a plain-text call isn't this gem's call
39
+ # to second-guess.
40
+ class GeminiProvider < Provider
41
+ API_BASE = "https://generativelanguage.googleapis.com/v1beta/models"
42
+
43
+ # Raised on any non-2xx response. Gemini's own error body (JSON, usually
44
+ # {"error": {"code":, "message":, "status":}}) is preserved verbatim in
45
+ # the message, same reasoning as AnthropicProvider::RequestError.
46
+ class RequestError < Error
47
+ attr_reader :status
48
+
49
+ def initialize(status:, body:)
50
+ @status = status
51
+ super("Gemini API request failed with status #{status}: #{body}")
52
+ end
53
+ end
54
+
55
+ # Raised when schema: was given but the response can't be trusted to
56
+ # contain valid, complete schema-matching JSON. reason: is "refusal" or
57
+ # "max_tokens" - see the class comment above for which finishReason maps
58
+ # to each.
59
+ class StructuredOutputError < Error
60
+ attr_reader :reason, :response
61
+
62
+ def initialize(reason:, response:)
63
+ @reason = reason
64
+ @response = response
65
+ super("Gemini did not return schema-matching output (reason: #{reason.inspect}): #{response.inspect}")
66
+ end
67
+ end
68
+
69
+ # pricing: {"gemini-..." => { input_per_million: <USD Float>, output_per_million: <USD Float> }, ...}
70
+ # Deliberately not pre-filled with Google's current rates - same
71
+ # reasoning as AnthropicProvider's own pricing: comment.
72
+ def initialize(api_key: ENV.fetch("GEMINI_API_KEY", nil), max_output_tokens: 4096, pricing: {})
73
+ super()
74
+ raise ArgumentError, "#{self.class} requires api_key: (or GEMINI_API_KEY)" if api_key.nil?
75
+
76
+ @api_key = api_key
77
+ @max_output_tokens = max_output_tokens
78
+ @pricing = pricing
79
+ end
80
+
81
+ def call(model:, prompt:, schema: nil)
82
+ started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
83
+ status, body = perform_request(model, request_body(prompt: prompt, schema: schema))
84
+ latency_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
85
+
86
+ raise RequestError.new(status: status, body: body) unless status.between?(200, 299)
87
+
88
+ build_response(JSON.parse(body), model: model, schema: schema, latency_ms: latency_ms)
89
+ end
90
+
91
+ private
92
+
93
+ def request_body(prompt:, schema:)
94
+ body = {
95
+ contents: [{ parts: [{ text: prompt }] }],
96
+ generationConfig: { maxOutputTokens: @max_output_tokens }
97
+ }
98
+ return body if schema.nil?
99
+
100
+ body[:generationConfig] = body[:generationConfig].merge(responseMimeType: "application/json",
101
+ responseSchema: schema)
102
+ body
103
+ end
104
+
105
+ def perform_request(model, body)
106
+ uri = URI("#{API_BASE}/#{model}:generateContent")
107
+ uri.query = URI.encode_www_form(key: @api_key)
108
+ response = Net::HTTP.post(uri, JSON.generate(body), { "content-type" => "application/json" })
109
+ [response.code.to_i, response.body]
110
+ end
111
+
112
+ def build_response(response, model:, schema:, latency_ms:)
113
+ usage = response["usageMetadata"] || {}
114
+ input_tokens, output_tokens = usage.values_at("promptTokenCount", "candidatesTokenCount")
115
+
116
+ ProviderResponse.new(
117
+ content: extract_content(response, schema: schema),
118
+ model: response["modelVersion"] || model,
119
+ input_tokens: input_tokens,
120
+ output_tokens: output_tokens,
121
+ cost_cents: cost_cents(model: model, input_tokens: input_tokens, output_tokens: output_tokens),
122
+ latency_ms: latency_ms
123
+ )
124
+ end
125
+
126
+ def extract_content(response, schema:)
127
+ schema ? extract_structured_output(response) : extract_text(response)
128
+ end
129
+
130
+ def extract_structured_output(response)
131
+ finish_reason = response.dig("candidates", 0, "finishReason")
132
+ if %w[SAFETY RECITATION].include?(finish_reason)
133
+ raise StructuredOutputError.new(reason: "refusal", response: response)
134
+ end
135
+ raise StructuredOutputError.new(reason: "max_tokens", response: response) if finish_reason == "MAX_TOKENS"
136
+
137
+ JSON.parse(extract_text(response))
138
+ end
139
+
140
+ def extract_text(response)
141
+ part = response.dig("candidates", 0, "content", "parts", 0)
142
+ raise Error, "Gemini response had no text content part" if part.nil? || part["text"].nil?
143
+
144
+ part["text"]
145
+ end
146
+
147
+ def cost_cents(model:, input_tokens:, output_tokens:)
148
+ return nil if input_tokens.nil? || output_tokens.nil?
149
+
150
+ rates = @pricing[model]
151
+ return nil if rates.nil?
152
+
153
+ input_cost = input_tokens * rates.fetch(:input_per_million)
154
+ output_cost = output_tokens * rates.fetch(:output_per_million)
155
+ ((input_cost + output_cost) / 1_000_000.0 * 100).round
156
+ end
157
+ end
158
+ end
@@ -0,0 +1,246 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "net/http"
4
+ require "json"
5
+ require "uri"
6
+
7
+ module Belay
8
+ # Calls OpenAI's API directly over Net::HTTP (stdlib) - same reasoning as
9
+ # AnthropicProvider: no SDK gem, so no second gem's own version matrix for
10
+ # this gem to track for a provider this central. POST a JSON body, read a
11
+ # JSON body back is small enough that stdlib Net::HTTP + JSON covers it.
12
+ #
13
+ # api: picks which of OpenAI's two current text-generation APIs to call -
14
+ # :responses (default, OpenAI's own forward-looking API, POST /v1/responses)
15
+ # or :chat_completions (the older, still fully supported POST
16
+ # /v1/chat/completions). Both are real, current APIs, not a deprecated/
17
+ # replacement pair, and their request/response shapes differ enough
18
+ # (structured-output config lives at text.format vs response_format; usage
19
+ # keys are input_tokens/output_tokens vs prompt_tokens/completion_tokens;
20
+ # truncation and refusal surface completely differently - see
21
+ # extract_responses_content/extract_chat_completions_content below) that
22
+ # this class branches internally on @api rather than guessing one and
23
+ # forcing callers who need the other to go write their own provider. The
24
+ # public interface (#call) and everything around it (errors, cost, latency)
25
+ # is identical either way.
26
+ #
27
+ # Endpoint and field names verified against OpenAI's own public OpenAPI spec
28
+ # (github.com/openai/openai-openapi) as of 2026-10-28.
29
+ #
30
+ # schema: on :responses sets text.format: {type: "json_schema", name:,
31
+ # schema:, strict: true}; on :chat_completions it's response_format:
32
+ # {type: "json_schema", json_schema: {name:, schema:, strict: true}}.
33
+ # Unlike Anthropic, OpenAI requires a name: on the schema either way - a
34
+ # fixed, gem-chosen value (SCHEMA_NAME) is used throughout, since nothing
35
+ # about the schema format itself needs that name to be caller-visible.
36
+ # strict: true is always passed (not left to OpenAI's own default: false) -
37
+ # a non-strict schema is advisory only, and this gem's own
38
+ # StructuredOutputError handling below assumes a schema that was actually
39
+ # enforced.
40
+ #
41
+ # refusal/truncation are each mapped to the same StructuredOutputError
42
+ # reason: values AnthropicProvider uses ("refusal"/"max_tokens"), so a
43
+ # caller can branch on reason: the same way regardless of which provider is
44
+ # configured:
45
+ # :responses - a "refusal" content item on the output message ->
46
+ # "refusal"; status: "incomplete" with
47
+ # incomplete_details.reason == "max_output_tokens" ->
48
+ # "max_tokens"
49
+ # :chat_completions - message.refusal present -> "refusal";
50
+ # finish_reason == "length" -> "max_tokens"
51
+ # Both checks, like AnthropicProvider's own, only run when schema: was
52
+ # given - a refusal or truncation on a plain-text call isn't this gem's
53
+ # problem to second-guess; it surfaces as whatever text (or lack of it)
54
+ # the author's own code then has to deal with, same as any other provider.
55
+ # Genuinely two providers' worth of request/response handling behind one #call
56
+ # interface (see api: above), not one class that grew unchecked - same
57
+ # ClassLength exemption reasoning as Executor::StepRunner (executor.rb).
58
+ class OpenAIProvider < Provider # rubocop:disable Metrics/ClassLength
59
+ API_URLS = {
60
+ responses: URI("https://api.openai.com/v1/responses"),
61
+ chat_completions: URI("https://api.openai.com/v1/chat/completions")
62
+ }.freeze
63
+
64
+ SCHEMA_NAME = "belay_response"
65
+
66
+ # Raised on any non-2xx response. OpenAI's own error body (JSON, usually
67
+ # {"error": {"type":, "message":}}) is preserved verbatim in the message,
68
+ # same reasoning as AnthropicProvider::RequestError.
69
+ class RequestError < Error
70
+ attr_reader :status
71
+
72
+ def initialize(status:, body:)
73
+ @status = status
74
+ super("OpenAI API request failed with status #{status}: #{body}")
75
+ end
76
+ end
77
+
78
+ # Raised when schema: was given but the response can't be trusted to
79
+ # contain valid, complete schema-matching JSON. reason: is "refusal" or
80
+ # "max_tokens" - see the class comment above for exactly what maps to
81
+ # each, per api:.
82
+ class StructuredOutputError < Error
83
+ attr_reader :reason, :response
84
+
85
+ def initialize(reason:, response:)
86
+ @reason = reason
87
+ @response = response
88
+ super("OpenAI did not return schema-matching output (reason: #{reason.inspect}): #{response.inspect}")
89
+ end
90
+ end
91
+
92
+ # pricing: {"gpt-..." => { input_per_million: <USD Float>, output_per_million: <USD Float> }, ...}
93
+ # Deliberately not pre-filled with OpenAI's current rates - same reasoning
94
+ # as AnthropicProvider's own pricing: comment.
95
+ def initialize(api_key: ENV.fetch("OPENAI_API_KEY", nil), api: :responses, max_tokens: 4096, pricing: {})
96
+ super()
97
+ raise ArgumentError, "#{self.class} requires api_key: (or OPENAI_API_KEY)" if api_key.nil?
98
+ raise ArgumentError, "api: must be one of #{API_URLS.keys.inspect}, got #{api.inspect}" unless API_URLS.key?(api)
99
+
100
+ @api_key = api_key
101
+ @api = api
102
+ @max_tokens = max_tokens
103
+ @pricing = pricing
104
+ end
105
+
106
+ def call(model:, prompt:, schema: nil)
107
+ started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
108
+ status, body = perform_request(request_body(model: model, prompt: prompt, schema: schema))
109
+ latency_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
110
+
111
+ raise RequestError.new(status: status, body: body) unless status.between?(200, 299)
112
+
113
+ build_response(JSON.parse(body), model: model, schema: schema, latency_ms: latency_ms)
114
+ end
115
+
116
+ private
117
+
118
+ def request_body(model:, prompt:, schema:)
119
+ return responses_request_body(model, prompt, schema) if @api == :responses
120
+
121
+ chat_completions_request_body(model, prompt, schema)
122
+ end
123
+
124
+ def responses_request_body(model, prompt, schema)
125
+ body = { model: model, input: prompt, max_output_tokens: @max_tokens }
126
+ return body if schema.nil?
127
+
128
+ body.merge(text: { format: { type: "json_schema", name: SCHEMA_NAME, schema: schema, strict: true } })
129
+ end
130
+
131
+ def chat_completions_request_body(model, prompt, schema)
132
+ body = { model: model, max_completion_tokens: @max_tokens, messages: [{ role: "user", content: prompt }] }
133
+ return body if schema.nil?
134
+
135
+ body.merge(response_format: { type: "json_schema",
136
+ json_schema: { name: SCHEMA_NAME, schema: schema, strict: true } })
137
+ end
138
+
139
+ def perform_request(body)
140
+ response = Net::HTTP.post(API_URLS.fetch(@api), JSON.generate(body), request_headers)
141
+ [response.code.to_i, response.body]
142
+ end
143
+
144
+ def request_headers
145
+ { "Authorization" => "Bearer #{@api_key}", "content-type" => "application/json" }
146
+ end
147
+
148
+ def build_response(response, model:, schema:, latency_ms:)
149
+ usage = response["usage"] || {}
150
+ input_tokens, output_tokens = usage_tokens(usage)
151
+
152
+ ProviderResponse.new(
153
+ content: extract_content(response, schema: schema),
154
+ model: response["model"] || model,
155
+ input_tokens: input_tokens,
156
+ output_tokens: output_tokens,
157
+ cost_cents: cost_cents(model: model, input_tokens: input_tokens, output_tokens: output_tokens),
158
+ latency_ms: latency_ms
159
+ )
160
+ end
161
+
162
+ def usage_tokens(usage)
163
+ keys = @api == :responses ? %w[input_tokens output_tokens] : %w[prompt_tokens completion_tokens]
164
+ usage.values_at(*keys)
165
+ end
166
+
167
+ def extract_content(response, schema:)
168
+ return extract_responses_content(response, schema: schema) if @api == :responses
169
+
170
+ extract_chat_completions_content(response, schema: schema)
171
+ end
172
+
173
+ def extract_responses_content(response, schema:)
174
+ message = find_output_message(response)
175
+ return extract_responses_structured_output(response, message) if schema
176
+
177
+ text = output_text(message)
178
+ raise Error, "OpenAI response had no output_text content" if text.nil?
179
+
180
+ text
181
+ end
182
+
183
+ def extract_responses_structured_output(response, message)
184
+ raise StructuredOutputError.new(reason: "refusal", response: response) if refusal?(message)
185
+ raise StructuredOutputError.new(reason: "max_tokens", response: response) if responses_truncated?(response)
186
+
187
+ text = output_text(message)
188
+ raise Error, "OpenAI response had no output_text content" if text.nil?
189
+
190
+ JSON.parse(text)
191
+ end
192
+
193
+ def find_output_message(response)
194
+ message = (response["output"] || []).find { |item| item["type"] == "message" }
195
+ raise Error, "OpenAI response had no output message" if message.nil?
196
+
197
+ message
198
+ end
199
+
200
+ def output_text(message)
201
+ text_block = (message["content"] || []).find { |c| c["type"] == "output_text" }
202
+ text_block && text_block["text"]
203
+ end
204
+
205
+ def refusal?(message)
206
+ (message["content"] || []).any? { |c| c["type"] == "refusal" }
207
+ end
208
+
209
+ def responses_truncated?(response)
210
+ response["status"] == "incomplete" && response.dig("incomplete_details", "reason") == "max_output_tokens"
211
+ end
212
+
213
+ def extract_chat_completions_content(response, schema:)
214
+ return extract_chat_completions_structured_output(response) if schema
215
+
216
+ content = response.dig("choices", 0, "message", "content")
217
+ raise Error, "OpenAI response had no message content" if content.nil?
218
+
219
+ content
220
+ end
221
+
222
+ def extract_chat_completions_structured_output(response)
223
+ message = response.dig("choices", 0, "message") || {}
224
+ raise StructuredOutputError.new(reason: "refusal", response: response) if message["refusal"]
225
+
226
+ finish_reason = response.dig("choices", 0, "finish_reason")
227
+ raise StructuredOutputError.new(reason: "max_tokens", response: response) if finish_reason == "length"
228
+
229
+ content = message["content"]
230
+ raise Error, "OpenAI response had no message content" if content.nil?
231
+
232
+ JSON.parse(content)
233
+ end
234
+
235
+ def cost_cents(model:, input_tokens:, output_tokens:)
236
+ return nil if input_tokens.nil? || output_tokens.nil?
237
+
238
+ rates = @pricing[model]
239
+ return nil if rates.nil?
240
+
241
+ input_cost = input_tokens * rates.fetch(:input_per_million)
242
+ output_cost = output_tokens * rates.fetch(:output_per_million)
243
+ ((input_cost + output_cost) / 1_000_000.0 * 100).round
244
+ end
245
+ end
246
+ end
data/lib/belay/run.rb CHANGED
@@ -36,15 +36,15 @@ module Belay
36
36
  validates :workflow_version, presence: true
37
37
 
38
38
  def start!
39
- transition!(from: "pending", to: "running", started_at: Time.current)
39
+ transition!(from: "pending", to: "running", event: "started", started_at: Time.current)
40
40
  end
41
41
 
42
42
  def pause!
43
- transition!(from: "running", to: "paused")
43
+ transition!(from: "running", to: "paused", event: "paused")
44
44
  end
45
45
 
46
46
  def resume!
47
- transition!(from: "paused", to: "running")
47
+ transition!(from: "paused", to: "running", event: "resumed")
48
48
  end
49
49
 
50
50
  # None of complete!/fail!/cancel! below clear lease_owner/lease_expires_at - a
@@ -53,15 +53,16 @@ module Belay
53
53
  # first, so a stale lease on a terminal run is never actually read. Would need
54
54
  # fixing if anything ever queries leases without that status filter.
55
55
  def complete!
56
- transition!(from: "running", to: "completed", finished_at: Time.current)
56
+ transition!(from: "running", to: "completed", event: "completed", finished_at: Time.current)
57
57
  end
58
58
 
59
59
  def fail!(error: nil)
60
- transition!(from: %w[pending running paused], to: "failed", finished_at: Time.current, error: error)
60
+ transition!(from: %w[pending running paused], to: "failed", event: "failed",
61
+ finished_at: Time.current, error: error)
61
62
  end
62
63
 
63
64
  def cancel!
64
- transition!(from: %w[pending running paused], to: "cancelled", finished_at: Time.current)
65
+ transition!(from: %w[pending running paused], to: "cancelled", event: "cancelled", finished_at: Time.current)
65
66
  end
66
67
 
67
68
  # At most one, by construction: Executor stops at the FIRST gate whose when:
@@ -81,6 +82,19 @@ module Belay
81
82
  steps.needs_review.first
82
83
  end
83
84
 
85
+ # SQL SUM, not steps.sum(&:cost_cents) in Ruby - a plain SUM already treats every
86
+ # NULL row (every non-llm_call step, and any llm_call step whose provider had no
87
+ # pricing: configured for that model) as 0 and returns 0, never nil, for a run
88
+ # with no costed steps at all - exactly the "we don't know the cost" vs. "the
89
+ # cost is zero" distinction Provider#cost_cents' own nil already draws at the
90
+ # single-step level, just rolled up: a run where NOTHING is priced costs "0
91
+ # cents we know about," the same honest answer a single unpriced step's own nil
92
+ # would give if summed the Ruby way (0, via Array#sum's +0 default) anyway - so
93
+ # there's no real information lost by using SQL's NULL-safe SUM here instead.
94
+ def cost_cents
95
+ steps.sum(:cost_cents)
96
+ end
97
+
84
98
  # DESIGN_DECISIONS #5: "a worker claims a run with a conditional UPDATE on
85
99
  # lease_owner/lease_expires_at." The condition - no owner yet, or the existing
86
100
  # lease has expired - is evaluated by Postgres in the same statement that claims
@@ -251,6 +265,13 @@ module Belay
251
265
  # answer wasn't quite right, which is the opposite of "safe to auto-settle
252
266
  # identically next time."
253
267
  record_rule_decision!(gate, approved: true) if edited_payload.nil?
268
+ # Last, same ordering principle as ResumeRunJob/record_rule_decision! above: a
269
+ # subscriber that raises (ActiveSupport::Notifications runs subscribers
270
+ # synchronously, in-process - a slow or broken one is this gem's problem the
271
+ # moment it's subscribed) must never be able to leave the gate settled with no
272
+ # job enqueued, which placing this any earlier would risk.
273
+ ActiveSupport::Notifications.instrument("belay.gate.approved", run_id: id, step_id: gate.id, name: gate.name,
274
+ by: by, edited: !edited_payload.nil?)
254
275
  end
255
276
 
256
277
  # Rejects this run's pending gate: records a rejected Belay::Approval, settles
@@ -273,6 +294,8 @@ module Belay
273
294
  end
274
295
 
275
296
  record_rule_decision!(gate, approved: false)
297
+ ActiveSupport::Notifications.instrument("belay.gate.rejected", run_id: id, step_id: gate.id, name: gate.name,
298
+ by: by)
276
299
  end
277
300
 
278
301
  # The two outcomes a human can give a needs_review step (flag_needs_review!,
@@ -615,7 +638,19 @@ module Belay
615
638
  # No concurrency guard: this reads status in memory, then does a plain update!. Two
616
639
  # processes racing on the same run (stale in-memory copies) can both pass the check.
617
640
  # Deferred to the leases work (Phase 1f) rather than solved here.
618
- def transition!(from:, to:, **attrs)
641
+ #
642
+ # event: is a separate argument from to:, not derived from it (e.g. "belay.run.#{to}") -
643
+ # to: "running" is reached by both start! and resume!, and those are two meaningfully
644
+ # different events for anything listening (ActiveSupport::Notifications.instrument
645
+ # below; see docs/instrumentation.md), the same way the comment above this method
646
+ # already explains why dispatch can't collapse them into one target-keyed graph either.
647
+ #
648
+ # Wrapping update! in .instrument, not instrumenting after the fact, is deliberate: a
649
+ # subscriber gets the real duration of the UPDATE for free (same convention
650
+ # ActiveRecord's own sql.active_record notification uses), and a failed update! never
651
+ # fires the event at all - nothing here is misreported as a transition that didn't
652
+ # actually happen.
653
+ def transition!(from:, to:, event:, **attrs)
619
654
  allowed_from = Array(from)
620
655
  unless allowed_from.include?(status)
621
656
  raise InvalidTransitionError,
@@ -623,7 +658,8 @@ module Belay
623
658
  "(expected current status to be #{allowed_from.inspect})"
624
659
  end
625
660
 
626
- update!(**attrs, status: to)
661
+ payload = { run_id: id, workflow: workflow, workflow_version: workflow_version, from: status, to: to }
662
+ ActiveSupport::Notifications.instrument("belay.run.#{event}", payload) { update!(**attrs, status: to) }
627
663
  end
628
664
  end
629
665
  end
data/lib/belay/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Belay
4
- VERSION = "0.1.0"
4
+ VERSION = "0.3.0"
5
5
  end
@@ -0,0 +1,110 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "active_job"
4
+ require "net/http"
5
+ require "json"
6
+ require "uri"
7
+ require "openssl"
8
+
9
+ module Belay
10
+ # POSTs a single belay.* event to Belay.config.webhook_url, enqueued by
11
+ # Belay::Webhooks.deliver (never called directly by application code). A separate
12
+ # job, not an inline HTTP call from inside the instrument block that fired the
13
+ # event, so a slow or down webhook receiver can never add latency to - or fail - a
14
+ # gate pausing, a step getting flagged needs_review, or a run completing. Same
15
+ # reasoning as ResumeRunJob/JanitorJob already being jobs rather than inline calls.
16
+ #
17
+ # event/payload, not the Run/Step records themselves: ActiveJob serializes a plain
18
+ # Hash (with its original String/Symbol keys preserved) without needing GlobalID,
19
+ # and every value already in an instrumentation payload (ids, strings, booleans) is
20
+ # plain-JSON-safe as-is - no serializer concern to design around. (Webhooks.deliver
21
+ # itself is responsible for never handing this job a payload containing the raw
22
+ # :exception_object ActiveSupport::Notifications::Instrumenter adds when a wrapped
23
+ # block raises - see that module's own comment - so this job never has to guard
24
+ # against an unserializable argument.)
25
+ #
26
+ # webhook_url/webhook_secret are both read here, at perform time, not captured at
27
+ # enqueue time - so clearing webhook_url between enqueue and perform (an app
28
+ # deliberately turning webhooks off mid-flight) makes this job a clean no-op rather
29
+ # than delivering to a URL the app just unset.
30
+ #
31
+ # OPEN_TIMEOUT/READ_TIMEOUT are set explicitly (unlike AnthropicProvider/
32
+ # OpenAIProvider/GeminiProvider's own Net::HTTP calls, which set neither): those
33
+ # three call a trusted vendor's own API; webhook_url is an arbitrary URL the host
34
+ # app configured, which could point at a receiver that hangs indefinitely without
35
+ # ever closing the connection. Net::HTTP's stdlib default (60s/60s) would otherwise
36
+ # let one hung receiver tie up a job worker for minutes across all 5 retry_on
37
+ # attempts; a short, fixed timeout here caps that per attempt.
38
+ #
39
+ # retry_on, not discard_on: a webhook receiver having a bad minute is exactly the
40
+ # case retrying helps with, and letting it eventually raise once attempts are
41
+ # exhausted surfaces the failure in whatever the host app's own job backend already
42
+ # does with a failed job (Sidekiq's dead set, GoodJob's errors, etc.) - silently
43
+ # swallowing a permanently-failing webhook forever would hide a real integration
44
+ # break from the one app that could actually fix it. A malformed webhook_url is
45
+ # deliberately NOT in this list, below - retrying the exact same unparseable string
46
+ # five times, with increasing delay, can never succeed; #perform raises a plain
47
+ # Belay::Error for that case instead, immediately, once.
48
+ class WebhookJob < ActiveJob::Base
49
+ OPEN_TIMEOUT = 5
50
+ READ_TIMEOUT = 5
51
+
52
+ # Raised on any non-2xx response. The receiving endpoint's own response body is
53
+ # preserved verbatim in the message, same reasoning as every Provider's own
54
+ # RequestError.
55
+ class RequestError < Error
56
+ attr_reader :status
57
+
58
+ def initialize(status:, body:)
59
+ @status = status
60
+ super("Belay webhook POST failed with status #{status}: #{body}")
61
+ end
62
+ end
63
+
64
+ retry_on RequestError, wait: :polynomially_longer, attempts: 5
65
+ retry_on Net::OpenTimeout, Net::ReadTimeout, SocketError, wait: :polynomially_longer, attempts: 5
66
+
67
+ def perform(event:, payload:)
68
+ url = Belay.config.webhook_url
69
+ return if url.nil?
70
+
71
+ body = JSON.generate(event: event, payload: payload)
72
+ response = post(parse_webhook_url!(url), body)
73
+ status = response.code.to_i
74
+ raise RequestError.new(status: status, body: response.body) unless status.between?(200, 299)
75
+ end
76
+
77
+ private
78
+
79
+ # URI() alone doesn't catch every malformed URL - a scheme-less string like
80
+ # "example.com/hook" parses successfully into a URI::Generic with host: nil
81
+ # (it's treated as a relative reference), rather than raising. Checking host
82
+ # explicitly catches that case too, so it fails the same clear, immediate,
83
+ # non-retried way as a string URI() itself rejects, instead of surfacing later
84
+ # as a cryptic Net::HTTP connection error against a nil host.
85
+ def parse_webhook_url!(url)
86
+ uri = URI(url)
87
+ raise URI::InvalidURIError, "no host" if uri.host.nil?
88
+
89
+ uri
90
+ rescue URI::InvalidURIError => e
91
+ raise Error, "Belay.config.webhook_url (#{url.inspect}) is not a valid URL: #{e.message}"
92
+ end
93
+
94
+ def post(uri, body)
95
+ Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == "https",
96
+ open_timeout: OPEN_TIMEOUT, read_timeout: READ_TIMEOUT) do |http|
97
+ request = Net::HTTP::Post.new(uri, request_headers(body))
98
+ request.body = body
99
+ http.request(request)
100
+ end
101
+ end
102
+
103
+ def request_headers(body)
104
+ headers = { "content-type" => "application/json" }
105
+ secret = Belay.config.webhook_secret
106
+ headers["X-Belay-Signature"] = OpenSSL::HMAC.hexdigest("SHA256", secret, body) if secret
107
+ headers
108
+ end
109
+ end
110
+ end
@@ -0,0 +1,42 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Belay
4
+ # Bridges ActiveSupport::Notifications (docs/instrumentation.md) to
5
+ # Belay::WebhookJob. .subscribe! is called exactly once, from Belay::Engine's own
6
+ # initializer, at process boot - not per-request, not re-subscribed if config
7
+ # changes later.
8
+ #
9
+ # Subscribes to every belay.* event unconditionally, then filters on
10
+ # Belay.config.webhook_events inside the single callback, rather than
11
+ # subscribing/unsubscribing per event as config.webhook_events changes - config can
12
+ # be (re)assigned any time after boot (a test's Belay.reset_config!, a console
13
+ # session), and re-deriving the subscription from a config value that might not
14
+ # exist yet at initializer time would make this fragile to initializer ordering.
15
+ # Checking config inside the callback instead means order never matters: by the
16
+ # time any belay.* event actually fires, config has long since been set.
17
+ module Webhooks
18
+ def self.subscribe!
19
+ ActiveSupport::Notifications.subscribe(/^belay\./) do |name, _start, _finish, _id, payload|
20
+ deliver(name, payload)
21
+ end
22
+ end
23
+
24
+ def self.deliver(name, payload)
25
+ return if Belay.config.webhook_url.nil?
26
+ return unless Belay.config.webhook_events.include?(name)
27
+ # Run#transition! wraps update! in ActiveSupport::Notifications.instrument(...) {
28
+ # }; if that block raises (e.g. a validation/locking error), Instrumenter still
29
+ # notifies every subscriber (the `ensure` in its own #instrument), but first
30
+ # mutates payload in place to add :exception (a 2-element String Array - fine)
31
+ # AND :exception_object (the raw exception instance - not ActiveJob-serializable,
32
+ # would raise ActiveJob::SerializationError from inside this very call). Skip it
33
+ # entirely rather than strip just the unserializable key: the transition this
34
+ # payload describes didn't actually happen, so forwarding it at all - "run
35
+ # completed," when update! raised - would be reporting something false, not
36
+ # just a serialization hazard.
37
+ return if payload.key?(:exception)
38
+
39
+ WebhookJob.perform_later(event: name, payload: payload)
40
+ end
41
+ end
42
+ end
data/lib/belay.rb CHANGED
@@ -20,9 +20,13 @@ module Belay
20
20
  require_relative "belay/provider"
21
21
  require_relative "belay/fake_provider"
22
22
  require_relative "belay/anthropic_provider"
23
+ require_relative "belay/openai_provider"
24
+ require_relative "belay/gemini_provider"
23
25
  require_relative "belay/executor"
24
26
  require_relative "belay/resume_run_job"
25
27
  require_relative "belay/janitor_job"
28
+ require_relative "belay/webhook_job"
29
+ require_relative "belay/webhooks"
26
30
  require_relative "belay/engine"
27
31
 
28
32
  # Raised by Run#start!/pause!/resume!/complete!/fail!/cancel! when the current
@@ -43,6 +43,17 @@ module Belay
43
43
  def create_belay_rules_migration
44
44
  migration_template "create_belay_rules.rb.tt", "db/migrate/create_belay_rules.rb"
45
45
  end
46
+
47
+ # Added after 0.2.0 shipped, as a new migration - never editing
48
+ # create_belay_runs.rb.tt above once it's shipped (see that file's own
49
+ # history). A host app upgrading the gem and re-running `rails generate
50
+ # belay:install` only gets this one copied in: migration_template skips
51
+ # every migration whose class already exists in db/migrate, so the five
52
+ # above are untouched no-ops on a re-run.
53
+ def add_budget_alert_sent_at_to_belay_runs_migration
54
+ migration_template "add_budget_alert_sent_at_to_belay_runs.rb.tt",
55
+ "db/migrate/add_budget_alert_sent_at_to_belay_runs.rb"
56
+ end
46
57
  end
47
58
  end
48
59
  end
@@ -0,0 +1,9 @@
1
+ class AddBudgetAlertSentAtToBelayRuns < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
2
+ def change
3
+ # Set the one time Executor fires belay.run.budget_exceeded for this run (once
4
+ # its total cost_cents crosses config.run_cost_budget_cents) - nil forever for
5
+ # a run that never has a budget configured, or never crosses it. Guards against
6
+ # firing the same event again on every later step that's still over budget.
7
+ add_column :belay_runs, :budget_alert_sent_at, :datetime
8
+ end
9
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: belay
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Swakhar Dey
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2026-10-27 00:00:00.000000000 Z
11
+ date: 2026-11-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: actionpack
@@ -164,6 +164,7 @@ files:
164
164
  - app/views/belay/runs/show.html.erb
165
165
  - app/views/belay/shared/_flash.html.erb
166
166
  - app/views/belay/shared/_sidebar.html.erb
167
+ - app/views/belay/stats/_budget_banner.html.erb
167
168
  - app/views/belay/stats/_chart.html.erb
168
169
  - app/views/belay/stats/_table.html.erb
169
170
  - app/views/belay/stats/index.html.erb
@@ -180,7 +181,9 @@ files:
180
181
  - lib/belay/engine.rb
181
182
  - lib/belay/executor.rb
182
183
  - lib/belay/fake_provider.rb
184
+ - lib/belay/gemini_provider.rb
183
185
  - lib/belay/janitor_job.rb
186
+ - lib/belay/openai_provider.rb
184
187
  - lib/belay/provider.rb
185
188
  - lib/belay/provider_response.rb
186
189
  - lib/belay/resume_run_job.rb
@@ -188,8 +191,11 @@ files:
188
191
  - lib/belay/run.rb
189
192
  - lib/belay/step.rb
190
193
  - lib/belay/version.rb
194
+ - lib/belay/webhook_job.rb
195
+ - lib/belay/webhooks.rb
191
196
  - lib/belay/workflow.rb
192
197
  - lib/generators/belay/install/install_generator.rb
198
+ - lib/generators/belay/install/templates/add_budget_alert_sent_at_to_belay_runs.rb.tt
193
199
  - lib/generators/belay/install/templates/create_belay_approvals.rb.tt
194
200
  - lib/generators/belay/install/templates/create_belay_cache_entries.rb.tt
195
201
  - lib/generators/belay/install/templates/create_belay_rules.rb.tt