batchwatch 0.2.1 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +165 -13
- data/lib/batchwatch/client.rb +161 -1
- data/lib/batchwatch/job.rb +741 -0
- data/lib/batchwatch.rb +1 -0
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d76f91aceeae04301676b3b43f48d94ee8cac41b83dede45b38a7d770c8aab1b
|
|
4
|
+
data.tar.gz: '0493dd22486b9768eb7910e8ed595fecf51efc6419111acd1f0ca8e4d3f8838e'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 64057bdf78674a4152d026ad69fecec08c549fb0dbd84cd8d04fd48dc858f2700a4fe76b9a2f9bbcf9eaede4ad6b963a8876931405dab47f5153e766d8ccead4
|
|
7
|
+
data.tar.gz: 2223af0f8679cb1219994f2f246af43616e4fc64f175cbed5e8f590e8739c18f585f8e5a7ad5c04cfccf98360536c4da67ca54065fbc4e8f22a79b37414983be
|
data/README.md
CHANGED
|
@@ -12,15 +12,11 @@ No dependencies. The standard library — `net/http`, `json`, `uri`, `socket`,
|
|
|
12
12
|
|
|
13
13
|
## Install
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
gem install batchwatch
|
|
16
16
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
```
|
|
21
|
-
|
|
22
|
-
or vendor the three files under `lib/` — they have no third-party imports. A
|
|
23
|
-
RubyGems release is on the way.
|
|
17
|
+
On [RubyGems](https://rubygems.org/gems/batchwatch), or add `gem "batchwatch"`
|
|
18
|
+
to your `Gemfile`. Or vendor the three files under `lib/` — they have no
|
|
19
|
+
third-party imports.
|
|
24
20
|
|
|
25
21
|
## Two lines
|
|
26
22
|
|
|
@@ -52,6 +48,118 @@ Get a key with no email and no card:
|
|
|
52
48
|
|
|
53
49
|
curl -X POST https://batchwatch.dev/v1/keys -d '{"label":"my pipeline"}'
|
|
54
50
|
|
|
51
|
+
## The deadline guard — batch when you can, sync when you must
|
|
52
|
+
|
|
53
|
+
Batch is half the price, but a queue that misses your deadline can take down a
|
|
54
|
+
product. `bw.batch(...)` gets you both: it runs your batch, watches the clock,
|
|
55
|
+
and if the batch has not finished by your deadline it cancels it and runs your
|
|
56
|
+
synchronous fallback instead — so your job always gets an answer, on time.
|
|
57
|
+
|
|
58
|
+
```ruby
|
|
59
|
+
job = bw.batch("gpt-5.6-sol", deadline: "15m",
|
|
60
|
+
on_deadline: -> { client.chat.completions.create(...) })
|
|
61
|
+
job.submit(-> { client.batches.create(...) })
|
|
62
|
+
result = job.result
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
You hand over two callables — the batch-create and the sync fallback — and the
|
|
66
|
+
client runs them. It never sees or builds your provider payload; there is no
|
|
67
|
+
field for it, exactly as with `track`. If the batch finishes in time you get its
|
|
68
|
+
result; if the deadline fires you get the fallback's result.
|
|
69
|
+
|
|
70
|
+
Every fallback is a **measured prediction outcome**: "batching would have missed
|
|
71
|
+
— running sync was right." It goes down the same accuracy path `should_batch`
|
|
72
|
+
already feeds, so the server can score how often the guard was needed. Nothing
|
|
73
|
+
new leaves the machine.
|
|
74
|
+
|
|
75
|
+
`deadline` speaks the same duration language as `should_batch`'s `max_wait` —
|
|
76
|
+
`"15m"`, `"6h"`, `"30s"`, a bare number of seconds, or `nil` for no guard. The
|
|
77
|
+
defaults duck-type the OpenAI / Anthropic batch shape; a different provider
|
|
78
|
+
passes `poll:` / `cancel:` / `result_of:` callables.
|
|
79
|
+
|
|
80
|
+
### The poll loop is ours, not yours
|
|
81
|
+
|
|
82
|
+
`result` owns the wait so you do not write the same sleep/backoff loop everyone
|
|
83
|
+
else does. It polls with **exponential backoff and jitter**, never faster than a
|
|
84
|
+
**rate-limit floor** (a naive one-second loop against a 24-hour job is 86,400
|
|
85
|
+
requests and an angry provider) and never slower than a ceiling. The **first**
|
|
86
|
+
interval is informed by the model's own **measured p50** — no reason to poll
|
|
87
|
+
every five seconds against a model whose median queue time is forty minutes —
|
|
88
|
+
and reading that p50 **fails open**: if batchwatch is unreachable, polling simply
|
|
89
|
+
continues on the fixed fallback schedule.
|
|
90
|
+
|
|
91
|
+
The 24-hour **expiry is a distinct terminal state**, never silently a timeout:
|
|
92
|
+
`job.expired` is `true` when the batch hit the cutoff, so you can tell "the queue
|
|
93
|
+
was slow" from "the batch failed".
|
|
94
|
+
|
|
95
|
+
Tune the cadence if you need to; the defaults are sane:
|
|
96
|
+
|
|
97
|
+
```ruby
|
|
98
|
+
job = bw.batch("gpt-5.6-sol", deadline: "6h",
|
|
99
|
+
on_deadline: -> { client.chat.completions.create(...) },
|
|
100
|
+
poll_base: 30, poll_floor: 5, poll_ceiling: 900)
|
|
101
|
+
job.submit(-> { client.batches.create(...) })
|
|
102
|
+
result = job.result
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Driving an async or worker loop yourself instead of blocking a thread? Step the
|
|
106
|
+
same state machine one poll at a time:
|
|
107
|
+
|
|
108
|
+
```ruby
|
|
109
|
+
state = job.poll_once # PollState::RUNNING / DONE / EXPIRED / FAILED
|
|
110
|
+
if state == Batchwatch::PollState::RUNNING
|
|
111
|
+
sleep(job.next_interval) # backoff + jitter, already applied
|
|
112
|
+
end
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### Partial completion — landed, failed, expired
|
|
116
|
+
|
|
117
|
+
A batch of 20,000 requests is not binary: some land, some fail per-request, and
|
|
118
|
+
some never return before the 24-hour expiry. `job.split` separates the three,
|
|
119
|
+
mapped back to your own objects **by `custom_id`** (never by index — provider
|
|
120
|
+
ordering is not guaranteed):
|
|
121
|
+
|
|
122
|
+
```ruby
|
|
123
|
+
result = job.split(downloaded_lines, every_submitted_id)
|
|
124
|
+
result.landed # count that came back clean
|
|
125
|
+
result.failed # count that failed per-request
|
|
126
|
+
result.expired # count still outstanding at the 24h cutoff
|
|
127
|
+
result.complete? # true only if EVERYTHING landed — never a silent success
|
|
128
|
+
result.result_for("my-request-42") # your object for one id, mapped correctly
|
|
129
|
+
|
|
130
|
+
# retry only what failed — idempotent, so a second call submits nothing new
|
|
131
|
+
child = job.retry_failed(->(failed_ids) { client.batches.create(...) })
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
An expired job is reported to batchwatch as `status="expired"`, which is recorded
|
|
135
|
+
but kept out of the percentiles (only `completed` rows count) — so a slow queue
|
|
136
|
+
neither pollutes p90 nor loses the "the queue was slow" signal. That measurement
|
|
137
|
+
rule lives with the ingest contract on the server, under `/v1/calls/complete` →
|
|
138
|
+
"Partial completion", not only here in the client.
|
|
139
|
+
|
|
140
|
+
## Subscribe to outcome alerts
|
|
141
|
+
|
|
142
|
+
Get told when a model's queue degrades, on your own webhook or Slack. All three
|
|
143
|
+
calls are keyed to your own token, so — like `my_calls` and unlike the
|
|
144
|
+
measurement path — they do **not** fail open: without a token they raise
|
|
145
|
+
`Batchwatch::AuthError` rather than silently pretend you subscribed.
|
|
146
|
+
|
|
147
|
+
```ruby
|
|
148
|
+
bw = Batchwatch::Client.new(token: "bw_...")
|
|
149
|
+
|
|
150
|
+
# webhook: omit the secret and the server mints one, returned ONCE — read it here
|
|
151
|
+
sub = bw.subscribe("webhook", "https://example.com/hook",
|
|
152
|
+
providers: "openai", min_severity: "severe")
|
|
153
|
+
puts sub["secret"] # shown only on creation, never again
|
|
154
|
+
|
|
155
|
+
# slack needs no secret
|
|
156
|
+
bw.subscribe("slack", "https://hooks.slack.com/services/...")
|
|
157
|
+
|
|
158
|
+
# list your active subscriptions (never the secret), then revoke one by id
|
|
159
|
+
bw.subscriptions.each { |s| puts [s["id"], s["channel"], s["target"]].join(" ") }
|
|
160
|
+
bw.unsubscribe(sub["id"])
|
|
161
|
+
```
|
|
162
|
+
|
|
55
163
|
## It fails open, always
|
|
56
164
|
|
|
57
165
|
If batchwatch is down, slow, or broken, your job must not notice. That is the
|
|
@@ -100,6 +208,35 @@ back labelled as a ceiling.
|
|
|
100
208
|
Passing `output_tokens: 0` really does send `0`: zero is a measurement, absence
|
|
101
209
|
is not.
|
|
102
210
|
|
|
211
|
+
## Read your own contributions and key status
|
|
212
|
+
|
|
213
|
+
Two read routes, both keyed to your own token. They are the readback for
|
|
214
|
+
`track()`: there is no route to a single call by id, so `my_calls` is how you
|
|
215
|
+
confirm a measurement landed.
|
|
216
|
+
|
|
217
|
+
```ruby
|
|
218
|
+
bw = Batchwatch::Client.new(token: "bw_...")
|
|
219
|
+
|
|
220
|
+
# everything this key has contributed
|
|
221
|
+
mine = bw.my_calls
|
|
222
|
+
puts "#{mine['count']} calls"
|
|
223
|
+
mine["calls"].each { |c| puts [c["model"], c["status"], c["duration_s"]].join(" ") }
|
|
224
|
+
|
|
225
|
+
# page through — pagination is keyed on started_server, never an offset, so a
|
|
226
|
+
# row arriving mid-walk cannot make you skip anything. Follow "next" (or pass
|
|
227
|
+
# after: <unix seconds>) until it is null.
|
|
228
|
+
mine = bw.my_calls(after: 1787666964, limit: 100)
|
|
229
|
+
|
|
230
|
+
# your tier, whether you are contributing, and your quota
|
|
231
|
+
status = bw.key_status
|
|
232
|
+
puts [status["tier"], status["contributing"], status["quota"]["calls_left"]].join(" ")
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Both require a key and, like `subscribe`, do **not** fail open — without a token
|
|
236
|
+
there is nothing to read, so they raise `Batchwatch::AuthError` rather than
|
|
237
|
+
return an empty answer that reads like "no contributions". The server's row comes
|
|
238
|
+
back verbatim; no key is renamed.
|
|
239
|
+
|
|
103
240
|
## Spooling
|
|
104
241
|
|
|
105
242
|
When a measurement cannot be delivered, the completed record is appended to a
|
|
@@ -152,6 +289,20 @@ in CI.
|
|
|
152
289
|
- `t.done(output_tokens: nil, status: "completed", ttfb_ms: nil)`
|
|
153
290
|
- `t.failed(status: "failed")`
|
|
154
291
|
- `t.started(input_tokens: ...)` when the count is only known after submission
|
|
292
|
+
- `batch(model, deadline: nil, on_deadline: nil, provider: "openai", **kw) -> BatchJob` — the deadline-guarded, self-polling job
|
|
293
|
+
- `job.submit(create)` — run the caller's batch-create callable (a block/proc/lambda), remember the handle
|
|
294
|
+
- `job.result` — block, polling with backoff + jitter, or fall back to `on_deadline` at the deadline
|
|
295
|
+
- `job.poll_once -> PollState` — one non-blocking step for an async/worker loop; `job.next_interval` is the seconds to sleep before the next
|
|
296
|
+
- `job.fell_back` — `true` once the guard has fired; `job.expired` — `true` if the batch hit the 24h expiry; `job.poll_count` — polls made
|
|
297
|
+
- `job.split(results = nil, custom_ids = nil) -> BatchResult` — split landed / failed / expired, mapped by `custom_id` (`result.complete?` / `result.result_for(id)`)
|
|
298
|
+
- `job.retry_failed(resubmit, result: nil) -> BatchJob | nil` — resubmit only the failed subset, idempotently
|
|
299
|
+
- poll cadence: `poll_base:` / `poll_floor:` / `poll_ceiling:` / `poll_backoff:` / `poll_jitter:` / `use_p50_cadence:`
|
|
300
|
+
- override the provider shape with `poll:` / `cancel:` / `result_of:` / `succeeded:` / `expired:`
|
|
301
|
+
- `subscribe(channel, target, secret: nil, providers: nil, models: nil, min_severity: nil) -> Hash` — outcome alerts for this key (`POST /v1/subscriptions`); requires a key
|
|
302
|
+
- `subscriptions -> Array` — this key's active subscriptions (`GET /v1/subscriptions`); requires a key
|
|
303
|
+
- `unsubscribe(sub_id) -> Hash` — revoke one of your own subscriptions (`DELETE /v1/subscriptions/{id}`); requires a key
|
|
304
|
+
- `my_calls(after: nil, limit: nil) -> Hash` — this key's own contributions (`GET /v1/calls/mine`); requires a key, follows `next`/`after` for pagination
|
|
305
|
+
- `key_status -> Hash` — this key's tier / contribution status / quota (`GET /v1/keys/current`); requires a key
|
|
155
306
|
- `flush(timeout: 5.0) -> true/false` — wait for outstanding submissions before exit
|
|
156
307
|
- `flush_spool(timeout: nil) -> Integer` — send what is on disk, returns accepted
|
|
157
308
|
|
|
@@ -161,11 +312,12 @@ in CI.
|
|
|
161
312
|
# or a single suite:
|
|
162
313
|
ruby -Ilib -Itest test/test_fail_open.rb
|
|
163
314
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
315
|
+
No network beyond loopback. They start real HTTP servers on ephemeral ports (raw
|
|
316
|
+
`TCPServer`, port 0) rather than stubbing `Net::HTTP`: the thing under test is
|
|
317
|
+
network behaviour, so the network should be in the test. The allowlist and
|
|
318
|
+
fail-open tests carry positive controls, so a client that sent nothing at all
|
|
319
|
+
would fail them rather than pass. The poll-loop tests inject a clock, sleep and
|
|
320
|
+
rng, so the backoff schedule is pinned exactly with no real sleeps.
|
|
169
321
|
|
|
170
322
|
## Requirements
|
|
171
323
|
|
data/lib/batchwatch/client.rb
CHANGED
|
@@ -40,7 +40,7 @@ require "time"
|
|
|
40
40
|
require_relative "spool"
|
|
41
41
|
|
|
42
42
|
module Batchwatch
|
|
43
|
-
VERSION = "0.2.
|
|
43
|
+
VERSION = "0.2.2"
|
|
44
44
|
|
|
45
45
|
# How often, at most, we try to drain the spool on our own.
|
|
46
46
|
SPOOL_INTERVAL_S = 60.0
|
|
@@ -215,6 +215,16 @@ module Batchwatch
|
|
|
215
215
|
end
|
|
216
216
|
end
|
|
217
217
|
|
|
218
|
+
# Raised when an action REQUIRES a key and none is set.
|
|
219
|
+
#
|
|
220
|
+
# The measurement path fails open - with no key it degrades silently, because
|
|
221
|
+
# a missing telemetry call must never stand in the way of the caller's job.
|
|
222
|
+
# But reading your own contributions is an EXPLICIT action against a per-key
|
|
223
|
+
# route: here a silent empty answer would be a lie - the caller reads "no
|
|
224
|
+
# contributions" when in truth there was nothing to read. So we say it out
|
|
225
|
+
# loud instead.
|
|
226
|
+
class AuthError < StandardError; end
|
|
227
|
+
|
|
218
228
|
# The client. Every submission is non-blocking and fails open.
|
|
219
229
|
#
|
|
220
230
|
# Args:
|
|
@@ -278,6 +288,7 @@ module Batchwatch
|
|
|
278
288
|
when "GET" then Net::HTTP::Get
|
|
279
289
|
when "POST" then Net::HTTP::Post
|
|
280
290
|
when "PATCH" then Net::HTTP::Patch
|
|
291
|
+
when "DELETE" then Net::HTTP::Delete
|
|
281
292
|
else raise ArgumentError, "unknown method #{method}"
|
|
282
293
|
end
|
|
283
294
|
req = req_class.new(uri)
|
|
@@ -417,6 +428,98 @@ module Batchwatch
|
|
|
417
428
|
@advice_lock.synchronize { @advice[model] }
|
|
418
429
|
end
|
|
419
430
|
|
|
431
|
+
# ---------------------------------------------------------- subscription
|
|
432
|
+
|
|
433
|
+
# Subscribe to outcome alerts for THIS key. Returns the row.
|
|
434
|
+
#
|
|
435
|
+
# +channel+ is "webhook" or "slack", +target+ an https URL. +secret+ applies
|
|
436
|
+
# only to webhook: if you omit it, the server makes one itself and returns
|
|
437
|
+
# the plaintext ONCE in +secret+ - read it here and configure your receiver,
|
|
438
|
+
# it never comes again. +providers+ and +models+ are comma-strings (or omit
|
|
439
|
+
# = all), +min_severity+ is "degraded" or "severe".
|
|
440
|
+
#
|
|
441
|
+
# Unlike the measurement path, this one does NOT fail open: subscribing is an
|
|
442
|
+
# explicit action against an authorized route, so without a key we raise
|
|
443
|
+
# AuthError rather than silently pretend.
|
|
444
|
+
def subscribe(channel, target, secret: nil, providers: nil, models: nil,
|
|
445
|
+
min_severity: nil)
|
|
446
|
+
require_key("subscribe")
|
|
447
|
+
body = { "channel" => channel, "target" => target }
|
|
448
|
+
{ "secret" => secret, "providers" => providers, "models" => models,
|
|
449
|
+
"min_severity" => min_severity }.each do |name, value|
|
|
450
|
+
body[name] = value unless value.nil?
|
|
451
|
+
end
|
|
452
|
+
call("/v1/subscriptions", method: "POST", body: body)
|
|
453
|
+
end
|
|
454
|
+
|
|
455
|
+
# THIS key's active subscriptions, as an array. Never the secret.
|
|
456
|
+
def subscriptions
|
|
457
|
+
require_key("subscriptions")
|
|
458
|
+
r = call("/v1/subscriptions")
|
|
459
|
+
(r || {}).fetch("subscriptions", [])
|
|
460
|
+
end
|
|
461
|
+
|
|
462
|
+
# Revoke one of your OWN subscriptions by numeric id. Returns the response.
|
|
463
|
+
def unsubscribe(sub_id)
|
|
464
|
+
require_key("unsubscribe")
|
|
465
|
+
call("/v1/subscriptions/#{sub_id}", method: "DELETE")
|
|
466
|
+
end
|
|
467
|
+
|
|
468
|
+
# --------------------------------------------------------------- job (#178)
|
|
469
|
+
|
|
470
|
+
# A high-level batch job: deadline guard (#178) + poll loop (#179) + partial
|
|
471
|
+
# completion (#180).
|
|
472
|
+
#
|
|
473
|
+
# The epic's promise: *we take the callable, never the payload*. You give us
|
|
474
|
+
# two callables - the batch-create and the synchronous fallback - and we run
|
|
475
|
+
# them, watch the deadline, poll the batch, and hand back a result. We never
|
|
476
|
+
# see or construct the provider payload, exactly as with track().
|
|
477
|
+
#
|
|
478
|
+
# job = bw.batch("gpt-5.6-sol", deadline: "15m",
|
|
479
|
+
# on_deadline: -> { client.chat.completions.create(...) })
|
|
480
|
+
# job.submit(-> { client.batches.create(...) })
|
|
481
|
+
# result = job.result
|
|
482
|
+
#
|
|
483
|
+
# If the batch finishes before +deadline+ you get its result. If the deadline
|
|
484
|
+
# arrives first we cancel the batch (best-effort), run +on_deadline+ and
|
|
485
|
+
# return that instead - so the caller's job always gets an answer, on time.
|
|
486
|
+
# Every fallback is reported down the existing #101 accuracy path
|
|
487
|
+
# (acted_verdict=false on a completion) so the server can score the
|
|
488
|
+
# prediction; nothing new is sent.
|
|
489
|
+
#
|
|
490
|
+
# See Batchwatch::BatchJob for the full contract (poll cadence, poll_once,
|
|
491
|
+
# split, retry_failed, and the injectable clock/sleep/rng seams).
|
|
492
|
+
def batch(model, deadline: nil, on_deadline: nil, provider: "openai", **kw)
|
|
493
|
+
BatchJob.new(self, model, deadline: deadline, on_deadline: on_deadline,
|
|
494
|
+
provider: provider, **kw)
|
|
495
|
+
end
|
|
496
|
+
|
|
497
|
+
# Report a deadline-guard outcome down the existing #101 path.
|
|
498
|
+
#
|
|
499
|
+
# The deadline guard is a should_batch-style prediction carried out: when it
|
|
500
|
+
# falls back, the outcome is the server's "run_sync" case ("batching would
|
|
501
|
+
# have missed"). #101 already carries exactly that - a completion measurement
|
|
502
|
+
# bearing acted_verdict and deadline_s, which the server judges against its
|
|
503
|
+
# own measured duration and rolls into /v1/accuracy. So we seed the same
|
|
504
|
+
# per-model advice slot should_batch writes and emit a completion through the
|
|
505
|
+
# SAME track() path - no second reporter, no new route.
|
|
506
|
+
#
|
|
507
|
+
# A completion in time carries acted_verdict=true (the batch held the
|
|
508
|
+
# deadline); a fallback carries false. Both sides are reported, which is what
|
|
509
|
+
# keeps the accuracy number honest. Percentiles are omitted, not invented as
|
|
510
|
+
# zero (rule #30): the guard did not quote a p90.
|
|
511
|
+
def record_job_outcome(model, provider, acted_verdict, deadline_s)
|
|
512
|
+
@advice_lock.synchronize do
|
|
513
|
+
@advice[model] = {
|
|
514
|
+
"acted_verdict" => acted_verdict,
|
|
515
|
+
"deadline_s" => deadline_s,
|
|
516
|
+
"quoted_p50_s" => nil,
|
|
517
|
+
"quoted_p90_s" => nil
|
|
518
|
+
}
|
|
519
|
+
end
|
|
520
|
+
track(model, provider: provider, mode: "batch") { |t| t.done }
|
|
521
|
+
end
|
|
522
|
+
|
|
420
523
|
# ------------------------------------------------------------- measurement
|
|
421
524
|
|
|
422
525
|
# Measure one call. The submission happens in the background.
|
|
@@ -447,6 +550,52 @@ module Batchwatch
|
|
|
447
550
|
end
|
|
448
551
|
end
|
|
449
552
|
|
|
553
|
+
# ------------------------------------------------------- read your own key
|
|
554
|
+
|
|
555
|
+
# The measurements THIS key has contributed. Requires a key.
|
|
556
|
+
#
|
|
557
|
+
# GET /v1/calls/mine returns everything the service holds that came from
|
|
558
|
+
# your key - the only readback there is: there is no route to a single call
|
|
559
|
+
# by id, and none to anyone else's rows. Use it to confirm a measurement
|
|
560
|
+
# landed after track() / flush_spool().
|
|
561
|
+
#
|
|
562
|
+
# Returns the server's row verbatim: {"label", "count", "next", "calls":
|
|
563
|
+
# [...], "note"}. +next+ is a ready-made URL for the following page (null on
|
|
564
|
+
# the last), keyed on started_server so a walk cannot skip a row that
|
|
565
|
+
# arrives mid-walk.
|
|
566
|
+
#
|
|
567
|
+
# after: unix seconds; only rows with a later started_server come back.
|
|
568
|
+
# Omit for the first page.
|
|
569
|
+
# limit: rows per page, clamped server-side to 1-1000 (default 500).
|
|
570
|
+
#
|
|
571
|
+
# Unlike the measurement path this does NOT fail open: the route is per-key,
|
|
572
|
+
# so without one we raise AuthError rather than pretend.
|
|
573
|
+
def my_calls(after: nil, limit: nil)
|
|
574
|
+
require_key("my_calls")
|
|
575
|
+
q = {}
|
|
576
|
+
q["after"] = after.to_i unless after.nil?
|
|
577
|
+
q["limit"] = limit.to_i unless limit.nil?
|
|
578
|
+
path = "/v1/calls/mine"
|
|
579
|
+
path += "?#{URI.encode_www_form(q)}" unless q.empty?
|
|
580
|
+
call(path)
|
|
581
|
+
end
|
|
582
|
+
|
|
583
|
+
# THIS key's tier, contribution status and quota. Requires a key.
|
|
584
|
+
#
|
|
585
|
+
# GET /v1/keys/current returns the row verbatim: {"label", "tier",
|
|
586
|
+
# "contributing", "recent_measurements", "required", "window_days",
|
|
587
|
+
# "delayed_by_s", "live", "quota": {...}}. +tier+ is the effective tier
|
|
588
|
+
# derived from your measurements (free / contributor; only paid is
|
|
589
|
+
# operator-assigned), and +quota+ reports the calls window for the gated
|
|
590
|
+
# routes.
|
|
591
|
+
#
|
|
592
|
+
# Unlike the measurement path this does NOT fail open: without a key there
|
|
593
|
+
# is no key to describe, so we raise AuthError.
|
|
594
|
+
def key_status
|
|
595
|
+
require_key("key_status")
|
|
596
|
+
call("/v1/keys/current")
|
|
597
|
+
end
|
|
598
|
+
|
|
450
599
|
# --------------------------------------------------------------- spool
|
|
451
600
|
|
|
452
601
|
# Send everything waiting on disk. Returns the number accepted.
|
|
@@ -516,6 +665,17 @@ module Batchwatch
|
|
|
516
665
|
|
|
517
666
|
private
|
|
518
667
|
|
|
668
|
+
# Fail loudly if there is no key. The read routes are per-key - without one
|
|
669
|
+
# there is nothing to read, and a silent empty answer would be misread as
|
|
670
|
+
# "no contributions".
|
|
671
|
+
def require_key(action)
|
|
672
|
+
return if @token
|
|
673
|
+
|
|
674
|
+
raise AuthError,
|
|
675
|
+
"batchwatch: #{action} requires a key - set token: ... or " \
|
|
676
|
+
"$BATCHWATCH_TOKEN"
|
|
677
|
+
end
|
|
678
|
+
|
|
519
679
|
def monotonic
|
|
520
680
|
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
521
681
|
end
|
|
@@ -0,0 +1,741 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# The high-level batch job - the client does the annoying parts.
|
|
4
|
+
#
|
|
5
|
+
# The epic's core promise is *"we take the callable, never the payload"*. Every
|
|
6
|
+
# other batchwatch surface answers a question and hands the decision back to
|
|
7
|
+
# you; this one carries it out. You hand us two callables - the batch-create and
|
|
8
|
+
# the synchronous fallback - and we run them, watch the deadline, poll the
|
|
9
|
+
# batch, and give you a result. We never see or construct the provider payload:
|
|
10
|
+
# there is no field for it, by construction, exactly as with track().
|
|
11
|
+
#
|
|
12
|
+
# job = bw.batch("gpt-5.6-sol", deadline: "15m",
|
|
13
|
+
# on_deadline: -> { client.chat.completions.create(...) })
|
|
14
|
+
# job.submit(-> { client.batches.create(...) })
|
|
15
|
+
# result = job.result
|
|
16
|
+
#
|
|
17
|
+
# What happens:
|
|
18
|
+
#
|
|
19
|
+
# * job.submit(fn) runs your batch-create callable and remembers the handle it
|
|
20
|
+
# returned and the time it started. We never look inside the handle beyond the
|
|
21
|
+
# questions we have to ask it - is it done, did it succeed, what did it return
|
|
22
|
+
# - and those go through callables you can override.
|
|
23
|
+
# * job.result waits for the batch. If it finishes before the deadline you get
|
|
24
|
+
# its result. If the deadline arrives first, we cancel the batch (best-effort),
|
|
25
|
+
# run on_deadline and give you *that* result instead - your job still gets an
|
|
26
|
+
# answer, on time.
|
|
27
|
+
# * **Every fallback is a measured prediction outcome.** A deadline miss is the
|
|
28
|
+
# server's #101 "run_sync" case, so we report it down the SAME accuracy path
|
|
29
|
+
# track() already uses: a completion carrying acted_verdict=false and the
|
|
30
|
+
# deadline_s we were given. No second reporter, no new route, no payload.
|
|
31
|
+
#
|
|
32
|
+
# The wait itself (#179) owns the poll loop so the caller does not: exponential
|
|
33
|
+
# backoff with jitter, a rate-limit floor so a naive one-second loop against a
|
|
34
|
+
# 24-hour job cannot fire 86,400 requests, a ceiling so the interval does not
|
|
35
|
+
# run away, and a first cadence informed by our own measured p50 for the model.
|
|
36
|
+
# All of it fails open: if batchwatch cannot tell us the p50, we poll on the
|
|
37
|
+
# fixed fallback schedule rather than stopping.
|
|
38
|
+
#
|
|
39
|
+
# Partial completion (#180) is the third promise: a real batch of 20,000
|
|
40
|
+
# requests comes back with some landed, some failed per-request, and some never
|
|
41
|
+
# returned before the 24-hour expiry. BatchJob#split hands you a BatchResult
|
|
42
|
+
# that separates those three, mapped back to your own objects by custom_id
|
|
43
|
+
# (never by index - provider ordering is not guaranteed), and
|
|
44
|
+
# BatchJob#retry_failed resubmits only the failed subset with a stable
|
|
45
|
+
# idempotency key so a retry cannot duplicate the job.
|
|
46
|
+
|
|
47
|
+
module Batchwatch
|
|
48
|
+
# Raised for a misuse of the job API that we must not paper over.
|
|
49
|
+
#
|
|
50
|
+
# The measurement path fails open, because a lost telemetry call must never
|
|
51
|
+
# stand in the way of the user's job. The job path is different: it IS the
|
|
52
|
+
# user's job. If we cannot run it as asked - no submit before a result, no
|
|
53
|
+
# fallback to fall back to - a silent no-op would strand the caller waiting on
|
|
54
|
+
# a result that never comes. So we say it out loud instead.
|
|
55
|
+
class BatchJobError < StandardError; end
|
|
56
|
+
|
|
57
|
+
# ------------------------------------------------------------- poll cadence
|
|
58
|
+
#
|
|
59
|
+
# The floor is the rate-limit guard: never poll faster than this, no matter
|
|
60
|
+
# how eager the schedule, because a one-second loop against a 24-hour job is
|
|
61
|
+
# 86,400 requests and an angry provider. The ceiling stops the backoff running
|
|
62
|
+
# away - once the interval reaches it we keep polling at that cadence rather
|
|
63
|
+
# than doubling forever. The base is where the schedule starts when we have no
|
|
64
|
+
# p50 to inform it. All three are seconds; every one is overridable on
|
|
65
|
+
# bw.batch(...).
|
|
66
|
+
POLL_FLOOR_S = 5.0 # rate-limit floor: never poll faster than this
|
|
67
|
+
POLL_BASE_S = 5.0 # first interval when we have no measured p50
|
|
68
|
+
POLL_CEILING_S = 300.0 # backoff ceiling: never poll slower than this
|
|
69
|
+
POLL_BACKOFF = 2.0 # multiply the interval by this each miss
|
|
70
|
+
POLL_JITTER = 0.5 # full-jitter fraction: sleep in [(1-j)*d, d]
|
|
71
|
+
|
|
72
|
+
# How much of a model's measured p50 we spend before the first poll. A median
|
|
73
|
+
# queue time of forty minutes means the batch is very unlikely to be done in
|
|
74
|
+
# the first few minutes, so polling then only burns rate limit. We wait a
|
|
75
|
+
# fraction of the p50, clamped to the ceiling, so a slow model gets a patient
|
|
76
|
+
# first poll and a fast one is not starved. This is the use of our own dataset
|
|
77
|
+
# that the card calls a genuine selling point - nobody without the
|
|
78
|
+
# measurements can do it.
|
|
79
|
+
POLL_P50_FRACTION = 0.5
|
|
80
|
+
|
|
81
|
+
# The outcome of a single non-blocking poll - the async/worker driver's view.
|
|
82
|
+
#
|
|
83
|
+
# BatchJob#poll_once returns one of these constants so a caller driving the
|
|
84
|
+
# loop from their own event loop (rather than letting #result block a thread)
|
|
85
|
+
# can decide what to do next without reading provider internals:
|
|
86
|
+
#
|
|
87
|
+
# * RUNNING - not terminal yet; sleep next_interval and poll again.
|
|
88
|
+
# * DONE - terminal and successful; call #result for the value.
|
|
89
|
+
# * EXPIRED - terminal via the 24h expiry; a DISTINCT state, never a silent
|
|
90
|
+
# timeout, so the caller can tell "the queue was slow" from "the batch
|
|
91
|
+
# failed".
|
|
92
|
+
# * FAILED - terminal but not successful (failed / cancelled).
|
|
93
|
+
#
|
|
94
|
+
# EXPIRED and FAILED both fall back the same way inside #result; they are kept
|
|
95
|
+
# separate here because the caller - and #180's measurement - must be able to
|
|
96
|
+
# tell them apart.
|
|
97
|
+
module PollState
|
|
98
|
+
RUNNING = "running"
|
|
99
|
+
DONE = "done"
|
|
100
|
+
EXPIRED = "expired"
|
|
101
|
+
FAILED = "failed"
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
# Terminal batch statuses across the providers we have seen. A job in one of
|
|
105
|
+
# these is not going to change again, so waiting longer is pointless. Mirrors
|
|
106
|
+
# the status maps in the backfill tool (completed / failed / expired /
|
|
107
|
+
# cancelled), plus OpenAI's "finalizing"-then-"completed" and Anthropic's
|
|
108
|
+
# "ended".
|
|
109
|
+
TERMINAL_STATUSES = %w[
|
|
110
|
+
completed complete succeeded success
|
|
111
|
+
failed error errored
|
|
112
|
+
expired cancelled canceled ended
|
|
113
|
+
].freeze
|
|
114
|
+
|
|
115
|
+
# The 24h expiry is a TERMINAL state, not "keep waiting" - and it is a
|
|
116
|
+
# distinct one. Counting an expired job as completed pollutes the accuracy
|
|
117
|
+
# number; counting it as a plain failure loses the signal that the queue was
|
|
118
|
+
# slow. So we name it separately (both here for the poll loop and in #180's
|
|
119
|
+
# split).
|
|
120
|
+
EXPIRED_STATUSES = %w[expired].freeze
|
|
121
|
+
|
|
122
|
+
SUCCESS_STATUSES = %w[completed complete succeeded success ended].freeze
|
|
123
|
+
|
|
124
|
+
# Markers for the coarse whole-batch split and an unmappable result line. Kept
|
|
125
|
+
# as frozen sentinels rather than magic strings so they cannot collide with a
|
|
126
|
+
# real caller custom_id.
|
|
127
|
+
WHOLE_MARKER = "__batchwatch_whole_batch__"
|
|
128
|
+
UNMAPPED_MARKER = "__batchwatch_unmapped__"
|
|
129
|
+
|
|
130
|
+
# The lower-cased status string of a handle, or "". Duck-typed.
|
|
131
|
+
#
|
|
132
|
+
# Reads handle.status (a method, or a [:status]/["status"] hash key) and
|
|
133
|
+
# nothing else. Never touches the payload.
|
|
134
|
+
def self.handle_status(handle)
|
|
135
|
+
status =
|
|
136
|
+
if handle.respond_to?(:status)
|
|
137
|
+
handle.status
|
|
138
|
+
elsif handle.respond_to?(:[])
|
|
139
|
+
handle[:status] || handle["status"]
|
|
140
|
+
end
|
|
141
|
+
status.nil? ? "" : status.to_s.downcase
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
# Is this batch finished? Duck-typed against the common provider shape. The
|
|
145
|
+
# OpenAI / Anthropic batch objects both expose a status string; we read it and
|
|
146
|
+
# nothing else. "Finished" means a terminal status; anything else means keep
|
|
147
|
+
# waiting.
|
|
148
|
+
def self.default_poll(handle)
|
|
149
|
+
TERMINAL_STATUSES.include?(handle_status(handle))
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
# Did the batch finish *successfully* (as opposed to failed/expired)?
|
|
153
|
+
def self.default_succeeded(handle)
|
|
154
|
+
SUCCESS_STATUSES.include?(handle_status(handle))
|
|
155
|
+
end
|
|
156
|
+
|
|
157
|
+
# Did the batch reach the 24h expiry? Kept separate from default_succeeded so
|
|
158
|
+
# an expiry is a distinct terminal state the caller and the measurement can
|
|
159
|
+
# see - never silently a timeout.
|
|
160
|
+
def self.default_expired(handle)
|
|
161
|
+
EXPIRED_STATUSES.include?(handle_status(handle))
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
# What the caller gets back for a completed batch: the handle itself. We do
|
|
165
|
+
# not download or parse the batch output - that is the caller's data and their
|
|
166
|
+
# provider SDK's job. Override with result_of: if you want something else.
|
|
167
|
+
def self.default_result(handle)
|
|
168
|
+
handle
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
# Best-effort cancel of a running batch. Never raises upward. Duck-typed
|
|
172
|
+
# against a handle.cancel method if the provider object carries one; otherwise
|
|
173
|
+
# a no-op. The deadline guard has already decided to fall back, so a cancel
|
|
174
|
+
# that fails to land only wastes provider spend on a batch nobody will read.
|
|
175
|
+
def self.default_cancel(handle)
|
|
176
|
+
handle.cancel if handle.respond_to?(:cancel)
|
|
177
|
+
end
|
|
178
|
+
|
|
179
|
+
# The custom_id of a provider result line, or nil. Duck-typed. Reads
|
|
180
|
+
# item["custom_id"] (or [:custom_id], or the attribute) - the field OpenAI and
|
|
181
|
+
# Anthropic both round-trip on a batch line. Nothing else is read.
|
|
182
|
+
def self.custom_id_of(item)
|
|
183
|
+
if item.respond_to?(:[])
|
|
184
|
+
return item["custom_id"] || item[:custom_id]
|
|
185
|
+
end
|
|
186
|
+
item.respond_to?(:custom_id) ? item.custom_id : nil
|
|
187
|
+
end
|
|
188
|
+
|
|
189
|
+
# Did this per-request line fail? Duck-typed against the provider shape. A
|
|
190
|
+
# batch output line carries an error (non-null) on failure, or a response with
|
|
191
|
+
# a non-2xx status_code. We read only those envelope fields - never the body.
|
|
192
|
+
def self.result_error?(item)
|
|
193
|
+
error =
|
|
194
|
+
if item.respond_to?(:[])
|
|
195
|
+
item["error"] || item[:error]
|
|
196
|
+
elsif item.respond_to?(:error)
|
|
197
|
+
item.error
|
|
198
|
+
end
|
|
199
|
+
return true if error
|
|
200
|
+
|
|
201
|
+
response =
|
|
202
|
+
if item.respond_to?(:[])
|
|
203
|
+
item["response"] || item[:response]
|
|
204
|
+
elsif item.respond_to?(:response)
|
|
205
|
+
item.response
|
|
206
|
+
end
|
|
207
|
+
if response.respond_to?(:[])
|
|
208
|
+
code = response["status_code"] || response[:status_code]
|
|
209
|
+
unless code.nil?
|
|
210
|
+
begin
|
|
211
|
+
n = Integer(code)
|
|
212
|
+
return !(n >= 200 && n < 300)
|
|
213
|
+
rescue ArgumentError, TypeError
|
|
214
|
+
return false
|
|
215
|
+
end
|
|
216
|
+
end
|
|
217
|
+
end
|
|
218
|
+
false
|
|
219
|
+
end
|
|
220
|
+
|
|
221
|
+
# One high-level batch job: deadline guard (#178), poll loop (#179), partial
|
|
222
|
+
# completion (#180).
|
|
223
|
+
#
|
|
224
|
+
# Created by Batchwatch::Client#batch. result blocks, polling with backoff and
|
|
225
|
+
# jitter until the batch is terminal or the deadline fires. For an async /
|
|
226
|
+
# worker context that does not want a blocked thread, poll_once drives the same
|
|
227
|
+
# state machine one step at a time.
|
|
228
|
+
#
|
|
229
|
+
# The clock / sleep / rng seams exist for deterministic tests: the backoff
|
|
230
|
+
# schedule and its jitter are pure functions of these, so a fake clock and a
|
|
231
|
+
# fake rng pin the exact poll count with no real sleeps. Production leaves them
|
|
232
|
+
# as the real time / random.
|
|
233
|
+
class BatchJob
|
|
234
|
+
def initialize(bw, model, deadline: nil, on_deadline: nil,
|
|
235
|
+
provider: "openai", poll: nil, cancel: nil, result_of: nil,
|
|
236
|
+
succeeded: nil, expired: nil,
|
|
237
|
+
poll_base: nil, poll_floor: nil,
|
|
238
|
+
poll_ceiling: POLL_CEILING_S, poll_backoff: POLL_BACKOFF,
|
|
239
|
+
poll_jitter: POLL_JITTER, use_p50_cadence: true,
|
|
240
|
+
poll_interval: nil, clock: nil, sleep: nil, rng: nil)
|
|
241
|
+
@bw = bw
|
|
242
|
+
@model = model
|
|
243
|
+
@provider = provider
|
|
244
|
+
@on_deadline = on_deadline
|
|
245
|
+
@deadline_s = Batchwatch.seconds(deadline)
|
|
246
|
+
@poll = poll || Batchwatch.method(:default_poll)
|
|
247
|
+
@cancel = cancel || Batchwatch.method(:default_cancel)
|
|
248
|
+
@result_of = result_of || Batchwatch.method(:default_result)
|
|
249
|
+
@succeeded = succeeded || Batchwatch.method(:default_succeeded)
|
|
250
|
+
@expired = expired || Batchwatch.method(:default_expired)
|
|
251
|
+
|
|
252
|
+
# poll_interval is #178's name for the (then-fixed) poll cadence. #179
|
|
253
|
+
# renamed it to poll_base (the FIRST interval before backoff) and split out
|
|
254
|
+
# an explicit poll_floor (the rate-limit guard). We honour the old name as
|
|
255
|
+
# an alias so #178's two-callable usage - and its tests, which set a tiny
|
|
256
|
+
# interval to make the deadline fire fast - keep working untouched (epic
|
|
257
|
+
# rule #4). When only poll_interval is given, a small value is also taken
|
|
258
|
+
# as the floor so the deadline-guard tests still poll as fast as they
|
|
259
|
+
# asked, not clamped up to the 5s default.
|
|
260
|
+
if poll_base.nil?
|
|
261
|
+
poll_base = poll_interval.nil? ? POLL_BASE_S : poll_interval
|
|
262
|
+
end
|
|
263
|
+
if poll_floor.nil?
|
|
264
|
+
poll_floor = poll_interval.nil? ? POLL_FLOOR_S : [poll_interval, POLL_FLOOR_S].min
|
|
265
|
+
end
|
|
266
|
+
|
|
267
|
+
# Poll cadence (#179). The floor and ceiling are the two guard rails; the
|
|
268
|
+
# base is stored raw because EVERY interval that reaches a sleep goes
|
|
269
|
+
# through clamp, which is the ONE place the floor and ceiling are applied
|
|
270
|
+
# (P7: one canonical site, so the rate-limit floor cannot be
|
|
271
|
+
# half-removed). A base below the floor therefore still polls at the floor.
|
|
272
|
+
@poll_floor = [0.001, poll_floor.to_f].max
|
|
273
|
+
@poll_ceiling = [@poll_floor, poll_ceiling.to_f].max
|
|
274
|
+
@poll_base = [0.0, poll_base.to_f].max
|
|
275
|
+
@poll_backoff = [1.0, poll_backoff.to_f].max
|
|
276
|
+
@poll_jitter = [[poll_jitter.to_f, 1.0].min, 0.0].max
|
|
277
|
+
@use_p50_cadence = use_p50_cadence
|
|
278
|
+
|
|
279
|
+
# Injectable seams for deterministic tests; real time/random otherwise.
|
|
280
|
+
@clock = clock || -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }
|
|
281
|
+
@sleep = sleep || ->(s) { Kernel.sleep(s) }
|
|
282
|
+
@rand = rng || -> { Kernel.rand }
|
|
283
|
+
|
|
284
|
+
@handle = nil # the provider's batch object
|
|
285
|
+
@submitted_at = nil # monotonic clock at submit
|
|
286
|
+
@result = nil
|
|
287
|
+
@fell_back = false # did the deadline guard fire?
|
|
288
|
+
@expired_flag = false # did the batch hit the 24h expiry?
|
|
289
|
+
@resolved = false # has result run to completion?
|
|
290
|
+
@interval = nil # current backoff interval, lazily seeded
|
|
291
|
+
@poll_count = 0 # how many times we asked "are you done?"
|
|
292
|
+
@retries = nil # lazy per-job retry registry (#180)
|
|
293
|
+
end
|
|
294
|
+
|
|
295
|
+
attr_reader :handle
|
|
296
|
+
|
|
297
|
+
# --------------------------------------------------------------- lifecycle
|
|
298
|
+
|
|
299
|
+
# Run the caller's batch-create callable and remember the handle.
|
|
300
|
+
#
|
|
301
|
+
# +create+ is a zero-argument callable (a block, proc, or lambda) that
|
|
302
|
+
# returns the provider's batch object - typically
|
|
303
|
+
# -> { client.batches.create(...) }. We call it, store what it returns, and
|
|
304
|
+
# start the deadline clock. We never inspect the payload it built; the handle
|
|
305
|
+
# is opaque to us except for the questions the poll/result callables ask.
|
|
306
|
+
#
|
|
307
|
+
# Raising from create is the caller's own error and is left untouched - the
|
|
308
|
+
# batch never started, so there is nothing for us to guard.
|
|
309
|
+
def submit(create = nil, &block)
|
|
310
|
+
create ||= block
|
|
311
|
+
raise BatchJobError, "batchwatch: submit() needs a callable" if create.nil?
|
|
312
|
+
|
|
313
|
+
unless @handle.nil?
|
|
314
|
+
raise BatchJobError,
|
|
315
|
+
"batchwatch: this job was already submitted - create a new " \
|
|
316
|
+
"bw.batch(...) for a second batch"
|
|
317
|
+
end
|
|
318
|
+
@handle = create.call
|
|
319
|
+
@submitted_at = @clock.call
|
|
320
|
+
@handle
|
|
321
|
+
end
|
|
322
|
+
|
|
323
|
+
# Block until the batch finishes, or the deadline fires.
|
|
324
|
+
#
|
|
325
|
+
# Returns the batch result if it completed in time, otherwise the result of
|
|
326
|
+
# on_deadline. Idempotent: a second call returns the same result without
|
|
327
|
+
# re-waiting or re-reporting.
|
|
328
|
+
#
|
|
329
|
+
# Raises BatchJobError if called before submit.
|
|
330
|
+
def result
|
|
331
|
+
return @result if @resolved
|
|
332
|
+
|
|
333
|
+
if @handle.nil?
|
|
334
|
+
raise BatchJobError,
|
|
335
|
+
"batchwatch: result() before submit() - call " \
|
|
336
|
+
"job.submit(-> { client.batches.create(...) }) first"
|
|
337
|
+
end
|
|
338
|
+
|
|
339
|
+
completed_in_time = wait_loop
|
|
340
|
+
if completed_in_time && @succeeded.call(@handle)
|
|
341
|
+
@result = @result_of.call(@handle)
|
|
342
|
+
report_outcome(fell_back: false)
|
|
343
|
+
else
|
|
344
|
+
@result = fallback(timed_out: !completed_in_time)
|
|
345
|
+
end
|
|
346
|
+
@resolved = true
|
|
347
|
+
@result
|
|
348
|
+
end
|
|
349
|
+
|
|
350
|
+
# Take one non-blocking step of the poll loop and report the state.
|
|
351
|
+
#
|
|
352
|
+
# The non-blocking variant (#179): an async or worker context can drive the
|
|
353
|
+
# wait itself instead of handing us a thread. Call it, read the returned
|
|
354
|
+
# PollState, and - if it is RUNNING - sleep next_interval (which we have
|
|
355
|
+
# already advanced) before the next call. result is still the way to get the
|
|
356
|
+
# value once a step returns a terminal state; this only reports where the
|
|
357
|
+
# batch is.
|
|
358
|
+
#
|
|
359
|
+
# Unlike result this never sleeps and never falls back - it is a single
|
|
360
|
+
# question. The deadline is still honoured by result's own loop.
|
|
361
|
+
def poll_once
|
|
362
|
+
if @handle.nil?
|
|
363
|
+
raise BatchJobError,
|
|
364
|
+
"batchwatch: poll_once() before submit() - call " \
|
|
365
|
+
"job.submit(-> { client.batches.create(...) }) first"
|
|
366
|
+
end
|
|
367
|
+
@interval = seed_interval if @interval.nil?
|
|
368
|
+
state = poll_state
|
|
369
|
+
@interval = advance_interval(@interval) if state == PollState::RUNNING
|
|
370
|
+
state
|
|
371
|
+
end
|
|
372
|
+
|
|
373
|
+
# Seconds to sleep before the next poll_once, jitter applied. Already
|
|
374
|
+
# includes the rate-limit floor and the jitter, so an async driver can sleep
|
|
375
|
+
# exactly this and match the blocking loop's cadence.
|
|
376
|
+
def next_interval
|
|
377
|
+
@interval = seed_interval if @interval.nil?
|
|
378
|
+
sleep_for
|
|
379
|
+
end
|
|
380
|
+
|
|
381
|
+
# How many times we have asked the batch whether it is done. This is the
|
|
382
|
+
# number that gets someone rate-limited, so it is exposed for a caller - and
|
|
383
|
+
# a test - to assert a bound on.
|
|
384
|
+
def poll_count
|
|
385
|
+
@poll_count
|
|
386
|
+
end
|
|
387
|
+
|
|
388
|
+
# true once the batch was seen to reach the 24h expiry. A distinct terminal
|
|
389
|
+
# state, never folded into a timeout.
|
|
390
|
+
def expired
|
|
391
|
+
@expired_flag
|
|
392
|
+
end
|
|
393
|
+
|
|
394
|
+
# true once the deadline guard has fired and the fallback was used.
|
|
395
|
+
def fell_back
|
|
396
|
+
@fell_back
|
|
397
|
+
end
|
|
398
|
+
|
|
399
|
+
# ------------------------------------------------- partial completion (#180)
|
|
400
|
+
|
|
401
|
+
# Split the batch into landed / failed / expired, mapped by custom_id.
|
|
402
|
+
#
|
|
403
|
+
# A real batch is not binary: a job of 20,000 requests can come back with
|
|
404
|
+
# some landed, some failed per-request, and some never returned before the
|
|
405
|
+
# 24h expiry. This is where silent data loss lives - a naive "the job
|
|
406
|
+
# completed" reads all three as success. BatchResult separates them, with
|
|
407
|
+
# counts and the per-request ids, so the caller can act.
|
|
408
|
+
#
|
|
409
|
+
# +results+ is the per-request outcomes the caller downloaded from THEIR
|
|
410
|
+
# provider (we never fetch content). Each item is a mapping carrying a
|
|
411
|
+
# custom_id and either a success or an error - the shape the OpenAI /
|
|
412
|
+
# Anthropic batch output lines already have. +custom_ids+ is the full set the
|
|
413
|
+
# caller SUBMITTED; any id in it that has no result line is expired
|
|
414
|
+
# (outstanding at the 24h cutoff). Mapping is by custom_id, never by index:
|
|
415
|
+
# provider result ordering is not guaranteed.
|
|
416
|
+
#
|
|
417
|
+
# Pass neither and we read the whole-batch terminal status off the handle
|
|
418
|
+
# instead - a whole-batch expiry becomes an all-expired result, a whole-batch
|
|
419
|
+
# failure an all-failed one.
|
|
420
|
+
def split(results = nil, custom_ids = nil)
|
|
421
|
+
BatchResult.from_results(self, results, custom_ids)
|
|
422
|
+
end
|
|
423
|
+
|
|
424
|
+
# Resubmit ONLY the failed subset, once, with an idempotency key.
|
|
425
|
+
#
|
|
426
|
+
# +resubmit+ is a callable the caller hands us - resubmit.call(failed_ids) -
|
|
427
|
+
# that recreates a batch over just those ids. Same epic rule: we take the
|
|
428
|
+
# callable, never the payload. We call it with the list of failed custom_ids
|
|
429
|
+
# and remember the handle it returns as a fresh child job.
|
|
430
|
+
#
|
|
431
|
+
# Idempotent by construction: calling it twice submits ONE retry. Duplicating
|
|
432
|
+
# a 20,000-request job is an expensive failure, so the key is the sorted set
|
|
433
|
+
# of failed ids - a pure function of WHAT is being retried, not of when. A
|
|
434
|
+
# second call with the same failure set returns the same child without
|
|
435
|
+
# resubmitting; a genuinely different failure set is a different retry and
|
|
436
|
+
# does submit.
|
|
437
|
+
#
|
|
438
|
+
# Returns the child BatchJob, or nil when there is nothing to retry (an empty
|
|
439
|
+
# failure set is a no-op, not an empty batch).
|
|
440
|
+
def retry_failed(resubmit, result: nil)
|
|
441
|
+
res = result.nil? ? split : result
|
|
442
|
+
failed_ids = res.failed_ids.sort
|
|
443
|
+
return nil if failed_ids.empty?
|
|
444
|
+
|
|
445
|
+
@retries ||= {}
|
|
446
|
+
key = failed_ids
|
|
447
|
+
return @retries[key] if @retries.key?(key)
|
|
448
|
+
|
|
449
|
+
child = @bw.batch(@model, deadline: nil, on_deadline: @on_deadline,
|
|
450
|
+
provider: @provider)
|
|
451
|
+
child.submit(-> { resubmit.call(failed_ids.dup) })
|
|
452
|
+
@retries[key] = child
|
|
453
|
+
child
|
|
454
|
+
end
|
|
455
|
+
|
|
456
|
+
# ------------------------------------------------------- context-manager use
|
|
457
|
+
|
|
458
|
+
# A block form so the partial-completion cleanup has an obvious home; the
|
|
459
|
+
# base job has nothing to do on the way out. Yields self and returns the
|
|
460
|
+
# block's value. We do NOT force a result the caller never asked for.
|
|
461
|
+
def with
|
|
462
|
+
yield self
|
|
463
|
+
end
|
|
464
|
+
|
|
465
|
+
private
|
|
466
|
+
|
|
467
|
+
# ------------------------------------------------------------- the poll loop
|
|
468
|
+
#
|
|
469
|
+
# #179 owns the wait. The contract result leans on:
|
|
470
|
+
# wait_loop -> true if the batch reached a terminal status before the
|
|
471
|
+
# deadline (result then asks succeeded/expired what to
|
|
472
|
+
# do with it)
|
|
473
|
+
# false if the deadline arrived first
|
|
474
|
+
# side: it must NOT cancel, report, or fall back. Deciding what a false
|
|
475
|
+
# means is result's job.
|
|
476
|
+
def wait_loop
|
|
477
|
+
@interval = seed_interval
|
|
478
|
+
loop do
|
|
479
|
+
state = poll_state
|
|
480
|
+
return true if state != PollState::RUNNING
|
|
481
|
+
return false if past_deadline?
|
|
482
|
+
|
|
483
|
+
@sleep.call(sleep_for)
|
|
484
|
+
@interval = advance_interval(@interval)
|
|
485
|
+
end
|
|
486
|
+
end
|
|
487
|
+
|
|
488
|
+
# Ask the batch its state ONCE and classify it. Counts the poll.
|
|
489
|
+
def poll_state
|
|
490
|
+
@poll_count += 1
|
|
491
|
+
return PollState::RUNNING unless @poll.call(@handle)
|
|
492
|
+
|
|
493
|
+
# Terminal. Which kind?
|
|
494
|
+
if @expired.call(@handle)
|
|
495
|
+
@expired_flag = true
|
|
496
|
+
return PollState::EXPIRED
|
|
497
|
+
end
|
|
498
|
+
return PollState::DONE if @succeeded.call(@handle)
|
|
499
|
+
|
|
500
|
+
PollState::FAILED
|
|
501
|
+
end
|
|
502
|
+
|
|
503
|
+
# ---------------------------------------------------------- cadence + backoff
|
|
504
|
+
|
|
505
|
+
# The first poll interval: p50-informed when we can, fixed otherwise. We
|
|
506
|
+
# spend a fraction of the model's measured p50 before the first poll. The
|
|
507
|
+
# read fails open (measured_p50 swallows everything and returns nil), so an
|
|
508
|
+
# unreachable or silent batchwatch drops us to the fixed base with no
|
|
509
|
+
# interruption. Clamped into [floor, ceiling] so a wild p50 cannot push us
|
|
510
|
+
# past either guard.
|
|
511
|
+
def seed_interval
|
|
512
|
+
base = @poll_base
|
|
513
|
+
if @use_p50_cadence
|
|
514
|
+
p50 = measured_p50
|
|
515
|
+
base = p50 * POLL_P50_FRACTION if p50 && p50 > 0
|
|
516
|
+
end
|
|
517
|
+
clamp(base)
|
|
518
|
+
end
|
|
519
|
+
|
|
520
|
+
# Exponential backoff, clamped to the ceiling.
|
|
521
|
+
def advance_interval(interval)
|
|
522
|
+
clamp(interval * @poll_backoff)
|
|
523
|
+
end
|
|
524
|
+
|
|
525
|
+
# The ONE place the floor and ceiling are applied (P7).
|
|
526
|
+
def clamp(interval)
|
|
527
|
+
[@poll_floor, [@poll_ceiling, interval.to_f].min].max
|
|
528
|
+
end
|
|
529
|
+
|
|
530
|
+
# The model's measured p50 queue time in seconds, or nil. Reuses the
|
|
531
|
+
# client's existing read of /v1/wait (wait_now) - no new route, no new
|
|
532
|
+
# payload field. Fails open by construction: wait_now already swallows every
|
|
533
|
+
# error and returns nil when it cannot answer, and we defend once more here
|
|
534
|
+
# so a malformed number can never take the poll loop down. A nil means "poll
|
|
535
|
+
# on the fixed schedule", never "stop polling".
|
|
536
|
+
def measured_p50
|
|
537
|
+
info = @bw.wait_now(@model, provider: @provider, mode: "batch")
|
|
538
|
+
return nil unless info
|
|
539
|
+
|
|
540
|
+
%w[p50_s observed_wait_s planning_wait_s].each do |key|
|
|
541
|
+
value = info[key]
|
|
542
|
+
return value.to_f unless value.nil?
|
|
543
|
+
end
|
|
544
|
+
nil
|
|
545
|
+
rescue StandardError
|
|
546
|
+
nil
|
|
547
|
+
end
|
|
548
|
+
|
|
549
|
+
def past_deadline?
|
|
550
|
+
return false if @deadline_s.nil?
|
|
551
|
+
|
|
552
|
+
(@clock.call - @submitted_at) >= @deadline_s
|
|
553
|
+
end
|
|
554
|
+
|
|
555
|
+
# Sleep before the next poll: jittered interval, clamped to the deadline.
|
|
556
|
+
#
|
|
557
|
+
# Full jitter: the actual sleep is drawn uniformly from
|
|
558
|
+
# [max(floor, (1-jitter)*interval), interval]. interval is already floored
|
|
559
|
+
# (it comes from clamp), and we floor the window bottom too, so jitter can
|
|
560
|
+
# spread a fleet that all submitted at once without any client ever dipping
|
|
561
|
+
# under the rate-limit floor. Then we clamp to the deadline so a coarse
|
|
562
|
+
# interval never overshoots the guard.
|
|
563
|
+
def sleep_for
|
|
564
|
+
interval = @interval.nil? ? clamp(@poll_base) : @interval
|
|
565
|
+
low = [@poll_floor, interval * (1.0 - @poll_jitter)].max
|
|
566
|
+
delay = low + (interval - low) * @rand.call
|
|
567
|
+
return delay if @deadline_s.nil?
|
|
568
|
+
|
|
569
|
+
remaining = @deadline_s - (@clock.call - @submitted_at)
|
|
570
|
+
[0.0, [delay, remaining].min].max
|
|
571
|
+
end
|
|
572
|
+
|
|
573
|
+
# ------------------------------------------------------------- fallback
|
|
574
|
+
|
|
575
|
+
# Cancel the batch, run the sync fallback, report the outcome. Reached when
|
|
576
|
+
# the batch missed its deadline OR finished in a non-success terminal state
|
|
577
|
+
# (failed / expired). Either way the caller still needs an answer.
|
|
578
|
+
def fallback(timed_out:)
|
|
579
|
+
@fell_back = true
|
|
580
|
+
# Best-effort cancel first, so we are not paying for a batch nobody will
|
|
581
|
+
# read. A cancel failure is swallowed - the fallback matters more.
|
|
582
|
+
begin
|
|
583
|
+
@cancel.call(@handle)
|
|
584
|
+
rescue StandardError
|
|
585
|
+
nil
|
|
586
|
+
end
|
|
587
|
+
|
|
588
|
+
if @on_deadline.nil?
|
|
589
|
+
raise BatchJobError,
|
|
590
|
+
"batchwatch: the batch #{why_fallback(timed_out)} but no " \
|
|
591
|
+
"on_deadline fallback was given - pass " \
|
|
592
|
+
"on_deadline: -> { client.chat.completions.create(...) } " \
|
|
593
|
+
"to bw.batch(...)"
|
|
594
|
+
end
|
|
595
|
+
|
|
596
|
+
# Report BEFORE running the fallback: the outcome is already known (the
|
|
597
|
+
# batch would have missed), and the report goes out on a background thread,
|
|
598
|
+
# so it costs the caller's own call nothing.
|
|
599
|
+
report_outcome(fell_back: true)
|
|
600
|
+
@on_deadline.call
|
|
601
|
+
end
|
|
602
|
+
|
|
603
|
+
# Record the prediction outcome down the existing #101 accuracy path. A
|
|
604
|
+
# deadline fallback IS the server's "run_sync" case; a batch that completed
|
|
605
|
+
# in time is the "run_batch" hit. Reporting both sides is what makes the
|
|
606
|
+
# accuracy number honest. Failing to report must never affect the caller's
|
|
607
|
+
# job, so this rides the same fail-open contract as track().
|
|
608
|
+
def report_outcome(fell_back:)
|
|
609
|
+
@bw.record_job_outcome(@model, @provider, !fell_back, @deadline_s)
|
|
610
|
+
rescue StandardError
|
|
611
|
+
nil
|
|
612
|
+
end
|
|
613
|
+
|
|
614
|
+
def why_fallback(timed_out)
|
|
615
|
+
return "missed its deadline" if timed_out
|
|
616
|
+
return "expired (24h)" if @expired_flag
|
|
617
|
+
|
|
618
|
+
"did not succeed"
|
|
619
|
+
end
|
|
620
|
+
end
|
|
621
|
+
|
|
622
|
+
# The split of a batch into landed / failed / expired (#180).
|
|
623
|
+
#
|
|
624
|
+
# A real batch is three sets, not one: requests that landed, requests that
|
|
625
|
+
# failed per-request, and requests still outstanding when the 24h expiry hit.
|
|
626
|
+
# Reporting only "completed" loses the last two, which is exactly where silent
|
|
627
|
+
# data loss lives. This object carries the three sets, their counts, and the
|
|
628
|
+
# per-request ids, and - because provider ordering is not guaranteed - is built
|
|
629
|
+
# by mapping on custom_id, never by index.
|
|
630
|
+
#
|
|
631
|
+
# The measurement rule for a partial job (#180): an expiry is reported with
|
|
632
|
+
# status="expired" (NOT completed, which would pollute p90; NOT failed, which
|
|
633
|
+
# would lose the "queue was slow" signal), a per-request-failed job as
|
|
634
|
+
# completed with the landed count, and a clean job as completed. The client
|
|
635
|
+
# does not invent a fourth reading.
|
|
636
|
+
class BatchResult
|
|
637
|
+
attr_reader :job, :landed_ids, :failed_ids, :expired_ids, :results_by_id
|
|
638
|
+
|
|
639
|
+
def initialize(job, landed_ids:, failed_ids:, expired_ids:,
|
|
640
|
+
results_by_id: nil)
|
|
641
|
+
@job = job
|
|
642
|
+
@landed_ids = landed_ids.dup
|
|
643
|
+
@failed_ids = failed_ids.dup
|
|
644
|
+
@expired_ids = expired_ids.dup
|
|
645
|
+
# The caller's own result objects, keyed by custom_id, for the landed ones
|
|
646
|
+
# - so map-back is a hash lookup, never a positional zip.
|
|
647
|
+
@results_by_id = (results_by_id || {}).dup
|
|
648
|
+
end
|
|
649
|
+
|
|
650
|
+
# ---- counts, so a caller can branch without .length on three arrays -------
|
|
651
|
+
|
|
652
|
+
def landed
|
|
653
|
+
@landed_ids.length
|
|
654
|
+
end
|
|
655
|
+
|
|
656
|
+
def failed
|
|
657
|
+
@failed_ids.length
|
|
658
|
+
end
|
|
659
|
+
|
|
660
|
+
def expired
|
|
661
|
+
@expired_ids.length
|
|
662
|
+
end
|
|
663
|
+
|
|
664
|
+
def total
|
|
665
|
+
landed + failed + expired
|
|
666
|
+
end
|
|
667
|
+
|
|
668
|
+
# true only when every submitted request landed. No silent success: a job
|
|
669
|
+
# with anything failed or outstanding at expiry is NOT complete. This is the
|
|
670
|
+
# method a caller checks before treating the batch as done.
|
|
671
|
+
def complete?
|
|
672
|
+
total > 0 && failed.zero? && expired.zero?
|
|
673
|
+
end
|
|
674
|
+
|
|
675
|
+
# The caller's own landed result for one custom_id, or nil. The map-back the
|
|
676
|
+
# naive zip gets wrong: a lookup by id, so out-of-order provider results
|
|
677
|
+
# still reach the right caller object.
|
|
678
|
+
def result_for(custom_id)
|
|
679
|
+
@results_by_id[custom_id]
|
|
680
|
+
end
|
|
681
|
+
|
|
682
|
+
# Build the split from per-request outcomes, or from the handle status.
|
|
683
|
+
#
|
|
684
|
+
# See BatchJob#split for the argument contract. The per-request path
|
|
685
|
+
# classifies each result line by whether it carries an error, treats any
|
|
686
|
+
# submitted id with no line as expired, and keys the landed results by
|
|
687
|
+
# custom_id. The handle-status fallback reads the whole-batch terminal state
|
|
688
|
+
# for a provider that reports no lines.
|
|
689
|
+
def self.from_results(job, results = nil, custom_ids = nil)
|
|
690
|
+
return from_handle(job) if results.nil? && custom_ids.nil?
|
|
691
|
+
|
|
692
|
+
results = Array(results)
|
|
693
|
+
seen = {}
|
|
694
|
+
landed = []
|
|
695
|
+
failed = []
|
|
696
|
+
results.each do |item|
|
|
697
|
+
cid = Batchwatch.custom_id_of(item)
|
|
698
|
+
if cid.nil?
|
|
699
|
+
# A line we cannot map is not silently dropped into "landed" - that
|
|
700
|
+
# would be the zip-by-index mistake in another guise. It counts as
|
|
701
|
+
# failed so the caller sees the reconciliation gap.
|
|
702
|
+
failed << UNMAPPED_MARKER
|
|
703
|
+
next
|
|
704
|
+
end
|
|
705
|
+
seen[cid] = item
|
|
706
|
+
if Batchwatch.result_error?(item)
|
|
707
|
+
failed << cid
|
|
708
|
+
else
|
|
709
|
+
landed << cid
|
|
710
|
+
end
|
|
711
|
+
end
|
|
712
|
+
|
|
713
|
+
# Anything submitted but never returned is outstanding at expiry.
|
|
714
|
+
submitted = custom_ids.nil? ? seen.keys : Array(custom_ids)
|
|
715
|
+
expired = submitted.reject { |cid| seen.key?(cid) }
|
|
716
|
+
|
|
717
|
+
results_by_id = {}
|
|
718
|
+
landed.each { |cid| results_by_id[cid] = seen[cid] unless cid == UNMAPPED_MARKER }
|
|
719
|
+
new(job, landed_ids: landed.reject { |c| c == UNMAPPED_MARKER },
|
|
720
|
+
failed_ids: failed, expired_ids: expired,
|
|
721
|
+
results_by_id: results_by_id)
|
|
722
|
+
end
|
|
723
|
+
|
|
724
|
+
# Coarse split from the whole-batch terminal status. For a provider that
|
|
725
|
+
# gives us no per-request lines: an expired handle is all-expired, a
|
|
726
|
+
# non-success terminal handle all-failed, a success all-landed. There are no
|
|
727
|
+
# ids to list at this granularity, so the sets carry a single whole-batch
|
|
728
|
+
# marker each and the counts read 1/0/0-style.
|
|
729
|
+
def self.from_handle(job)
|
|
730
|
+
handle = job.handle
|
|
731
|
+
if Batchwatch.default_expired(handle) || job.expired
|
|
732
|
+
return new(job, landed_ids: [], failed_ids: [], expired_ids: [WHOLE_MARKER])
|
|
733
|
+
end
|
|
734
|
+
if Batchwatch.default_succeeded(handle)
|
|
735
|
+
return new(job, landed_ids: [WHOLE_MARKER], failed_ids: [], expired_ids: [])
|
|
736
|
+
end
|
|
737
|
+
|
|
738
|
+
new(job, landed_ids: [], failed_ids: [WHOLE_MARKER], expired_ids: [])
|
|
739
|
+
end
|
|
740
|
+
end
|
|
741
|
+
end
|
data/lib/batchwatch.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: batchwatch
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andreas Graae
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-08-
|
|
11
|
+
date: 2026-08-31 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: minitest
|
|
@@ -51,6 +51,7 @@ files:
|
|
|
51
51
|
- README.md
|
|
52
52
|
- lib/batchwatch.rb
|
|
53
53
|
- lib/batchwatch/client.rb
|
|
54
|
+
- lib/batchwatch/job.rb
|
|
54
55
|
- lib/batchwatch/spool.rb
|
|
55
56
|
homepage: https://batchwatch.dev
|
|
56
57
|
licenses:
|