coolhand 0.7.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/AGENTS.harness.md +49 -14
- data/CHANGELOG.md +42 -1
- data/README.md +60 -2
- data/Rakefile +7 -0
- data/docs/configuration.md +19 -0
- data/docs/declared_workloads.md +156 -0
- data/docs/feedback-links.md +48 -0
- data/docs/log-search.md +102 -0
- data/docs/openai.md +5 -1
- data/docs/template-search.md +46 -4
- data/docs/vertex.md +1 -0
- data/docs/workload-search.md +116 -0
- data/lib/coolhand/api_service.rb +9 -4
- data/lib/coolhand/base_interceptor.rb +7 -5
- data/lib/coolhand/configuration.rb +16 -2
- data/lib/coolhand/declarable.rb +178 -0
- data/lib/coolhand/declared_context.rb +106 -0
- data/lib/coolhand/errors.rb +2 -1
- data/lib/coolhand/log_queue.rb +179 -0
- data/lib/coolhand/log_service.rb +61 -0
- data/lib/coolhand/logger_service.rb +10 -18
- data/lib/coolhand/net_http_interceptor.rb +145 -139
- data/lib/coolhand/open_ai/batch_result_processor.rb +30 -29
- data/lib/coolhand/open_ai/webhook_validator.rb +27 -15
- data/lib/coolhand/optimization_feedback_link_service.rb +85 -0
- data/lib/coolhand/rails_auto_declare.rb +55 -0
- data/lib/coolhand/railtie.rb +11 -0
- data/lib/coolhand/read_query.rb +75 -0
- data/lib/coolhand/read_requests.rb +26 -13
- data/lib/coolhand/response_warnings.rb +39 -0
- data/lib/coolhand/template_service.rb +33 -33
- data/lib/coolhand/version.rb +1 -1
- data/lib/coolhand/vertex/batch_result_processor.rb +9 -1
- data/lib/coolhand/workload_service.rb +50 -0
- data/lib/coolhand/write_requests.rb +61 -0
- data/lib/coolhand.rb +63 -12
- metadata +17 -2
data/docs/openai.md
CHANGED
|
@@ -14,7 +14,7 @@ Requires Rails — `Coolhand::WebhookInterceptor` and `Coolhand::OpenAi::BatchRe
|
|
|
14
14
|
- Ensure you skip CSRF for the webhook endpoint: `skip_before_action :verify_authenticity_token`
|
|
15
15
|
- Override the `webhook_secret` method to return your OpenAI webhook secret
|
|
16
16
|
|
|
17
|
-
`intercept_batch_request` already validates the webhook and, for `batch.completed`/`batch.failed`/`batch.expired`/`batch.cancelled` events, calls `Coolhand::OpenAi::BatchResultProcessor`
|
|
17
|
+
`intercept_batch_request` already validates the webhook and, for `batch.completed`/`batch.failed`/`batch.expired`/`batch.cancelled` events, calls `Coolhand::OpenAi::BatchResultProcessor` — that part requires no code in your controller action. The batch's status is checked inline in the `before_action`, but for a completed batch the two JSONL file downloads and the one-request-log-per-item sends run on Coolhand's [background log thread](configuration.md#asynchronous-logging), so a large batch doesn't hold up your webhook response. (With `config.async_logging = false` this all runs inline, and a large batch can trip your webhook endpoint's timeout and OpenAI's retry behavior.) The example below shows a controller action doing additional, app-specific work on top of that (e.g. updating your own `BatchApiRequest` record and enqueuing your own background job), which is optional.
|
|
18
18
|
|
|
19
19
|
## Minimal example
|
|
20
20
|
|
|
@@ -59,6 +59,10 @@ end
|
|
|
59
59
|
# ...existing code...
|
|
60
60
|
```
|
|
61
61
|
|
|
62
|
+
## Signature header and secret rotation
|
|
63
|
+
|
|
64
|
+
The `webhook-signature` header may carry several space-separated signatures (`v1,sigA v1,sigB`), as senders produce while a webhook secret is being rotated. `Coolhand::OpenAi::WebhookValidator` accepts the request if any `v1,` entry matches your secret, comparing each in constant time and ignoring entries with any other version prefix. The timestamp and replay checks below still run, and only after a signature matches.
|
|
65
|
+
|
|
62
66
|
## Replay protection
|
|
63
67
|
|
|
64
68
|
`Coolhand::OpenAi::WebhookValidator` rejects requests whose `webhook-timestamp` is more than **300 seconds** away from the current time, and rejects a `webhook-id` it has already seen within that window — this prevents a captured request (from a log aggregator, APM trace, or proxy log) from being replayed to re-trigger `Coolhand::OpenAi::BatchResultProcessor`.
|
data/docs/template-search.md
CHANGED
|
@@ -41,7 +41,7 @@ This is **not** a port of the `search_templates` MCP tool, and the two do not ag
|
|
|
41
41
|
Template *creation, update and deprecation* stay on the MCP surface. This REST surface is
|
|
42
42
|
read-only, and there is no version-history sub-resource.
|
|
43
43
|
|
|
44
|
-
## `search_templates(search: nil, workload_id: nil, status: nil, include_deprecated: nil, include_system: nil, page: nil, per: nil)`
|
|
44
|
+
## `search_templates(search: nil, workload_id: nil, status: nil, include_deprecated: nil, include_system: nil, include_metrics: nil, days_back: nil, since: nil, until: nil, page: nil, per: nil)`
|
|
45
45
|
|
|
46
46
|
Search is a *parameter* on the list endpoint rather than a route of its own, so this is one method
|
|
47
47
|
rather than a list/search pair.
|
|
@@ -56,6 +56,10 @@ and this API's already agree, so there is no second spelling to translate throug
|
|
|
56
56
|
| `status` | String | `"draft"`, `"published"` or `"failure"`. Any other non-empty value is a `422` from the server; empty is treated as no filter. |
|
|
57
57
|
| `include_deprecated` | Boolean | Include templates with a non-null `deprecated_at`. Defaults to `false` server-side. |
|
|
58
58
|
| `include_system` | Boolean | Include the `Unmatched` / `Ignored API Calls` buckets. Defaults to `false` server-side. |
|
|
59
|
+
| `include_metrics` | Boolean | Adds a `:metrics` Hash to every row. See [Metrics](#metrics). |
|
|
60
|
+
| `days_back` | Integer | Rolling metrics window in days (server default 28, max 365). Ignored unless metrics are requested, and when `since` is given. |
|
|
61
|
+
| `since` | Time, DateTime, Date or String | Metrics window start, inclusive. Overrides `days_back`. |
|
|
62
|
+
| `until` | Time, DateTime, Date or String | Metrics window end, exclusive. Defaults to now. |
|
|
59
63
|
| `page` | Integer | 1-based. |
|
|
60
64
|
| `per` | Integer | Default 25, max 100, both enforced server-side. |
|
|
61
65
|
|
|
@@ -138,11 +142,14 @@ unmatched = Coolhand.template_service
|
|
|
138
142
|
puts unmatched[:log_count]
|
|
139
143
|
```
|
|
140
144
|
|
|
141
|
-
## `get_template(id)`
|
|
145
|
+
## `get_template(id, include_metrics: nil, days_back: nil, since: nil, until: nil)`
|
|
142
146
|
|
|
143
147
|
`id` is the template hashid — the `:id` field from `search_templates`.
|
|
144
148
|
|
|
145
|
-
Returns a Hash with every list field above, **plus** the full untruncated regexes the list omits
|
|
149
|
+
Returns a Hash with every list field above, **plus** the full untruncated regexes the list omits, and
|
|
150
|
+
a `:metrics` Hash. Unlike the list, metrics are **on by default** here: pass `include_metrics: false` to
|
|
151
|
+
omit them, which also skips the window validation. `days_back`, `since` and `until` set the window
|
|
152
|
+
exactly as they do on the list, and an invalid one is a `422` unless metrics are off.
|
|
146
153
|
|
|
147
154
|
| field | type |
|
|
148
155
|
|---|---|
|
|
@@ -153,6 +160,41 @@ Unlike the list, this filters on nothing but client ownership: a deprecated or s
|
|
|
153
160
|
reachable by id with **no opt-in flag**, because inspecting one of those is the usual reason to
|
|
154
161
|
fetch a template directly.
|
|
155
162
|
|
|
163
|
+
## Metrics
|
|
164
|
+
|
|
165
|
+
With `include_metrics: true` every row (and by default the `get_template` result) carries `:metrics`,
|
|
166
|
+
computed by the same SQL as the dashboard, so tiered pricing, cached-token discounts and reasoning tokens
|
|
167
|
+
are applied. `Coolhand.workload_service` returns the same object per workload, rolled up across its
|
|
168
|
+
templates ([Reading Workloads](workload-search.md)).
|
|
169
|
+
|
|
170
|
+
Averages and counters cover non-failed, directly-collected client logs inside the window. Keys are the
|
|
171
|
+
wire names, as Symbols.
|
|
172
|
+
|
|
173
|
+
| field | type | notes |
|
|
174
|
+
|---|---|---|
|
|
175
|
+
| `:days_back` | Integer or nil | `nil` when `since` defined the window. |
|
|
176
|
+
| `:since`, `:until` | String | The resolved window, ISO-8601 UTC. |
|
|
177
|
+
| `:request_count` | Integer | |
|
|
178
|
+
| `:failure_count` | Integer | Failed logs. |
|
|
179
|
+
| `:error_rate` | Float or nil | `nil` when `failure_count` is 0. |
|
|
180
|
+
| `:error_rate_change` | Float or nil | Against the window of the same length immediately before. |
|
|
181
|
+
| `:avg_cost_per_request` | Float or nil | `nil` when no log in the window could be priced. |
|
|
182
|
+
| `:total_cost` | Float or nil | |
|
|
183
|
+
| `:priced_request_count` | Integer | The logs `total_cost` covers: non-failed, with tokens, model has pricing. |
|
|
184
|
+
| `:long_context_request_count` | Integer | Priced logs that crossed their model's input-token pricing tier. |
|
|
185
|
+
| `:total_input_tokens`, `:total_output_tokens` | Integer | Raw token columns summed over non-failed logs, not cache-adjusted. |
|
|
186
|
+
| `:avg_input_tokens`, `:avg_output_tokens` | Integer or nil | |
|
|
187
|
+
| `:avg_latency_ms` | Float or nil | |
|
|
188
|
+
| `:correctness_score`, `:sentiment_score`, `:revision_score` | Float or nil | `sentiment_score` and `revision_score` are all-time, not windowed. |
|
|
189
|
+
| `:first_request_at`, `:last_request_at` | String or nil | Lifetime, not windowed. |
|
|
190
|
+
|
|
191
|
+
Summing the per-log `cost` from [Reading Logs](log-search.md) can exceed `total_cost`, because the log
|
|
192
|
+
`cost` is also computed for failed logs and logs with a zero token count.
|
|
193
|
+
|
|
194
|
+
Windows follow the rules in [Reading Workloads](workload-search.md#windows-since-and-until). A window
|
|
195
|
+
error is a `422` on the `since` or `until` key. Metrics add aggregate statements, so expect a slower
|
|
196
|
+
response, and a `504` on a large page.
|
|
197
|
+
|
|
156
198
|
## Errors
|
|
157
199
|
|
|
158
200
|
The write methods in this gem (`create_feedback`, `create_log`, `send_llm_request_log`) log a
|
|
@@ -186,7 +228,7 @@ end
|
|
|
186
228
|
|---|---|
|
|
187
229
|
| `401` | Missing, invalid, or public API key. |
|
|
188
230
|
| `404` | `get_template` only. Unknown id, **or** one belonging to another client — existence is not disclosed, so this is never a `403`. |
|
|
189
|
-
| `422` | Unrecognised `status`,
|
|
231
|
+
| `422` | Unrecognised `status`, a `workload_id` that does not decode or belongs to another client, or a bad `days_back`, `since` or `until` when metrics are requested. |
|
|
190
232
|
| `504` | See below. |
|
|
191
233
|
|
|
192
234
|
### `504` is expected, and retryable
|
data/docs/vertex.md
CHANGED
|
@@ -15,6 +15,7 @@ require "coolhand/vertex/batch_result_processor"
|
|
|
15
15
|
```
|
|
16
16
|
|
|
17
17
|
- Call the `Coolhand::Vertex::BatchResultProcessor` service with `batch_info` and the downloaded batch results.
|
|
18
|
+
- For a succeeded job, the per-item request logs are sent from Coolhand's [background log thread](configuration.md#asynchronous-logging) as a single job, so `call` returns without waiting on the Coolhand API (set `config.async_logging = false` to send inline).
|
|
18
19
|
- Optionally pass `model:` to populate the logged `model` field. It falls back to a `"model"` key in `batch_info` if present, and is omitted from the payload entirely when neither is available. If the value (from either source) looks like a Vertex model resource path (`publishers/<publisher>/models/<id>` or `projects/<project>/locations/<location>/models/<id>`), it's normalized down to just the trailing id/version before being logged; any other string is sent as-is.
|
|
19
20
|
- `batch_info["name"]` must be the exact Vertex job resource name (`projects/<project>/locations/<location>/batchPredictionJobs/<job-id>`), and `batch_info["startTime"]`/`["endTime"]` must be valid ISO 8601 timestamps — a batch with a missing or malformed resource name or timestamps is skipped entirely (logged as an error) rather than sent with a broken URL.
|
|
20
21
|
- If `endTime` precedes `startTime` (e.g. clock skew), the batch is still logged — `duration_ms` is clamped to `0` and a warning is logged — rather than dropping the results.
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
# Reading Workloads (Search)
|
|
2
|
+
|
|
3
|
+
`Coolhand::WorkloadService#search_workloads` lists your workloads, the groups of prompt templates that
|
|
4
|
+
make up one task or agent, optionally with a cost and performance `metrics` rollup per workload. It
|
|
5
|
+
wraps `GET /api/v2/workloads`. It is read-only, and there is no per-workload endpoint.
|
|
6
|
+
|
|
7
|
+
**Auth:** requires your **private** API key. The public key is write-only on this API and is rejected
|
|
8
|
+
exactly like an invalid key.
|
|
9
|
+
|
|
10
|
+
**IDs:** a workload's `:id` is its **hashid**, a String. It is what the `workload_id` filter on
|
|
11
|
+
`search_templates` and `search_logs` expects.
|
|
12
|
+
|
|
13
|
+
**Scoping:** the client is always derived from your API key. There is deliberately no `client_id`
|
|
14
|
+
keyword, so passing one raises `ArgumentError`.
|
|
15
|
+
|
|
16
|
+
## `search_workloads`
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
require "coolhand"
|
|
20
|
+
|
|
21
|
+
Coolhand.configure do |config|
|
|
22
|
+
config.api_key = ENV.fetch("COOLHAND_PRIVATE_API_KEY")
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
result = Coolhand.workload_service.search_workloads(
|
|
26
|
+
include_metrics: true,
|
|
27
|
+
since: Time.utc(2026, 9, 1),
|
|
28
|
+
until: Time.utc(2026, 10, 1),
|
|
29
|
+
per: 50
|
|
30
|
+
)
|
|
31
|
+
|
|
32
|
+
result.workloads.each do |workload|
|
|
33
|
+
puts "#{workload[:name]}: $#{workload[:metrics][:total_cost]}"
|
|
34
|
+
end
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
### Keywords
|
|
38
|
+
|
|
39
|
+
All optional, and named exactly like the wire parameters.
|
|
40
|
+
|
|
41
|
+
| keyword | type | notes |
|
|
42
|
+
|---|---|---|
|
|
43
|
+
| `search` | String | Case-insensitive substring match on the workload name. |
|
|
44
|
+
| `include_archived` | Boolean | Defaults to `false` server-side. |
|
|
45
|
+
| `include_system` | Boolean | `Unmatched`, `Embedding Requests` and the like. Defaults to `false` server-side. |
|
|
46
|
+
| `include_templates` | Boolean | Adds each workload's active templates and their routing patterns as `:templates`. |
|
|
47
|
+
| `include_metrics` | Boolean | Adds a `:metrics` rollup across all of the workload's templates. |
|
|
48
|
+
| `days_back` | Integer | Rolling metrics window ending now (server default 28, max 365). Ignored when `since` is given. |
|
|
49
|
+
| `since` | Time, DateTime, Date or String | Metrics window start, inclusive. Overrides `days_back`. |
|
|
50
|
+
| `until` | Time, DateTime, Date or String | Metrics window end, exclusive. Defaults to now. |
|
|
51
|
+
| `page` | Integer | 1-based. |
|
|
52
|
+
| `per` | Integer | Default 25, max 100, both enforced server-side. This gem only sends `per`, never its `per_page` alias. |
|
|
53
|
+
|
|
54
|
+
`false` and `0` are sent as given; only an unset (`nil`) keyword is left off the request.
|
|
55
|
+
|
|
56
|
+
### Windows: `since` and `until`
|
|
57
|
+
|
|
58
|
+
`Time`, `DateTime` and `Date` values are serialised as UTC ISO8601 (a `Date` as a date alone, which the
|
|
59
|
+
server reads as midnight UTC). A String is passed through untouched, so the server stays the one
|
|
60
|
+
validator. Anything else raises `ArgumentError` before a request is made.
|
|
61
|
+
|
|
62
|
+
```ruby
|
|
63
|
+
search_workloads(include_metrics: true, since: "2026-09-01") # date alone: midnight UTC
|
|
64
|
+
search_workloads(include_metrics: true, since: "2026-09-01T12:00:00+02:00") # the + is encoded as %2B for you
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Server-side rules, which only apply when `include_metrics` is set:
|
|
68
|
+
|
|
69
|
+
- A missing offset means UTC. `since` is inclusive and `until` is exclusive.
|
|
70
|
+
- With `since`, `days_back` is ignored and `metrics[:days_back]` is `nil`. With only `until`, the window
|
|
71
|
+
is `days_back` long ending there.
|
|
72
|
+
- The prior window behind `error_rate_change` is the same length, immediately before the window.
|
|
73
|
+
- A malformed value, a `since` not before `until`, or a window over 365 days is a `422` with the error on
|
|
74
|
+
the `since` or `until` key.
|
|
75
|
+
|
|
76
|
+
### Return value
|
|
77
|
+
|
|
78
|
+
A `Coolhand::WorkloadSearchResult`, a `Struct` with `#workloads` and `#pagination`. Rows are plain
|
|
79
|
+
Hashes with Symbol keys, handed back as the API rendered them so a field the server adds later reaches
|
|
80
|
+
you.
|
|
81
|
+
|
|
82
|
+
| field | type | notes |
|
|
83
|
+
|---|---|---|
|
|
84
|
+
| `:id` | String | Hashid. |
|
|
85
|
+
| `:name` | String | |
|
|
86
|
+
| `:description` | String or nil | |
|
|
87
|
+
| `:archived`, `:system`, `:merged` | Boolean | |
|
|
88
|
+
| `:template_count` | Integer | Active templates. |
|
|
89
|
+
| `:draft_template_count` | Integer | |
|
|
90
|
+
| `:log_count` | Integer | Every log on any of its templates, all generators, so it can exceed the sum of the templates' own `log_count`. |
|
|
91
|
+
| `:last_activity` | String or nil | ISO-8601 UTC. |
|
|
92
|
+
| `:templates` | Array | Only with `include_templates`. Each has `:id`, `:name`, `:status`, `:user_prompt_pattern`, `:system_prompt_pattern`. |
|
|
93
|
+
| `:metrics` | Hash | Only with `include_metrics`. See below. |
|
|
94
|
+
|
|
95
|
+
Rows are ordered by name. `pagination` is a `Coolhand::Pagination` built from the `X-Page`,
|
|
96
|
+
`X-Per-Page`, `X-Total-Count` and `X-Total-Pages` response headers, which this endpoint always sends,
|
|
97
|
+
never from the length of the array.
|
|
98
|
+
|
|
99
|
+
### `metrics`
|
|
100
|
+
|
|
101
|
+
The same object `search_templates` returns per template, computed by the same SQL as the dashboard, so
|
|
102
|
+
tiered pricing, cached-token discounts and reasoning tokens are applied. See
|
|
103
|
+
[Metrics](template-search.md#metrics) for every field.
|
|
104
|
+
|
|
105
|
+
## Errors
|
|
106
|
+
|
|
107
|
+
The read methods raise where the write methods log and return `nil`. `Coolhand::HttpError` carries
|
|
108
|
+
`#status` and `#body` (`{"errors": {"<field>": ["msg"]}}` on a `422`).
|
|
109
|
+
|
|
110
|
+
| status | when |
|
|
111
|
+
|---|---|
|
|
112
|
+
| `401` | No API key, an invalid key, or the public key. |
|
|
113
|
+
| `422` | A bad `days_back`, `since` or `until`. Only checked when `include_metrics` is set. |
|
|
114
|
+
| `504` | An aggregate exceeded the server's statement timeout. Retryable: narrow with `search` or a smaller `per`. |
|
|
115
|
+
|
|
116
|
+
`Coolhand::Error` covers no API key configured, a transport failure, and a body that is not a JSON array.
|
data/lib/coolhand/api_service.rb
CHANGED
|
@@ -6,12 +6,14 @@ require "json"
|
|
|
6
6
|
require_relative "collector"
|
|
7
7
|
require_relative "errors"
|
|
8
8
|
require_relative "read_requests"
|
|
9
|
+
require_relative "response_warnings"
|
|
9
10
|
|
|
10
11
|
module Coolhand
|
|
11
12
|
class ApiService
|
|
12
13
|
# The GET half of this class. See Coolhand::ReadRequests for why reads raise where writes
|
|
13
14
|
# log and return nil.
|
|
14
15
|
include ReadRequests
|
|
16
|
+
include ResponseWarnings
|
|
15
17
|
|
|
16
18
|
# Caps how much of a failed response body is interpolated into a log line or an exception
|
|
17
19
|
# message. Without it an oversized body from a proxy or gateway becomes the message.
|
|
@@ -36,7 +38,9 @@ module Coolhand
|
|
|
36
38
|
log JSON.pretty_generate(sanitize_payload_for_json(payload))
|
|
37
39
|
nil
|
|
38
40
|
else
|
|
39
|
-
send_request(payload, "✅ Successfully sent request metadata")
|
|
41
|
+
result = send_request(payload, "✅ Successfully sent request metadata")
|
|
42
|
+
log_response_warnings(result)
|
|
43
|
+
result
|
|
40
44
|
end
|
|
41
45
|
end
|
|
42
46
|
|
|
@@ -80,9 +84,10 @@ module Coolhand
|
|
|
80
84
|
uri = URI.parse(@api_endpoint)
|
|
81
85
|
http = Net::HTTP.new(uri.host, uri.port)
|
|
82
86
|
http.use_ssl = (uri.scheme == "https")
|
|
83
|
-
# Bound worst-case latency: this call happens inline in
|
|
84
|
-
# request's path
|
|
85
|
-
#
|
|
87
|
+
# Bound worst-case latency: with async_logging off this call happens inline in
|
|
88
|
+
# the intercepted request's path (otherwise it ties up the background log
|
|
89
|
+
# worker), so a slow/unreachable Coolhand backend must not hang for Ruby's
|
|
90
|
+
# ~60s Net::HTTP defaults.
|
|
86
91
|
http.open_timeout = 5
|
|
87
92
|
http.read_timeout = 5
|
|
88
93
|
|
|
@@ -149,7 +149,8 @@ module Coolhand
|
|
|
149
149
|
end
|
|
150
150
|
|
|
151
151
|
def send_complete_request_log(request_id:, method:, url:, request_headers:, request_body:, response_headers:,
|
|
152
|
-
response_body:, status_code:, start_time:, end_time:, duration_ms:, is_streaming:, source_api: nil, model: nil
|
|
152
|
+
response_body:, status_code:, start_time:, end_time:, duration_ms:, is_streaming:, source_api: nil, model: nil,
|
|
153
|
+
declared: nil)
|
|
153
154
|
raw_request = {
|
|
154
155
|
id: request_id,
|
|
155
156
|
timestamp: start_time.iso8601,
|
|
@@ -168,11 +169,12 @@ module Coolhand
|
|
|
168
169
|
raw_request[:model] = model if Coolhand.required_field?(model)
|
|
169
170
|
|
|
170
171
|
request_data = { raw_request: raw_request }
|
|
172
|
+
request_data[:declared] = declared if declared && !declared.empty?
|
|
171
173
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
174
|
+
LogQueue.submit do
|
|
175
|
+
Coolhand::ApiService.new.send_llm_request_log(request_data)
|
|
176
|
+
Coolhand.log "📤 Sent complete request/response log for #{request_id} (duration: #{duration_ms}ms)"
|
|
177
|
+
end
|
|
176
178
|
rescue StandardError => e
|
|
177
179
|
Coolhand.log "❌ Error sending complete request log: #{e.message}"
|
|
178
180
|
end
|
|
@@ -29,8 +29,8 @@ module Coolhand
|
|
|
29
29
|
DEFAULT_MAX_CAPTURED_BODY_BYTES = 1_000_000
|
|
30
30
|
|
|
31
31
|
attr_accessor :api_key, :environment, :silent, :debug_mode, :capture, :exclude_api_patterns, :enabled,
|
|
32
|
-
:max_captured_body_bytes, :webhook_replay_tolerance_seconds, :webhook_id_store
|
|
33
|
-
attr_reader :intercept_addresses, :intercept_path_patterns, :base_url
|
|
32
|
+
:max_captured_body_bytes, :webhook_replay_tolerance_seconds, :webhook_id_store, :async_logging
|
|
33
|
+
attr_reader :intercept_addresses, :intercept_path_patterns, :base_url, :auto_declare
|
|
34
34
|
|
|
35
35
|
def initialize
|
|
36
36
|
# Set defaults
|
|
@@ -44,6 +44,8 @@ module Coolhand
|
|
|
44
44
|
@capture = true
|
|
45
45
|
@exclude_api_patterns = DEFAULT_EXCLUDE_API_PATTERNS.dup
|
|
46
46
|
@enabled = true
|
|
47
|
+
@async_logging = true
|
|
48
|
+
@auto_declare = []
|
|
47
49
|
@max_captured_body_bytes = DEFAULT_MAX_CAPTURED_BODY_BYTES
|
|
48
50
|
@webhook_replay_tolerance_seconds = 300
|
|
49
51
|
@webhook_id_store = Coolhand::OpenAi::WebhookIdStore.new
|
|
@@ -73,6 +75,18 @@ module Coolhand
|
|
|
73
75
|
@intercept_path_patterns = value.is_a?(Array) ? value : [value]
|
|
74
76
|
end
|
|
75
77
|
|
|
78
|
+
# Rails entry points to declare a workload/template for automatically: any of
|
|
79
|
+
# RailsAutoDeclare::TARGETS (:active_job, :controller). Off by default. See docs/declared_workloads.md.
|
|
80
|
+
def auto_declare=(value)
|
|
81
|
+
targets = Array(value).map { |target| target.to_s.to_sym }
|
|
82
|
+
unknown = targets - RailsAutoDeclare::TARGETS
|
|
83
|
+
unless unknown.empty?
|
|
84
|
+
raise Error, "auto_declare supports #{RailsAutoDeclare::TARGETS.inspect}, got #{unknown.inspect}"
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
@auto_declare = targets.uniq
|
|
88
|
+
end
|
|
89
|
+
|
|
76
90
|
def base_url=(value)
|
|
77
91
|
stripped = value&.sub(%r{/+\z}, "")
|
|
78
92
|
raise Error, BASE_URL_ERROR_MSG unless stripped.nil? || valid_base_url?(stripped)
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Coolhand
|
|
4
|
+
# Declares the workload/template of the LLM calls a class makes, once, on the class, so passive
|
|
5
|
+
# capture picks them up without a `Coolhand.with_context` block at every call site. Wrapped methods
|
|
6
|
+
# run inside `with_context`, so nesting and precedence are exactly those documented in
|
|
7
|
+
# docs/declared_workloads.md. Settings are inherited by subclasses.
|
|
8
|
+
#
|
|
9
|
+
# class ApplicationService
|
|
10
|
+
# include Coolhand::Declarable
|
|
11
|
+
# coolhand_workload reporter: "my-app"
|
|
12
|
+
# coolhand_declare_all
|
|
13
|
+
# end
|
|
14
|
+
#
|
|
15
|
+
# class Parser < ApplicationService
|
|
16
|
+
# def call = client.chat(...) # workload "Parser#call", template "Parser#call"
|
|
17
|
+
# end
|
|
18
|
+
module Declarable
|
|
19
|
+
def self.included(base)
|
|
20
|
+
raise ArgumentError, "Coolhand::Declarable can only be included in a class" unless base.is_a?(Class)
|
|
21
|
+
|
|
22
|
+
base.extend(ClassMethods)
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Never an LLM call site, and wrapping them would tax every call or declare a junk workload.
|
|
26
|
+
SKIPPED_METHODS = %i[method_missing respond_to? respond_to_missing? == === eql? equal? hash to_s inspect
|
|
27
|
+
to_str to_ary to_proc coerce <=>].freeze
|
|
28
|
+
ATTRIBUTE_MACROS = %w[attr attr_reader attr_writer attr_accessor].freeze
|
|
29
|
+
private_constant :SKIPPED_METHODS, :ATTRIBUTE_MACROS
|
|
30
|
+
|
|
31
|
+
module ClassMethods
|
|
32
|
+
# Class-level defaults, inherited field by field (the nearest class that sets a field wins). With
|
|
33
|
+
# no `key`, each wrapped method declares its own workload, "ClassName#method"; a `key` makes every
|
|
34
|
+
# method share one workload. The class name is the runtime class, so an inherited method declares
|
|
35
|
+
# under the subclass.
|
|
36
|
+
def coolhand_workload(key: nil, name: nil, description: nil, reporter: nil)
|
|
37
|
+
workload = DeclaredContext.normalize_part(:workload, { key: key, name: name, description: description }.compact,
|
|
38
|
+
DeclaredContext::WORKLOAD_FIELDS)
|
|
39
|
+
@coolhand_settings = { workload: workload, reporter: DeclaredContext.normalize_reporter(reporter) }.compact
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# Wraps the named instance methods (now, or when they are defined later). `template:` is a key or a
|
|
43
|
+
# `{ key:, name: }` Hash; by default the template key is "ClassName#method".
|
|
44
|
+
def coolhand_declare(*names, template: nil)
|
|
45
|
+
# Checked here, at class definition, so a bad value never surfaces inside the host's method.
|
|
46
|
+
template = DeclaredContext.normalize_part(:template, template.is_a?(String) ? { key: template } : template,
|
|
47
|
+
DeclaredContext::TEMPLATE_FIELDS)
|
|
48
|
+
names.flatten.each do |name|
|
|
49
|
+
name = name.to_sym
|
|
50
|
+
(@coolhand_declared ||= {})[name] = { template: template }
|
|
51
|
+
coolhand_wrap(name) if method_defined?(name) || private_method_defined?(name)
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
# Wraps every public instance method defined in this class and its subclasses after this call, so
|
|
56
|
+
# put it at the top of the class body. Private and protected methods, attribute readers/writers
|
|
57
|
+
# and object-protocol methods (see SKIPPED_METHODS) are left alone; `coolhand_declare` wraps one of
|
|
58
|
+
# those if you name it.
|
|
59
|
+
def coolhand_declare_all
|
|
60
|
+
@coolhand_declare_all = true
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# @api private
|
|
64
|
+
def coolhand_identity(method_name)
|
|
65
|
+
key = "#{name}##{method_name}"
|
|
66
|
+
template = coolhand_template_for(method_name)
|
|
67
|
+
{
|
|
68
|
+
reporter: coolhand_setting(:reporter),
|
|
69
|
+
workload: { key: key }.merge(coolhand_setting(:workload) || {}),
|
|
70
|
+
template: template || { key: key }
|
|
71
|
+
}
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# @api private
|
|
75
|
+
def coolhand_declare_call?(method_name)
|
|
76
|
+
return false if name.nil? || !Coolhand.configuration.enabled
|
|
77
|
+
return false unless DeclaredContext.valid_key?("#{name}##{method_name}")
|
|
78
|
+
|
|
79
|
+
public_method_defined?(method_name) || !coolhand_declaration(method_name).nil?
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# @api private
|
|
83
|
+
def coolhand_own_state
|
|
84
|
+
[@coolhand_settings, @coolhand_declared, @coolhand_declare_all]
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
private
|
|
88
|
+
|
|
89
|
+
def method_added(method_name)
|
|
90
|
+
super
|
|
91
|
+
coolhand_wrap(method_name) if !@coolhand_wrapping && coolhand_wrap?(method_name)
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def coolhand_wrap?(method_name)
|
|
95
|
+
return true if coolhand_declaration(method_name)
|
|
96
|
+
|
|
97
|
+
public_method_defined?(method_name) && !SKIPPED_METHODS.include?(method_name) &&
|
|
98
|
+
coolhand_declare_all? && !attribute_definition?
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
# True while `attr_reader`/`attr_writer`/`attr_accessor` is what is defining the method. Read from
|
|
102
|
+
# the stack rather than by overriding those macros, which would lose the class body's
|
|
103
|
+
# `private`/`protected` default visibility. A window of frames, not one fixed depth, so a
|
|
104
|
+
# `method_added` override that calls `super` does not hide the macro.
|
|
105
|
+
def attribute_definition?
|
|
106
|
+
caller_locations(1, 8)&.any? { |frame| ATTRIBUTE_MACROS.include?(frame.base_label) } || false
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
def coolhand_declare_all?
|
|
110
|
+
coolhand_each_class { |klass| return true if klass.coolhand_own_state[2] }
|
|
111
|
+
false
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
# Redefines the method in the same class, rather than prepending a wrapper, so a later
|
|
115
|
+
# `private :name` (or `private def name`) applies to the wrapper like it would to the original.
|
|
116
|
+
def coolhand_wrap(method_name)
|
|
117
|
+
original = instance_method(method_name)
|
|
118
|
+
wrappers = (@coolhand_wrappers ||= {})
|
|
119
|
+
return if wrappers[method_name] == original
|
|
120
|
+
|
|
121
|
+
visibility = coolhand_visibility(method_name)
|
|
122
|
+
@coolhand_wrapping = true
|
|
123
|
+
define_method(method_name) do |*args, **kwargs, &block|
|
|
124
|
+
klass = self.class
|
|
125
|
+
next original.bind_call(self, *args, **kwargs, &block) unless klass.coolhand_declare_call?(method_name)
|
|
126
|
+
|
|
127
|
+
Coolhand.with_context(**klass.coolhand_identity(method_name)) do
|
|
128
|
+
original.bind_call(self, *args, **kwargs, &block)
|
|
129
|
+
end
|
|
130
|
+
end
|
|
131
|
+
send(visibility, method_name)
|
|
132
|
+
wrappers[method_name] = instance_method(method_name)
|
|
133
|
+
ensure
|
|
134
|
+
@coolhand_wrapping = false
|
|
135
|
+
end
|
|
136
|
+
|
|
137
|
+
# Includes inherited methods: wrapping one a parent made private must not make it public here.
|
|
138
|
+
def coolhand_visibility(method_name)
|
|
139
|
+
return :private if private_method_defined?(method_name)
|
|
140
|
+
return :protected if protected_method_defined?(method_name)
|
|
141
|
+
|
|
142
|
+
:public
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
# Yields this class and its superclasses that include Declarable, nearest first. A loop rather
|
|
146
|
+
# than a collected Array, since the lookups below run on every wrapped call.
|
|
147
|
+
def coolhand_each_class
|
|
148
|
+
klass = self
|
|
149
|
+
while klass.respond_to?(:coolhand_own_state)
|
|
150
|
+
yield klass
|
|
151
|
+
klass = klass.superclass
|
|
152
|
+
end
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
def coolhand_declaration(method_name)
|
|
156
|
+
coolhand_nearest(1, method_name)
|
|
157
|
+
end
|
|
158
|
+
|
|
159
|
+
def coolhand_template_for(method_name)
|
|
160
|
+
coolhand_declaration(method_name)&.fetch(:template)
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
def coolhand_setting(field)
|
|
164
|
+
coolhand_nearest(0, field)
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
# The value for `key` in the nearest class whose own state (slot `slot` of `coolhand_own_state`)
|
|
168
|
+
# sets it, or nil.
|
|
169
|
+
def coolhand_nearest(slot, key)
|
|
170
|
+
coolhand_each_class do |klass|
|
|
171
|
+
state = klass.coolhand_own_state[slot]
|
|
172
|
+
return state[key] if state&.key?(key)
|
|
173
|
+
end
|
|
174
|
+
nil
|
|
175
|
+
end
|
|
176
|
+
end
|
|
177
|
+
end
|
|
178
|
+
end
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Coolhand
|
|
4
|
+
# Ambient workload/template identity for captured calls, set with {Coolhand.with_context}. The
|
|
5
|
+
# interceptor snapshots {current} on the calling thread before a log is queued, because the
|
|
6
|
+
# background log worker does not inherit thread-locals. See docs/declared_workloads.md.
|
|
7
|
+
module DeclaredContext
|
|
8
|
+
THREAD_KEY = :coolhand_declared_context
|
|
9
|
+
|
|
10
|
+
# The fields the server accepts under `declared.workload` / `declared.template`; anything else
|
|
11
|
+
# is dropped here rather than sent.
|
|
12
|
+
WORKLOAD_FIELDS = %i[id key name description].freeze
|
|
13
|
+
TEMPLATE_FIELDS = %i[id key name].freeze
|
|
14
|
+
|
|
15
|
+
module_function
|
|
16
|
+
|
|
17
|
+
# A frozen Hash, or nil when no context is active.
|
|
18
|
+
def current
|
|
19
|
+
Thread.current[THREAD_KEY]
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def clear!
|
|
23
|
+
Thread.current[THREAD_KEY] = nil
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
# Each field is replaced wholesale by the innermost block that sets it. Hashes are not merged
|
|
27
|
+
# key by key: `{ id: x }` merged with `{ key: y }` would declare both, and the server prefers `id`.
|
|
28
|
+
# A template belongs to a workload, so a block that declares a workload without a template does
|
|
29
|
+
# not inherit the outer one; it would pair the new workload with a template from another.
|
|
30
|
+
def with(reporter: nil, workload: nil, template: nil)
|
|
31
|
+
previous = current
|
|
32
|
+
declared = {
|
|
33
|
+
reporter: normalize_reporter(reporter),
|
|
34
|
+
workload: normalize_part(:workload, workload, WORKLOAD_FIELDS),
|
|
35
|
+
template: normalize_part(:template, template, TEMPLATE_FIELDS)
|
|
36
|
+
}.compact
|
|
37
|
+
inherited = previous || {}
|
|
38
|
+
inherited = inherited.except(:template) if declared.key?(:workload) && !declared.key?(:template)
|
|
39
|
+
merged = inherited.merge(declared)
|
|
40
|
+
ensure_template_has_workload!(merged)
|
|
41
|
+
Thread.current[THREAD_KEY] = merged.empty? ? nil : merged.freeze
|
|
42
|
+
yield
|
|
43
|
+
ensure
|
|
44
|
+
Thread.current[THREAD_KEY] = previous
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def normalize_reporter(reporter)
|
|
48
|
+
return nil if reporter.nil?
|
|
49
|
+
|
|
50
|
+
reporter.to_s
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# Mirrors the server's key rules: at most 255 characters, no control or invisible characters, no
|
|
54
|
+
# leading or trailing whitespace, and no leading, trailing or doubled "/". The server answers a
|
|
55
|
+
# bad key with a 422 that rejects the whole log, so it is checked here, before the call is made.
|
|
56
|
+
MAX_KEY_LENGTH = 255
|
|
57
|
+
KEY_FORMAT = %r{
|
|
58
|
+
\A(?!/)(?!.*//)
|
|
59
|
+
[^\p{Cntrl}\p{Cf}[:space:]]
|
|
60
|
+
(?:[^\p{Cntrl}\p{Cf}]*[^\p{Cntrl}\p{Cf}[:space:]])?
|
|
61
|
+
(?<!/)\z
|
|
62
|
+
}x
|
|
63
|
+
|
|
64
|
+
def valid_key?(key)
|
|
65
|
+
key.is_a?(String) && key.length <= MAX_KEY_LENGTH && KEY_FORMAT.match?(key)
|
|
66
|
+
rescue ArgumentError, EncodingError
|
|
67
|
+
false # a string with a broken or incompatible encoding is not a valid key
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
# A template given by key, with no workload to create it in, is a 422 that drops the log. A
|
|
71
|
+
# template `id` refers to an existing record and needs no workload.
|
|
72
|
+
def ensure_template_has_workload!(declared)
|
|
73
|
+
template = declared[:template]
|
|
74
|
+
return if template.nil? || template.key?(:id) || declared.key?(:workload)
|
|
75
|
+
|
|
76
|
+
raise ArgumentError, "template with a :key needs a workload; declare one in this or an enclosing block"
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
# Validated up front: the server answers a non-object `workload`/`template` with a 422 for the
|
|
80
|
+
# whole log, which would drop the call from Coolhand rather than just its identity.
|
|
81
|
+
def normalize_part(name, value, fields)
|
|
82
|
+
return nil if value.nil?
|
|
83
|
+
raise ArgumentError, "#{name} must be a Hash, got #{value.class}" unless value.is_a?(Hash)
|
|
84
|
+
|
|
85
|
+
part = value.each_with_object({}) do |(k, v), acc|
|
|
86
|
+
field = k.respond_to?(:to_sym) ? k.to_sym : nil
|
|
87
|
+
next unless fields.include?(field) && !v.nil?
|
|
88
|
+
# A blank id refers to nothing and would 422 the log, so it counts as not given.
|
|
89
|
+
next if field == :id && v.to_s.strip.empty?
|
|
90
|
+
|
|
91
|
+
acc[field] = v.to_s.dup.freeze
|
|
92
|
+
end
|
|
93
|
+
return nil if part.empty?
|
|
94
|
+
|
|
95
|
+
if part.key?(:key) && !valid_key?(part[:key])
|
|
96
|
+
raise ArgumentError, "#{name} key must be at most #{MAX_KEY_LENGTH} printable characters with no " \
|
|
97
|
+
"surrounding whitespace and no leading, trailing or doubled \"/\""
|
|
98
|
+
end
|
|
99
|
+
unless part.key?(:id) || part.key?(:key)
|
|
100
|
+
raise ArgumentError, "#{name} needs an :id or a :key, got #{value.keys.inspect}"
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
part.freeze
|
|
104
|
+
end
|
|
105
|
+
end
|
|
106
|
+
end
|
data/lib/coolhand/errors.rb
CHANGED
|
@@ -3,7 +3,8 @@
|
|
|
3
3
|
module Coolhand
|
|
4
4
|
class Error < StandardError; end
|
|
5
5
|
|
|
6
|
-
# Raised by the gem's
|
|
6
|
+
# Raised by the gem's raising API methods (template reads, feedback links) when the Coolhand API
|
|
7
|
+
# answers with a non-2xx status.
|
|
7
8
|
#
|
|
8
9
|
# `status` is carried so callers can branch on it (404 vs retryable 504) without matching the
|
|
9
10
|
# message text.
|