coolhand 0.6.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/AGENTS.harness.md +49 -14
- data/CHANGELOG.md +48 -0
- data/README.md +45 -4
- data/docs/configuration.md +27 -2
- data/docs/feedback-links.md +48 -0
- data/docs/log-search.md +102 -0
- data/docs/openai.md +1 -1
- data/docs/template-search.md +46 -4
- data/docs/vertex.md +3 -0
- data/docs/workload-search.md +116 -0
- data/lib/coolhand/api_service.rb +4 -3
- data/lib/coolhand/base_interceptor.rb +42 -7
- data/lib/coolhand/configuration.rb +2 -1
- data/lib/coolhand/default_exclude_api_patterns.yml +11 -0
- data/lib/coolhand/default_intercept_addresses.yml +54 -0
- data/lib/coolhand/errors.rb +2 -1
- data/lib/coolhand/log_queue.rb +179 -0
- data/lib/coolhand/log_service.rb +67 -0
- data/lib/coolhand/net_http_interceptor.rb +144 -109
- data/lib/coolhand/open_ai/batch_result_processor.rb +30 -29
- data/lib/coolhand/open_ai/webhook_validator.rb +2 -1
- data/lib/coolhand/optimization_feedback_link_service.rb +85 -0
- data/lib/coolhand/read_query.rb +61 -0
- data/lib/coolhand/read_requests.rb +26 -13
- data/lib/coolhand/template_service.rb +13 -29
- data/lib/coolhand/version.rb +1 -1
- data/lib/coolhand/vertex/batch_result_processor.rb +9 -1
- data/lib/coolhand/workload_service.rb +46 -0
- data/lib/coolhand/write_requests.rb +47 -0
- data/lib/coolhand.rb +33 -0
- metadata +11 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 256c9237619b3f7116aa9751f78f0d0e367381b12afa694742da07ed1d7d3123
|
|
4
|
+
data.tar.gz: 149ddf8c4cd1800b105382ade5d2a2e9e57a360884e34e1b988b926739e90f30
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2ccbf54c2787a8df93178120583d83666553454a1b86385ec1c4ea10e228b859b995a35bed1450e9ab6d38dbb5d145d9b7a269be89f5562870d51a63630e5653
|
|
7
|
+
data.tar.gz: 4e6db88c4530737370030809a9d8e094d791761a6306a63701edc4413ed81f61a716d975953799c3d46b0cebbb3ddd9e1d172b22ee5d7b29c18bdd65ffe9e25c
|
data/AGENTS.harness.md
CHANGED
|
@@ -39,8 +39,8 @@ node <workspaceRoot>/coolhand/harness/harness.mjs my-issue --run <RUN_DIR> --rep
|
|
|
39
39
|
2. `<workspaceRoot>/coolhand/harness/RESIST_RULES.md` — the refuse list.
|
|
40
40
|
3. The API definition at `specPath`. It is your only source of truth **for the endpoint's
|
|
41
41
|
contract** — paths, params, response fields, status codes.
|
|
42
|
-
4. `coolhand-ruby/
|
|
43
|
-
disagrees with this harness file,
|
|
42
|
+
4. `coolhand-ruby/AGENTS.md` — this repo's own rulebook. **It is authoritative.** Where it
|
|
43
|
+
disagrees with this harness file, AGENTS.md wins — follow it and say so in your PR.
|
|
44
44
|
|
|
45
45
|
**Your issue links node's PR as the reference implementation. Use it for structure, not
|
|
46
46
|
for facts.** Node went first so you do not have to rediscover how a REST method fits into
|
|
@@ -70,7 +70,7 @@ inside the current architecture, escalate and STOP.
|
|
|
70
70
|
Require it inside the file that actually uses it, executed only when that provider's
|
|
71
71
|
functionality is accessed. Clients may not use every provider and must not be forced to
|
|
72
72
|
install gems they do not need — apps consuming this as a path gem break on missing optional
|
|
73
|
-
dependencies. See `coolhand-ruby/
|
|
73
|
+
dependencies. See `coolhand-ruby/AGENTS.md`.
|
|
74
74
|
|
|
75
75
|
## 3. Prove it against the real server
|
|
76
76
|
|
|
@@ -107,13 +107,44 @@ node <workspaceRoot>/coolhand/harness/harness.mjs wait --run <RUN_DIR> --channel
|
|
|
107
107
|
|
|
108
108
|
Name the rule number (`R1`–`R5`). Do not guess. Do not stub. Do not work around it.
|
|
109
109
|
|
|
110
|
-
## 5.
|
|
111
|
-
|
|
112
|
-
**Mandatory, before you push.**
|
|
113
|
-
→
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
110
|
+
## 5. Get your review — you cannot run it yourself
|
|
111
|
+
|
|
112
|
+
**Mandatory, before you push. You do not run this section yourself.** You are three
|
|
113
|
+
levels deep in this tree (server → node → you), and the platform does not let an agent at
|
|
114
|
+
this depth spawn the reviewer subagent the review skill needs — see
|
|
115
|
+
`<workspaceRoot>/coolhand/harness/RESIST_RULES.md` → R8's depth table. This is not something
|
|
116
|
+
to confirm by trying; go straight to the request below.
|
|
117
|
+
|
|
118
|
+
1. Commit locally first — build, prove it against the local server, run `bundle exec rake`,
|
|
119
|
+
all of section 3, all done before this step.
|
|
120
|
+
2. Send an R8 escalation to node naming your repo path, branch, and `HEAD` sha:
|
|
121
|
+
```
|
|
122
|
+
node <workspaceRoot>/coolhand/harness/harness.mjs send --run <RUN_DIR> --channel ruby \
|
|
123
|
+
--from ruby --to node --kind escalation \
|
|
124
|
+
--text "R8: ready for review. repo=<workspaceRoot>/coolhand-ruby branch=<branch> sha=<sha>"
|
|
125
|
+
```
|
|
126
|
+
3. `wait --after` the `messageId` your `send` in step 2 just printed. A timeout is silence,
|
|
127
|
+
not permission — call `wait` again, `--after` that same id, rather than pushing
|
|
128
|
+
(`RESIST_RULES.md` → "What escalate and stop means, mechanically"). If you loop back here
|
|
129
|
+
from step 5, use the fresh escalation's `messageId` instead.
|
|
130
|
+
4. **The first reply is an `ack`, not the answer** — it means node has started, not that it
|
|
131
|
+
is done. `wait` again, `--after` that ack's `messageId`, for the `resolution` that
|
|
132
|
+
actually follows.
|
|
133
|
+
5. **If what follows is a second `ack` instead of a `resolution`, node is asking you to do
|
|
134
|
+
something before it can start — most often that your checkout is not clean** (see
|
|
135
|
+
`RESIST_RULES.md` → R8 → "Serving a review for a child" step 2). Do what it asks, then
|
|
136
|
+
send a fresh R8 escalation with your current `HEAD` sha (it may have changed) and go back
|
|
137
|
+
to step 3. Do not just `wait` again — nothing arrives until you re-escalate.
|
|
138
|
+
6. **The `resolution` that eventually arrives always carries the full Iteration Breakdown
|
|
139
|
+
table plus an explicit `CLEAN`/`capped` verdict** — the same `result` value node has
|
|
140
|
+
already recorded (it records before it replies) — from a review node ran against your
|
|
141
|
+
actual repo. Not just one or the other, not your own read of the diff, and not an ack.
|
|
142
|
+
This is what you post as your own PR comment below, unedited plus the "Reviewed by node
|
|
143
|
+
against `<sha>`" line.
|
|
144
|
+
7. **Then move on to section 6 (Open your PR).** If node's review applied fixes,
|
|
145
|
+
it already committed them directly into your checkout while you waited — whatever is at
|
|
146
|
+
`HEAD` when you push there already includes them, and you do not need to reproduce or
|
|
147
|
+
look for them separately.
|
|
117
148
|
|
|
118
149
|
## 6. Open your PR — then STOP
|
|
119
150
|
|
|
@@ -123,8 +154,11 @@ for, escalate to node (R8) and STOP rather than substitute a self-review.
|
|
|
123
154
|
2. Body must reference your issue with `Closes #N` so it auto-closes on merge, and must
|
|
124
155
|
say: **depends on the server PR — deploy that first.**
|
|
125
156
|
3. Record it: `node <workspaceRoot>/coolhand/harness/harness.mjs pr --run <RUN_DIR> --repo ruby --url <url>`
|
|
126
|
-
4. Post
|
|
127
|
-
|
|
157
|
+
4. Post node's Iteration Breakdown table (section 5) as a comment on this PR — see
|
|
158
|
+
`RESIST_RULES.md` → "After the loop exits" — and add one line the table itself does not
|
|
159
|
+
carry: `Reviewed by node against <sha>`, since you did not run it yourself. **Do not call
|
|
160
|
+
`harness.mjs loop-review` here — node already recorded it**, against the sha that
|
|
161
|
+
resulted from its own fix commits, not the one you originally escalated with.
|
|
128
162
|
5. **Stop.** You launch no one. The tree ends with you on this branch.
|
|
129
163
|
|
|
130
164
|
## 7. Done means
|
|
@@ -133,7 +167,8 @@ for, escalate to node (R8) and STOP rather than substitute a self-review.
|
|
|
133
167
|
- [ ] `bundle exec rake` passes (RSpec + RuboCop)
|
|
134
168
|
- [ ] At least one spec hit the real local server, not a mock
|
|
135
169
|
- [ ] PR opened, recorded, references its issue, states its dependency on the server PR
|
|
136
|
-
- [ ]
|
|
137
|
-
is posted as a comment on your PR
|
|
170
|
+
- [ ] Node ran your review for real (not approximated by you) and its Iteration Breakdown
|
|
171
|
+
table is posted as a comment on your PR, naming node and the reviewed sha, and
|
|
172
|
+
recorded with `harness.mjs loop-review`
|
|
138
173
|
- [ ] Every field and status code came from the definition, not from node's code
|
|
139
174
|
- [ ] You launched no child agents
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,54 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.8.0] - 2026-10-09
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- **Read workloads and logs, with cost and performance metrics** ([#138](https://github.com/Coolhand-Labs/coolhand-ruby/pull/138)) — `Coolhand.workload_service.search_workloads` lists workloads (`GET /api/v2/workloads`) and `Coolhand.log_service` reads logs back out: `search_logs` (filters including `since`, `until`, `min_cost`, `order`, `project_path`, `source_application`, `include_total`, `sort`) and `get_log`, which returns `cost` and `cost_breakdown`. `search_templates` and `get_template` gain `include_metrics`, `days_back`, `since` and `until`. `since:`/`until:` accept a `Time`, `DateTime`, `Date` or ISO8601 String; anything else raises `ArgumentError` before a request is made. These need your **private** API key and raise `Coolhand::Error` (`Coolhand::HttpError` for non-2xx responses) rather than logging and returning `nil`. **Deploy note:** requires the matching Coolhand server release (Coolhand-Labs/coolhand#1753). See [Reading Workloads](docs/workload-search.md) and [Reading Logs](docs/log-search.md).
|
|
14
|
+
- **Link feedback to an optimization** ([#133](https://github.com/Coolhand-Labs/coolhand-ruby/pull/133)) — `Coolhand.optimization_feedback_link_service` provides `link_feedback`, `bulk_link_feedback` (sent in batches of 100, results summed) and `unlink_feedback` for `/api/v2/optimizations/:id/feedback_links`. Private API key required; failures raise `Coolhand::Error` / `Coolhand::HttpError`. See [Linking Feedback](docs/feedback-links.md).
|
|
15
|
+
- **Logs are sent from a background queue by default** ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148)) — the Coolhand POST no longer runs inline in the intercepted LLM call, so a slow or unreachable Coolhand backend can no longer add up to about 10s to each call. `Coolhand::LogQueue` holds up to 1,000 jobs per lane and drops the oldest when full, restarts itself in forked processes, and flushes at exit (waiting up to 5s). OpenAI and Vertex batch results are processed as one background job per batch, on their own lane so a long batch cannot delay or evict per-request logs. New `config.async_logging` (default `true`) and `Coolhand.flush(timeout: 5)`; `examples/benchmark_async_logging.rb` is a pass/fail benchmark. See [Asynchronous Logging](docs/configuration.md#asynchronous-logging). **Migration note:** anything that relied on the log POST finishing before the intercepted call returned (tests asserting on a stubbed `ApiService` inline, scripts that end with `exit!`, a process killed by `kill -9` or the OOM killer, which loses queued logs) should set `config.async_logging = false` or call `Coolhand.flush`. `debug_mode` always runs inline.
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
- Repo guidance moved from `CLAUDE.md` to `AGENTS.md` ([#130](https://github.com/Coolhand-Labs/coolhand-ruby/pull/130)), and the agent-harness review handshake in `AGENTS.harness.md` was rewritten ([#134](https://github.com/Coolhand-Labs/coolhand-ruby/pull/134)). Neither ships in the gem, and `AGENTS.md` is excluded from the gem package in place of `CLAUDE.md`.
|
|
19
|
+
- CI now uses `ruby/setup-ruby` 1.327.0 ([#135](https://github.com/Coolhand-Labs/coolhand-ruby/pull/135)).
|
|
20
|
+
- Shared read-URL building moved into `Coolhand::ReadQuery`, and POST/DELETE helpers that raise on failure into `Coolhand::WriteRequests`; `TemplateService` now uses `ReadQuery`. Internal refactor with no behavior change to `search_templates`/`get_template` beyond the new options.
|
|
21
|
+
- `Coolhand::NetHttpInterceptor` no longer includes `BaseInterceptor` ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148)); its helpers moved to `Coolhand::NetHttpInterceptor::Helpers`, so `Net::HTTP` gains only `#request` instead of about 15 generically named private methods (`intercept?`, `parse_json`, `sanitize_url`, ...). **Migration note:** internal API, but if you called those helpers on a `Net::HTTP` instance, use `Coolhand::NetHttpInterceptor::Helpers` or `Coolhand::BaseInterceptor`.
|
|
22
|
+
|
|
23
|
+
### Fixed
|
|
24
|
+
- Disabling the Net::HTTP interceptor logged a stale "Faraday monitoring disabled" message ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148)); it now says "Net::HTTP interceptor unpatched (capture stopped)".
|
|
25
|
+
|
|
26
|
+
### Security
|
|
27
|
+
- **Release-time hardening from a whole-package security review** ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148), plus fixes made on the release branch):
|
|
28
|
+
- Requests forwarded by the OpenAI batch webhook handler now go through the same request-body redaction as live interception, so Azure OpenAI "On Your Data" datastore credentials in a batch request body are replaced with `[REDACTED]` before being sent to Coolhand.
|
|
29
|
+
- The OpenAI batch handler now matches requests to results with a one-pass index instead of a scan per result, so a large batch no longer takes quadratic time. When a `custom_id` is duplicated the first request still wins, as before.
|
|
30
|
+
|
|
31
|
+
## [0.7.0] - 2026-09-20
|
|
32
|
+
|
|
33
|
+
### Added
|
|
34
|
+
- **Seven more providers are monitored out of the box** ([#117](https://github.com/Coolhand-Labs/coolhand-ruby/pull/117)) — DeepSeek (`api.deepseek.com`), Mistral (`api.mistral.ai`), Perplexity (`api.perplexity.ai`) and xAI (`api.x.ai`) are captured host-wide. Cohere is captured only on its chat and embed endpoints (`/v2/chat`, `/v1/embed`, `/v2/embed`, on `api.cohere.com` and `api.cohere.ai`), TypeSafe Jev only on `api.typesafe.ai/v1/systemone`, and Ollama only on port 11434 for `/api/chat`, `/api/generate`, `/api/embed` and `/api/embeddings` (`localhost`, `127.0.0.1`, or a bare single-label host such as a Docker Compose service). No configuration required. **Migration note:** if your app already calls any of these, those calls will now start being logged to Coolhand after upgrading. To opt out, override `config.intercept_addresses` with your own list.
|
|
35
|
+
- **Azure inference endpoints are monitored out of the box** ([#115](https://github.com/Coolhand-Labs/coolhand-ruby/pull/115)) — Azure OpenAI dedicated hosts (`openai.azure.com`/`.us`/`.cn`), Azure AI Services / Foundry multi-service hosts (`cognitiveservices.azure.*`, `services.ai.azure.*`, captured only on their `/openai/` and `/models/` paths), serverless/MaaS hosts (`inference.ai.azure.com`, `models.ai.azure.com`) and Azure ML managed online endpoints (`inference.ml.azure.com`/`.us`). The Azure ML entries are deliberately not path-anchored, because every managed online endpoint scores at `/score`, so non-LLM deployments there will also be captured. Remove those two entries from `intercept_addresses` if that over-captures for you. Default `exclude_api_patterns` now also skips Azure OpenAI control-plane paths (`/openai/files`, `/batches`, `/fine_tuning`, `/models`, and their `/openai/v1/` spellings). **Migration note:** anyone who set `exclude_api_patterns` explicitly keeps their own list and will not get the new Azure exclusions.
|
|
36
|
+
- **`intercept_addresses` entries can pin a port and/or a path prefix** ([#117](https://github.com/Coolhand-Labs/coolhand-ruby/pull/117), [#115](https://github.com/Coolhand-Labs/coolhand-ruby/pull/115)) — `host:port`, `host/path` and `host:port/path` are all accepted. A request must match every part, and the path matches on a segment boundary (`host/openai` matches `/openai` and `/openai/x`, not `/openaiz`). Plain-host entries behave exactly as before. See [Configuration](docs/configuration.md#custom-intercept-addresses).
|
|
37
|
+
|
|
38
|
+
### Changed
|
|
39
|
+
- Ruby 4.0 was added to the CI matrix ([#113](https://github.com/Coolhand-Labs/coolhand-ruby/pull/113)) and then removed again before release: its job could not `bundle install` against the locked dependencies (`BUNDLED WITH 2.3.6`, the conditional `pry-byebug`/`debug` Gemfile entry and `unicode-emoji 4.1.0`). CI covers Ruby 3.1–3.4 until the lock can support 4.0.
|
|
40
|
+
|
|
41
|
+
### Fixed
|
|
42
|
+
- `examples/anthropic_example.rb` could not run against the pinned `anthropic` 0.4.x gem: it relied on the client reading `ANTHROPIC_API_KEY` from the environment and on object-style response access, and it targeted a retired model. It now passes `access_token:` explicitly, reads the response as a Hash, and uses `claude-haiku-4-5-20251001`. Found by running it against a live key during release validation.
|
|
43
|
+
|
|
44
|
+
### Security
|
|
45
|
+
- **Azure OpenAI "On Your Data" datastore credentials are now redacted from captured request bodies** ([#115](https://github.com/Coolhand-Labs/coolhand-ruby/pull/115)) — a request's `data_sources`/`dataSources` entries can carry live credentials (Azure AI Search API keys, Cosmos/Mongo connection strings, Elasticsearch encoded keys) that header and URL sanitization never looked at. `BaseInterceptor.sanitize_body` now replaces any value under those entries whose key name contains `key`, `token`, `secret`, `password`, `credential`, `connectionstring` or `signature` (separators and case ignored) with `[REDACTED]` before the log is forwarded. Message content and tool definitions outside `data_sources` are untouched.
|
|
46
|
+
- **Release-time hardening from a whole-package security review** ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148), plus fixes made on the release branch):
|
|
47
|
+
- `Coolhand.log` no longer raises when stdout is unwritable. Previously a failing `puts` inside the interceptor's `ensure` block could replace the host application's own LLM response with an `IOError`.
|
|
48
|
+
- A bad `intercept_addresses`/`exclude_api_patterns` value (e.g. a String or Symbol instead of an array of Strings) now degrades to "don't capture" instead of raising into every intercepted request.
|
|
49
|
+
- `BaseInterceptor.sanitize_url` now fails closed on an unparseable URL, dropping userinfo, query and fragment rather than returning the raw URL with credentials intact.
|
|
50
|
+
- Header redaction now also matches names containing `auth`, `passw`, `credential`, `bearer`, `jwt` or `session` (e.g. `X-Auth`, `X-Session-Id`, `X-Jwt`). **Migration note:** headers such as `x-session-id` now appear as `[REDACTED]` in Coolhand.
|
|
51
|
+
- `sanitize_body` matches the `data_sources` key case- and separator-insensitively, and also redacts `pwd`, `passw*`, `bearer` and `jwt` credential keys.
|
|
52
|
+
- Hosts with a trailing dot (`api.openai.com.`) are now matched like their dotless form.
|
|
53
|
+
- `OpenAi::WebhookValidator` rejects a bare `whsec_` secret, which decodes to an empty HMAC key and would make signatures forgeable.
|
|
54
|
+
|
|
55
|
+
### Documentation
|
|
56
|
+
- Clarified that Google's "Gemini Enterprise Agent Platform" is the new marketing name for Vertex AI and does not change `aiplatform.googleapis.com` or `Coolhand::Vertex` ([#118](https://github.com/Coolhand-Labs/coolhand-ruby/pull/118)).
|
|
57
|
+
|
|
10
58
|
## [0.6.0] - 2026-09-12
|
|
11
59
|
|
|
12
60
|
### Added
|
data/README.md
CHANGED
|
@@ -103,6 +103,35 @@ Search is a parameter on the list endpoint, not a route of its own, and the `Unm
|
|
|
103
103
|
|
|
104
104
|
For the full filter reference, pagination, and error handling, see [Reading Templates →](docs/template-search.md).
|
|
105
105
|
|
|
106
|
+
### Cost and performance metrics
|
|
107
|
+
|
|
108
|
+
Pass `include_metrics: true` to `search_templates` (or `search_workloads`) for the dashboard's cost and performance numbers, over a rolling `days_back` window or an explicit `since:` / `until:` one. `search_logs` carries each log's `cost` and can sort and filter by it.
|
|
109
|
+
|
|
110
|
+
```ruby
|
|
111
|
+
window = { since: Time.utc(2026, 9, 1), until: Time.utc(2026, 10, 1) }
|
|
112
|
+
|
|
113
|
+
Coolhand.workload_service.search_workloads(include_metrics: true, **window).workloads
|
|
114
|
+
.map { |w| [w[:name], w[:metrics][:total_cost]] }
|
|
115
|
+
|
|
116
|
+
Coolhand.log_service.search_logs(order: 'cost_desc', min_cost: 1.0, per: 10, **window).logs
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
See [Reading Workloads →](docs/workload-search.md) and [Reading Logs →](docs/log-search.md).
|
|
120
|
+
|
|
121
|
+
## Linking Feedback to an Optimization
|
|
122
|
+
|
|
123
|
+
Attach feedback to an optimization as evidence, one at a time or in bulk. These require your **private** API key and **raise** `Coolhand::Error` on failure (`Coolhand::HttpError` for non-2xx responses).
|
|
124
|
+
|
|
125
|
+
```ruby
|
|
126
|
+
links = Coolhand.optimization_feedback_link_service
|
|
127
|
+
|
|
128
|
+
link = links.link_feedback('optimizationHashid', 'feedbackHashid', note: 'why')
|
|
129
|
+
links.bulk_link_feedback('optimizationHashid', %w[fb1 fb2 fb3]) # => { linked:, already_linked:, errored:, not_found: }
|
|
130
|
+
links.unlink_feedback('optimizationHashid', link[:id])
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
For batching, error semantics, and details, see [Linking Feedback →](docs/feedback-links.md).
|
|
134
|
+
|
|
106
135
|
## Rails Integration
|
|
107
136
|
|
|
108
137
|
### Configuration
|
|
@@ -120,8 +149,8 @@ Coolhand.configure do |config|
|
|
|
120
149
|
config.silent = Rails.env.production?
|
|
121
150
|
|
|
122
151
|
# Specify which LLM endpoints to intercept (array of strings)
|
|
123
|
-
# Optional - defaults to
|
|
124
|
-
# config.intercept_addresses = ["api.openai.com", "api.anthropic.com", "api.cohere.
|
|
152
|
+
# Optional - defaults to every provider listed under "Supported Libraries" below
|
|
153
|
+
# config.intercept_addresses = ["api.openai.com", "api.anthropic.com", "api.cohere.com/v2/chat"]
|
|
125
154
|
end
|
|
126
155
|
```
|
|
127
156
|
|
|
@@ -178,10 +207,11 @@ end
|
|
|
178
207
|
| `enabled` | Boolean | `true` | Set to `false` to disable all patching and validation (e.g. `Rails.env.production?`) |
|
|
179
208
|
| `capture` | Boolean | `true` | Whether to capture and forward intercepted requests. Set to `false` to monitor without forwarding, then use [`Coolhand.with_capture`](#selective-capture) to re-enable selectively |
|
|
180
209
|
| `silent` | Boolean | `false` | Whether to suppress console output |
|
|
181
|
-
| `intercept_addresses` | Array |
|
|
210
|
+
| `intercept_addresses` | Array | All [supported providers](#supported-libraries) (`DEFAULT_INTERCEPT_ADDRESSES`) | Array of API hosts to monitor (matched by host, optionally pinned to a port and/or path prefix). This is a required allow-list — `[]` is ignored (a warning is logged) rather than disabling capture; use `enabled` or `capture` for that. See [Configuration](docs/configuration.md) |
|
|
182
211
|
| `intercept_path_patterns` | Array | `[":generateContent", ":streamGenerateContent"]` | Path patterns to additionally monitor on Google API hosts — see [Configuration](docs/configuration.md) |
|
|
183
212
|
| `exclude_api_patterns` | Array | `["/batchPredictionJobs/"]` | Deny-list checked after `intercept_addresses`; matching paths are skipped. Unlike `intercept_addresses`, `exclude_api_patterns = []` genuinely disables exclusion. See [Configuration](docs/configuration.md) |
|
|
184
213
|
| `max_captured_body_bytes` | Integer | `1_000_000` | Maximum size of a captured JSON request body — oversized bodies are replaced with a placeholder. Non-JSON bodies (e.g. file/audio uploads) are always skipped regardless of size — see [Advanced Configuration](docs/configuration.md) |
|
|
214
|
+
| `async_logging` | Boolean | `true` | Send logs to Coolhand from a background thread so a slow Coolhand API never delays your LLM calls. Set to `false` to send inline — see [Advanced Configuration](docs/configuration.md#asynchronous-logging) |
|
|
185
215
|
|
|
186
216
|
## Usage Examples
|
|
187
217
|
|
|
@@ -337,10 +367,18 @@ The monitor works with multiple transport layers and Ruby libraries:
|
|
|
337
367
|
**Other providers monitored out of the box:**
|
|
338
368
|
- Google Gemini (`generativelanguage.googleapis.com`)
|
|
339
369
|
- Google Vertex AI (`aiplatform.googleapis.com`) — see the [Vertex AI batch result logging guide](docs/vertex.md) for async batch jobs
|
|
370
|
+
- Azure OpenAI Service, Azure AI Foundry, and Azure Machine Learning managed online endpoints (dedicated hosts, multi-service Azure AI Services hosts, and serverless/MaaS deployments) — see [Configuration](docs/configuration.md) for the full list of covered hosts
|
|
340
371
|
- AWS Bedrock (OpenAI-compatible endpoint)
|
|
341
372
|
- Cloudflare AI Gateway
|
|
342
373
|
- OpenRouter
|
|
343
374
|
- OpenCode Zen (`opencode.ai`)
|
|
375
|
+
- DeepSeek (`api.deepseek.com`)
|
|
376
|
+
- Mistral (`api.mistral.ai`)
|
|
377
|
+
- Perplexity (`api.perplexity.ai`)
|
|
378
|
+
- xAI (`api.x.ai`)
|
|
379
|
+
- Cohere (chat and embed endpoints only, on `api.cohere.com` and `api.cohere.ai`)
|
|
380
|
+
- TypeSafe Jev / System One (`api.typesafe.ai/v1/systemone`)
|
|
381
|
+
- Ollama (self-hosted, default port 11434 — chat, generate and embed endpoints) — see [Configuration](docs/configuration.md) for non-default hosts
|
|
344
382
|
|
|
345
383
|
**Universal Coverage**: Since most Ruby HTTP libraries use Net::HTTP under the hood, Coolhand's single interceptor provides comprehensive monitoring without needing library-specific integrations.
|
|
346
384
|
|
|
@@ -426,13 +464,16 @@ The monitor handles errors gracefully:
|
|
|
426
464
|
- Failed API logging attempts are logged to console but don't interrupt your application
|
|
427
465
|
- Invalid API keys will be reported but won't crash your app
|
|
428
466
|
- Network issues are handled with appropriate error messages
|
|
429
|
-
- The read methods (`search_templates`, `get_template`) are the exception — they raise `Coolhand::HttpError
|
|
467
|
+
- The read methods (`search_templates`, `get_template`, `search_workloads`, `search_logs`, `get_log`) and the feedback-link methods are the exception — they raise `Coolhand::Error` (`Coolhand::HttpError` for non-2xx responses); see [Reading Templates →](docs/template-search.md), [Reading Workloads →](docs/workload-search.md), [Reading Logs →](docs/log-search.md) and [Linking Feedback →](docs/feedback-links.md)
|
|
430
468
|
|
|
431
469
|
## Documentation
|
|
432
470
|
|
|
433
471
|
- **[Configuration](docs/configuration.md)** — Self-hosted deployments, base_url rules, debug mode, custom intercept addresses
|
|
434
472
|
- **[Feedback API](docs/feedback.md)** — Full field reference, matching strategies, sentiment values
|
|
435
473
|
- **[Reading Templates](docs/template-search.md)** — Search LLM request templates and fetch a single one, prompt patterns included, using the private API key
|
|
474
|
+
- **[Reading Workloads](docs/workload-search.md)** — List workloads with optional cost and performance metrics over a rolling or explicit time window
|
|
475
|
+
- **[Reading Logs](docs/log-search.md)** — Search logs with per-log cost, filter and sort by cost, and fetch one log with its cost breakdown
|
|
476
|
+
- **[Linking Feedback](docs/feedback-links.md)** — Link feedback to an optimization as evidence, singly or in bulk, using the private API key
|
|
436
477
|
- **[Anthropic Integration](docs/anthropic.md)** — Official and community Anthropic Ruby gems, streaming, dual gem handling, and troubleshooting
|
|
437
478
|
- **[ElevenLabs Integration](docs/elevenlabs.md)** — Webhook capture, feedback submission, and Rails integration
|
|
438
479
|
- **[OpenAI Batch Webhook Handler](docs/openai.md)** — Handle OpenAI batch job completion events via webhook interception
|
data/docs/configuration.md
CHANGED
|
@@ -39,6 +39,25 @@ end
|
|
|
39
39
|
|
|
40
40
|
Because it forces capture unconditionally and prints full request/response bodies to the console, only enable `debug_mode` in development — never in production or in an environment handling real user data.
|
|
41
41
|
|
|
42
|
+
## Asynchronous Logging
|
|
43
|
+
|
|
44
|
+
By default Coolhand ships each captured request to its API from background worker threads (one for per-request logs, a separate one for batch jobs), so a slow or unreachable Coolhand backend never adds latency to your own LLM calls. The request-path work is limited to capturing and sanitizing the payload.
|
|
45
|
+
|
|
46
|
+
```ruby
|
|
47
|
+
Coolhand.configure do |config|
|
|
48
|
+
config.async_logging = false # default: true — send inline, in the intercepted call's request path
|
|
49
|
+
end
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
**How it behaves:**
|
|
53
|
+
- The queue is bounded at 1,000 pending logs. If Coolhand is down long enough to fill it, the *oldest* pending log is dropped (with a log line) rather than blocking your app or growing memory.
|
|
54
|
+
- The bound is on the *number* of pending logs, not their size: each holds its captured response body in memory until sent, so a long Coolhand outage under heavy traffic with large responses can hold a lot of memory. Batch jobs use their own lane, so a long-running batch never delays or evicts per-request logs.
|
|
55
|
+
- A forked process (Puma/Unicorn workers, Spring) starts its own worker automatically.
|
|
56
|
+
- Pending logs are flushed at process exit (up to 5 seconds in total, so a batch job still running at exit may be only partly logged). In a short-lived script, or to wait explicitly, call `Coolhand.flush(timeout: 5)` — it returns `true` once everything has been sent.
|
|
57
|
+
- OpenAI and Vertex batch results are processed as a single background job per batch.
|
|
58
|
+
- `config.debug_mode` always runs inline so printed payloads appear in order.
|
|
59
|
+
- Logs queued but not yet sent are lost if the process is killed hard (`kill -9`, OOM). Set `async_logging = false` if you need delivery before each call returns.
|
|
60
|
+
|
|
42
61
|
## Request Body Capture Limits
|
|
43
62
|
|
|
44
63
|
To avoid holding large uploads (batch JSONL files, fine-tune corpora, audio for transcription) in memory and shipping them to Coolhand in full, `config.max_captured_body_bytes` caps how much of a request body gets captured:
|
|
@@ -74,7 +93,13 @@ Setting `intercept_addresses` **replaces** the default list entirely, so include
|
|
|
74
93
|
|
|
75
94
|
The default list can be found in `Coolhand::Configuration::DEFAULT_INTERCEPT_ADDRESSES`.
|
|
76
95
|
|
|
77
|
-
Each entry is matched against the request's parsed host — an exact match, or a dot-boundary suffix match (`my-llm-proxy.internal` also matches `eu.my-llm-proxy.internal`) — case-insensitively. A single `*` in an entry matches exactly one host label, e.g. `bedrock-runtime.*.amazonaws.com`. Entries are never matched against the full URL, so query strings
|
|
96
|
+
Each entry is matched against the request's parsed host (optionally with a port and path prefix, see below) — an exact match, or a dot-boundary suffix match (`my-llm-proxy.internal` also matches `eu.my-llm-proxy.internal`) — case-insensitively. A single `*` in an entry matches exactly one host label, e.g. `bedrock-runtime.*.amazonaws.com`. Entries are never matched against the full URL, so query strings that merely mention a host can't trigger a false match.
|
|
97
|
+
|
|
98
|
+
An entry can also pin a port (`host:port`) and/or a path (`host/path`, or `host:port/path`), and then a request must match all of them. Cohere, TypeSafe and Ollama are defaulted this way: `api.cohere.com/v2/chat`, `api.typesafe.ai/v1/systemone`, and `*:11434/api/chat` (plus `/api/generate`, `/api/embed`, `/api/embeddings`, also for `127.0.0.1`). Ollama has no fixed host, so its defaults require Ollama's default port *and* an Ollama path — an app's own unrelated `/api/chat` on another host or port is not captured. To capture an Ollama server on a dotted host or IP, or a non-default port, add entries such as `192.168.1.5:11434/api/chat` or `ollama.internal:8080/api/chat`. An IPv6 loopback server (`[::1]:11434`) is already covered by the defaults.
|
|
99
|
+
|
|
100
|
+
Path-anchoring is also how the Azure defaults work: multi-service hosts like `cognitiveservices.azure.com/openai/` (Azure AI Services, where Speech, Vision, Language and Content Safety share a domain with LLM inference) only match paths under the given prefix, after the host itself has matched.
|
|
101
|
+
|
|
102
|
+
**Note on Azure ML managed online endpoints:** `inference.ml.azure.com` and `inference.ml.azure.us` are intentionally captured *without* a path anchor. Every Azure ML managed online endpoint scores at `/score` regardless of what model sits behind it, so there's no path that distinguishes an LLM deployment from a tabular or vision one — this gem chooses to capture broadly rather than silently miss LLM deployments. If that over-captures for your setup, override `intercept_addresses` to a custom list omitting these two entries.
|
|
78
103
|
|
|
79
104
|
A small number of Google API endpoints use a colon-suffixed path (e.g. `.../gemini-pro:generateContent`) with no host of their own. Those live in the separate `intercept_path_patterns` setting (default: `Coolhand::Configuration::DEFAULT_INTERCEPT_PATH_PATTERNS`) and, unlike `intercept_addresses`, are only checked against the path of requests whose host is already `googleapis.com` or a subdomain of it — and only when `intercept_addresses` still includes at least one `googleapis.com` host. If you override `intercept_addresses` to a list with no Google hosts, `intercept_path_patterns` has no effect and Google API traffic is not captured at all.
|
|
80
105
|
|
|
@@ -82,7 +107,7 @@ A small number of Google API endpoints use a colon-suffixed path (e.g. `.../gemi
|
|
|
82
107
|
|
|
83
108
|
## Excluding API Patterns
|
|
84
109
|
|
|
85
|
-
`config.exclude_api_patterns` is a deny-list checked *after* `intercept_addresses` allows a request through — a match against the request's path is skipped and never forwarded as an `llm_request_log`. It defaults to `
|
|
110
|
+
`config.exclude_api_patterns` is a deny-list checked *after* `intercept_addresses` allows a request through — a match against the request's path is skipped and never forwarded as an `llm_request_log`. It defaults to a small built-in list (see `Coolhand::Configuration::DEFAULT_EXCLUDE_API_PATTERNS`) that suppresses Vertex AI batch job management noise (`/batchPredictionJobs/`) and Azure OpenAI control-plane paths (file, batch, and fine-tuning management under `/openai/` and `/openai/v1/`) that aren't inference traffic.
|
|
86
111
|
|
|
87
112
|
```ruby
|
|
88
113
|
Coolhand.configure do |config|
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Linking feedback to an optimization
|
|
2
|
+
|
|
3
|
+
Attach feedback to an optimization as supporting evidence. `Coolhand::OptimizationFeedbackLinkService`
|
|
4
|
+
wraps `POST /api/v2/optimizations/{optimization_id}/feedback_links` (single and bulk share the route,
|
|
5
|
+
chosen by the body) and `DELETE /api/v2/optimizations/{optimization_id}/feedback_links/{id}`.
|
|
6
|
+
|
|
7
|
+
These methods require your **private** API key; the public key gets a `401`.
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
require "coolhand"
|
|
11
|
+
|
|
12
|
+
Coolhand.configure do |config|
|
|
13
|
+
config.api_key = ENV.fetch("COOLHAND_PRIVATE_API_KEY")
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
links = Coolhand.optimization_feedback_link_service
|
|
17
|
+
|
|
18
|
+
# One feedback. link[:id] is the link's hashid, not the feedback's.
|
|
19
|
+
link = links.link_feedback("optimizationHashid", "feedbackHashid", note: "why")
|
|
20
|
+
|
|
21
|
+
# Many feedbacks.
|
|
22
|
+
result = links.bulk_link_feedback("optimizationHashid", %w[fb1 fb2 fb3])
|
|
23
|
+
result # => { linked: 2, already_linked: 1, errored: 0, not_found: [] }
|
|
24
|
+
|
|
25
|
+
# Remove a link.
|
|
26
|
+
links.unlink_feedback("optimizationHashid", link[:id])
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
## Bulk behavior
|
|
30
|
+
|
|
31
|
+
- The server accepts at most 100 ids per request. `bulk_link_feedback` splits longer lists into
|
|
32
|
+
batches of 100, sums `linked` / `already_linked` / `errored` and concatenates `not_found`.
|
|
33
|
+
- Already-linked ids count as `already_linked`, not errors. Unknown, malformed and other-client ids
|
|
34
|
+
are all reported in `not_found`.
|
|
35
|
+
- If a batch fails, the call raises and earlier batches stay applied. Repeating the call is safe.
|
|
36
|
+
- `link_feedback` is not retry-safe: after a transport error, a `422` "already linked" on retry may mean
|
|
37
|
+
the first attempt succeeded. Use `bulk_link_feedback` with a single id for retry-safe semantics.
|
|
38
|
+
- An empty list or a blank id raises `Coolhand::Error` before any request is made.
|
|
39
|
+
- Ids are trimmed of surrounding whitespace but not de-duplicated client-side.
|
|
40
|
+
|
|
41
|
+
## Errors
|
|
42
|
+
|
|
43
|
+
Unlike the logging writes, all three methods raise. A non-2xx response raises `Coolhand::HttpError`
|
|
44
|
+
whose `status` is the HTTP code: `401` missing or public key, `404` unknown optimization, feedback or
|
|
45
|
+
link, `422` invalid input, an already-linked feedback (single mode) or a `note` that is too long.
|
|
46
|
+
Client-side validation and transport failures raise `Coolhand::Error`.
|
|
47
|
+
|
|
48
|
+
`config.debug_mode` does not suppress these calls: they change server state, so they are always sent.
|
data/docs/log-search.md
ADDED
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# Reading Logs (Search + Get)
|
|
2
|
+
|
|
3
|
+
`Coolhand::LogService` reads logs back out of Coolhand, with the per-log `cost` the dashboard shows.
|
|
4
|
+
The write side, `Coolhand.logger_service`, is a different class. It wraps two read-only endpoints:
|
|
5
|
+
|
|
6
|
+
| method | endpoint |
|
|
7
|
+
|---|---|
|
|
8
|
+
| `search_logs` | `GET /api/v2/llm_request_logs` |
|
|
9
|
+
| `get_log` | `GET /api/v2/llm_request_logs/{id}` |
|
|
10
|
+
|
|
11
|
+
Both require your **private** API key. The public key is write-only on this API and is rejected exactly
|
|
12
|
+
like an invalid one.
|
|
13
|
+
|
|
14
|
+
```ruby
|
|
15
|
+
require "coolhand"
|
|
16
|
+
|
|
17
|
+
Coolhand.configure do |config|
|
|
18
|
+
config.api_key = ENV.fetch("COOLHAND_PRIVATE_API_KEY")
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
logs = Coolhand.log_service
|
|
22
|
+
|
|
23
|
+
# The ten most expensive logs of the last 30 days
|
|
24
|
+
result = logs.search_logs(since: Time.now.utc - (30 * 86_400), order: "cost_desc", per: 10)
|
|
25
|
+
result.logs.each { |log| puts "#{log[:id]} #{log[:model]} $#{log[:cost]}" }
|
|
26
|
+
|
|
27
|
+
detail = logs.get_log(result.logs.first[:id])
|
|
28
|
+
detail[:cost_breakdown][:input_cost]
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## `search_logs`
|
|
32
|
+
|
|
33
|
+
All keywords are optional, and named after the wire parameters (`sort` maps to `q[s]`). Two wire
|
|
34
|
+
parameters are not exposed: the `q[source_api_in][]` array filter, so use the single-value `source_api`
|
|
35
|
+
keyword, and the `per_page` alias, since this gem only sends `per`.
|
|
36
|
+
|
|
37
|
+
| keyword | type | notes |
|
|
38
|
+
|---|---|---|
|
|
39
|
+
| `template_id`, `workload_id` | String | Hashids. One that does not decode, or that belongs to another client, is a `422`. |
|
|
40
|
+
| `system_prompt_contains`, `user_prompt_contains` | String | Case-insensitive substring. |
|
|
41
|
+
| `model`, `source_api`, `source_api_result`, `source_application` | String | Exact match. `source_api_result` is `success`, `failed`, `operational` or `unmatched`. |
|
|
42
|
+
| `project_path` | String | Exact match against `metadata.project_path`. |
|
|
43
|
+
| `unmatched_only` | Boolean | Only logs with no template. |
|
|
44
|
+
| `days_back` | Integer | Last N days. Unset means unrestricted. Ignored when `since` is given; with only `until`, the window is `days_back` long ending at `until`. |
|
|
45
|
+
| `since`, `until` | Time, DateTime, Date or String | Bounds on `created_at`: `since` inclusive, `until` exclusive. See [Windows](workload-search.md#windows-since-and-until). |
|
|
46
|
+
| `min_cost` | Numeric | Only logs whose `cost` (USD) is at least this. `0` is sent. |
|
|
47
|
+
| `order` | String | `"cost_desc"` sorts by `cost`, highest first, replacing `sort`. Any other value is a `422` from the server. |
|
|
48
|
+
| `include_prompts` | Boolean | Adds `:system_prompt` and `:user_prompt`, each truncated to 500 characters. |
|
|
49
|
+
| `include_total` | Boolean | Asks the server for the `X-Total-*` headers. Off by default there (it skips a `COUNT(*)`). |
|
|
50
|
+
| `sort` | String | Ransack sort expression, sent as `q[s]`, e.g. `"created_at desc"`. |
|
|
51
|
+
| `page`, `per` | Integer | 1-based; `per` defaults to 25, max 100. |
|
|
52
|
+
|
|
53
|
+
`min_cost` and `order: "cost_desc"` only consider logs that can be priced (non-nil `cost`). The cost is
|
|
54
|
+
computed per log, so on a client with a lot of history combine them with `since` or `days_back`, or the
|
|
55
|
+
query can time out (`504`).
|
|
56
|
+
|
|
57
|
+
### Return value
|
|
58
|
+
|
|
59
|
+
A `Coolhand::LogSearchResult`, a `Struct` with `#logs` and `#pagination`. The response body is a bare
|
|
60
|
+
array, never wrapped. Rows are Symbol-keyed Hashes: `:id`, `:collector`, `:source_api`,
|
|
61
|
+
`:source_application`, `:metadata`, `:source_api_result`, `:model`, `:template_id`, `:template_name`,
|
|
62
|
+
`:input_tokens`, `:output_tokens`, `:latency_ms`, `:created_at`, `:updated_at`, `:ingest_evidence` and
|
|
63
|
+
`:cost`.
|
|
64
|
+
|
|
65
|
+
`:cost` is a Float in USD, or `nil` when the log has no tokens or its model has no pricing.
|
|
66
|
+
|
|
67
|
+
**Pagination without totals.** `X-Page` and `X-Per-Page` are always sent, but `X-Total-Count` and
|
|
68
|
+
`X-Total-Pages` only come with `include_total: true`. Without it, `pagination.total_count` and
|
|
69
|
+
`total_pages` are lower bounds the gem derives, not counts, and `has_next_page` is true when the page
|
|
70
|
+
came back full. Pass `include_total: true` when you need real totals.
|
|
71
|
+
|
|
72
|
+
## `get_log(id, section: nil, max_chars: nil, search_query: nil, include_thinking: nil)`
|
|
73
|
+
|
|
74
|
+
`id` is the log hashid. Returns the full log as a Symbol-keyed Hash: the list fields plus the prompts and
|
|
75
|
+
`:output`, and:
|
|
76
|
+
|
|
77
|
+
| field | type |
|
|
78
|
+
|---|---|
|
|
79
|
+
| `:url` | String: the log's dashboard path (`/c/<client>/llm_request_logs/<id>`) |
|
|
80
|
+
| `:cost` | Float or nil |
|
|
81
|
+
| `:cost_breakdown` | Hash or nil: `:total_cost`, `:input_cost`, `:output_cost`, `:cached_input_cost`, `:cache_creation_input_cost`, `:reasoning_output_cost` |
|
|
82
|
+
|
|
83
|
+
`section` (`"full"`, `"beginning"`, `"end"`), `max_chars` and `search_query` bound or search the content;
|
|
84
|
+
`include_thinking` adds `:thinking_response`. Truncation and search are reported back as `:truncated`,
|
|
85
|
+
`:total_chars`, `:search_query` and `:matches`. Only directly-collected logs are fetchable; internally
|
|
86
|
+
generated records are a `404`.
|
|
87
|
+
|
|
88
|
+
## Errors
|
|
89
|
+
|
|
90
|
+
The read methods raise where the write methods log and return `nil`. `Coolhand::HttpError` carries
|
|
91
|
+
`#status` and `#body`.
|
|
92
|
+
|
|
93
|
+
| status | when |
|
|
94
|
+
|---|---|
|
|
95
|
+
| `401` | No API key, an invalid key, or the public key. |
|
|
96
|
+
| `404` | `get_log` only: unknown id, another client's log, or an internally generated record. |
|
|
97
|
+
| `422` | Bad `template_id`, `workload_id`, `days_back`, `since`, `until`, `min_cost`, `order` or `max_chars`. |
|
|
98
|
+
| `504` | The search exceeded the statement timeout. Narrow the window and retry. |
|
|
99
|
+
|
|
100
|
+
`Coolhand::Error` covers no API key configured, a transport failure, a body that is not valid JSON, a
|
|
101
|
+
non-array list body, and a blank `id` passed to `get_log`. An unsupported `since`/`until` type raises
|
|
102
|
+
`ArgumentError` before any request.
|
data/docs/openai.md
CHANGED
|
@@ -14,7 +14,7 @@ Requires Rails — `Coolhand::WebhookInterceptor` and `Coolhand::OpenAi::BatchRe
|
|
|
14
14
|
- Ensure you skip CSRF for the webhook endpoint: `skip_before_action :verify_authenticity_token`
|
|
15
15
|
- Override the `webhook_secret` method to return your OpenAI webhook secret
|
|
16
16
|
|
|
17
|
-
`intercept_batch_request` already validates the webhook and, for `batch.completed`/`batch.failed`/`batch.expired`/`batch.cancelled` events, calls `Coolhand::OpenAi::BatchResultProcessor`
|
|
17
|
+
`intercept_batch_request` already validates the webhook and, for `batch.completed`/`batch.failed`/`batch.expired`/`batch.cancelled` events, calls `Coolhand::OpenAi::BatchResultProcessor` — that part requires no code in your controller action. The batch's status is checked inline in the `before_action`, but for a completed batch the two JSONL file downloads and the one-request-log-per-item sends run on Coolhand's [background log thread](configuration.md#asynchronous-logging), so a large batch doesn't hold up your webhook response. (With `config.async_logging = false` this all runs inline, and a large batch can trip your webhook endpoint's timeout and OpenAI's retry behavior.) The example below shows a controller action doing additional, app-specific work on top of that (e.g. updating your own `BatchApiRequest` record and enqueuing your own background job), which is optional.
|
|
18
18
|
|
|
19
19
|
## Minimal example
|
|
20
20
|
|
data/docs/template-search.md
CHANGED
|
@@ -41,7 +41,7 @@ This is **not** a port of the `search_templates` MCP tool, and the two do not ag
|
|
|
41
41
|
Template *creation, update and deprecation* stay on the MCP surface. This REST surface is
|
|
42
42
|
read-only, and there is no version-history sub-resource.
|
|
43
43
|
|
|
44
|
-
## `search_templates(search: nil, workload_id: nil, status: nil, include_deprecated: nil, include_system: nil, page: nil, per: nil)`
|
|
44
|
+
## `search_templates(search: nil, workload_id: nil, status: nil, include_deprecated: nil, include_system: nil, include_metrics: nil, days_back: nil, since: nil, until: nil, page: nil, per: nil)`
|
|
45
45
|
|
|
46
46
|
Search is a *parameter* on the list endpoint rather than a route of its own, so this is one method
|
|
47
47
|
rather than a list/search pair.
|
|
@@ -56,6 +56,10 @@ and this API's already agree, so there is no second spelling to translate throug
|
|
|
56
56
|
| `status` | String | `"draft"`, `"published"` or `"failure"`. Any other non-empty value is a `422` from the server; empty is treated as no filter. |
|
|
57
57
|
| `include_deprecated` | Boolean | Include templates with a non-null `deprecated_at`. Defaults to `false` server-side. |
|
|
58
58
|
| `include_system` | Boolean | Include the `Unmatched` / `Ignored API Calls` buckets. Defaults to `false` server-side. |
|
|
59
|
+
| `include_metrics` | Boolean | Adds a `:metrics` Hash to every row. See [Metrics](#metrics). |
|
|
60
|
+
| `days_back` | Integer | Rolling metrics window in days (server default 28, max 365). Ignored unless metrics are requested, and when `since` is given. |
|
|
61
|
+
| `since` | Time, DateTime, Date or String | Metrics window start, inclusive. Overrides `days_back`. |
|
|
62
|
+
| `until` | Time, DateTime, Date or String | Metrics window end, exclusive. Defaults to now. |
|
|
59
63
|
| `page` | Integer | 1-based. |
|
|
60
64
|
| `per` | Integer | Default 25, max 100, both enforced server-side. |
|
|
61
65
|
|
|
@@ -138,11 +142,14 @@ unmatched = Coolhand.template_service
|
|
|
138
142
|
puts unmatched[:log_count]
|
|
139
143
|
```
|
|
140
144
|
|
|
141
|
-
## `get_template(id)`
|
|
145
|
+
## `get_template(id, include_metrics: nil, days_back: nil, since: nil, until: nil)`
|
|
142
146
|
|
|
143
147
|
`id` is the template hashid — the `:id` field from `search_templates`.
|
|
144
148
|
|
|
145
|
-
Returns a Hash with every list field above, **plus** the full untruncated regexes the list omits
|
|
149
|
+
Returns a Hash with every list field above, **plus** the full untruncated regexes the list omits, and
|
|
150
|
+
a `:metrics` Hash. Unlike the list, metrics are **on by default** here: pass `include_metrics: false` to
|
|
151
|
+
omit them, which also skips the window validation. `days_back`, `since` and `until` set the window
|
|
152
|
+
exactly as they do on the list, and an invalid one is a `422` unless metrics are off.
|
|
146
153
|
|
|
147
154
|
| field | type |
|
|
148
155
|
|---|---|
|
|
@@ -153,6 +160,41 @@ Unlike the list, this filters on nothing but client ownership: a deprecated or s
|
|
|
153
160
|
reachable by id with **no opt-in flag**, because inspecting one of those is the usual reason to
|
|
154
161
|
fetch a template directly.
|
|
155
162
|
|
|
163
|
+
## Metrics
|
|
164
|
+
|
|
165
|
+
With `include_metrics: true` every row (and by default the `get_template` result) carries `:metrics`,
|
|
166
|
+
computed by the same SQL as the dashboard, so tiered pricing, cached-token discounts and reasoning tokens
|
|
167
|
+
are applied. `Coolhand.workload_service` returns the same object per workload, rolled up across its
|
|
168
|
+
templates ([Reading Workloads](workload-search.md)).
|
|
169
|
+
|
|
170
|
+
Averages and counters cover non-failed, directly-collected client logs inside the window. Keys are the
|
|
171
|
+
wire names, as Symbols.
|
|
172
|
+
|
|
173
|
+
| field | type | notes |
|
|
174
|
+
|---|---|---|
|
|
175
|
+
| `:days_back` | Integer or nil | `nil` when `since` defined the window. |
|
|
176
|
+
| `:since`, `:until` | String | The resolved window, ISO-8601 UTC. |
|
|
177
|
+
| `:request_count` | Integer | |
|
|
178
|
+
| `:failure_count` | Integer | Failed logs. |
|
|
179
|
+
| `:error_rate` | Float or nil | `nil` when `failure_count` is 0. |
|
|
180
|
+
| `:error_rate_change` | Float or nil | Against the window of the same length immediately before. |
|
|
181
|
+
| `:avg_cost_per_request` | Float or nil | `nil` when no log in the window could be priced. |
|
|
182
|
+
| `:total_cost` | Float or nil | |
|
|
183
|
+
| `:priced_request_count` | Integer | The logs `total_cost` covers: non-failed, with tokens, model has pricing. |
|
|
184
|
+
| `:long_context_request_count` | Integer | Priced logs that crossed their model's input-token pricing tier. |
|
|
185
|
+
| `:total_input_tokens`, `:total_output_tokens` | Integer | Raw token columns summed over non-failed logs, not cache-adjusted. |
|
|
186
|
+
| `:avg_input_tokens`, `:avg_output_tokens` | Integer or nil | |
|
|
187
|
+
| `:avg_latency_ms` | Float or nil | |
|
|
188
|
+
| `:correctness_score`, `:sentiment_score`, `:revision_score` | Float or nil | `sentiment_score` and `revision_score` are all-time, not windowed. |
|
|
189
|
+
| `:first_request_at`, `:last_request_at` | String or nil | Lifetime, not windowed. |
|
|
190
|
+
|
|
191
|
+
Summing the per-log `cost` from [Reading Logs](log-search.md) can exceed `total_cost`, because the log
|
|
192
|
+
`cost` is also computed for failed logs and logs with a zero token count.
|
|
193
|
+
|
|
194
|
+
Windows follow the rules in [Reading Workloads](workload-search.md#windows-since-and-until). A window
|
|
195
|
+
error is a `422` on the `since` or `until` key. Metrics add aggregate statements, so expect a slower
|
|
196
|
+
response, and a `504` on a large page.
|
|
197
|
+
|
|
156
198
|
## Errors
|
|
157
199
|
|
|
158
200
|
The write methods in this gem (`create_feedback`, `create_log`, `send_llm_request_log`) log a
|
|
@@ -186,7 +228,7 @@ end
|
|
|
186
228
|
|---|---|
|
|
187
229
|
| `401` | Missing, invalid, or public API key. |
|
|
188
230
|
| `404` | `get_template` only. Unknown id, **or** one belonging to another client — existence is not disclosed, so this is never a `403`. |
|
|
189
|
-
| `422` | Unrecognised `status`,
|
|
231
|
+
| `422` | Unrecognised `status`, a `workload_id` that does not decode or belongs to another client, or a bad `days_back`, `since` or `until` when metrics are requested. |
|
|
190
232
|
| `504` | See below. |
|
|
191
233
|
|
|
192
234
|
### `504` is expected, and retryable
|
data/docs/vertex.md
CHANGED
|
@@ -6,6 +6,8 @@ For monitoring regular (non-batch) Vertex AI calls, no extra setup is required b
|
|
|
6
6
|
|
|
7
7
|
Requires Rails — `Coolhand::Vertex::BatchResultProcessor` logs via `Rails.logger` internally. `config.capture = false` and `Coolhand.without_capture` do not suppress these logs: unlike the passive Net::HTTP interceptor, calling this processor is an explicit, deliberate act, so it always sends.
|
|
8
8
|
|
|
9
|
+
> **Note:** As of May 2026, Google markets this product as "Gemini Enterprise Agent Platform" rather than "Vertex AI." The API endpoint (`aiplatform.googleapis.com`) and Ruby SDK are unchanged — it's a console/marketing rename, not an SDK rename — so `Coolhand::Vertex` naming stays accurate here. It's also still a distinct integration from the separate Gemini Developer API (`generativelanguage.googleapis.com`), which Coolhand intercepts independently.
|
|
10
|
+
|
|
9
11
|
## Usage
|
|
10
12
|
|
|
11
13
|
```ruby
|
|
@@ -13,6 +15,7 @@ require "coolhand/vertex/batch_result_processor"
|
|
|
13
15
|
```
|
|
14
16
|
|
|
15
17
|
- Call the `Coolhand::Vertex::BatchResultProcessor` service with `batch_info` and the downloaded batch results.
|
|
18
|
+
- For a succeeded job, the per-item request logs are sent from Coolhand's [background log thread](configuration.md#asynchronous-logging) as a single job, so `call` returns without waiting on the Coolhand API (set `config.async_logging = false` to send inline).
|
|
16
19
|
- Optionally pass `model:` to populate the logged `model` field. It falls back to a `"model"` key in `batch_info` if present, and is omitted from the payload entirely when neither is available. If the value (from either source) looks like a Vertex model resource path (`publishers/<publisher>/models/<id>` or `projects/<project>/locations/<location>/models/<id>`), it's normalized down to just the trailing id/version before being logged; any other string is sent as-is.
|
|
17
20
|
- `batch_info["name"]` must be the exact Vertex job resource name (`projects/<project>/locations/<location>/batchPredictionJobs/<job-id>`), and `batch_info["startTime"]`/`["endTime"]` must be valid ISO 8601 timestamps — a batch with a missing or malformed resource name or timestamps is skipped entirely (logged as an error) rather than sent with a broken URL.
|
|
18
21
|
- If `endTime` precedes `startTime` (e.g. clock skew), the batch is still logged — `duration_ms` is clamped to `0` and a warning is logged — rather than dropping the results.
|