coolhand 0.6.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a3c1e767ea245f56520ee634e48cd2f12c6dc06be79b3998537fea9e9b8ea9ff
4
- data.tar.gz: aad0485d474aa1e3f796f2af30083304a1ed704ede8428ddc4b0c9a4ff32678e
3
+ metadata.gz: 256c9237619b3f7116aa9751f78f0d0e367381b12afa694742da07ed1d7d3123
4
+ data.tar.gz: 149ddf8c4cd1800b105382ade5d2a2e9e57a360884e34e1b988b926739e90f30
5
5
  SHA512:
6
- metadata.gz: 2b497e1401c64e9eb6a6b1bbb061ecb359985a39f8dd95ce8018406d36e3c49ba061e5557299d4e6ee00498f8f8bf91ac9f3b91e56a32f5b08af599f5ef16768
7
- data.tar.gz: 84d13bff930f045607270849ad02fd5c8548b15b39f2043636c868bcfe857e04fab4c0ae113fe4e5872ca299d197c83b04c2f0adce5bffe3723c4bacd5890b03
6
+ metadata.gz: 2ccbf54c2787a8df93178120583d83666553454a1b86385ec1c4ea10e228b859b995a35bed1450e9ab6d38dbb5d145d9b7a269be89f5562870d51a63630e5653
7
+ data.tar.gz: 4e6db88c4530737370030809a9d8e094d791761a6306a63701edc4413ed81f61a716d975953799c3d46b0cebbb3ddd9e1d172b22ee5d7b29c18bdd65ffe9e25c
data/AGENTS.harness.md CHANGED
@@ -39,8 +39,8 @@ node <workspaceRoot>/coolhand/harness/harness.mjs my-issue --run <RUN_DIR> --rep
39
39
  2. `<workspaceRoot>/coolhand/harness/RESIST_RULES.md` — the refuse list.
40
40
  3. The API definition at `specPath`. It is your only source of truth **for the endpoint's
41
41
  contract** — paths, params, response fields, status codes.
42
- 4. `coolhand-ruby/CLAUDE.md` — this repo's own rulebook. **It is authoritative.** Where it
43
- disagrees with this harness file, CLAUDE.md wins — follow it and say so in your PR.
42
+ 4. `coolhand-ruby/AGENTS.md` — this repo's own rulebook. **It is authoritative.** Where it
43
+ disagrees with this harness file, AGENTS.md wins — follow it and say so in your PR.
44
44
 
45
45
  **Your issue links node's PR as the reference implementation. Use it for structure, not
46
46
  for facts.** Node went first so you do not have to rediscover how a REST method fits into
@@ -70,7 +70,7 @@ inside the current architecture, escalate and STOP.
70
70
  Require it inside the file that actually uses it, executed only when that provider's
71
71
  functionality is accessed. Clients may not use every provider and must not be forced to
72
72
  install gems they do not need — apps consuming this as a path gem break on missing optional
73
- dependencies. See `coolhand-ruby/CLAUDE.md`.
73
+ dependencies. See `coolhand-ruby/AGENTS.md`.
74
74
 
75
75
  ## 3. Prove it against the real server
76
76
 
@@ -107,13 +107,44 @@ node <workspaceRoot>/coolhand/harness/harness.mjs wait --run <RUN_DIR> --channel
107
107
 
108
108
  Name the rule number (`R1`–`R5`). Do not guess. Do not stub. Do not work around it.
109
109
 
110
- ## 5. Run your review skill
111
-
112
- **Mandatory, before you push.** Follow `<workspaceRoot>/coolhand/harness/RESIST_RULES.md`
113
- → "Before you push: run your repo's review skill" (R8) — load
114
- `.claude/skills/loop-review/SKILL.md` off disk and run it for real against your diff. Do
115
- not approximate its steps manually; if you cannot spawn the reviewer subagent it calls
116
- for, escalate to node (R8) and STOP rather than substitute a self-review.
110
+ ## 5. Get your review — you cannot run it yourself
111
+
112
+ **Mandatory, before you push. You do not run this section yourself.** You are three
113
+ levels deep in this tree (server → node → you), and the platform does not let an agent at
114
+ this depth spawn the reviewer subagent the review skill needs — see
115
+ `<workspaceRoot>/coolhand/harness/RESIST_RULES.md` → R8's depth table. This is not something
116
+ to confirm by trying; go straight to the request below.
117
+
118
+ 1. Commit locally first — build, prove it against the local server, run `bundle exec rake`,
119
+ all of section 3, all done before this step.
120
+ 2. Send an R8 escalation to node naming your repo path, branch, and `HEAD` sha:
121
+ ```
122
+ node <workspaceRoot>/coolhand/harness/harness.mjs send --run <RUN_DIR> --channel ruby \
123
+ --from ruby --to node --kind escalation \
124
+ --text "R8: ready for review. repo=<workspaceRoot>/coolhand-ruby branch=<branch> sha=<sha>"
125
+ ```
126
+ 3. `wait --after` the `messageId` your `send` in step 2 just printed. A timeout is silence,
127
+ not permission — call `wait` again, `--after` that same id, rather than pushing
128
+ (`RESIST_RULES.md` → "What escalate and stop means, mechanically"). If you loop back here
129
+ from step 5, use the fresh escalation's `messageId` instead.
130
+ 4. **The first reply is an `ack`, not the answer** — it means node has started, not that it
131
+ is done. `wait` again, `--after` that ack's `messageId`, for the `resolution` that
132
+ actually follows.
133
+ 5. **If what follows is a second `ack` instead of a `resolution`, node is asking you to do
134
+ something before it can start — most often that your checkout is not clean** (see
135
+ `RESIST_RULES.md` → R8 → "Serving a review for a child" step 2). Do what it asks, then
136
+ send a fresh R8 escalation with your current `HEAD` sha (it may have changed) and go back
137
+ to step 3. Do not just `wait` again — nothing arrives until you re-escalate.
138
+ 6. **The `resolution` that eventually arrives always carries the full Iteration Breakdown
139
+ table plus an explicit `CLEAN`/`capped` verdict** — the same `result` value node has
140
+ already recorded (it records before it replies) — from a review node ran against your
141
+ actual repo. Not just one or the other, not your own read of the diff, and not an ack.
142
+ This is what you post as your own PR comment below, unedited plus the "Reviewed by node
143
+ against `<sha>`" line.
144
+ 7. **Then move on to section 6 (Open your PR).** If node's review applied fixes,
145
+ it already committed them directly into your checkout while you waited — whatever is at
146
+ `HEAD` when you push there already includes them, and you do not need to reproduce or
147
+ look for them separately.
117
148
 
118
149
  ## 6. Open your PR — then STOP
119
150
 
@@ -123,8 +154,11 @@ for, escalate to node (R8) and STOP rather than substitute a self-review.
123
154
  2. Body must reference your issue with `Closes #N` so it auto-closes on merge, and must
124
155
  say: **depends on the server PR — deploy that first.**
125
156
  3. Record it: `node <workspaceRoot>/coolhand/harness/harness.mjs pr --run <RUN_DIR> --repo ruby --url <url>`
126
- 4. Post your review skill's Iteration Breakdown table (section 5) as a comment on this PR
127
- — see `RESIST_RULES.md` → "After the loop exits."
157
+ 4. Post node's Iteration Breakdown table (section 5) as a comment on this PR — see
158
+ `RESIST_RULES.md` → "After the loop exits" — and add one line the table itself does not
159
+ carry: `Reviewed by node against <sha>`, since you did not run it yourself. **Do not call
160
+ `harness.mjs loop-review` here — node already recorded it**, against the sha that
161
+ resulted from its own fix commits, not the one you originally escalated with.
128
162
  5. **Stop.** You launch no one. The tree ends with you on this branch.
129
163
 
130
164
  ## 7. Done means
@@ -133,7 +167,8 @@ for, escalate to node (R8) and STOP rather than substitute a self-review.
133
167
  - [ ] `bundle exec rake` passes (RSpec + RuboCop)
134
168
  - [ ] At least one spec hit the real local server, not a mock
135
169
  - [ ] PR opened, recorded, references its issue, states its dependency on the server PR
136
- - [ ] Your review skill ran for real (not approximated) and its Iteration Breakdown table
137
- is posted as a comment on your PR
170
+ - [ ] Node ran your review for real (not approximated by you) and its Iteration Breakdown
171
+ table is posted as a comment on your PR, naming node and the reviewed sha, and
172
+ recorded with `harness.mjs loop-review`
138
173
  - [ ] Every field and status code came from the definition, not from node's code
139
174
  - [ ] You launched no child agents
data/CHANGELOG.md CHANGED
@@ -7,6 +7,54 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.8.0] - 2026-10-09
11
+
12
+ ### Added
13
+ - **Read workloads and logs, with cost and performance metrics** ([#138](https://github.com/Coolhand-Labs/coolhand-ruby/pull/138)) — `Coolhand.workload_service.search_workloads` lists workloads (`GET /api/v2/workloads`) and `Coolhand.log_service` reads logs back out: `search_logs` (filters including `since`, `until`, `min_cost`, `order`, `project_path`, `source_application`, `include_total`, `sort`) and `get_log`, which returns `cost` and `cost_breakdown`. `search_templates` and `get_template` gain `include_metrics`, `days_back`, `since` and `until`. `since:`/`until:` accept a `Time`, `DateTime`, `Date` or ISO8601 String; anything else raises `ArgumentError` before a request is made. These need your **private** API key and raise `Coolhand::Error` (`Coolhand::HttpError` for non-2xx responses) rather than logging and returning `nil`. **Deploy note:** requires the matching Coolhand server release (Coolhand-Labs/coolhand#1753). See [Reading Workloads](docs/workload-search.md) and [Reading Logs](docs/log-search.md).
14
+ - **Link feedback to an optimization** ([#133](https://github.com/Coolhand-Labs/coolhand-ruby/pull/133)) — `Coolhand.optimization_feedback_link_service` provides `link_feedback`, `bulk_link_feedback` (sent in batches of 100, results summed) and `unlink_feedback` for `/api/v2/optimizations/:id/feedback_links`. Private API key required; failures raise `Coolhand::Error` / `Coolhand::HttpError`. See [Linking Feedback](docs/feedback-links.md).
15
+ - **Logs are sent from a background queue by default** ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148)) — the Coolhand POST no longer runs inline in the intercepted LLM call, so a slow or unreachable Coolhand backend can no longer add up to about 10s to each call. `Coolhand::LogQueue` holds up to 1,000 jobs per lane and drops the oldest when full, restarts itself in forked processes, and flushes at exit (waiting up to 5s). OpenAI and Vertex batch results are processed as one background job per batch, on their own lane so a long batch cannot delay or evict per-request logs. New `config.async_logging` (default `true`) and `Coolhand.flush(timeout: 5)`; `examples/benchmark_async_logging.rb` is a pass/fail benchmark. See [Asynchronous Logging](docs/configuration.md#asynchronous-logging). **Migration note:** anything that relied on the log POST finishing before the intercepted call returned (tests asserting on a stubbed `ApiService` inline, scripts that end with `exit!`, a process killed by `kill -9` or the OOM killer, which loses queued logs) should set `config.async_logging = false` or call `Coolhand.flush`. `debug_mode` always runs inline.
16
+
17
+ ### Changed
18
+ - Repo guidance moved from `CLAUDE.md` to `AGENTS.md` ([#130](https://github.com/Coolhand-Labs/coolhand-ruby/pull/130)), and the agent-harness review handshake in `AGENTS.harness.md` was rewritten ([#134](https://github.com/Coolhand-Labs/coolhand-ruby/pull/134)). Neither ships in the gem, and `AGENTS.md` is excluded from the gem package in place of `CLAUDE.md`.
19
+ - CI now uses `ruby/setup-ruby` 1.327.0 ([#135](https://github.com/Coolhand-Labs/coolhand-ruby/pull/135)).
20
+ - Shared read-URL building moved into `Coolhand::ReadQuery`, and POST/DELETE helpers that raise on failure into `Coolhand::WriteRequests`; `TemplateService` now uses `ReadQuery`. Internal refactor with no behavior change to `search_templates`/`get_template` beyond the new options.
21
+ - `Coolhand::NetHttpInterceptor` no longer includes `BaseInterceptor` ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148)); its helpers moved to `Coolhand::NetHttpInterceptor::Helpers`, so `Net::HTTP` gains only `#request` instead of about 15 generically named private methods (`intercept?`, `parse_json`, `sanitize_url`, ...). **Migration note:** internal API, but if you called those helpers on a `Net::HTTP` instance, use `Coolhand::NetHttpInterceptor::Helpers` or `Coolhand::BaseInterceptor`.
22
+
23
+ ### Fixed
24
+ - Disabling the Net::HTTP interceptor logged a stale "Faraday monitoring disabled" message ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148)); it now says "Net::HTTP interceptor unpatched (capture stopped)".
25
+
26
+ ### Security
27
+ - **Release-time hardening from a whole-package security review** ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148), plus fixes made on the release branch):
28
+ - Requests forwarded by the OpenAI batch webhook handler now go through the same request-body redaction as live interception, so Azure OpenAI "On Your Data" datastore credentials in a batch request body are replaced with `[REDACTED]` before being sent to Coolhand.
29
+ - The OpenAI batch handler now matches requests to results with a one-pass index instead of a scan per result, so a large batch no longer takes quadratic time. When a `custom_id` is duplicated the first request still wins, as before.
30
+
31
+ ## [0.7.0] - 2026-09-20
32
+
33
+ ### Added
34
+ - **Seven more providers are monitored out of the box** ([#117](https://github.com/Coolhand-Labs/coolhand-ruby/pull/117)) — DeepSeek (`api.deepseek.com`), Mistral (`api.mistral.ai`), Perplexity (`api.perplexity.ai`) and xAI (`api.x.ai`) are captured host-wide. Cohere is captured only on its chat and embed endpoints (`/v2/chat`, `/v1/embed`, `/v2/embed`, on `api.cohere.com` and `api.cohere.ai`), TypeSafe Jev only on `api.typesafe.ai/v1/systemone`, and Ollama only on port 11434 for `/api/chat`, `/api/generate`, `/api/embed` and `/api/embeddings` (`localhost`, `127.0.0.1`, or a bare single-label host such as a Docker Compose service). No configuration required. **Migration note:** if your app already calls any of these, those calls will now start being logged to Coolhand after upgrading. To opt out, override `config.intercept_addresses` with your own list.
35
+ - **Azure inference endpoints are monitored out of the box** ([#115](https://github.com/Coolhand-Labs/coolhand-ruby/pull/115)) — Azure OpenAI dedicated hosts (`openai.azure.com`/`.us`/`.cn`), Azure AI Services / Foundry multi-service hosts (`cognitiveservices.azure.*`, `services.ai.azure.*`, captured only on their `/openai/` and `/models/` paths), serverless/MaaS hosts (`inference.ai.azure.com`, `models.ai.azure.com`) and Azure ML managed online endpoints (`inference.ml.azure.com`/`.us`). The Azure ML entries are deliberately not path-anchored, because every managed online endpoint scores at `/score`, so non-LLM deployments there will also be captured. Remove those two entries from `intercept_addresses` if that over-captures for you. Default `exclude_api_patterns` now also skips Azure OpenAI control-plane paths (`/openai/files`, `/batches`, `/fine_tuning`, `/models`, and their `/openai/v1/` spellings). **Migration note:** anyone who set `exclude_api_patterns` explicitly keeps their own list and will not get the new Azure exclusions.
36
+ - **`intercept_addresses` entries can pin a port and/or a path prefix** ([#117](https://github.com/Coolhand-Labs/coolhand-ruby/pull/117), [#115](https://github.com/Coolhand-Labs/coolhand-ruby/pull/115)) — `host:port`, `host/path` and `host:port/path` are all accepted. A request must match every part, and the path matches on a segment boundary (`host/openai` matches `/openai` and `/openai/x`, not `/openaiz`). Plain-host entries behave exactly as before. See [Configuration](docs/configuration.md#custom-intercept-addresses).
37
+
38
+ ### Changed
39
+ - Ruby 4.0 was added to the CI matrix ([#113](https://github.com/Coolhand-Labs/coolhand-ruby/pull/113)) and then removed again before release: its job could not `bundle install` against the locked dependencies (`BUNDLED WITH 2.3.6`, the conditional `pry-byebug`/`debug` Gemfile entry and `unicode-emoji 4.1.0`). CI covers Ruby 3.1–3.4 until the lock can support 4.0.
40
+
41
+ ### Fixed
42
+ - `examples/anthropic_example.rb` could not run against the pinned `anthropic` 0.4.x gem: it relied on the client reading `ANTHROPIC_API_KEY` from the environment and on object-style response access, and it targeted a retired model. It now passes `access_token:` explicitly, reads the response as a Hash, and uses `claude-haiku-4-5-20251001`. Found by running it against a live key during release validation.
43
+
44
+ ### Security
45
+ - **Azure OpenAI "On Your Data" datastore credentials are now redacted from captured request bodies** ([#115](https://github.com/Coolhand-Labs/coolhand-ruby/pull/115)) — a request's `data_sources`/`dataSources` entries can carry live credentials (Azure AI Search API keys, Cosmos/Mongo connection strings, Elasticsearch encoded keys) that header and URL sanitization never looked at. `BaseInterceptor.sanitize_body` now replaces any value under those entries whose key name contains `key`, `token`, `secret`, `password`, `credential`, `connectionstring` or `signature` (separators and case ignored) with `[REDACTED]` before the log is forwarded. Message content and tool definitions outside `data_sources` are untouched.
46
+ - **Release-time hardening from a whole-package security review** ([#148](https://github.com/Coolhand-Labs/coolhand-ruby/pull/148), plus fixes made on the release branch):
47
+ - `Coolhand.log` no longer raises when stdout is unwritable. Previously a failing `puts` inside the interceptor's `ensure` block could replace the host application's own LLM response with an `IOError`.
48
+ - A bad `intercept_addresses`/`exclude_api_patterns` value (e.g. a String or Symbol instead of an array of Strings) now degrades to "don't capture" instead of raising into every intercepted request.
49
+ - `BaseInterceptor.sanitize_url` now fails closed on an unparseable URL, dropping userinfo, query and fragment rather than returning the raw URL with credentials intact.
50
+ - Header redaction now also matches names containing `auth`, `passw`, `credential`, `bearer`, `jwt` or `session` (e.g. `X-Auth`, `X-Session-Id`, `X-Jwt`). **Migration note:** headers such as `x-session-id` now appear as `[REDACTED]` in Coolhand.
51
+ - `sanitize_body` matches the `data_sources` key case- and separator-insensitively, and also redacts `pwd`, `passw*`, `bearer` and `jwt` credential keys.
52
+ - Hosts with a trailing dot (`api.openai.com.`) are now matched like their dotless form.
53
+ - `OpenAi::WebhookValidator` rejects a bare `whsec_` secret, which decodes to an empty HMAC key and would make signatures forgeable.
54
+
55
+ ### Documentation
56
+ - Clarified that Google's "Gemini Enterprise Agent Platform" is the new marketing name for Vertex AI and does not change `aiplatform.googleapis.com` or `Coolhand::Vertex` ([#118](https://github.com/Coolhand-Labs/coolhand-ruby/pull/118)).
57
+
10
58
  ## [0.6.0] - 2026-09-12
11
59
 
12
60
  ### Added
data/README.md CHANGED
@@ -103,6 +103,35 @@ Search is a parameter on the list endpoint, not a route of its own, and the `Unm
103
103
 
104
104
  For the full filter reference, pagination, and error handling, see [Reading Templates →](docs/template-search.md).
105
105
 
106
+ ### Cost and performance metrics
107
+
108
+ Pass `include_metrics: true` to `search_templates` (or `search_workloads`) for the dashboard's cost and performance numbers, over a rolling `days_back` window or an explicit `since:` / `until:` one. `search_logs` carries each log's `cost` and can sort and filter by it.
109
+
110
+ ```ruby
111
+ window = { since: Time.utc(2026, 9, 1), until: Time.utc(2026, 10, 1) }
112
+
113
+ Coolhand.workload_service.search_workloads(include_metrics: true, **window).workloads
114
+ .map { |w| [w[:name], w[:metrics][:total_cost]] }
115
+
116
+ Coolhand.log_service.search_logs(order: 'cost_desc', min_cost: 1.0, per: 10, **window).logs
117
+ ```
118
+
119
+ See [Reading Workloads →](docs/workload-search.md) and [Reading Logs →](docs/log-search.md).
120
+
121
+ ## Linking Feedback to an Optimization
122
+
123
+ Attach feedback to an optimization as evidence, one at a time or in bulk. These require your **private** API key and **raise** `Coolhand::Error` on failure (`Coolhand::HttpError` for non-2xx responses).
124
+
125
+ ```ruby
126
+ links = Coolhand.optimization_feedback_link_service
127
+
128
+ link = links.link_feedback('optimizationHashid', 'feedbackHashid', note: 'why')
129
+ links.bulk_link_feedback('optimizationHashid', %w[fb1 fb2 fb3]) # => { linked:, already_linked:, errored:, not_found: }
130
+ links.unlink_feedback('optimizationHashid', link[:id])
131
+ ```
132
+
133
+ For batching, error semantics, and details, see [Linking Feedback →](docs/feedback-links.md).
134
+
106
135
  ## Rails Integration
107
136
 
108
137
  ### Configuration
@@ -120,8 +149,8 @@ Coolhand.configure do |config|
120
149
  config.silent = Rails.env.production?
121
150
 
122
151
  # Specify which LLM endpoints to intercept (array of strings)
123
- # Optional - defaults to OpenAI, Anthropic, ElevenLabs, Google Gemini, and GitHub Models
124
- # config.intercept_addresses = ["api.openai.com", "api.anthropic.com", "api.cohere.ai"]
152
+ # Optional - defaults to every provider listed under "Supported Libraries" below
153
+ # config.intercept_addresses = ["api.openai.com", "api.anthropic.com", "api.cohere.com/v2/chat"]
125
154
  end
126
155
  ```
127
156
 
@@ -178,10 +207,11 @@ end
178
207
  | `enabled` | Boolean | `true` | Set to `false` to disable all patching and validation (e.g. `Rails.env.production?`) |
179
208
  | `capture` | Boolean | `true` | Whether to capture and forward intercepted requests. Set to `false` to monitor without forwarding, then use [`Coolhand.with_capture`](#selective-capture) to re-enable selectively |
180
209
  | `silent` | Boolean | `false` | Whether to suppress console output |
181
- | `intercept_addresses` | Array | `["api.openai.com", "api.anthropic.com"]` | Array of API hosts to monitor (matched by host, not full URL). This is a required allow-list — `[]` is ignored (a warning is logged) rather than disabling capture; use `enabled` or `capture` for that. See [Configuration](docs/configuration.md) |
210
+ | `intercept_addresses` | Array | All [supported providers](#supported-libraries) (`DEFAULT_INTERCEPT_ADDRESSES`) | Array of API hosts to monitor (matched by host, optionally pinned to a port and/or path prefix). This is a required allow-list — `[]` is ignored (a warning is logged) rather than disabling capture; use `enabled` or `capture` for that. See [Configuration](docs/configuration.md) |
182
211
  | `intercept_path_patterns` | Array | `[":generateContent", ":streamGenerateContent"]` | Path patterns to additionally monitor on Google API hosts — see [Configuration](docs/configuration.md) |
183
212
  | `exclude_api_patterns` | Array | `["/batchPredictionJobs/"]` | Deny-list checked after `intercept_addresses`; matching paths are skipped. Unlike `intercept_addresses`, `exclude_api_patterns = []` genuinely disables exclusion. See [Configuration](docs/configuration.md) |
184
213
  | `max_captured_body_bytes` | Integer | `1_000_000` | Maximum size of a captured JSON request body — oversized bodies are replaced with a placeholder. Non-JSON bodies (e.g. file/audio uploads) are always skipped regardless of size — see [Advanced Configuration](docs/configuration.md) |
214
+ | `async_logging` | Boolean | `true` | Send logs to Coolhand from a background thread so a slow Coolhand API never delays your LLM calls. Set to `false` to send inline — see [Advanced Configuration](docs/configuration.md#asynchronous-logging) |
185
215
 
186
216
  ## Usage Examples
187
217
 
@@ -337,10 +367,18 @@ The monitor works with multiple transport layers and Ruby libraries:
337
367
  **Other providers monitored out of the box:**
338
368
  - Google Gemini (`generativelanguage.googleapis.com`)
339
369
  - Google Vertex AI (`aiplatform.googleapis.com`) — see the [Vertex AI batch result logging guide](docs/vertex.md) for async batch jobs
370
+ - Azure OpenAI Service, Azure AI Foundry, and Azure Machine Learning managed online endpoints (dedicated hosts, multi-service Azure AI Services hosts, and serverless/MaaS deployments) — see [Configuration](docs/configuration.md) for the full list of covered hosts
340
371
  - AWS Bedrock (OpenAI-compatible endpoint)
341
372
  - Cloudflare AI Gateway
342
373
  - OpenRouter
343
374
  - OpenCode Zen (`opencode.ai`)
375
+ - DeepSeek (`api.deepseek.com`)
376
+ - Mistral (`api.mistral.ai`)
377
+ - Perplexity (`api.perplexity.ai`)
378
+ - xAI (`api.x.ai`)
379
+ - Cohere (chat and embed endpoints only, on `api.cohere.com` and `api.cohere.ai`)
380
+ - TypeSafe Jev / System One (`api.typesafe.ai/v1/systemone`)
381
+ - Ollama (self-hosted, default port 11434 — chat, generate and embed endpoints) — see [Configuration](docs/configuration.md) for non-default hosts
344
382
 
345
383
  **Universal Coverage**: Since most Ruby HTTP libraries use Net::HTTP under the hood, Coolhand's single interceptor provides comprehensive monitoring without needing library-specific integrations.
346
384
 
@@ -426,13 +464,16 @@ The monitor handles errors gracefully:
426
464
  - Failed API logging attempts are logged to console but don't interrupt your application
427
465
  - Invalid API keys will be reported but won't crash your app
428
466
  - Network issues are handled with appropriate error messages
429
- - The read methods (`search_templates`, `get_template`) are the exception — they raise `Coolhand::HttpError`; see [Reading Templates →](docs/template-search.md)
467
+ - The read methods (`search_templates`, `get_template`, `search_workloads`, `search_logs`, `get_log`) and the feedback-link methods are the exception — they raise `Coolhand::Error` (`Coolhand::HttpError` for non-2xx responses); see [Reading Templates →](docs/template-search.md), [Reading Workloads →](docs/workload-search.md), [Reading Logs →](docs/log-search.md) and [Linking Feedback →](docs/feedback-links.md)
430
468
 
431
469
  ## Documentation
432
470
 
433
471
  - **[Configuration](docs/configuration.md)** — Self-hosted deployments, base_url rules, debug mode, custom intercept addresses
434
472
  - **[Feedback API](docs/feedback.md)** — Full field reference, matching strategies, sentiment values
435
473
  - **[Reading Templates](docs/template-search.md)** — Search LLM request templates and fetch a single one, prompt patterns included, using the private API key
474
+ - **[Reading Workloads](docs/workload-search.md)** — List workloads with optional cost and performance metrics over a rolling or explicit time window
475
+ - **[Reading Logs](docs/log-search.md)** — Search logs with per-log cost, filter and sort by cost, and fetch one log with its cost breakdown
476
+ - **[Linking Feedback](docs/feedback-links.md)** — Link feedback to an optimization as evidence, singly or in bulk, using the private API key
436
477
  - **[Anthropic Integration](docs/anthropic.md)** — Official and community Anthropic Ruby gems, streaming, dual gem handling, and troubleshooting
437
478
  - **[ElevenLabs Integration](docs/elevenlabs.md)** — Webhook capture, feedback submission, and Rails integration
438
479
  - **[OpenAI Batch Webhook Handler](docs/openai.md)** — Handle OpenAI batch job completion events via webhook interception
@@ -39,6 +39,25 @@ end
39
39
 
40
40
  Because it forces capture unconditionally and prints full request/response bodies to the console, only enable `debug_mode` in development — never in production or in an environment handling real user data.
41
41
 
42
+ ## Asynchronous Logging
43
+
44
+ By default Coolhand ships each captured request to its API from background worker threads (one for per-request logs, a separate one for batch jobs), so a slow or unreachable Coolhand backend never adds latency to your own LLM calls. The request-path work is limited to capturing and sanitizing the payload.
45
+
46
+ ```ruby
47
+ Coolhand.configure do |config|
48
+ config.async_logging = false # default: true — send inline, in the intercepted call's request path
49
+ end
50
+ ```
51
+
52
+ **How it behaves:**
53
+ - The queue is bounded at 1,000 pending logs. If Coolhand is down long enough to fill it, the *oldest* pending log is dropped (with a log line) rather than blocking your app or growing memory.
54
+ - The bound is on the *number* of pending logs, not their size: each holds its captured response body in memory until sent, so a long Coolhand outage under heavy traffic with large responses can hold a lot of memory. Batch jobs use their own lane, so a long-running batch never delays or evicts per-request logs.
55
+ - A forked process (Puma/Unicorn workers, Spring) starts its own worker automatically.
56
+ - Pending logs are flushed at process exit (up to 5 seconds in total, so a batch job still running at exit may be only partly logged). In a short-lived script, or to wait explicitly, call `Coolhand.flush(timeout: 5)` — it returns `true` once everything has been sent.
57
+ - OpenAI and Vertex batch results are processed as a single background job per batch.
58
+ - `config.debug_mode` always runs inline so printed payloads appear in order.
59
+ - Logs queued but not yet sent are lost if the process is killed hard (`kill -9`, OOM). Set `async_logging = false` if you need delivery before each call returns.
60
+
42
61
  ## Request Body Capture Limits
43
62
 
44
63
  To avoid holding large uploads (batch JSONL files, fine-tune corpora, audio for transcription) in memory and shipping them to Coolhand in full, `config.max_captured_body_bytes` caps how much of a request body gets captured:
@@ -74,7 +93,13 @@ Setting `intercept_addresses` **replaces** the default list entirely, so include
74
93
 
75
94
  The default list can be found in `Coolhand::Configuration::DEFAULT_INTERCEPT_ADDRESSES`.
76
95
 
77
- Each entry is matched against the request's parsed host — an exact match, or a dot-boundary suffix match (`my-llm-proxy.internal` also matches `eu.my-llm-proxy.internal`) — case-insensitively. A single `*` in an entry matches exactly one host label, e.g. `bedrock-runtime.*.amazonaws.com`. Entries are never matched against the full URL, so query strings or paths that merely mention a host can't trigger a false match.
96
+ Each entry is matched against the request's parsed host (optionally with a port and path prefix, see below) — an exact match, or a dot-boundary suffix match (`my-llm-proxy.internal` also matches `eu.my-llm-proxy.internal`) — case-insensitively. A single `*` in an entry matches exactly one host label, e.g. `bedrock-runtime.*.amazonaws.com`. Entries are never matched against the full URL, so query strings that merely mention a host can't trigger a false match.
97
+
98
+ An entry can also pin a port (`host:port`) and/or a path (`host/path`, or `host:port/path`), and then a request must match all of them. Cohere, TypeSafe and Ollama are defaulted this way: `api.cohere.com/v2/chat`, `api.typesafe.ai/v1/systemone`, and `*:11434/api/chat` (plus `/api/generate`, `/api/embed`, `/api/embeddings`, also for `127.0.0.1`). Ollama has no fixed host, so its defaults require Ollama's default port *and* an Ollama path — an app's own unrelated `/api/chat` on another host or port is not captured. To capture an Ollama server on a dotted host or IP, or a non-default port, add entries such as `192.168.1.5:11434/api/chat` or `ollama.internal:8080/api/chat`. An IPv6 loopback server (`[::1]:11434`) is already covered by the defaults.
99
+
100
+ Path-anchoring is also how the Azure defaults work: multi-service hosts like `cognitiveservices.azure.com/openai/` (Azure AI Services, where Speech, Vision, Language and Content Safety share a domain with LLM inference) only match paths under the given prefix, after the host itself has matched.
101
+
102
+ **Note on Azure ML managed online endpoints:** `inference.ml.azure.com` and `inference.ml.azure.us` are intentionally captured *without* a path anchor. Every Azure ML managed online endpoint scores at `/score` regardless of what model sits behind it, so there's no path that distinguishes an LLM deployment from a tabular or vision one — this gem chooses to capture broadly rather than silently miss LLM deployments. If that over-captures for your setup, override `intercept_addresses` to a custom list omitting these two entries.
78
103
 
79
104
  A small number of Google API endpoints use a colon-suffixed path (e.g. `.../gemini-pro:generateContent`) with no host of their own. Those live in the separate `intercept_path_patterns` setting (default: `Coolhand::Configuration::DEFAULT_INTERCEPT_PATH_PATTERNS`) and, unlike `intercept_addresses`, are only checked against the path of requests whose host is already `googleapis.com` or a subdomain of it — and only when `intercept_addresses` still includes at least one `googleapis.com` host. If you override `intercept_addresses` to a list with no Google hosts, `intercept_path_patterns` has no effect and Google API traffic is not captured at all.
80
105
 
@@ -82,7 +107,7 @@ A small number of Google API endpoints use a colon-suffixed path (e.g. `.../gemi
82
107
 
83
108
  ## Excluding API Patterns
84
109
 
85
- `config.exclude_api_patterns` is a deny-list checked *after* `intercept_addresses` allows a request through — a match against the request's path is skipped and never forwarded as an `llm_request_log`. It defaults to `["/batchPredictionJobs/"]`, to suppress Vertex AI batch job management noise.
110
+ `config.exclude_api_patterns` is a deny-list checked *after* `intercept_addresses` allows a request through — a match against the request's path is skipped and never forwarded as an `llm_request_log`. It defaults to a small built-in list (see `Coolhand::Configuration::DEFAULT_EXCLUDE_API_PATTERNS`) that suppresses Vertex AI batch job management noise (`/batchPredictionJobs/`) and Azure OpenAI control-plane paths (file, batch, and fine-tuning management under `/openai/` and `/openai/v1/`) that aren't inference traffic.
86
111
 
87
112
  ```ruby
88
113
  Coolhand.configure do |config|
@@ -0,0 +1,48 @@
1
+ # Linking feedback to an optimization
2
+
3
+ Attach feedback to an optimization as supporting evidence. `Coolhand::OptimizationFeedbackLinkService`
4
+ wraps `POST /api/v2/optimizations/{optimization_id}/feedback_links` (single and bulk share the route,
5
+ chosen by the body) and `DELETE /api/v2/optimizations/{optimization_id}/feedback_links/{id}`.
6
+
7
+ These methods require your **private** API key; the public key gets a `401`.
8
+
9
+ ```ruby
10
+ require "coolhand"
11
+
12
+ Coolhand.configure do |config|
13
+ config.api_key = ENV.fetch("COOLHAND_PRIVATE_API_KEY")
14
+ end
15
+
16
+ links = Coolhand.optimization_feedback_link_service
17
+
18
+ # One feedback. link[:id] is the link's hashid, not the feedback's.
19
+ link = links.link_feedback("optimizationHashid", "feedbackHashid", note: "why")
20
+
21
+ # Many feedbacks.
22
+ result = links.bulk_link_feedback("optimizationHashid", %w[fb1 fb2 fb3])
23
+ result # => { linked: 2, already_linked: 1, errored: 0, not_found: [] }
24
+
25
+ # Remove a link.
26
+ links.unlink_feedback("optimizationHashid", link[:id])
27
+ ```
28
+
29
+ ## Bulk behavior
30
+
31
+ - The server accepts at most 100 ids per request. `bulk_link_feedback` splits longer lists into
32
+ batches of 100, sums `linked` / `already_linked` / `errored` and concatenates `not_found`.
33
+ - Already-linked ids count as `already_linked`, not errors. Unknown, malformed and other-client ids
34
+ are all reported in `not_found`.
35
+ - If a batch fails, the call raises and earlier batches stay applied. Repeating the call is safe.
36
+ - `link_feedback` is not retry-safe: after a transport error, a `422` "already linked" on retry may mean
37
+ the first attempt succeeded. Use `bulk_link_feedback` with a single id for retry-safe semantics.
38
+ - An empty list or a blank id raises `Coolhand::Error` before any request is made.
39
+ - Ids are trimmed of surrounding whitespace but not de-duplicated client-side.
40
+
41
+ ## Errors
42
+
43
+ Unlike the logging writes, all three methods raise. A non-2xx response raises `Coolhand::HttpError`
44
+ whose `status` is the HTTP code: `401` missing or public key, `404` unknown optimization, feedback or
45
+ link, `422` invalid input, an already-linked feedback (single mode) or a `note` that is too long.
46
+ Client-side validation and transport failures raise `Coolhand::Error`.
47
+
48
+ `config.debug_mode` does not suppress these calls: they change server state, so they are always sent.
@@ -0,0 +1,102 @@
1
+ # Reading Logs (Search + Get)
2
+
3
+ `Coolhand::LogService` reads logs back out of Coolhand, with the per-log `cost` the dashboard shows.
4
+ The write side, `Coolhand.logger_service`, is a different class. It wraps two read-only endpoints:
5
+
6
+ | method | endpoint |
7
+ |---|---|
8
+ | `search_logs` | `GET /api/v2/llm_request_logs` |
9
+ | `get_log` | `GET /api/v2/llm_request_logs/{id}` |
10
+
11
+ Both require your **private** API key. The public key is write-only on this API and is rejected exactly
12
+ like an invalid one.
13
+
14
+ ```ruby
15
+ require "coolhand"
16
+
17
+ Coolhand.configure do |config|
18
+ config.api_key = ENV.fetch("COOLHAND_PRIVATE_API_KEY")
19
+ end
20
+
21
+ logs = Coolhand.log_service
22
+
23
+ # The ten most expensive logs of the last 30 days
24
+ result = logs.search_logs(since: Time.now.utc - (30 * 86_400), order: "cost_desc", per: 10)
25
+ result.logs.each { |log| puts "#{log[:id]} #{log[:model]} $#{log[:cost]}" }
26
+
27
+ detail = logs.get_log(result.logs.first[:id])
28
+ detail[:cost_breakdown][:input_cost]
29
+ ```
30
+
31
+ ## `search_logs`
32
+
33
+ All keywords are optional, and named after the wire parameters (`sort` maps to `q[s]`). Two wire
34
+ parameters are not exposed: the `q[source_api_in][]` array filter, so use the single-value `source_api`
35
+ keyword, and the `per_page` alias, since this gem only sends `per`.
36
+
37
+ | keyword | type | notes |
38
+ |---|---|---|
39
+ | `template_id`, `workload_id` | String | Hashids. One that does not decode, or that belongs to another client, is a `422`. |
40
+ | `system_prompt_contains`, `user_prompt_contains` | String | Case-insensitive substring. |
41
+ | `model`, `source_api`, `source_api_result`, `source_application` | String | Exact match. `source_api_result` is `success`, `failed`, `operational` or `unmatched`. |
42
+ | `project_path` | String | Exact match against `metadata.project_path`. |
43
+ | `unmatched_only` | Boolean | Only logs with no template. |
44
+ | `days_back` | Integer | Last N days. Unset means unrestricted. Ignored when `since` is given; with only `until`, the window is `days_back` long ending at `until`. |
45
+ | `since`, `until` | Time, DateTime, Date or String | Bounds on `created_at`: `since` inclusive, `until` exclusive. See [Windows](workload-search.md#windows-since-and-until). |
46
+ | `min_cost` | Numeric | Only logs whose `cost` (USD) is at least this. `0` is sent. |
47
+ | `order` | String | `"cost_desc"` sorts by `cost`, highest first, replacing `sort`. Any other value is a `422` from the server. |
48
+ | `include_prompts` | Boolean | Adds `:system_prompt` and `:user_prompt`, each truncated to 500 characters. |
49
+ | `include_total` | Boolean | Asks the server for the `X-Total-*` headers. Off by default there (it skips a `COUNT(*)`). |
50
+ | `sort` | String | Ransack sort expression, sent as `q[s]`, e.g. `"created_at desc"`. |
51
+ | `page`, `per` | Integer | 1-based; `per` defaults to 25, max 100. |
52
+
53
+ `min_cost` and `order: "cost_desc"` only consider logs that can be priced (non-nil `cost`). The cost is
54
+ computed per log, so on a client with a lot of history combine them with `since` or `days_back`, or the
55
+ query can time out (`504`).
56
+
57
+ ### Return value
58
+
59
+ A `Coolhand::LogSearchResult`, a `Struct` with `#logs` and `#pagination`. The response body is a bare
60
+ array, never wrapped. Rows are Symbol-keyed Hashes: `:id`, `:collector`, `:source_api`,
61
+ `:source_application`, `:metadata`, `:source_api_result`, `:model`, `:template_id`, `:template_name`,
62
+ `:input_tokens`, `:output_tokens`, `:latency_ms`, `:created_at`, `:updated_at`, `:ingest_evidence` and
63
+ `:cost`.
64
+
65
+ `:cost` is a Float in USD, or `nil` when the log has no tokens or its model has no pricing.
66
+
67
+ **Pagination without totals.** `X-Page` and `X-Per-Page` are always sent, but `X-Total-Count` and
68
+ `X-Total-Pages` only come with `include_total: true`. Without it, `pagination.total_count` and
69
+ `total_pages` are lower bounds the gem derives, not counts, and `has_next_page` is true when the page
70
+ came back full. Pass `include_total: true` when you need real totals.
71
+
72
+ ## `get_log(id, section: nil, max_chars: nil, search_query: nil, include_thinking: nil)`
73
+
74
+ `id` is the log hashid. Returns the full log as a Symbol-keyed Hash: the list fields plus the prompts and
75
+ `:output`, and:
76
+
77
+ | field | type |
78
+ |---|---|
79
+ | `:url` | String: the log's dashboard path (`/c/<client>/llm_request_logs/<id>`) |
80
+ | `:cost` | Float or nil |
81
+ | `:cost_breakdown` | Hash or nil: `:total_cost`, `:input_cost`, `:output_cost`, `:cached_input_cost`, `:cache_creation_input_cost`, `:reasoning_output_cost` |
82
+
83
+ `section` (`"full"`, `"beginning"`, `"end"`), `max_chars` and `search_query` bound or search the content;
84
+ `include_thinking` adds `:thinking_response`. Truncation and search are reported back as `:truncated`,
85
+ `:total_chars`, `:search_query` and `:matches`. Only directly-collected logs are fetchable; internally
86
+ generated records are a `404`.
87
+
88
+ ## Errors
89
+
90
+ The read methods raise where the write methods log and return `nil`. `Coolhand::HttpError` carries
91
+ `#status` and `#body`.
92
+
93
+ | status | when |
94
+ |---|---|
95
+ | `401` | No API key, an invalid key, or the public key. |
96
+ | `404` | `get_log` only: unknown id, another client's log, or an internally generated record. |
97
+ | `422` | Bad `template_id`, `workload_id`, `days_back`, `since`, `until`, `min_cost`, `order` or `max_chars`. |
98
+ | `504` | The search exceeded the statement timeout. Narrow the window and retry. |
99
+
100
+ `Coolhand::Error` covers no API key configured, a transport failure, a body that is not valid JSON, a
101
+ non-array list body, and a blank `id` passed to `get_log`. An unsupported `since`/`until` type raises
102
+ `ArgumentError` before any request.
data/docs/openai.md CHANGED
@@ -14,7 +14,7 @@ Requires Rails — `Coolhand::WebhookInterceptor` and `Coolhand::OpenAi::BatchRe
14
14
  - Ensure you skip CSRF for the webhook endpoint: `skip_before_action :verify_authenticity_token`
15
15
  - Override the `webhook_secret` method to return your OpenAI webhook secret
16
16
 
17
- `intercept_batch_request` already validates the webhook and, for `batch.completed`/`batch.failed`/`batch.expired`/`batch.cancelled` events, calls `Coolhand::OpenAi::BatchResultProcessor` **synchronously**, inline in the `before_action` — that part requires no code in your controller action. It downloads two JSONL files from OpenAI and sends one request log per batch item, so a large batch can take long enough to trip your webhook endpoint's timeout (and OpenAI's retry behavior); consider that when sizing batches. The example below shows a controller action doing additional, app-specific work on top of that (e.g. updating your own `BatchApiRequest` record and enqueuing your own background job), which is optional.
17
+ `intercept_batch_request` already validates the webhook and, for `batch.completed`/`batch.failed`/`batch.expired`/`batch.cancelled` events, calls `Coolhand::OpenAi::BatchResultProcessor` — that part requires no code in your controller action. The batch's status is checked inline in the `before_action`, but for a completed batch the two JSONL file downloads and the one-request-log-per-item sends run on Coolhand's [background log thread](configuration.md#asynchronous-logging), so a large batch doesn't hold up your webhook response. (With `config.async_logging = false` this all runs inline, and a large batch can trip your webhook endpoint's timeout and OpenAI's retry behavior.) The example below shows a controller action doing additional, app-specific work on top of that (e.g. updating your own `BatchApiRequest` record and enqueuing your own background job), which is optional.
18
18
 
19
19
  ## Minimal example
20
20
 
@@ -41,7 +41,7 @@ This is **not** a port of the `search_templates` MCP tool, and the two do not ag
41
41
  Template *creation, update and deprecation* stay on the MCP surface. This REST surface is
42
42
  read-only, and there is no version-history sub-resource.
43
43
 
44
- ## `search_templates(search: nil, workload_id: nil, status: nil, include_deprecated: nil, include_system: nil, page: nil, per: nil)`
44
+ ## `search_templates(search: nil, workload_id: nil, status: nil, include_deprecated: nil, include_system: nil, include_metrics: nil, days_back: nil, since: nil, until: nil, page: nil, per: nil)`
45
45
 
46
46
  Search is a *parameter* on the list endpoint rather than a route of its own, so this is one method
47
47
  rather than a list/search pair.
@@ -56,6 +56,10 @@ and this API's already agree, so there is no second spelling to translate throug
56
56
  | `status` | String | `"draft"`, `"published"` or `"failure"`. Any other non-empty value is a `422` from the server; empty is treated as no filter. |
57
57
  | `include_deprecated` | Boolean | Include templates with a non-null `deprecated_at`. Defaults to `false` server-side. |
58
58
  | `include_system` | Boolean | Include the `Unmatched` / `Ignored API Calls` buckets. Defaults to `false` server-side. |
59
+ | `include_metrics` | Boolean | Adds a `:metrics` Hash to every row. See [Metrics](#metrics). |
60
+ | `days_back` | Integer | Rolling metrics window in days (server default 28, max 365). Ignored unless metrics are requested, and when `since` is given. |
61
+ | `since` | Time, DateTime, Date or String | Metrics window start, inclusive. Overrides `days_back`. |
62
+ | `until` | Time, DateTime, Date or String | Metrics window end, exclusive. Defaults to now. |
59
63
  | `page` | Integer | 1-based. |
60
64
  | `per` | Integer | Default 25, max 100, both enforced server-side. |
61
65
 
@@ -138,11 +142,14 @@ unmatched = Coolhand.template_service
138
142
  puts unmatched[:log_count]
139
143
  ```
140
144
 
141
- ## `get_template(id)`
145
+ ## `get_template(id, include_metrics: nil, days_back: nil, since: nil, until: nil)`
142
146
 
143
147
  `id` is the template hashid — the `:id` field from `search_templates`.
144
148
 
145
- Returns a Hash with every list field above, **plus** the full untruncated regexes the list omits:
149
+ Returns a Hash with every list field above, **plus** the full untruncated regexes the list omits, and
150
+ a `:metrics` Hash. Unlike the list, metrics are **on by default** here: pass `include_metrics: false` to
151
+ omit them, which also skips the window validation. `days_back`, `since` and `until` set the window
152
+ exactly as they do on the list, and an invalid one is a `422` unless metrics are off.
146
153
 
147
154
  | field | type |
148
155
  |---|---|
@@ -153,6 +160,41 @@ Unlike the list, this filters on nothing but client ownership: a deprecated or s
153
160
  reachable by id with **no opt-in flag**, because inspecting one of those is the usual reason to
154
161
  fetch a template directly.
155
162
 
163
+ ## Metrics
164
+
165
+ With `include_metrics: true` every row (and by default the `get_template` result) carries `:metrics`,
166
+ computed by the same SQL as the dashboard, so tiered pricing, cached-token discounts and reasoning tokens
167
+ are applied. `Coolhand.workload_service` returns the same object per workload, rolled up across its
168
+ templates ([Reading Workloads](workload-search.md)).
169
+
170
+ Averages and counters cover non-failed, directly-collected client logs inside the window. Keys are the
171
+ wire names, as Symbols.
172
+
173
+ | field | type | notes |
174
+ |---|---|---|
175
+ | `:days_back` | Integer or nil | `nil` when `since` defined the window. |
176
+ | `:since`, `:until` | String | The resolved window, ISO-8601 UTC. |
177
+ | `:request_count` | Integer | |
178
+ | `:failure_count` | Integer | Failed logs. |
179
+ | `:error_rate` | Float or nil | `nil` when `failure_count` is 0. |
180
+ | `:error_rate_change` | Float or nil | Against the window of the same length immediately before. |
181
+ | `:avg_cost_per_request` | Float or nil | `nil` when no log in the window could be priced. |
182
+ | `:total_cost` | Float or nil | |
183
+ | `:priced_request_count` | Integer | The logs `total_cost` covers: non-failed, with tokens, model has pricing. |
184
+ | `:long_context_request_count` | Integer | Priced logs that crossed their model's input-token pricing tier. |
185
+ | `:total_input_tokens`, `:total_output_tokens` | Integer | Raw token columns summed over non-failed logs, not cache-adjusted. |
186
+ | `:avg_input_tokens`, `:avg_output_tokens` | Integer or nil | |
187
+ | `:avg_latency_ms` | Float or nil | |
188
+ | `:correctness_score`, `:sentiment_score`, `:revision_score` | Float or nil | `sentiment_score` and `revision_score` are all-time, not windowed. |
189
+ | `:first_request_at`, `:last_request_at` | String or nil | Lifetime, not windowed. |
190
+
191
+ Summing the per-log `cost` from [Reading Logs](log-search.md) can exceed `total_cost`, because the log
192
+ `cost` is also computed for failed logs and logs with a zero token count.
193
+
194
+ Windows follow the rules in [Reading Workloads](workload-search.md#windows-since-and-until). A window
195
+ error is a `422` on the `since` or `until` key. Metrics add aggregate statements, so expect a slower
196
+ response, and a `504` on a large page.
197
+
156
198
  ## Errors
157
199
 
158
200
  The write methods in this gem (`create_feedback`, `create_log`, `send_llm_request_log`) log a
@@ -186,7 +228,7 @@ end
186
228
  |---|---|
187
229
  | `401` | Missing, invalid, or public API key. |
188
230
  | `404` | `get_template` only. Unknown id, **or** one belonging to another client — existence is not disclosed, so this is never a `403`. |
189
- | `422` | Unrecognised `status`, or a `workload_id` that does not decode or belongs to another client. |
231
+ | `422` | Unrecognised `status`, a `workload_id` that does not decode or belongs to another client, or a bad `days_back`, `since` or `until` when metrics are requested. |
190
232
  | `504` | See below. |
191
233
 
192
234
  ### `504` is expected, and retryable
data/docs/vertex.md CHANGED
@@ -6,6 +6,8 @@ For monitoring regular (non-batch) Vertex AI calls, no extra setup is required b
6
6
 
7
7
  Requires Rails — `Coolhand::Vertex::BatchResultProcessor` logs via `Rails.logger` internally. `config.capture = false` and `Coolhand.without_capture` do not suppress these logs: unlike the passive Net::HTTP interceptor, calling this processor is an explicit, deliberate act, so it always sends.
8
8
 
9
+ > **Note:** As of May 2026, Google markets this product as "Gemini Enterprise Agent Platform" rather than "Vertex AI." The API endpoint (`aiplatform.googleapis.com`) and Ruby SDK are unchanged — it's a console/marketing rename, not an SDK rename — so `Coolhand::Vertex` naming stays accurate here. It's also still a distinct integration from the separate Gemini Developer API (`generativelanguage.googleapis.com`), which Coolhand intercepts independently.
10
+
9
11
  ## Usage
10
12
 
11
13
  ```ruby
@@ -13,6 +15,7 @@ require "coolhand/vertex/batch_result_processor"
13
15
  ```
14
16
 
15
17
  - Call the `Coolhand::Vertex::BatchResultProcessor` service with `batch_info` and the downloaded batch results.
18
+ - For a succeeded job, the per-item request logs are sent from Coolhand's [background log thread](configuration.md#asynchronous-logging) as a single job, so `call` returns without waiting on the Coolhand API (set `config.async_logging = false` to send inline).
16
19
  - Optionally pass `model:` to populate the logged `model` field. It falls back to a `"model"` key in `batch_info` if present, and is omitted from the payload entirely when neither is available. If the value (from either source) looks like a Vertex model resource path (`publishers/<publisher>/models/<id>` or `projects/<project>/locations/<location>/models/<id>`), it's normalized down to just the trailing id/version before being logged; any other string is sent as-is.
17
20
  - `batch_info["name"]` must be the exact Vertex job resource name (`projects/<project>/locations/<location>/batchPredictionJobs/<job-id>`), and `batch_info["startTime"]`/`["endTime"]` must be valid ISO 8601 timestamps — a batch with a missing or malformed resource name or timestamps is skipped entirely (logged as an error) rather than sent with a broken URL.
18
21
  - If `endTime` precedes `startTime` (e.g. clock skew), the batch is still logged — `duration_ms` is clamped to `0` and a warning is logged — rather than dropping the results.