coolhand 0.10.1 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +21 -1
- data/README.md +19 -4
- data/docs/configuration.md +15 -2
- data/docs/log-search.md +54 -7
- data/docs/step-templates.md +85 -0
- data/lib/coolhand/api_service.rb +2 -1
- data/lib/coolhand/base_interceptor.rb +2 -1
- data/lib/coolhand/default_exclude_api_patterns.yml +2 -1
- data/lib/coolhand/default_intercept_addresses.yml +2 -1
- data/lib/coolhand/log_service.rb +16 -4
- data/lib/coolhand/net_http_interceptor.rb +28 -5
- data/lib/coolhand/read_query.rb +11 -4
- data/lib/coolhand/step_template_service.rb +45 -0
- data/lib/coolhand/stream_read.rb +44 -0
- data/lib/coolhand/version.rb +1 -1
- data/lib/coolhand.rb +7 -0
- metadata +5 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 0643c994a7f5032e47c8f5abd9276dafdb6b1dfe74fd9ec0c0b82c4a62fe59a2
|
|
4
|
+
data.tar.gz: f1dc8be18a069e2ded6808153db13480df36e11ba03fbbb94dd220531baa055e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 8880da761ef592e427b577676d1699059bbb5705a4e0f7ffe6086cce5d7a7b0fecd3f7f90bba0b4a476f8d75d8938a33a1fe868ade248b4b70a114a8a4f1b3bf
|
|
7
|
+
data.tar.gz: aafbecd72485fecb263599ad2370094d0818d6abef830bceaaceb3005cbb165c40f6077b7545ecfc6cb58889bad420f4bd6f5158463efcf8daa1b5d6674897c3
|
data/CHANGELOG.md
CHANGED
|
@@ -7,10 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.11.0] - 2026-11-06
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- **Abandoned streamed responses are now flagged** ([#205](https://github.com/Coolhand-Labs/coolhand-ruby/pull/205)). A streamed response your code stops reading before it ends (the `read_body` block `break`s, `throw`s or raises, or the connection drops mid-body) is logged with the partial body and `raw_request.stream_incomplete: true`; a response read to its end is sent without the field. Coolhand's backend then nulls latency, and tokens unless the provider's final usage event is in the body. Nothing is marked when no part of the body had arrived yet. See [Abandoned Streams](docs/configuration.md).
|
|
14
|
+
- `Coolhand.log_service.failure_summary(days_back: nil)` ([#204](https://github.com/Coolhand-Labs/coolhand-ruby/pull/204)) rolls failed requests up by error pattern and by template (`GET /api/v2/llm_request_logs/failure_summary`). It returns one Symbol-keyed Hash, not a paginated result; `days_back` is sent as given, so zero or a negative value reaches the server's 422 and raises `Coolhand::HttpError`. `search_logs(error_contains:)` filters by a case-insensitive substring of the failed request's error text, so a pattern's `:error_pattern` lists the logs behind it. See [Reading Logs](docs/log-search.md).
|
|
15
|
+
- Step template reads ([#202](https://github.com/Coolhand-Labs/coolhand-ruby/pull/202)). `Coolhand.step_template_service` has `list_step_templates(search:, step_type:, include_metrics:, page:, per:)` (returns a `StepTemplateSearchResult` with pagination) and `get_step_template(id, include_metrics:)` for the skills and recurring steps Coolhand detects in captured agent sessions. `get_step_template` includes `metrics` by default, so pass `include_metrics: false` to leave them out. `get_log(id, include_turn_metrics: true)` adds a per-turn breakdown as `turn_metrics` (nil for a log that is not a captured agent session). See [Reading Step Templates](docs/step-templates.md).
|
|
16
|
+
- `examples/abandoned_stream_example.rb`: a smoke test for `stream_incomplete`. A local scenario with fake servers needs no keys; a live scenario abandons a real OpenAI and/or Anthropic stream. It runs under `bin/live-examples`.
|
|
17
|
+
|
|
18
|
+
### Changed
|
|
19
|
+
- **Behavior change:** when your streaming block raises mid-stream after part of the body arrived, the log now holds the partial body, `stream_incomplete: true` and the response's status code ([#205](https://github.com/Coolhand-Labs/coolhand-ruby/pull/205)). The host exception's class and message are no longer recorded for those cases, and the status is never derived from the error. When nothing had been buffered yet, the error record is unchanged and the log is not marked. **Migration note:** if you read the error class out of logs for a stream your own block aborted, look for `stream_incomplete: true` instead.
|
|
20
|
+
- The POST that ships a log to Coolhand now also has a 5 second `write_timeout`, matching its existing 5 second open and read timeouts, so a stalled upload cannot hang the caller (inline with `async_logging = false`, or the background log worker).
|
|
21
|
+
- Docs: the Amazon Bedrock host list in `docs/configuration.md` now names every covered runtime host and the shapes that are deliberately not covered (VPC interface endpoint DNS names, the aws-cn dual-stack shape, `bedrock-agent-runtime`), and notes that the `/openai/*` exclude patterns apply to Azure OpenAI Service and Amazon Bedrock alike ([#199](https://github.com/Coolhand-Labs/coolhand-ruby/pull/199)). The README Bedrock bullet now links there. A spec covers a commercial-region FIPS host.
|
|
22
|
+
- Docs: the `0.10.1` Bedrock FIPS note below no longer calls the FIPS hosts "the common case for GovCloud callers"; they exist in commercial regions too ([#199](https://github.com/Coolhand-Labs/coolhand-ruby/pull/199)).
|
|
23
|
+
- Docs: [Abandoned Streams](docs/configuration.md) now notes that a bare `break` out of a `read_body` block can leave Net::HTTP itself waiting out its read timeout when it lands partway through a chunk (plain Net::HTTP does the same without Coolhand), and that `throw` or `return` out of the whole request is the way to stop reading a live stream.
|
|
24
|
+
- The release skills (`/prep-release`, `/finish-release`) credit outside and first-time contributors in the changelog and release notes ([#207](https://github.com/Coolhand-Labs/coolhand-ruby/pull/207)). `/loop-review` no longer asks feature branches for a changelog entry, matching `AGENTS.md`.
|
|
25
|
+
|
|
26
|
+
### Dependencies
|
|
27
|
+
- `ruby/setup-ruby` bumped to 1.336.0 in CI and the publish workflow ([#206](https://github.com/Coolhand-Labs/coolhand-ruby/pull/206)).
|
|
28
|
+
- `addressable` bumped from 2.9.0 to 2.9.2 in `Gemfile.lock` ([#204](https://github.com/Coolhand-Labs/coolhand-ruby/pull/204)), which clears the `bundler-audit` advisory that was failing CI.
|
|
29
|
+
|
|
10
30
|
## [0.10.1] - 2026-10-26
|
|
11
31
|
|
|
12
32
|
### Added
|
|
13
|
-
- **Amazon Bedrock FIPS, dual-stack and China-partition endpoints are now captured** ([#168](https://github.com/Coolhand-Labs/coolhand-ruby/pull/168)). The default intercept list held only `bedrock-runtime.*.amazonaws.com`, so calls to the FIPS hosts (`bedrock-runtime-fips.<region>.amazonaws.com`,
|
|
33
|
+
- **Amazon Bedrock FIPS, dual-stack and China-partition endpoints are now captured** ([#168](https://github.com/Coolhand-Labs/coolhand-ruby/pull/168)). The default intercept list held only `bedrock-runtime.*.amazonaws.com`, so calls to the FIPS hosts (`bedrock-runtime-fips.<region>.amazonaws.com`, which exist in GovCloud and commercial regions), the dual-stack IPv6 hosts (`bedrock-runtime.<region>.api.aws` and the FIPS variant) and the China partition (`bedrock-runtime.<region>.amazonaws.com.cn`) were silently not captured. A wildcard entry is an exact host match with no implicit subdomains, so each variant needs its own entry.
|
|
14
34
|
- Credentials in request and response bodies are redacted ([#178](https://github.com/Coolhand-Labs/coolhand-ruby/pull/178)). String values under `authorization`, `authorization_token`, `access_token`, `refresh_token`, `id_token`, `api_key`, `client_secret`, `private_key`, `secret_key` and `password` (any depth, matched by exact name) become `[REDACTED]`, as do the string values in the `headers` of an MCP tool and the ephemeral key returned by a Realtime `client_secret` response. `response_body` is now redacted too, not just `request_body`. **Migration note:** logged bodies that carry such fields now show `[REDACTED]` in Coolhand. See [Body Redaction](docs/configuration.md).
|
|
15
35
|
- `max_captured_body_bytes` now also caps **response** bodies ([#177](https://github.com/Coolhand-Labs/coolhand-ruby/pull/177)). A response larger than the cap (default 1 MB) is logged as a `{"_coolhand_capture_skipped": "body_too_large", ...}` marker instead of in full; your app always receives the complete response. A `body_stream` request that declares a `Content-Length` over the cap is no longer read for capture, and a non-Integer or negative cap falls back to the default instead of raising. See [Body Capture Limits](docs/configuration.md).
|
|
16
36
|
- `WebhookIdStore#release!(id)` and `WebhookValidator#release_claim!` ([#176](https://github.com/Coolhand-Labs/coolhand-ruby/pull/176)). A custom `webhook_id_store` may optionally implement `release!(id)`; stores that only implement `claim!` keep working unchanged.
|
data/README.md
CHANGED
|
@@ -103,9 +103,23 @@ Search is a parameter on the list endpoint, not a route of its own, and the `Unm
|
|
|
103
103
|
|
|
104
104
|
For the full filter reference, pagination, and error handling, see [Reading Templates →](docs/template-search.md).
|
|
105
105
|
|
|
106
|
+
### Step templates
|
|
107
|
+
|
|
108
|
+
`Coolhand.step_template_service` reads the skills and other recurring steps Coolhand detects in your captured agent sessions, with the same private API key. `include_metrics: true` adds run, turn, tool and efficiency figures.
|
|
109
|
+
|
|
110
|
+
```ruby
|
|
111
|
+
steps = Coolhand.step_template_service
|
|
112
|
+
|
|
113
|
+
result = steps.list_step_templates(step_type: 'prompt_step', include_metrics: true)
|
|
114
|
+
step = steps.get_step_template(result.step_templates.first[:id])
|
|
115
|
+
step[:metrics][:notes]
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
For the parameters, `metrics` fields and errors, see [Reading Step Templates →](docs/step-templates.md).
|
|
119
|
+
|
|
106
120
|
### Cost and performance metrics
|
|
107
121
|
|
|
108
|
-
Pass `include_metrics: true` to `search_templates` (or `search_workloads`) for the dashboard's cost and performance numbers, over a rolling `days_back` window or an explicit `since:` / `until:` one. `search_logs` carries each log's `cost` and can sort and filter by it.
|
|
122
|
+
Pass `include_metrics: true` to `search_templates` (or `search_workloads`) for the dashboard's cost and performance numbers, over a rolling `days_back` window or an explicit `since:` / `until:` one. `search_logs` carries each log's `cost` and can sort and filter by it. `failure_summary` rolls failed requests up by error pattern and template, and `search_logs(error_contains:)` lists the logs behind a pattern.
|
|
109
123
|
|
|
110
124
|
```ruby
|
|
111
125
|
window = { since: Time.utc(2026, 9, 1), until: Time.utc(2026, 10, 1) }
|
|
@@ -411,7 +425,7 @@ The monitor works with multiple transport layers and Ruby libraries:
|
|
|
411
425
|
- Google Gemini (`generativelanguage.googleapis.com`)
|
|
412
426
|
- Google Vertex AI (`aiplatform.googleapis.com`) — see the [Vertex AI batch result logging guide](docs/vertex.md) for async batch jobs
|
|
413
427
|
- Azure OpenAI Service, Azure AI Foundry, and Azure Machine Learning managed online endpoints (dedicated hosts, multi-service Azure AI Services hosts, and serverless/MaaS deployments) — see [Configuration](docs/configuration.md) for the full list of covered hosts
|
|
414
|
-
-
|
|
428
|
+
- Amazon Bedrock — the standard, FIPS, dual-stack and China-partition runtime hosts; see [Configuration](docs/configuration.md) for the full list of covered hosts
|
|
415
429
|
- Cloudflare AI Gateway
|
|
416
430
|
- OpenRouter
|
|
417
431
|
- OpenCode Zen (`opencode.ai`)
|
|
@@ -440,7 +454,7 @@ When a request matches configured LLM endpoints:
|
|
|
440
454
|
|
|
441
455
|
1. The original request executes normally
|
|
442
456
|
2. Request and response data (body, headers, status) are captured by the interceptor
|
|
443
|
-
3. For streaming requests, the complete accumulated response is captured (not individual chunks)
|
|
457
|
+
3. For streaming requests, the complete accumulated response is captured (not individual chunks); a stream your code abandons before it ends is logged with the partial body and `stream_incomplete: true` ([details](docs/configuration.md#abandoned-streams-stream_incomplete))
|
|
444
458
|
4. Data is sent to the Coolhand API inline, after the original response is available (bounded by a 5-second connect/read timeout so a slow or unreachable Coolhand backend can't hang your request indefinitely)
|
|
445
459
|
5. Your application continues once the log is sent (or times out) — the original request/response is never blocked on Coolhand being reachable
|
|
446
460
|
|
|
@@ -507,13 +521,14 @@ The monitor handles errors gracefully:
|
|
|
507
521
|
- Failed API logging attempts are logged to console but don't interrupt your application
|
|
508
522
|
- Invalid API keys will be reported but won't crash your app
|
|
509
523
|
- Network issues are handled with appropriate error messages
|
|
510
|
-
- The read methods (`search_templates`, `get_template`, `search_workloads`, `search_logs`, `get_log`, `search_help_articles`, `get_help_article`, `list_help_categories`), the feedback-link methods and the `upsert` methods are the exception — they raise `Coolhand::Error` (`Coolhand::HttpError` for non-2xx responses); see [Reading Templates →](docs/template-search.md), [Reading Workloads →](docs/workload-search.md), [Reading Logs →](docs/log-search.md), [Linking Feedback →](docs/feedback-links.md) and [Declared Workloads →](docs/declared_workloads.md)
|
|
524
|
+
- The read methods (`search_templates`, `get_template`, `list_step_templates`, `get_step_template`, `search_workloads`, `search_logs`, `get_log`, `failure_summary`, `search_help_articles`, `get_help_article`, `list_help_categories`), the feedback-link methods and the `upsert` methods are the exception — they raise `Coolhand::Error` (`Coolhand::HttpError` for non-2xx responses); see [Reading Templates →](docs/template-search.md), [Reading Step Templates →](docs/step-templates.md), [Reading Workloads →](docs/workload-search.md), [Reading Logs →](docs/log-search.md), [Linking Feedback →](docs/feedback-links.md) and [Declared Workloads →](docs/declared_workloads.md)
|
|
511
525
|
|
|
512
526
|
## Documentation
|
|
513
527
|
|
|
514
528
|
- **[Configuration](docs/configuration.md)** — Self-hosted deployments, base_url rules, debug mode, custom intercept addresses
|
|
515
529
|
- **[Feedback API](docs/feedback.md)** — Full field reference, matching strategies, sentiment values
|
|
516
530
|
- **[Reading Templates](docs/template-search.md)** — Search LLM request templates and fetch a single one, prompt patterns included, using the private API key
|
|
531
|
+
- **[Reading Step Templates](docs/step-templates.md)** — List the skills and recurring steps detected in captured agent sessions, with run and efficiency metrics, using the private API key
|
|
517
532
|
- **[Reading Workloads](docs/workload-search.md)** — List workloads with optional cost and performance metrics over a rolling or explicit time window
|
|
518
533
|
- **[Reading Logs](docs/log-search.md)** — Search logs with per-log cost, filter and sort by cost, and fetch one log with its cost breakdown
|
|
519
534
|
- **[Reading the Help Center](docs/help-center.md)** — Search and read Coolhand's help center articles and list its sections, with no API key required
|
data/docs/configuration.md
CHANGED
|
@@ -98,6 +98,19 @@ end
|
|
|
98
98
|
|
|
99
99
|
In every case, only the *captured log entry* is affected — the real request to the LLM provider always sends the complete, untruncated body, and your code always receives the complete response (every streamed chunk is passed to your block untouched).
|
|
100
100
|
|
|
101
|
+
## Abandoned Streams (`stream_incomplete`)
|
|
102
|
+
|
|
103
|
+
A streamed response (a `Net::HTTP#request` block that reads the body through a block-form `read_body`, which is what the OpenAI, Anthropic and Gemini Ruby clients use under the hood) that your code stops reading before it ends is logged with whatever had arrived, and `raw_request.stream_incomplete: true`. A response read to its end is sent without the field. The backend then nulls latency, and nulls token counts unless the provider's final usage event is in the captured body; Coolhand never computes tokens, cost or latency for a partial body.
|
|
104
|
+
|
|
105
|
+
A stream is marked when the body was not read to its end because:
|
|
106
|
+
- your `read_body` block (or the SDK's) `break`s or `throw`s out of it, including right after the provider's terminator (`data: [DONE]`): only reading the body to its end counts as complete
|
|
107
|
+
- your streaming block raises
|
|
108
|
+
- the connection drops or times out mid-body (`EOFError`, `Errno::ECONNRESET`, `Net::ReadTimeout`, ...); the error still propagates to your code as before
|
|
109
|
+
|
|
110
|
+
Nothing is marked when no part of the body had arrived yet (for example the connection drops before the first chunk, or the block is left before reading): there is no partial body, so the log is the same as before, including the error record for a failure. When a stream fails mid-body (your streaming block raises, or the connection drops), the log holds the partial body and the status of the response, not the exception: its class and message are not recorded. Only one record is sent per call. The `body_too_large` cap is independent of the marker: a body truncated by the cap but read to its end is not marked, and one that is both truncated and abandoned is. The capture buffer is released as soon as the record is sent.
|
|
111
|
+
|
|
112
|
+
Limits: a bare `break` out of a `read_body` block can leave Net::HTTP itself waiting out its read timeout (60 seconds by default) when it lands partway through a chunk of a keep-alive connection, because Net::HTTP then tries to drain a half-read body. This happens with plain Net::HTTP too, without Coolhand loaded; to stop reading a live stream, `throw` or `return` out of the whole request so the connection is closed. By default (`ignore_eof`) Net::HTTP does not raise when a response with a `Content-Length` (or one that ends by closing the connection) is cut short, so those are logged as complete. Chunks are captured as Net::HTTP delivers them (already decompressed), so a cut-off gzip body is logged as the text received so far.
|
|
113
|
+
|
|
101
114
|
## Custom Intercept Addresses
|
|
102
115
|
|
|
103
116
|
By default Coolhand captures requests to a built-in list of LLM API hosts (OpenAI, Anthropic, Google Gemini, ElevenLabs, GitHub Models, and more). To capture a custom endpoint — an internal proxy, a self-hosted model server, or a third-party gateway — override `intercept_addresses`:
|
|
@@ -117,7 +130,7 @@ Setting `intercept_addresses` **replaces** the default list entirely, so include
|
|
|
117
130
|
|
|
118
131
|
The default list can be found in `Coolhand::Configuration::DEFAULT_INTERCEPT_ADDRESSES`.
|
|
119
132
|
|
|
120
|
-
Each entry is matched against the request's parsed host (optionally with a port and path prefix, see below) — an exact match, or a dot-boundary suffix match (`my-llm-proxy.internal` also matches `eu.my-llm-proxy.internal`) — case-insensitively. A single `*` in an entry matches exactly one host label, e.g. `bedrock-runtime.*.amazonaws.com`. A wildcard entry is an exact host match with no implicit subdomains, so each Bedrock
|
|
133
|
+
Each entry is matched against the request's parsed host (optionally with a port and path prefix, see below) — an exact match, or a dot-boundary suffix match (`my-llm-proxy.internal` also matches `eu.my-llm-proxy.internal`) — case-insensitively. A single `*` in an entry matches exactly one host label, e.g. `bedrock-runtime.*.amazonaws.com`. A wildcard entry is an exact host match with no implicit subdomains, so each Amazon Bedrock runtime host is listed separately: `bedrock-runtime.*.amazonaws.com`, `bedrock-runtime.*.amazonaws.com.cn`, `bedrock-runtime.*.api.aws` (dual-stack/IPv6), and the FIPS hosts `bedrock-runtime-fips.*.amazonaws.com` and `bedrock-runtime-fips.*.api.aws`. Hosts outside that set need their own entry — the endpoint-specific DNS name of a VPC interface endpoint (`vpce-….bedrock-runtime.…`; PrivateLink callers with private DNS enabled still address the standard regional host and are already covered), the aws-cn dual-stack shape (`bedrock-runtime.*.api.amazonwebservices.com.cn`) and the Agents for Amazon Bedrock runtime (`bedrock-agent-runtime.*.amazonaws.com`, which serves `InvokeAgent` and `RetrieveAndGenerate`) are not covered by the defaults. Entries are never matched against the full URL, so query strings that merely mention a host can't trigger a false match.
|
|
121
134
|
|
|
122
135
|
An entry can also pin a port (`host:port`) and/or a path (`host/path`, or `host:port/path`), and then a request must match all of them. Cohere, TypeSafe and Ollama are defaulted this way: `api.cohere.com/v2/chat`, `api.typesafe.ai/v1/systemone`, and `*:11434/api/chat` (plus `/api/generate`, `/api/embed`, `/api/embeddings`, also for `127.0.0.1`). Ollama has no fixed host, so its defaults require Ollama's default port *and* an Ollama path — an app's own unrelated `/api/chat` on another host or port is not captured. To capture an Ollama server on a dotted host or IP, or a non-default port, add entries such as `192.168.1.5:11434/api/chat` or `ollama.internal:8080/api/chat`. An IPv6 loopback server (`[::1]:11434`) is already covered by the defaults.
|
|
123
136
|
|
|
@@ -131,7 +144,7 @@ A small number of Google API endpoints use a colon-suffixed path (e.g. `.../gemi
|
|
|
131
144
|
|
|
132
145
|
## Excluding API Patterns
|
|
133
146
|
|
|
134
|
-
`config.exclude_api_patterns` is a deny-list checked *after* `intercept_addresses` allows a request through — a match against the request's path is skipped and never forwarded as an `llm_request_log`. It defaults to a small built-in list (see `Coolhand::Configuration::DEFAULT_EXCLUDE_API_PATTERNS`) that suppresses Vertex AI batch job management noise (`/batchPredictionJobs/`) and
|
|
147
|
+
`config.exclude_api_patterns` is a deny-list checked *after* `intercept_addresses` allows a request through — a match against the request's path is skipped and never forwarded as an `llm_request_log`. It defaults to a small built-in list (see `Coolhand::Configuration::DEFAULT_EXCLUDE_API_PATTERNS`) that suppresses Vertex AI batch job management noise (`/batchPredictionJobs/`) and OpenAI-compatible control-plane paths (file, batch, fine-tuning and model-listing management under `/openai/` and `/openai/v1/`) that aren't inference traffic. Those `/openai/` patterns apply to every intercepted host that exposes an OpenAI-compatible surface, Azure OpenAI Service and Amazon Bedrock alike, so narrowing them for one provider narrows them for both.
|
|
135
148
|
|
|
136
149
|
```ruby
|
|
137
150
|
Coolhand.configure do |config|
|
data/docs/log-search.md
CHANGED
|
@@ -1,14 +1,15 @@
|
|
|
1
1
|
# Reading Logs (Search + Get)
|
|
2
2
|
|
|
3
3
|
`Coolhand::LogService` reads logs back out of Coolhand, with the per-log `cost` the dashboard shows.
|
|
4
|
-
The write side, `Coolhand.logger_service`, is a different class. It wraps
|
|
4
|
+
The write side, `Coolhand.logger_service`, is a different class. It wraps three read-only endpoints:
|
|
5
5
|
|
|
6
6
|
| method | endpoint |
|
|
7
7
|
|---|---|
|
|
8
8
|
| `search_logs` | `GET /api/v2/llm_request_logs` |
|
|
9
9
|
| `get_log` | `GET /api/v2/llm_request_logs/{id}` |
|
|
10
|
+
| `failure_summary` | `GET /api/v2/llm_request_logs/failure_summary` |
|
|
10
11
|
|
|
11
|
-
|
|
12
|
+
All three require your **private** API key. The public key is write-only on this API and is rejected exactly
|
|
12
13
|
like an invalid one.
|
|
13
14
|
|
|
14
15
|
```ruby
|
|
@@ -38,6 +39,7 @@ keyword, and the `per_page` alias, since this gem only sends `per`.
|
|
|
38
39
|
|---|---|---|
|
|
39
40
|
| `template_id`, `workload_id` | String | Hashids. One that does not decode, or that belongs to another client, is a `422`. |
|
|
40
41
|
| `system_prompt_contains`, `user_prompt_contains` | String | Case-insensitive substring. |
|
|
42
|
+
| `error_contains` | String | Case-insensitive substring of the failed request's error text. Pass a `failure_summary` `:error_pattern` to list the logs behind that pattern. |
|
|
41
43
|
| `model`, `source_api`, `source_api_result`, `source_application` | String | Exact match. `source_api_result` is `success`, `failed`, `operational` or `unmatched`. |
|
|
42
44
|
| `project_path` | String | Exact match against `metadata.project_path`. |
|
|
43
45
|
| `unmatched_only` | Boolean | Only logs with no template. |
|
|
@@ -69,7 +71,7 @@ array, never wrapped. Rows are Symbol-keyed Hashes: `:id`, `:collector`, `:sourc
|
|
|
69
71
|
`total_pages` are lower bounds the gem derives, not counts, and `has_next_page` is true when the page
|
|
70
72
|
came back full. Pass `include_total: true` when you need real totals.
|
|
71
73
|
|
|
72
|
-
## `get_log(id, section: nil, max_chars: nil, search_query: nil, include_thinking: nil)`
|
|
74
|
+
## `get_log(id, section: nil, max_chars: nil, search_query: nil, include_thinking: nil, include_turn_metrics: nil)`
|
|
73
75
|
|
|
74
76
|
`id` is the log hashid. Returns the full log as a Symbol-keyed Hash: the list fields plus the prompts and
|
|
75
77
|
`:output`, and:
|
|
@@ -81,9 +83,54 @@ came back full. Pass `include_total: true` when you need real totals.
|
|
|
81
83
|
| `:cost_breakdown` | Hash or nil: `:total_cost`, `:input_cost`, `:output_cost`, `:cached_input_cost`, `:cache_creation_input_cost`, `:reasoning_output_cost` |
|
|
82
84
|
|
|
83
85
|
`section` (`"full"`, `"beginning"`, `"end"`), `max_chars` and `search_query` bound or search the content;
|
|
84
|
-
`include_thinking` adds `:thinking_response
|
|
85
|
-
`:total_chars`, `:search_query` and `:matches`.
|
|
86
|
-
generated records are a `404`.
|
|
86
|
+
`include_thinking` adds `:thinking_response`, and `include_turn_metrics: true` adds `:turn_metrics`.
|
|
87
|
+
Truncation and search are reported back as `:truncated`, `:total_chars`, `:search_query` and `:matches`.
|
|
88
|
+
Only directly-collected logs are fetchable; internally generated records are a `404`.
|
|
89
|
+
|
|
90
|
+
`:turn_metrics` is the per-turn breakdown of a captured agent session: an Array with one Hash per user
|
|
91
|
+
message, `{ index, human, started_at, ended_at, assistant_requests, tool_calls, input_tokens,
|
|
92
|
+
output_tokens, cache_read_tokens, cache_creation_tokens }`. The timestamps and the four token counts
|
|
93
|
+
are nil when unknown. It is `nil` for a log that is not a captured session, and the key is absent
|
|
94
|
+
unless you ask for it. It holds no content, so it is still returned after the content scrub.
|
|
95
|
+
|
|
96
|
+
## `failure_summary(days_back: nil)`
|
|
97
|
+
|
|
98
|
+
```ruby
|
|
99
|
+
summary = Coolhand.log_service.failure_summary(days_back: 30)
|
|
100
|
+
summary[:error_breakdown].each { |p| puts "#{p[:category]} #{p[:count]} #{p[:error_pattern]}" }
|
|
101
|
+
|
|
102
|
+
# The logs behind the most frequent pattern
|
|
103
|
+
top = summary[:error_breakdown].first
|
|
104
|
+
Coolhand.log_service.search_logs(error_contains: top[:error_pattern]).logs if top
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Rolls this client's recent failed requests up by error pattern (error text and `source_api`) and by
|
|
108
|
+
template, and sorts each pattern into whose problem it is. Failures inside a confirmed provider outage
|
|
109
|
+
are left out and counted in `:provider_outage_excluded_count`. Only directly-collected logs whose content
|
|
110
|
+
has not been removed by retention are counted.
|
|
111
|
+
|
|
112
|
+
`days_back` is the lookback in days (server default 7, max 36500). It is sent only when given and not
|
|
113
|
+
checked here, so `0` or a negative number is a `422` from the server.
|
|
114
|
+
|
|
115
|
+
Returns the response as one Symbol-keyed Hash, not paginated:
|
|
116
|
+
|
|
117
|
+
| key | type |
|
|
118
|
+
|---|---|
|
|
119
|
+
| `:client_id`, `:client_name` | String |
|
|
120
|
+
| `:days_analyzed`, `:total_failed_count`, `:provider_outage_excluded_count` | Integer |
|
|
121
|
+
| `:error_breakdown` | Array of the 50 most frequent patterns (below) |
|
|
122
|
+
| `:by_template` | Array of `:template_id` (String or nil), `:template_name`, `:failed_count`, `:error_patterns`, `:example_log_ids`, `:example_log_urls` |
|
|
123
|
+
| `:actionable_summary` | Hash: `:client_fixable_count`, `:server_side_count` |
|
|
124
|
+
|
|
125
|
+
Each `:error_breakdown` item has `:error_pattern`, `:source_api` (String or nil), `:count`,
|
|
126
|
+
`:first_seen_at`, `:last_seen_at` (ISO8601 Strings), `:category`, `:example_error_details`,
|
|
127
|
+
`:example_log_ids` (up to 3) and `:example_log_urls`. `:category` is `"client_error"` (a 4xx the caller
|
|
128
|
+
can fix), `"rate_limit"` (429), `"server_error"` (5xx), `"provider_error"` (no HTTP status) or
|
|
129
|
+
`"relayed_upstream"` (an upstream vendor's account or key error surfaced through a relay).
|
|
130
|
+
|
|
131
|
+
The totals and `:actionable_summary` count every pattern, not just the 50 listed. The API definition
|
|
132
|
+
types each `:by_template` `:error_patterns` item only as an open object; today it is `:category`,
|
|
133
|
+
`:count`, `:error` and `:source_api`, and any other key is optional.
|
|
87
134
|
|
|
88
135
|
## Errors
|
|
89
136
|
|
|
@@ -95,7 +142,7 @@ The read methods raise where the write methods log and return `nil`. `Coolhand::
|
|
|
95
142
|
| `401` | No API key, an invalid key, or the public key. |
|
|
96
143
|
| `404` | `get_log` only: unknown id, another client's log, or an internally generated record. |
|
|
97
144
|
| `422` | Bad `template_id`, `workload_id`, `days_back`, `since`, `until`, `min_cost`, `order` or `max_chars`. |
|
|
98
|
-
| `504` | The search exceeded the statement timeout. Narrow the window and retry. |
|
|
145
|
+
| `504` | The search or summary exceeded the statement timeout. Narrow the window (for `failure_summary`, a smaller `days_back`) and retry. |
|
|
99
146
|
|
|
100
147
|
`Coolhand::Error` covers no API key configured, a transport failure, a body that is not valid JSON, a
|
|
101
148
|
non-array list body, and a blank `id` passed to `get_log`. An unsupported `since`/`until` type raises
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Reading Step Templates (List + Get)
|
|
2
|
+
|
|
3
|
+
`Coolhand::StepTemplateService` reads the step templates Coolhand detects in captured agent sessions.
|
|
4
|
+
A `prompt_step` matches a user prompt such as a slash command; a `tool_step` matches tool usage. It
|
|
5
|
+
wraps two read-only endpoints; there are no write endpoints:
|
|
6
|
+
|
|
7
|
+
| method | endpoint |
|
|
8
|
+
|---|---|
|
|
9
|
+
| `list_step_templates` | `GET /api/v2/step_templates` |
|
|
10
|
+
| `get_step_template` | `GET /api/v2/step_templates/{id}` |
|
|
11
|
+
|
|
12
|
+
Both require your **private** API key; the public key returns `401`. The client is derived from the
|
|
13
|
+
key, so there is no `client_id` keyword.
|
|
14
|
+
|
|
15
|
+
```ruby
|
|
16
|
+
require "coolhand"
|
|
17
|
+
|
|
18
|
+
Coolhand.configure do |config|
|
|
19
|
+
config.api_key = ENV.fetch("COOLHAND_PRIVATE_API_KEY")
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
steps = Coolhand.step_template_service
|
|
23
|
+
|
|
24
|
+
result = steps.list_step_templates(step_type: "prompt_step", include_metrics: true, page: 1)
|
|
25
|
+
result.step_templates.each { |step| puts "#{step[:name]} (#{step[:sessions_count]} sessions)" }
|
|
26
|
+
|
|
27
|
+
lean = steps.get_step_template(result.step_templates.first[:id], include_metrics: false)
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## `list_step_templates(search: nil, step_type: nil, include_metrics: nil, page: nil, per: nil)`
|
|
31
|
+
|
|
32
|
+
| keyword | type | notes |
|
|
33
|
+
|---|---|---|
|
|
34
|
+
| `search` | String | Case-insensitive substring match on the name. |
|
|
35
|
+
| `step_type` | String | `"prompt_step"` or `"tool_step"`. Anything else is a `422` from the server. |
|
|
36
|
+
| `include_metrics` | Boolean | Adds a `:metrics` Hash per row. Off by default; when set the server caps `per` at 25, and `pagination.per_page` reports the size applied. |
|
|
37
|
+
| `page` | Integer | 1-based. |
|
|
38
|
+
| `per` | Integer | Default 25, max 100, both enforced server-side. |
|
|
39
|
+
|
|
40
|
+
Returns a `Coolhand::StepTemplateSearchResult` (`#step_templates`, `#pagination`), newest first. The
|
|
41
|
+
endpoint renders a bare array; `pagination` is a `Coolhand::Pagination` read from the `X-Page`,
|
|
42
|
+
`X-Per-Page`, `X-Total-Count` and `X-Total-Pages` response headers, the same as `search_templates`
|
|
43
|
+
and `search_logs`.
|
|
44
|
+
|
|
45
|
+
## `get_step_template(id, include_metrics: nil)`
|
|
46
|
+
|
|
47
|
+
`id` is the Integer `:id` from the list (not a hashid). Anything that is not a positive Integer raises
|
|
48
|
+
`Coolhand::Error` before any request. Unlike the list, `:metrics` is included by default; pass
|
|
49
|
+
`include_metrics: false` to skip the metrics queries.
|
|
50
|
+
|
|
51
|
+
## Fields
|
|
52
|
+
|
|
53
|
+
Rows are plain Symbol-keyed Hashes, handed back as the API rendered them. Every row has `:id`,
|
|
54
|
+
`:name`, `:step_type`, `:status`, `:user_turn_pattern` (nil-able), `:created_at`, `:updated_at`,
|
|
55
|
+
`:sessions_count` (distinct sessions it appeared in), `:occurrences_count` and `:last_seen_at`
|
|
56
|
+
(nil-able). `:metrics` is present only when requested, or by default on `get_step_template`:
|
|
57
|
+
|
|
58
|
+
| `:metrics` field | Meaning |
|
|
59
|
+
|---|---|
|
|
60
|
+
| `:total_runs` | Runs counted; excludes scrubbed prompt-step sessions. |
|
|
61
|
+
| `:tool_stats` | Array of `{ name:, pct_of_runs:, avg_per_run: }`, or nil when there is nothing to show. |
|
|
62
|
+
| `:turn_stats` | `{ avg:, median:, high:, single_turn_pct: }` (`single_turn_pct` nil-able), or nil. |
|
|
63
|
+
| `:efficiency` | `{ interactive:, automated:, all:, unavailable_runs: }`, or nil for tool steps and step templates with no runs. Each cohort is nil when it has no runs. |
|
|
64
|
+
| `:notes` | Array of plain-sentence explanations of each data gap that applies. |
|
|
65
|
+
|
|
66
|
+
A cohort is `{ runs:, avg_turns_after:, avg_tool_calls:, avg_requests:, token_runs:, tokens_available:,
|
|
67
|
+
median_tokens:, p90_tokens:, tokens_per_tool_call: }`. The last three are nil, and `tokens_available`
|
|
68
|
+
is `false`, when no run in the cohort has token data (expected today).
|
|
69
|
+
|
|
70
|
+
## Errors
|
|
71
|
+
|
|
72
|
+
Both methods raise on failure, like the other read methods. A non-2xx response is a
|
|
73
|
+
`Coolhand::HttpError` whose `#status` holds the HTTP status:
|
|
74
|
+
|
|
75
|
+
| status | when |
|
|
76
|
+
|---|---|
|
|
77
|
+
| `401` | Missing, invalid or public API key. |
|
|
78
|
+
| `404` | `get_step_template` only: unknown id, a draft or failed step template, or another client's. |
|
|
79
|
+
| `422` | `list_step_templates` only: a `step_type` other than `prompt_step` / `tool_step`. |
|
|
80
|
+
| `504` | A query exceeded the statement timeout. Retryable: narrow with `search` or a smaller `per`. |
|
|
81
|
+
|
|
82
|
+
## Turn metrics on a log
|
|
83
|
+
|
|
84
|
+
`Coolhand.log_service.get_log(id, include_turn_metrics: true)` adds `:turn_metrics` to the log; see
|
|
85
|
+
[Reading Logs](log-search.md).
|
data/lib/coolhand/api_service.rb
CHANGED
|
@@ -87,9 +87,10 @@ module Coolhand
|
|
|
87
87
|
# Bound worst-case latency: with async_logging off this call happens inline in
|
|
88
88
|
# the intercepted request's path (otherwise it ties up the background log
|
|
89
89
|
# worker), so a slow/unreachable Coolhand backend must not hang for Ruby's
|
|
90
|
-
# ~60s Net::HTTP defaults.
|
|
90
|
+
# ~60s Net::HTTP defaults (a stalled upload hits write_timeout).
|
|
91
91
|
http.open_timeout = 5
|
|
92
92
|
http.read_timeout = 5
|
|
93
|
+
http.write_timeout = 5
|
|
93
94
|
|
|
94
95
|
request = Net::HTTP::Post.new(uri.request_uri)
|
|
95
96
|
apply_headers(request, create_request_options(payload))
|
|
@@ -247,7 +247,7 @@ module Coolhand
|
|
|
247
247
|
|
|
248
248
|
def send_complete_request_log(request_id:, method:, url:, request_headers:, request_body:, response_headers:,
|
|
249
249
|
response_body:, status_code:, start_time:, end_time:, duration_ms:, is_streaming:, source_api: nil, model: nil,
|
|
250
|
-
declared: nil)
|
|
250
|
+
declared: nil, stream_incomplete: false)
|
|
251
251
|
raw_request = {
|
|
252
252
|
id: request_id,
|
|
253
253
|
timestamp: start_time.iso8601,
|
|
@@ -264,6 +264,7 @@ module Coolhand
|
|
|
264
264
|
}
|
|
265
265
|
raw_request[:source_api] = source_api if Coolhand.required_field?(source_api)
|
|
266
266
|
raw_request[:model] = model if Coolhand.required_field?(model)
|
|
267
|
+
raw_request[:stream_incomplete] = true if stream_incomplete
|
|
267
268
|
|
|
268
269
|
request_data = { raw_request: raw_request }
|
|
269
270
|
request_data[:declared] = declared if declared && !declared.empty?
|
|
@@ -8,7 +8,8 @@
|
|
|
8
8
|
# Users can disable: c.exclude_api_patterns = []
|
|
9
9
|
|
|
10
10
|
- "/batchPredictionJobs/"
|
|
11
|
-
#
|
|
11
|
+
# OpenAI-compatible control-plane paths (not inference), on Azure OpenAI Service
|
|
12
|
+
# and Amazon Bedrock alike — both spellings are needed
|
|
12
13
|
# since substring matching is contiguous: "/openai/files" does not match
|
|
13
14
|
# "/openai/v1/files".
|
|
14
15
|
- "/openai/files"
|
|
@@ -26,7 +26,8 @@
|
|
|
26
26
|
# Amazon Bedrock. Every endpoint variant needs its own entry: a wildcard entry is
|
|
27
27
|
# an exact host match with no implicit subdomains, and "amazonaws.com.cn" is not a
|
|
28
28
|
# dot-boundary suffix of "amazonaws.com". "api.aws" hosts are the dual-stack (IPv6)
|
|
29
|
-
# endpoints; "-fips" hosts are the FIPS 140-3 ones, used by GovCloud
|
|
29
|
+
# endpoints; "-fips" hosts are the FIPS 140-3 ones, used by GovCloud and other
|
|
30
|
+
# FIPS-constrained callers (they exist in commercial regions too).
|
|
30
31
|
- "bedrock-runtime.*.amazonaws.com"
|
|
31
32
|
- "bedrock-runtime.*.amazonaws.com.cn"
|
|
32
33
|
- "bedrock-runtime-fips.*.amazonaws.com"
|
data/lib/coolhand/log_service.rb
CHANGED
|
@@ -14,6 +14,7 @@ module Coolhand
|
|
|
14
14
|
include ReadQuery
|
|
15
15
|
|
|
16
16
|
ERROR_NOUN = "Log"
|
|
17
|
+
FAILURE_SUMMARY_PATH = "failure_summary"
|
|
17
18
|
BLANK_ID_MESSAGE = "get_log: id must be a non-empty log hashid"
|
|
18
19
|
|
|
19
20
|
def initialize
|
|
@@ -22,14 +23,15 @@ module Coolhand
|
|
|
22
23
|
|
|
23
24
|
# Filters, cost ordering, error semantics and server behaviour: docs/log-search.md.
|
|
24
25
|
def search_logs(template_id: nil, workload_id: nil, system_prompt_contains: nil, user_prompt_contains: nil,
|
|
25
|
-
|
|
26
|
-
unmatched_only: nil, days_back: nil, since: nil, until: nil, min_cost: nil, order: nil,
|
|
26
|
+
error_contains: nil, model: nil, source_api: nil, source_api_result: nil, source_application: nil,
|
|
27
|
+
project_path: nil, unmatched_only: nil, days_back: nil, since: nil, until: nil, min_cost: nil, order: nil,
|
|
27
28
|
include_prompts: nil, include_total: nil, sort: nil, page: nil, per: nil)
|
|
28
29
|
query = {
|
|
29
30
|
template_id: template_id,
|
|
30
31
|
workload_id: workload_id,
|
|
31
32
|
system_prompt_contains: system_prompt_contains,
|
|
32
33
|
user_prompt_contains: user_prompt_contains,
|
|
34
|
+
error_contains: error_contains,
|
|
33
35
|
model: model,
|
|
34
36
|
source_api: source_api,
|
|
35
37
|
source_api_result: source_api_result,
|
|
@@ -50,11 +52,21 @@ module Coolhand
|
|
|
50
52
|
fetch_list(query, LogSearchResult, :logs, page: page, per: per)
|
|
51
53
|
end
|
|
52
54
|
|
|
55
|
+
# Failed requests rolled up by error pattern and by template. One object, not paginated. Pass a
|
|
56
|
+
# pattern's `:error_pattern` as `search_logs(error_contains:)` to list its logs. `days_back` is sent
|
|
57
|
+
# as given, so the server answers 422 for zero or negative. docs/log-search.md.
|
|
58
|
+
def failure_summary(days_back: nil)
|
|
59
|
+
get_json(action_url(FAILURE_SUMMARY_PATH, { days_back: days_back }.compact), ERROR_NOUN)
|
|
60
|
+
end
|
|
61
|
+
|
|
53
62
|
# The full log, with `cost` and `cost_breakdown`. Both are nil when the log has no tokens or its
|
|
54
63
|
# model has no pricing.
|
|
55
|
-
|
|
64
|
+
# `turn_metrics` is added only when `include_turn_metrics: true`; it is nil for a log that is not a
|
|
65
|
+
# captured agent session.
|
|
66
|
+
def get_log(id, section: nil, max_chars: nil, search_query: nil, include_thinking: nil,
|
|
67
|
+
include_turn_metrics: nil)
|
|
56
68
|
query = { section: section, max_chars: max_chars, search_query: search_query,
|
|
57
|
-
include_thinking: include_thinking }.compact
|
|
69
|
+
include_thinking: include_thinking, include_turn_metrics: include_turn_metrics }.compact
|
|
58
70
|
get_json(resource_url(id, BLANK_ID_MESSAGE, query), ERROR_NOUN)
|
|
59
71
|
end
|
|
60
72
|
end
|
|
@@ -8,6 +8,7 @@ require "zlib"
|
|
|
8
8
|
|
|
9
9
|
require_relative "body_stream_tap"
|
|
10
10
|
require_relative "path_normalizer"
|
|
11
|
+
require_relative "stream_read"
|
|
11
12
|
|
|
12
13
|
module Coolhand
|
|
13
14
|
module NetHttpInterceptor
|
|
@@ -22,9 +23,18 @@ module Coolhand
|
|
|
22
23
|
# freed, and leaks one request's chunks into another's log.
|
|
23
24
|
return super unless block && Thread.current[:coolhand_capturing_stream].equal?(self)
|
|
24
25
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
26
|
+
finished = false
|
|
27
|
+
begin
|
|
28
|
+
result = super do |chunk|
|
|
29
|
+
Helpers.buffer_stream_chunk(chunk)
|
|
30
|
+
yield(chunk)
|
|
31
|
+
end
|
|
32
|
+
finished = true
|
|
33
|
+
result
|
|
34
|
+
ensure
|
|
35
|
+
# Also runs when the host's block breaks, throws or raises: read_body then never reached the
|
|
36
|
+
# end of the body, which is what marks the log stream_incomplete.
|
|
37
|
+
StreamRead.record(finished)
|
|
28
38
|
end
|
|
29
39
|
end
|
|
30
40
|
end
|
|
@@ -125,8 +135,10 @@ module Coolhand
|
|
|
125
135
|
# chunks, mixing one request's content into another's log.
|
|
126
136
|
previous_stream_buffer = Thread.current[:coolhand_stream_buffer]
|
|
127
137
|
previous_capturing_stream = Thread.current[:coolhand_capturing_stream]
|
|
138
|
+
previous_stream_read = Thread.current[:coolhand_stream_read]
|
|
128
139
|
Thread.current[:coolhand_stream_buffer] = nil
|
|
129
140
|
Thread.current[:coolhand_capturing_stream] = nil
|
|
141
|
+
Thread.current[:coolhand_stream_read] = nil
|
|
130
142
|
host_block_errors = []
|
|
131
143
|
tracked_block = Helpers.track_streaming_block(block, host_block_errors)
|
|
132
144
|
|
|
@@ -139,9 +151,18 @@ module Coolhand
|
|
|
139
151
|
ensure
|
|
140
152
|
# First, so the host's own stream is back on its request whatever fails below.
|
|
141
153
|
request_body = Helpers.finalize_request_body(captured_body, req)
|
|
154
|
+
stream_incomplete = StreamRead.incomplete?
|
|
155
|
+
# A stream abandoned by break, throw or raise: what was buffered so far is logged (cleaned of a
|
|
156
|
+
# split multibyte character) instead of the error, or of the capture above after a break.
|
|
157
|
+
logged_response = response
|
|
158
|
+
if stream_incomplete
|
|
159
|
+
logged_response = Thread.current[:coolhand_capturing_stream]
|
|
160
|
+
status_code, response_body = StreamRead.partial_response(logged_response)
|
|
161
|
+
end
|
|
142
162
|
active.delete(self)
|
|
143
163
|
Thread.current[:coolhand_stream_buffer] = previous_stream_buffer
|
|
144
164
|
Thread.current[:coolhand_capturing_stream] = previous_capturing_stream
|
|
165
|
+
Thread.current[:coolhand_stream_read] = previous_stream_read
|
|
145
166
|
end_time = Time.now
|
|
146
167
|
duration_ms = ((end_time - start_time) * 1000).round(2)
|
|
147
168
|
|
|
@@ -153,13 +174,14 @@ module Coolhand
|
|
|
153
174
|
url: url,
|
|
154
175
|
request_headers: BaseInterceptor.sanitize_headers(req),
|
|
155
176
|
request_body: request_body,
|
|
156
|
-
response_headers: BaseInterceptor.sanitize_headers(
|
|
177
|
+
response_headers: BaseInterceptor.sanitize_headers(logged_response),
|
|
157
178
|
response_body: response_body,
|
|
158
179
|
status_code: status_code,
|
|
159
180
|
start_time: start_time,
|
|
160
181
|
end_time: end_time,
|
|
161
182
|
duration_ms: duration_ms,
|
|
162
|
-
is_streaming: !!block
|
|
183
|
+
is_streaming: !!block,
|
|
184
|
+
stream_incomplete: stream_incomplete
|
|
163
185
|
)
|
|
164
186
|
end
|
|
165
187
|
|
|
@@ -190,6 +212,7 @@ module Coolhand
|
|
|
190
212
|
|
|
191
213
|
proc do |res|
|
|
192
214
|
Thread.current[:coolhand_capturing_stream] = res
|
|
215
|
+
Thread.current[:coolhand_stream_read] = nil
|
|
193
216
|
block.call(res)
|
|
194
217
|
rescue StandardError => e
|
|
195
218
|
host_block_errors << e unless transport_error?(e)
|
data/lib/coolhand/read_query.rb
CHANGED
|
@@ -13,9 +13,12 @@ module Coolhand
|
|
|
13
13
|
protected
|
|
14
14
|
|
|
15
15
|
def list_url(query)
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
16
|
+
url_with_query(api_endpoint, query)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# A fixed collection action such as `failure_summary`; unlike `resource_url` there is no id to validate.
|
|
20
|
+
def action_url(action, query = {})
|
|
21
|
+
url_with_query("#{api_endpoint}/#{action}", query)
|
|
19
22
|
end
|
|
20
23
|
|
|
21
24
|
# GETs a list endpoint and wraps the JSON array and its pagination headers in `result_class`,
|
|
@@ -39,7 +42,11 @@ module Coolhand
|
|
|
39
42
|
# retargets the request at another path. Neither would 404, so both are rejected here.
|
|
40
43
|
raise Error, blank_message if trimmed.empty? || [".", ".."].include?(trimmed)
|
|
41
44
|
|
|
42
|
-
|
|
45
|
+
url_with_query("#{api_endpoint}/#{escape_path_segment(trimmed)}", query)
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def url_with_query(url, query)
|
|
49
|
+
uri = URI.parse(url)
|
|
43
50
|
uri.query = URI.encode_www_form(query) unless query.empty?
|
|
44
51
|
uri
|
|
45
52
|
end
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "api_service"
|
|
4
|
+
require_relative "pagination"
|
|
5
|
+
require_relative "read_query"
|
|
6
|
+
|
|
7
|
+
module Coolhand
|
|
8
|
+
# Rows stay plain Symbol-keyed Hashes so a field the server adds later is not dropped in transit.
|
|
9
|
+
StepTemplateSearchResult = Struct.new(:step_templates, :pagination, keyword_init: true)
|
|
10
|
+
|
|
11
|
+
# Reads the skills and other recurring steps Coolhand detects in captured agent sessions. Read-only;
|
|
12
|
+
# there are no write endpoints. Requires the client's **private** key — the public key is rejected
|
|
13
|
+
# like an invalid one. See docs/step-templates.md.
|
|
14
|
+
class StepTemplateService < ApiService
|
|
15
|
+
include ReadQuery
|
|
16
|
+
|
|
17
|
+
ERROR_NOUN = "Step template"
|
|
18
|
+
INVALID_ID_MESSAGE = "get_step_template: id must be a positive Integer"
|
|
19
|
+
|
|
20
|
+
def initialize
|
|
21
|
+
super("v2/step_templates")
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
# Filters, `metrics` and error semantics: docs/step-templates.md.
|
|
25
|
+
def list_step_templates(search: nil, step_type: nil, include_metrics: nil, page: nil, per: nil)
|
|
26
|
+
query = {
|
|
27
|
+
search: search,
|
|
28
|
+
step_type: step_type,
|
|
29
|
+
include_metrics: include_metrics,
|
|
30
|
+
page: page,
|
|
31
|
+
per: per
|
|
32
|
+
}.compact
|
|
33
|
+
|
|
34
|
+
fetch_list(query, StepTemplateSearchResult, :step_templates, page: page, per: per)
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# Carries `metrics` unless `include_metrics: false`, the opposite default from the list.
|
|
38
|
+
def get_step_template(id, include_metrics: nil)
|
|
39
|
+
raise Error, INVALID_ID_MESSAGE unless id.is_a?(Integer) && id.positive?
|
|
40
|
+
|
|
41
|
+
query = { include_metrics: include_metrics }.compact
|
|
42
|
+
get_json(resource_url(id.to_s, INVALID_ID_MESSAGE, query), ERROR_NOUN)
|
|
43
|
+
end
|
|
44
|
+
end
|
|
45
|
+
end
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Coolhand
|
|
4
|
+
module NetHttpInterceptor
|
|
5
|
+
# Whether the host read a streamed response body to its end, tracked in thread-local state that
|
|
6
|
+
# #request saves and restores around each call (like the stream buffer).
|
|
7
|
+
module StreamRead
|
|
8
|
+
module_function
|
|
9
|
+
|
|
10
|
+
# Records how the host's first block-form read of the body ended; a later one cannot undo it
|
|
11
|
+
# (Net::HTTP raises IOError "called twice" on a second read, which would otherwise look abandoned).
|
|
12
|
+
def record(finished)
|
|
13
|
+
Thread.current[:coolhand_stream_read] ||= finished ? :complete : :abandoned
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
# True when a response was handed to the host's streaming block and some of its body was
|
|
17
|
+
# buffered, but it was not read to the end: the read was cut short (break, throw, raise, dropped
|
|
18
|
+
# connection). With nothing buffered there is no partial body to mark, so the log is unchanged.
|
|
19
|
+
# A body Net::HTTP read to its end is complete even if the host then raised.
|
|
20
|
+
def incomplete?
|
|
21
|
+
thread = Thread.current
|
|
22
|
+
return false unless thread[:coolhand_capturing_stream] && thread[:coolhand_stream_buffer]
|
|
23
|
+
|
|
24
|
+
thread[:coolhand_stream_read] == :abandoned
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
# [status_code, response_body] for an abandoned stream: what was buffered so far.
|
|
28
|
+
# Runs in #request's ensure, so it must not raise.
|
|
29
|
+
def partial_response(response)
|
|
30
|
+
buffered = Thread.current[:coolhand_stream_buffer]
|
|
31
|
+
# The cut can land inside a multibyte character, and invalid UTF-8 makes JSON.generate fail,
|
|
32
|
+
# which would lose the whole record.
|
|
33
|
+
if buffered.is_a?(String)
|
|
34
|
+
buffered = buffered.dup.force_encoding(Encoding::UTF_8)
|
|
35
|
+
buffered.scrub!("") unless buffered.valid_encoding?
|
|
36
|
+
end
|
|
37
|
+
[response.code.to_i, NetHttpInterceptor::Helpers.response_body_for_log(buffered, response)]
|
|
38
|
+
rescue StandardError => e
|
|
39
|
+
Coolhand.log "❌ Error capturing partial response: #{e.message}"
|
|
40
|
+
[nil, nil]
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
end
|
data/lib/coolhand/version.rb
CHANGED
data/lib/coolhand.rb
CHANGED
|
@@ -23,6 +23,7 @@ require_relative "coolhand/api_service"
|
|
|
23
23
|
require_relative "coolhand/logger_service"
|
|
24
24
|
require_relative "coolhand/feedback_service"
|
|
25
25
|
require_relative "coolhand/template_service"
|
|
26
|
+
require_relative "coolhand/step_template_service"
|
|
26
27
|
require_relative "coolhand/optimization_feedback_link_service"
|
|
27
28
|
require_relative "coolhand/workload_service"
|
|
28
29
|
require_relative "coolhand/help_center_service"
|
|
@@ -139,6 +140,12 @@ module Coolhand
|
|
|
139
140
|
TemplateService.new
|
|
140
141
|
end
|
|
141
142
|
|
|
143
|
+
# Creates a new StepTemplateService instance, for reading the skills and recurring steps detected
|
|
144
|
+
# in captured agent sessions. Requires the private API key.
|
|
145
|
+
def step_template_service
|
|
146
|
+
StepTemplateService.new
|
|
147
|
+
end
|
|
148
|
+
|
|
142
149
|
# Creates a new WorkloadService instance, for listing workloads (with optional cost and
|
|
143
150
|
# performance metrics). Needs the private API key.
|
|
144
151
|
def workload_service
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: coolhand
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.11.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Michael Carroll
|
|
@@ -9,7 +9,7 @@ authors:
|
|
|
9
9
|
autorequire:
|
|
10
10
|
bindir: exe
|
|
11
11
|
cert_chain: []
|
|
12
|
-
date: 2026-
|
|
12
|
+
date: 2026-11-08 00:00:00.000000000 Z
|
|
13
13
|
dependencies:
|
|
14
14
|
- !ruby/object:Gem::Dependency
|
|
15
15
|
name: base64
|
|
@@ -54,6 +54,7 @@ files:
|
|
|
54
54
|
- docs/help-center.md
|
|
55
55
|
- docs/log-search.md
|
|
56
56
|
- docs/openai.md
|
|
57
|
+
- docs/step-templates.md
|
|
57
58
|
- docs/template-search.md
|
|
58
59
|
- docs/vertex.md
|
|
59
60
|
- docs/workload-search.md
|
|
@@ -88,6 +89,8 @@ files:
|
|
|
88
89
|
- lib/coolhand/read_query.rb
|
|
89
90
|
- lib/coolhand/read_requests.rb
|
|
90
91
|
- lib/coolhand/response_warnings.rb
|
|
92
|
+
- lib/coolhand/step_template_service.rb
|
|
93
|
+
- lib/coolhand/stream_read.rb
|
|
91
94
|
- lib/coolhand/string_body_redactor.rb
|
|
92
95
|
- lib/coolhand/template_service.rb
|
|
93
96
|
- lib/coolhand/version.rb
|