coolhand 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/AGENTS.harness.md +139 -0
- data/CHANGELOG.md +64 -0
- data/README.md +46 -105
- data/docs/configuration.md +40 -0
- data/docs/openai.md +91 -0
- data/docs/template-search.md +218 -0
- data/docs/vertex.md +54 -0
- data/lib/coolhand/api_service.rb +56 -16
- data/lib/coolhand/base_interceptor.rb +58 -38
- data/lib/coolhand/configuration.rb +39 -6
- data/lib/coolhand/default_exclude_api_patterns.yml +3 -2
- data/lib/coolhand/default_intercept_addresses.yml +10 -5
- data/lib/coolhand/default_intercept_path_patterns.yml +12 -0
- data/lib/coolhand/errors.rb +19 -0
- data/lib/coolhand/logger_service.rb +1 -1
- data/lib/coolhand/net_http_interceptor.rb +154 -24
- data/lib/coolhand/open_ai/batch_result_processor.rb +3 -1
- data/lib/coolhand/open_ai/webhook_id_store.rb +45 -0
- data/lib/coolhand/open_ai/webhook_validator.rb +50 -12
- data/lib/coolhand/pagination.rb +82 -0
- data/lib/coolhand/read_requests.rb +77 -0
- data/lib/coolhand/template_service.rb +76 -0
- data/lib/coolhand/version.rb +1 -1
- data/lib/coolhand/vertex/batch_result_processor.rb +97 -40
- data/lib/coolhand/webhook_interceptor.rb +11 -0
- data/lib/coolhand.rb +10 -5
- metadata +16 -9
- data/.claude/skills/loop-review/SKILL.md +0 -112
- data/.claude/skills/prep-release/SKILL.md +0 -160
- data/.idea/coolhand-ruby.iml +0 -6
- data/CLAUDE.md +0 -47
|
@@ -0,0 +1,218 @@
|
|
|
1
|
+
# Reading Templates (Search + Get)
|
|
2
|
+
|
|
3
|
+
`Coolhand::TemplateService` reads back the LLM request templates your logs are matched against.
|
|
4
|
+
It wraps two read-only endpoints:
|
|
5
|
+
|
|
6
|
+
| method | endpoint |
|
|
7
|
+
|---|---|
|
|
8
|
+
| `search_templates` | `GET /api/v2/llm_request_templates` |
|
|
9
|
+
| `get_template` | `GET /api/v2/llm_request_templates/{id}` |
|
|
10
|
+
|
|
11
|
+
Both require your **private** API key. The public key is write-only on this API and is rejected
|
|
12
|
+
exactly like an invalid one.
|
|
13
|
+
|
|
14
|
+
```ruby
|
|
15
|
+
require "coolhand"
|
|
16
|
+
|
|
17
|
+
Coolhand.configure do |config|
|
|
18
|
+
config.api_key = ENV.fetch("COOLHAND_PRIVATE_API_KEY")
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
templates = Coolhand.template_service
|
|
22
|
+
|
|
23
|
+
result = templates.search_templates(search: "summar", status: "published")
|
|
24
|
+
result.templates.each { |template| puts "#{template[:name]} (#{template[:log_count]} logs)" }
|
|
25
|
+
|
|
26
|
+
detail = templates.get_template(result.templates.first[:id])
|
|
27
|
+
puts detail[:user_prompt_pattern]
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## What this is not
|
|
31
|
+
|
|
32
|
+
This is **not** a port of the `search_templates` MCP tool, and the two do not agree:
|
|
33
|
+
|
|
34
|
+
- `log_count` here counts only directly-collected client logs — the same records
|
|
35
|
+
`GET /api/v2/llm_request_logs?template_id=…` returns. Evals, bakeoff comparisons and synthetic
|
|
36
|
+
logs are excluded, so this number is often lower than the MCP tool's.
|
|
37
|
+
- Templates whose workload has been archived are **returned** here, not hidden, so the list agrees
|
|
38
|
+
with `get_template`, which can always fetch such a template by id. Narrow with `workload_id`
|
|
39
|
+
instead.
|
|
40
|
+
|
|
41
|
+
Template *creation, update and deprecation* stay on the MCP surface. This REST surface is
|
|
42
|
+
read-only, and there is no version-history sub-resource.
|
|
43
|
+
|
|
44
|
+
## `search_templates(search: nil, workload_id: nil, status: nil, include_deprecated: nil, include_system: nil, page: nil, per: nil)`
|
|
45
|
+
|
|
46
|
+
Search is a *parameter* on the list endpoint rather than a route of its own, so this is one method
|
|
47
|
+
rather than a list/search pair.
|
|
48
|
+
|
|
49
|
+
All filters are optional keyword arguments, and their names are the wire names — Ruby's convention
|
|
50
|
+
and this API's already agree, so there is no second spelling to translate through.
|
|
51
|
+
|
|
52
|
+
| keyword | type | notes |
|
|
53
|
+
|---|---|---|
|
|
54
|
+
| `search` | String | Case-insensitive **literal** substring match on the template name. `%` and `_` are escaped server-side and match themselves — do not escape them again. |
|
|
55
|
+
| `workload_id` | String | Workload hashid. One that does not decode, or that belongs to another client, is a `422` rather than an empty list. |
|
|
56
|
+
| `status` | String | `"draft"`, `"published"` or `"failure"`. Any other non-empty value is a `422` from the server; empty is treated as no filter. |
|
|
57
|
+
| `include_deprecated` | Boolean | Include templates with a non-null `deprecated_at`. Defaults to `false` server-side. |
|
|
58
|
+
| `include_system` | Boolean | Include the `Unmatched` / `Ignored API Calls` buckets. Defaults to `false` server-side. |
|
|
59
|
+
| `page` | Integer | 1-based. |
|
|
60
|
+
| `per` | Integer | Default 25, max 100, both enforced server-side. |
|
|
61
|
+
|
|
62
|
+
Two things it deliberately does **not** do:
|
|
63
|
+
|
|
64
|
+
- **There is no `client_id` keyword.** The client is always derived from the authenticating API key
|
|
65
|
+
and cannot be supplied, so passing one raises `ArgumentError` rather than reaching the wire.
|
|
66
|
+
- **`status` is not checked against a client-side allowlist.** The API definition enumerates the
|
|
67
|
+
values on the *query parameter*, but leaves the `status` field on the *response* an unconstrained
|
|
68
|
+
string. Sending an unrecognised value gets you the server's `422`; a status the server gains
|
|
69
|
+
later works without a gem release.
|
|
70
|
+
|
|
71
|
+
`per_page` is accepted on the wire as an alias for `per`. This gem only ever sends `per` — one knob
|
|
72
|
+
is enough.
|
|
73
|
+
|
|
74
|
+
### Return value
|
|
75
|
+
|
|
76
|
+
A `Coolhand::TemplateSearchResult` — a `Struct`, so `#templates`, `#pagination`, `#to_h` and
|
|
77
|
+
pattern matching all work:
|
|
78
|
+
|
|
79
|
+
```ruby
|
|
80
|
+
result = Coolhand.template_service.search_templates(include_system: true)
|
|
81
|
+
|
|
82
|
+
result.templates # => [{ id: "kp9npvc8qq2q", name: "Unmatched", ... }, ...]
|
|
83
|
+
result.pagination # => #<struct Coolhand::Pagination current_page=1, ...>
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
**Rows are plain Hashes with Symbol keys**, the same shape `create_feedback` and `create_log`
|
|
87
|
+
already return. They are handed back exactly as the API rendered them rather than being copied into
|
|
88
|
+
a value object, so a field the server adds later reaches you instead of being silently dropped on
|
|
89
|
+
the way through. That is also why there is no `system_template?` predicate: read the wire field,
|
|
90
|
+
`template[:system_template]`.
|
|
91
|
+
|
|
92
|
+
Rows are ordered newest first (`created_at DESC`, with a primary-key tiebreaker so paging is stable
|
|
93
|
+
across requests). Each carries:
|
|
94
|
+
|
|
95
|
+
| field | type | notes |
|
|
96
|
+
|---|---|---|
|
|
97
|
+
| `:id` | String | Hashid, never the integer primary key. |
|
|
98
|
+
| `:name` | String | Never null, but may be blank on a draft. |
|
|
99
|
+
| `:status` | String or nil | `draft` / `published` / `failure`. |
|
|
100
|
+
| `:version` | String or nil | |
|
|
101
|
+
| `:group` | String or nil | `chat`, `user_prompt`, `user_prompt_with_system_prompt`, `embedding`, `other`. |
|
|
102
|
+
| `:workload_id` | String | Workload hashid; never null. |
|
|
103
|
+
| `:workload_name` | String | Never null. |
|
|
104
|
+
| `:system_template` | Boolean | True for the `Unmatched` / `Ignored API Calls` buckets. |
|
|
105
|
+
| `:deprecated_at` | String or nil | ISO-8601 UTC. Non-null means the template has been superseded. |
|
|
106
|
+
| `:log_count` | Integer | Directly-collected client logs only — see [What this is not](#what-this-is-not). |
|
|
107
|
+
| `:created_at` | String | ISO-8601 UTC. |
|
|
108
|
+
| `:updated_at` | String | ISO-8601 UTC. |
|
|
109
|
+
|
|
110
|
+
**Prompt patterns are not in the list.** They come from `get_template` only.
|
|
111
|
+
|
|
112
|
+
`pagination` is a `Coolhand::Pagination` built from the endpoint's `X-Page`, `X-Per-Page`,
|
|
113
|
+
`X-Total-Count` and `X-Total-Pages` response headers — never from the size of the array, which only
|
|
114
|
+
ever describes the page in hand. Unlike `/llm_request_logs` there is no `include_total` opt-out
|
|
115
|
+
here; the headers are always sent.
|
|
116
|
+
|
|
117
|
+
| field | type |
|
|
118
|
+
|---|---|
|
|
119
|
+
| `current_page` | Integer |
|
|
120
|
+
| `per_page` | Integer |
|
|
121
|
+
| `total_count` | Integer |
|
|
122
|
+
| `total_pages` | Integer |
|
|
123
|
+
| `has_next_page` | Boolean |
|
|
124
|
+
| `has_prev_page` | Boolean |
|
|
125
|
+
|
|
126
|
+
### System templates and the empty default list
|
|
127
|
+
|
|
128
|
+
Every Coolhand client is created with two system buckets, `Unmatched` and `Ignored API Calls`. They
|
|
129
|
+
are hidden unless you pass `include_system: true`, so a client with no templates of its own gets an
|
|
130
|
+
empty array rather than those two rows. `Unmatched` is what you inspect when logs are misrouting:
|
|
131
|
+
|
|
132
|
+
```ruby
|
|
133
|
+
unmatched = Coolhand.template_service
|
|
134
|
+
.search_templates(include_system: true, search: "unmatched")
|
|
135
|
+
.templates
|
|
136
|
+
.first
|
|
137
|
+
|
|
138
|
+
puts unmatched[:log_count]
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
## `get_template(id)`
|
|
142
|
+
|
|
143
|
+
`id` is the template hashid — the `:id` field from `search_templates`.
|
|
144
|
+
|
|
145
|
+
Returns a Hash with every list field above, **plus** the full untruncated regexes the list omits:
|
|
146
|
+
|
|
147
|
+
| field | type |
|
|
148
|
+
|---|---|
|
|
149
|
+
| `:user_prompt_pattern` | String or nil |
|
|
150
|
+
| `:system_prompt_pattern` | String or nil |
|
|
151
|
+
|
|
152
|
+
Unlike the list, this filters on nothing but client ownership: a deprecated or system template is
|
|
153
|
+
reachable by id with **no opt-in flag**, because inspecting one of those is the usual reason to
|
|
154
|
+
fetch a template directly.
|
|
155
|
+
|
|
156
|
+
## Errors
|
|
157
|
+
|
|
158
|
+
The write methods in this gem (`create_feedback`, `create_log`, `send_llm_request_log`) log a
|
|
159
|
+
failure and return `nil` — instrumentation must never be the reason a host app falls over. **The
|
|
160
|
+
read methods are the opposite and raise**, because a caller that asked for data has to be able to
|
|
161
|
+
tell a `404` from a timeout from a genuinely empty result.
|
|
162
|
+
|
|
163
|
+
| raised | when |
|
|
164
|
+
|---|---|
|
|
165
|
+
| `Coolhand::HttpError` | The server answered with a non-2xx status. `#status` is the HTTP status code and `#body` the response body. |
|
|
166
|
+
| `Coolhand::Error` | No API key configured, a transport failure, a body that is not valid JSON, or a blank `id` passed to `get_template`. |
|
|
167
|
+
|
|
168
|
+
`Coolhand::HttpError` is a `Coolhand::Error`, so `rescue Coolhand::Error` catches both.
|
|
169
|
+
|
|
170
|
+
**Branch on `#status`, never on the message:**
|
|
171
|
+
|
|
172
|
+
```ruby
|
|
173
|
+
begin
|
|
174
|
+
result = Coolhand.template_service.search_templates
|
|
175
|
+
rescue Coolhand::HttpError => e
|
|
176
|
+
case e.status
|
|
177
|
+
when 401 then warn "Private API key required — the public key cannot read"
|
|
178
|
+
when 422 then warn "Bad filter: #{e.body}"
|
|
179
|
+
when 504 then warn "Timed out aggregating log_count — narrow the query and retry"
|
|
180
|
+
else raise
|
|
181
|
+
end
|
|
182
|
+
end
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
| status | meaning |
|
|
186
|
+
|---|---|
|
|
187
|
+
| `401` | Missing, invalid, or public API key. |
|
|
188
|
+
| `404` | `get_template` only. Unknown id, **or** one belonging to another client — existence is not disclosed, so this is never a `403`. |
|
|
189
|
+
| `422` | Unrecognised `status`, or a `workload_id` that does not decode or belongs to another client. |
|
|
190
|
+
| `504` | See below. |
|
|
191
|
+
|
|
192
|
+
### `504` is expected, and retryable
|
|
193
|
+
|
|
194
|
+
`log_count` aggregates over `llm_request_logs`, so its cost scales with how many logs the matched
|
|
195
|
+
templates hold — the `Unmatched` bucket can hold every log that never matched a template. Every
|
|
196
|
+
query behind these responses is bounded by a 10-second statement timeout, and exceeding it returns
|
|
197
|
+
`504` rather than hanging. It reaches you as an `HttpError` with `status == 504`, not folded into a
|
|
198
|
+
generic server error, precisely so you can narrow and retry:
|
|
199
|
+
|
|
200
|
+
```ruby
|
|
201
|
+
templates = Coolhand.template_service
|
|
202
|
+
|
|
203
|
+
begin
|
|
204
|
+
templates.search_templates(include_system: true)
|
|
205
|
+
rescue Coolhand::HttpError => e
|
|
206
|
+
raise unless e.status == 504
|
|
207
|
+
|
|
208
|
+
# Narrow the aggregate: one workload at a time, smaller pages.
|
|
209
|
+
templates.search_templates(include_system: true, workload_id: workload_id, per: 10)
|
|
210
|
+
end
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
This is also why the read path waits far longer than the write path before giving up. A write
|
|
214
|
+
times out after 5 seconds because it runs inline in your app's own request. A read allows 60: the
|
|
215
|
+
server bounds each *statement* behind a response at 10 seconds, but one response runs several, so
|
|
216
|
+
a slow-but-working `include_system=true` call measured 7-15 seconds against a development database
|
|
217
|
+
while returning `200` every time. Giving up sooner would report a working endpoint as a transport
|
|
218
|
+
failure, and would pre-empt the `504` you are meant to see and retry.
|
data/docs/vertex.md
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# Google Vertex AI Batch Result Logging
|
|
2
|
+
|
|
3
|
+
Log completed Google Vertex AI batch prediction job results the same way regular synchronous calls are logged. Unlike the [OpenAI batch handler](openai.md), this is not webhook-driven — you call `Coolhand::Vertex::BatchResultProcessor` directly from your own batch-job callback/polling code.
|
|
4
|
+
|
|
5
|
+
For monitoring regular (non-batch) Vertex AI calls, no extra setup is required beyond `Coolhand.configure` — `aiplatform.googleapis.com` is intercepted by default.
|
|
6
|
+
|
|
7
|
+
Requires Rails — `Coolhand::Vertex::BatchResultProcessor` logs via `Rails.logger` internally. `config.capture = false` and `Coolhand.without_capture` do not suppress these logs: unlike the passive Net::HTTP interceptor, calling this processor is an explicit, deliberate act, so it always sends.
|
|
8
|
+
|
|
9
|
+
## Usage
|
|
10
|
+
|
|
11
|
+
```ruby
|
|
12
|
+
require "coolhand/vertex/batch_result_processor"
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
- Call the `Coolhand::Vertex::BatchResultProcessor` service with `batch_info` and the downloaded batch results.
|
|
16
|
+
- Optionally pass `model:` to populate the logged `model` field. It falls back to a `"model"` key in `batch_info` if present, and is omitted from the payload entirely when neither is available. If the value (from either source) looks like a Vertex model resource path (`publishers/<publisher>/models/<id>` or `projects/<project>/locations/<location>/models/<id>`), it's normalized down to just the trailing id/version before being logged; any other string is sent as-is.
|
|
17
|
+
- `batch_info["name"]` must be the exact Vertex job resource name (`projects/<project>/locations/<location>/batchPredictionJobs/<job-id>`), and `batch_info["startTime"]`/`["endTime"]` must be valid ISO 8601 timestamps — a batch with a missing or malformed resource name or timestamps is skipped entirely (logged as an error) rather than sent with a broken URL.
|
|
18
|
+
- If `endTime` precedes `startTime` (e.g. clock skew), the batch is still logged — `duration_ms` is clamped to `0` and a warning is logged — rather than dropping the results.
|
|
19
|
+
- Each element of the `batch_results` array passed to `.call` must be a `Hash` with a `"request"` and/or `"response"` key (the request/response body to log for that item — either key alone is accepted). Any other shape is treated as malformed and skipped, with the skip counted in a `Rails.logger.warn` summary rather than raised.
|
|
20
|
+
|
|
21
|
+
## Minimal example
|
|
22
|
+
|
|
23
|
+
Only the key lines are shown — wire this into your own batch callback service.
|
|
24
|
+
|
|
25
|
+
```ruby
|
|
26
|
+
class Vertex::BatchCallbackProcessor < BaseService
|
|
27
|
+
option :batch_request, model: BatchApiRequest
|
|
28
|
+
option :batch_info
|
|
29
|
+
|
|
30
|
+
def call
|
|
31
|
+
case batch_info["state"]
|
|
32
|
+
when "JOB_STATE_PENDING"
|
|
33
|
+
nil
|
|
34
|
+
when "JOB_STATE_RUNNING", "JOB_STATE_QUEUED"
|
|
35
|
+
batch_request.update!(status: "processing")
|
|
36
|
+
|
|
37
|
+
Coolhand::Vertex::BatchResultProcessor.new(batch_info:).call
|
|
38
|
+
when "JOB_STATE_SUCCEEDED"
|
|
39
|
+
output_file_id = batch_info["outputInfo"]["gcsOutputDirectory"]
|
|
40
|
+
results = download_batch_results(output_file_id)
|
|
41
|
+
results.each { |batch_item| process_batch_result(batch_item) }
|
|
42
|
+
|
|
43
|
+
batch_request.update!(status: "completed", completed_at: Time.current, output_file_id:)
|
|
44
|
+
|
|
45
|
+
Coolhand::Vertex::BatchResultProcessor.new(batch_info:, model: batch_request.llm_model).call(results)
|
|
46
|
+
|
|
47
|
+
# Clean up GCS files after successful processing
|
|
48
|
+
cleanup_gcs_files(output_file_id)
|
|
49
|
+
when "JOB_STATE_FAILED"
|
|
50
|
+
handle_failed_batch(batch_info["error"]["message"])
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
```
|
data/lib/coolhand/api_service.rb
CHANGED
|
@@ -4,9 +4,19 @@ require "net/http"
|
|
|
4
4
|
require "uri"
|
|
5
5
|
require "json"
|
|
6
6
|
require_relative "collector"
|
|
7
|
+
require_relative "errors"
|
|
8
|
+
require_relative "read_requests"
|
|
7
9
|
|
|
8
10
|
module Coolhand
|
|
9
11
|
class ApiService
|
|
12
|
+
# The GET half of this class. See Coolhand::ReadRequests for why reads raise where writes
|
|
13
|
+
# log and return nil.
|
|
14
|
+
include ReadRequests
|
|
15
|
+
|
|
16
|
+
# Caps how much of a failed response body is interpolated into a log line or an exception
|
|
17
|
+
# message. Without it an oversized body from a proxy or gateway becomes the message.
|
|
18
|
+
ERROR_BODY_LIMIT = 2000
|
|
19
|
+
|
|
10
20
|
attr_reader :api_endpoint
|
|
11
21
|
|
|
12
22
|
def initialize(endpoint = "v2/llm_request_logs")
|
|
@@ -77,12 +87,7 @@ module Coolhand
|
|
|
77
87
|
http.read_timeout = 5
|
|
78
88
|
|
|
79
89
|
request = Net::HTTP::Post.new(uri.request_uri)
|
|
80
|
-
|
|
81
|
-
headers.each do |key, value|
|
|
82
|
-
# Ensure header values are UTF-8 encoded
|
|
83
|
-
encoded_value = value.is_a?(String) ? value.dup.force_encoding("UTF-8") : value
|
|
84
|
-
request[key] = encoded_value
|
|
85
|
-
end
|
|
90
|
+
apply_headers(request, create_request_options(payload))
|
|
86
91
|
|
|
87
92
|
# Clean payload and ensure UTF-8 encoding before JSON generation
|
|
88
93
|
cleaned_payload = sanitize_payload_for_json(payload)
|
|
@@ -92,21 +97,20 @@ module Coolhand
|
|
|
92
97
|
request.body = json_body.force_encoding("UTF-8")
|
|
93
98
|
|
|
94
99
|
begin
|
|
95
|
-
|
|
100
|
+
# This request goes through the same patched Net::HTTP as the host
|
|
101
|
+
# app's real LLM calls. Without this, a base_url/intercept_addresses
|
|
102
|
+
# configuration that also matches Coolhand's own API host would
|
|
103
|
+
# re-intercept this log-shipping call, generating a second log
|
|
104
|
+
# request that itself gets intercepted, and so on — unbounded
|
|
105
|
+
# request amplification.
|
|
106
|
+
response = Coolhand.without_capture { http.request(request) }
|
|
96
107
|
|
|
97
108
|
if response.is_a?(Net::HTTPSuccess)
|
|
98
109
|
result = JSON.parse(response.body, symbolize_names: true)
|
|
99
110
|
log success_message
|
|
100
111
|
result
|
|
101
112
|
else
|
|
102
|
-
|
|
103
|
-
# Only show first part of HTML error pages
|
|
104
|
-
error_msg = if body&.include?("<!DOCTYPE html>")
|
|
105
|
-
"#{body[0..200]}... [HTML error page truncated]"
|
|
106
|
-
else
|
|
107
|
-
body
|
|
108
|
-
end
|
|
109
|
-
log "❌ Request failed: #{response.code} - #{error_msg}"
|
|
113
|
+
log "❌ Request failed: #{response.code} - #{format_error_body(response.body)}"
|
|
110
114
|
nil
|
|
111
115
|
end
|
|
112
116
|
rescue StandardError => e
|
|
@@ -204,6 +208,26 @@ module Coolhand
|
|
|
204
208
|
|
|
205
209
|
private
|
|
206
210
|
|
|
211
|
+
# Shared by the write path's failure log and the read path's raised message, so a large
|
|
212
|
+
# response body never dumps a whole document into either. An HTML error page (a proxy's 502,
|
|
213
|
+
# say) is cut short hard; anything else keeps enough to diagnose from and no more.
|
|
214
|
+
def format_error_body(body)
|
|
215
|
+
return nil if body.nil?
|
|
216
|
+
|
|
217
|
+
text = body.dup.force_encoding("UTF-8")
|
|
218
|
+
return "#{text[0..200]}... [HTML error page truncated]" if text.include?("<!DOCTYPE html>")
|
|
219
|
+
return text if text.length <= ERROR_BODY_LIMIT
|
|
220
|
+
|
|
221
|
+
"#{text[0, ERROR_BODY_LIMIT]}... [truncated]"
|
|
222
|
+
end
|
|
223
|
+
|
|
224
|
+
def apply_headers(request, headers)
|
|
225
|
+
headers.each do |key, value|
|
|
226
|
+
# Net::HTTP rejects header values that are not UTF-8.
|
|
227
|
+
request[key] = value.is_a?(String) ? value.dup.force_encoding("UTF-8") : value
|
|
228
|
+
end
|
|
229
|
+
end
|
|
230
|
+
|
|
207
231
|
def missing_api_key?
|
|
208
232
|
return false if Coolhand.required_field?(api_key)
|
|
209
233
|
|
|
@@ -275,8 +299,24 @@ module Coolhand
|
|
|
275
299
|
return if silent
|
|
276
300
|
|
|
277
301
|
puts "\n🎉 LOGGING OpenAI API Call #{@api_endpoint}"
|
|
278
|
-
|
|
302
|
+
|
|
303
|
+
if debug_mode?
|
|
304
|
+
puts captured_data
|
|
305
|
+
else
|
|
306
|
+
puts request_body_summary(captured_data)
|
|
307
|
+
end
|
|
308
|
+
|
|
279
309
|
puts "📤 Sending to: #{@api_endpoint}"
|
|
280
310
|
end
|
|
311
|
+
|
|
312
|
+
def request_body_summary(captured_data)
|
|
313
|
+
return "captured_data: (unavailable)" unless captured_data.is_a?(Hash)
|
|
314
|
+
|
|
315
|
+
id = captured_data[:id] || captured_data["id"] || "N/A"
|
|
316
|
+
body = captured_data[:request_body] || captured_data["request_body"]
|
|
317
|
+
"id: #{id}, request_body: #{body.to_json.bytesize} bytes"
|
|
318
|
+
rescue StandardError
|
|
319
|
+
"captured_data: (unavailable)"
|
|
320
|
+
end
|
|
281
321
|
end
|
|
282
322
|
end
|
|
@@ -7,10 +7,11 @@ module Coolhand
|
|
|
7
7
|
|
|
8
8
|
# Matches any header whose *name* signals sensitive content, regardless of
|
|
9
9
|
# provider — covers known keys (x-api-key, x-goog-api-key, openai-api-key),
|
|
10
|
-
# AWS SigV4 session tokens (x-amz-security-token),
|
|
11
|
-
#
|
|
12
|
-
#
|
|
13
|
-
|
|
10
|
+
# AWS SigV4 session tokens (x-amz-security-token), session/CSRF cookies
|
|
11
|
+
# (cookie, set-cookie), and future/unknown providers using a
|
|
12
|
+
# similarly-named header. Shared with LoggerService so the two logging
|
|
13
|
+
# paths (interceptor + webhook forwarding) stay consistent.
|
|
14
|
+
SENSITIVE_HEADER_PATTERN = /key|token|secret|signature|authorization|cookie/i
|
|
14
15
|
|
|
15
16
|
def sanitize_headers(headers)
|
|
16
17
|
return {} if headers.nil?
|
|
@@ -22,7 +23,9 @@ module Coolhand
|
|
|
22
23
|
begin
|
|
23
24
|
headers.to_hash.transform_keys(&:to_s).transform_values { |v| normalize_header_value(v) }
|
|
24
25
|
rescue StandardError
|
|
25
|
-
#
|
|
26
|
+
# Deliberately fails closed to an empty hash rather than trying each_header/each on
|
|
27
|
+
# this object next — an object whose own to_hash raises is untrustworthy enough that
|
|
28
|
+
# guessing at another enumeration strategy isn't worth the risk of a second failure.
|
|
26
29
|
nil
|
|
27
30
|
end
|
|
28
31
|
elsif headers.respond_to?(:each_header)
|
|
@@ -69,50 +72,67 @@ module Coolhand
|
|
|
69
72
|
end
|
|
70
73
|
end
|
|
71
74
|
|
|
75
|
+
# Matches query param *names* that signal sensitive content, the same way
|
|
76
|
+
# SENSITIVE_HEADER_PATTERN does for headers — covers exact params this
|
|
77
|
+
# gem already knew about (key/token/secret) as well as presigned-URL
|
|
78
|
+
# credential params used by AWS (X-Amz-Signature, X-Amz-Credential,
|
|
79
|
+
# X-Amz-Security-Token) and Google Cloud (X-Goog-Signature,
|
|
80
|
+
# X-Goog-Credential) storage APIs.
|
|
81
|
+
SENSITIVE_QUERY_PARAM_PATTERN = /key|token|secret|sig|credential|password|auth/i
|
|
82
|
+
|
|
72
83
|
def sanitize_url(url)
|
|
73
84
|
uri = URI.parse(url)
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
redacted = true
|
|
82
|
-
[n, "[REDACTED]"]
|
|
83
|
-
else
|
|
84
|
-
[n, v]
|
|
85
|
-
end
|
|
85
|
+
modified = false
|
|
86
|
+
|
|
87
|
+
if uri.userinfo
|
|
88
|
+
# URI userinfo syntax disallows "[" / "]", so this can't reuse the [REDACTED]
|
|
89
|
+
# placeholder used elsewhere.
|
|
90
|
+
uri.userinfo = "REDACTED"
|
|
91
|
+
modified = true
|
|
86
92
|
end
|
|
87
93
|
|
|
88
|
-
if
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
94
|
+
if uri.query
|
|
95
|
+
params = URI.decode_www_form(uri.query)
|
|
96
|
+
redacted_query = false
|
|
97
|
+
params.map! do |n, v|
|
|
98
|
+
if n.match?(SENSITIVE_QUERY_PARAM_PATTERN)
|
|
99
|
+
redacted_query = true
|
|
100
|
+
[n, "[REDACTED]"]
|
|
101
|
+
else
|
|
102
|
+
[n, v]
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
if redacted_query
|
|
106
|
+
uri.query = URI.encode_www_form(params)
|
|
107
|
+
modified = true
|
|
108
|
+
end
|
|
93
109
|
end
|
|
110
|
+
|
|
111
|
+
modified ? uri.to_s : url
|
|
94
112
|
rescue URI::InvalidURIError
|
|
95
113
|
url
|
|
96
114
|
end
|
|
97
115
|
|
|
98
116
|
def send_complete_request_log(request_id:, method:, url:, request_headers:, request_body:, response_headers:,
|
|
99
|
-
response_body:, status_code:, start_time:, end_time:, duration_ms:, is_streaming:)
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
is_streaming: is_streaming
|
|
114
|
-
}
|
|
117
|
+
response_body:, status_code:, start_time:, end_time:, duration_ms:, is_streaming:, source_api: nil, model: nil)
|
|
118
|
+
raw_request = {
|
|
119
|
+
id: request_id,
|
|
120
|
+
timestamp: start_time.iso8601,
|
|
121
|
+
method: method.to_s.downcase,
|
|
122
|
+
url: sanitize_url(url),
|
|
123
|
+
headers: sanitize_headers(request_headers),
|
|
124
|
+
request_body: request_body,
|
|
125
|
+
response_headers: sanitize_headers(response_headers),
|
|
126
|
+
response_body: response_body,
|
|
127
|
+
status_code: status_code,
|
|
128
|
+
duration_ms: duration_ms,
|
|
129
|
+
completed_at: end_time.iso8601,
|
|
130
|
+
is_streaming: is_streaming
|
|
115
131
|
}
|
|
132
|
+
raw_request[:source_api] = source_api if Coolhand.required_field?(source_api)
|
|
133
|
+
raw_request[:model] = model if Coolhand.required_field?(model)
|
|
134
|
+
|
|
135
|
+
request_data = { raw_request: raw_request }
|
|
116
136
|
|
|
117
137
|
api_service = Coolhand::ApiService.new
|
|
118
138
|
api_service.send_llm_request_log(request_data)
|
|
@@ -3,22 +3,34 @@
|
|
|
3
3
|
require "yaml"
|
|
4
4
|
require "uri"
|
|
5
5
|
|
|
6
|
+
require_relative "open_ai/webhook_id_store"
|
|
7
|
+
|
|
6
8
|
module Coolhand
|
|
7
9
|
# Handles all configuration settings for the gem.
|
|
8
10
|
class Configuration
|
|
9
|
-
DEFAULT_EXCLUDE_API_PATTERNS = YAML.
|
|
11
|
+
DEFAULT_EXCLUDE_API_PATTERNS = YAML.safe_load_file(
|
|
10
12
|
File.join(__dir__, "default_exclude_api_patterns.yml")
|
|
11
13
|
).freeze
|
|
12
14
|
|
|
13
|
-
DEFAULT_INTERCEPT_ADDRESSES = YAML.
|
|
15
|
+
DEFAULT_INTERCEPT_ADDRESSES = YAML.safe_load_file(
|
|
14
16
|
File.join(__dir__, "default_intercept_addresses.yml")
|
|
15
17
|
).freeze
|
|
16
18
|
|
|
19
|
+
DEFAULT_INTERCEPT_PATH_PATTERNS = YAML.safe_load_file(
|
|
20
|
+
File.join(__dir__, "default_intercept_path_patterns.yml")
|
|
21
|
+
).freeze
|
|
22
|
+
|
|
17
23
|
BASE_URL_ERROR_MSG = "base_url must use https:// (or http://localhost / http://127.0.0.1 for local dev)"
|
|
18
24
|
LOOPBACK_HOSTS = %w[localhost 127.0.0.1 ::1].freeze
|
|
19
25
|
|
|
20
|
-
|
|
21
|
-
|
|
26
|
+
# 1 MB — generous for real chat/completion payloads, small enough to stop
|
|
27
|
+
# a multi-MB file upload from being logged in full when its content-type
|
|
28
|
+
# happens to look JSON-ish (or is unset).
|
|
29
|
+
DEFAULT_MAX_CAPTURED_BODY_BYTES = 1_000_000
|
|
30
|
+
|
|
31
|
+
attr_accessor :api_key, :environment, :silent, :debug_mode, :capture, :exclude_api_patterns, :enabled,
|
|
32
|
+
:max_captured_body_bytes, :webhook_replay_tolerance_seconds, :webhook_id_store
|
|
33
|
+
attr_reader :intercept_addresses, :intercept_path_patterns, :base_url
|
|
22
34
|
|
|
23
35
|
def initialize
|
|
24
36
|
# Set defaults
|
|
@@ -26,20 +38,41 @@ module Coolhand
|
|
|
26
38
|
@api_key = nil
|
|
27
39
|
@silent = false
|
|
28
40
|
@intercept_addresses = DEFAULT_INTERCEPT_ADDRESSES.dup
|
|
41
|
+
@intercept_path_patterns = DEFAULT_INTERCEPT_PATH_PATTERNS.dup
|
|
29
42
|
self.base_url = "https://coolhandlabs.com/api"
|
|
30
43
|
@debug_mode = false
|
|
31
44
|
@capture = true
|
|
32
45
|
@exclude_api_patterns = DEFAULT_EXCLUDE_API_PATTERNS.dup
|
|
33
46
|
@enabled = true
|
|
47
|
+
@max_captured_body_bytes = DEFAULT_MAX_CAPTURED_BODY_BYTES
|
|
48
|
+
@webhook_replay_tolerance_seconds = 300
|
|
49
|
+
@webhook_id_store = Coolhand::OpenAi::WebhookIdStore.new
|
|
34
50
|
end
|
|
35
51
|
|
|
36
|
-
#
|
|
52
|
+
# intercept_addresses is a required allow-list: NetHttpInterceptor#intercept? only
|
|
53
|
+
# captures a request whose URL matches an entry here, so an empty list would mean
|
|
54
|
+
# "never capture anything" (validate! deliberately rejects that). Unlike
|
|
55
|
+
# exclude_api_patterns, `= []` is not a supported way to disable it — use
|
|
56
|
+
# config.enabled = false or config.capture = false to disable capture entirely.
|
|
57
|
+
# nil/empty here is treated as "leave the current value alone" rather than cleared.
|
|
37
58
|
def intercept_addresses=(value)
|
|
38
|
-
|
|
59
|
+
if value.nil? || (value.is_a?(Array) && value.empty?)
|
|
60
|
+
Coolhand.log "⚠️ Coolhand: intercept_addresses = #{value.inspect} is ignored " \
|
|
61
|
+
"(would disable capture entirely) — keeping the current value. " \
|
|
62
|
+
"Use config.enabled = false or config.capture = false to disable capture."
|
|
63
|
+
return
|
|
64
|
+
end
|
|
39
65
|
|
|
40
66
|
@intercept_addresses = value.is_a?(Array) ? value : [value]
|
|
41
67
|
end
|
|
42
68
|
|
|
69
|
+
# Custom setter that preserves defaults when nil/empty array is provided
|
|
70
|
+
def intercept_path_patterns=(value)
|
|
71
|
+
return if value.nil? || (value.is_a?(Array) && value.empty?)
|
|
72
|
+
|
|
73
|
+
@intercept_path_patterns = value.is_a?(Array) ? value : [value]
|
|
74
|
+
end
|
|
75
|
+
|
|
43
76
|
def base_url=(value)
|
|
44
77
|
stripped = value&.sub(%r{/+\z}, "")
|
|
45
78
|
raise Error, BASE_URL_ERROR_MSG unless stripped.nil? || valid_base_url?(stripped)
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# Coolhand default exclude API patterns
|
|
2
|
-
# These substrings are matched against request
|
|
3
|
-
#
|
|
2
|
+
# These substrings are matched against the request's path (not the full URL or
|
|
3
|
+
# query string). A request whose path matches is skipped and not forwarded as
|
|
4
|
+
# an llm_request_log, regardless of whether it would otherwise match intercept_addresses.
|
|
4
5
|
#
|
|
5
6
|
# Users can extend defaults: c.exclude_api_patterns << "/myOperationalPath/"
|
|
6
7
|
# Users can override entirely: c.exclude_api_patterns = ["/only_this/"]
|
|
@@ -1,9 +1,15 @@
|
|
|
1
1
|
# Coolhand default intercept addresses
|
|
2
|
-
# These
|
|
3
|
-
#
|
|
2
|
+
# These are matched against the request's parsed host (exact match, or a
|
|
3
|
+
# dot-boundary suffix match, case-insensitive) to decide whether a request
|
|
4
|
+
# should be captured and forwarded as an llm_request_log. A single "*" in an
|
|
5
|
+
# entry matches exactly one host label — e.g. "bedrock-runtime.*.amazonaws.com"
|
|
6
|
+
# matches "bedrock-runtime.us-east-1.amazonaws.com" but nothing else.
|
|
4
7
|
#
|
|
5
8
|
# Users can extend defaults: c.intercept_addresses << "my.custom.api.com"
|
|
6
9
|
# Users can override entirely: c.intercept_addresses = ["only.this.com"]
|
|
10
|
+
# This list cannot be emptied — it's required (unlike exclude_api_patterns, which
|
|
11
|
+
# can be disabled with `= []`). To disable capture entirely, use
|
|
12
|
+
# c.enabled = false or c.capture = false instead.
|
|
7
13
|
|
|
8
14
|
- "api.openai.com"
|
|
9
15
|
- "api.anthropic.com"
|
|
@@ -11,9 +17,8 @@
|
|
|
11
17
|
- "generativelanguage.googleapis.com"
|
|
12
18
|
- "models.github.ai"
|
|
13
19
|
- "models.inference.ai.azure.com"
|
|
14
|
-
- ":generateContent"
|
|
15
|
-
- ":streamGenerateContent"
|
|
16
20
|
- "aiplatform.googleapis.com"
|
|
17
21
|
- "gateway.ai.cloudflare.com"
|
|
18
|
-
- "bedrock-runtime"
|
|
22
|
+
- "bedrock-runtime.*.amazonaws.com"
|
|
19
23
|
- "openrouter.ai"
|
|
24
|
+
- "opencode.ai"
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
# Colon-action path suffixes from Google's API Discovery convention
|
|
2
|
+
# (e.g. ".../v1beta/models/gemini-pro:generateContent"). These have no host
|
|
3
|
+
# component of their own, so — unlike intercept_addresses — they are only
|
|
4
|
+
# ever matched against the *path* of requests whose host is already
|
|
5
|
+
# googleapis.com or a googleapis.com subdomain. They can never match on
|
|
6
|
+
# their own against an unrelated/attacker-controlled host.
|
|
7
|
+
#
|
|
8
|
+
# Users can extend defaults: c.intercept_path_patterns << ":myAction"
|
|
9
|
+
# Users can override entirely: c.intercept_path_patterns = [":onlyThis"]
|
|
10
|
+
|
|
11
|
+
- ":generateContent"
|
|
12
|
+
- ":streamGenerateContent"
|