coolhand 0.5.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +33 -0
- data/README.md +16 -104
- data/docs/openai.md +60 -0
- data/docs/vertex.md +54 -0
- data/lib/coolhand/api_service.rb +7 -1
- data/lib/coolhand/base_interceptor.rb +32 -22
- data/lib/coolhand/configuration.rb +2 -2
- data/lib/coolhand/net_http_interceptor.rb +27 -4
- data/lib/coolhand/open_ai/batch_result_processor.rb +2 -0
- data/lib/coolhand/open_ai/webhook_validator.rb +6 -2
- data/lib/coolhand/version.rb +1 -1
- data/lib/coolhand/vertex/batch_result_processor.rb +97 -40
- data/lib/coolhand/webhook_interceptor.rb +11 -0
- data/lib/coolhand.rb +1 -1
- metadata +8 -9
- data/.claude/skills/loop-review/SKILL.md +0 -112
- data/.claude/skills/prep-release/SKILL.md +0 -160
- data/.idea/coolhand-ruby.iml +0 -6
- data/CLAUDE.md +0 -47
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 25393afdf95625b624ca2910cb3017685705529e9628a5d15cb3bdffdf122d52
|
|
4
|
+
data.tar.gz: 3e478c3b8494d5fef5bc87896fbf69cb9f93d088029b9a9745563c9a789b4473
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ad635f9377f6cb81967fc24b0c2aa9f9938c41e308c6b98511ee0139b9938e01ea766baf72a085700866c69ac50207da11e29f04d30a94ad3b487d3e22243d9e
|
|
7
|
+
data.tar.gz: d6ef0baf4e37d29b36a0d7a2d6a41822f551a0a753a2dc963c0da9eed60b90bf29b5f0015caa8a628c6f340ec9cb2b3ef2c83ecb9b59fbcec909d70f396603df
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.5.1] - 2026-08-02
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- **`Coolhand::Vertex::BatchResultProcessor.new` accepts an optional `model:` keyword** (falls back to a `"model"` key in `batch_info` when omitted) to populate the `model` field on batch-result logs — included only when one of those is available and non-blank, and omitted from the payload entirely otherwise (#76).
|
|
14
|
+
- **`Coolhand::BaseInterceptor.send_complete_request_log` accepts optional `source_api:`/`model:` keywords**, for synthetic/batch log types (like the Vertex processor above) that aren't backed by a real intercepted HTTP request (#76).
|
|
15
|
+
|
|
16
|
+
### Changed
|
|
17
|
+
- **Vertex batch results with a missing or malformed job resource name or unparseable `startTime`/`endTime` are no longer logged at all.** Previously the processor still sent a request log using whatever was in `batch_info["name"]` (even if blank), just with a broken URL. It now validates these fields and skips the log entirely — with a `Rails.logger.error` explaining why — rather than sending something unusable (#76).
|
|
18
|
+
- **A batch whose `endTime` precedes its `startTime` (e.g. clock skew) is still logged**, with `duration_ms` clamped to `0` and a warning logged, rather than dropped — unlike the missing/malformed cases above, the request/response content itself is still valid here (#76).
|
|
19
|
+
- **A Vertex batch result item that isn't a `Hash` with a `"request"` and/or `"response"` key is now rejected and logged as an error**, instead of being sent as a log with `nil` request/response bodies (#76).
|
|
20
|
+
- **The published gem no longer packages `.claude/`, `.idea/`, or `CLAUDE.md`** — these are repo/agent tooling, not part of the library.
|
|
21
|
+
|
|
22
|
+
### Security
|
|
23
|
+
- **A webhook processing error no longer leaves the request authenticated by default.** `Coolhand::WebhookInterceptor#intercept_batch_request` previously logged and swallowed *any* exception (a malformed webhook secret, a non-Hash JSON payload, a bug in `process_event`), returning normally without halting the Rails `before_action` chain — so the controller action still ran for a request whose signature was never confirmed valid. It now calls `head :unauthorized` in that rescue, matching the existing invalid-signature path, and validates the parsed webhook payload is a `Hash` before use.
|
|
24
|
+
- **A blank or whitespace-only OpenAI webhook secret is no longer usable as the HMAC key.** `Coolhand::OpenAi::WebhookValidator` previously only rejected `nil`/`false`, so `webhook_secret = ""` (e.g. an unset `ENV` var with a blank default) silently signed and verified requests with an empty key — a much weaker, easy-to-guess bypass than the already-known "secret not configured" path. It now uses the same `Coolhand.required_field?` blank-check used elsewhere in the gem.
|
|
25
|
+
- **`Cookie`/`Set-Cookie` headers are now redacted**, alongside the existing key/token/secret/signature/authorization patterns. Forwarding a Rails session cookie (a bearer credential) to Coolhand — e.g. via `LoggerService#forward_webhook(headers: request.headers)`, the pattern this gem's own docs show — was previously possible.
|
|
26
|
+
- **`BaseInterceptor.sanitize_url` now redacts a broader set of credential-shaped query parameters** (`sig`, `credential`, `password`, `auth`, in addition to the existing `key`/`token`/`secret`), matching the pattern-based approach already used for headers instead of a small exact-match list. This covers AWS SigV4 (`X-Amz-Signature`, `X-Amz-Credential`) and Google Cloud (`X-Goog-Signature`, `X-Goog-Credential`) presigned-URL parameters that were previously logged in full.
|
|
27
|
+
- **A failure capturing an intercepted request's body (e.g. an already-consumed `body_stream`) no longer prevents the real request from being attempted.** The capture happened before any error handling was in place, so an exception there escaped `NetHttpInterceptor#request` entirely — the host app's actual LLM call never happened. It's now rescued, logged, and treated as "body unavailable," and the real request still proceeds.
|
|
28
|
+
- **Streamed response bodies are no longer buffered into a thread-local for requests this gem never intercepted.** `ResponseInterceptor#read_body` previously buffered every block-form `read_body` call process-wide, regardless of whether the enclosing request matched `intercept_addresses` — an unbounded memory growth path on any thread that streams large non-LLM responses (file downloads, etc.). Buffering now only happens while a `NetHttpInterceptor#request` call is actively capturing.
|
|
29
|
+
- **A request made from inside another intercepted request's streaming callback no longer corrupts the outer request's logged response body.** The per-thread stream buffer is now saved and restored around each intercepted request instead of unconditionally cleared, so nested/reentrant LLM calls (e.g. a second provider call triggered from a chunk handler) no longer mix one request's content into another's log entry.
|
|
30
|
+
- **Coolhand's own request to its backend API is no longer re-intercepted and re-logged.** If a self-hosted `base_url` or custom `intercept_addresses` entry happens to match Coolhand's own API host, the log-shipping request itself would previously be captured and logged like any other request — which itself gets logged, recursively, without bound. It's now wrapped in `Coolhand.without_capture`.
|
|
31
|
+
- `Coolhand::OpenAi::WebhookValidator#webhook_secret` is no longer a public reader (it's only used internally) — reduces the chance of the raw secret ending up in error-tracker breadcrumbs or view-rendered controller state.
|
|
32
|
+
- `YAML.load_file` → `YAML.safe_load_file` for the gem's own bundled default-addresses/patterns YAML files — defense-in-depth against unsafe deserialization, consistent with modern Psych defaults.
|
|
33
|
+
- Added missing `require "openssl"` (`WebhookValidator`) and `require "stringio"` (`NetHttpInterceptor`) instead of relying on both being transitively loaded by something else first.
|
|
34
|
+
|
|
35
|
+
### Fixed
|
|
36
|
+
- **Vertex batch result logging** — `Coolhand::Vertex::BatchResultProcessor` now sends a fully-qualified URL (`https://aiplatform.googleapis.com/v1/<resource name>`) instead of the bare Vertex job resource name, plus explicit `source_api` and `model` values, all inside `raw_request`. The Coolhand API treats `raw_request` as free-form JSON, so this improves the data available to its classification without depending on new top-level payload fields being accepted. Also fixed: `timestamp`/`completed_at` are now sent as proper ISO 8601 strings instead of raw `Time` objects, and the URL is now run through the same `sanitize_url` helper used by every other logging path in the gem, for consistency (a Vertex batch URL can't actually carry a redactable query string today, so this is defense-in-depth rather than a fix for a live leak) (#76).
|
|
37
|
+
- **`require "coolhand"` no longer requires `faraday`.** The gem hasn't used Faraday directly since the 0.4.0 unified `NetHttpInterceptor`, but a stale top-level `require "faraday"` meant loading the gem could raise `LoadError` for anyone without faraday installed — it was never a declared dependency. If your app relied on `require "coolhand"` transitively loading Faraday, require it yourself.
|
|
38
|
+
- `Coolhand::Vertex::BatchResultProcessor`, `Coolhand::OpenAi::BatchResultProcessor`, and `Coolhand::OpenAi::WebhookValidator` can now each be required independently (e.g. `require "coolhand/vertex/batch_result_processor"`) without first requiring `"coolhand"` — previously this could raise `NameError` depending on load order.
|
|
39
|
+
- `Coolhand::BaseInterceptor.send_complete_request_log` now runs `request_headers`/`response_headers` through `sanitize_headers` itself, rather than trusting every caller to have pre-sanitized them. `NetHttpInterceptor` already sanitized before calling it (`sanitize_headers` is idempotent, so no behavior change there); this closes the gap for any future caller that forgets.
|
|
40
|
+
- `Coolhand::Vertex::BatchResultProcessor#handle_failed_batch` no longer raises `NoMethodError` when a failed batch's `error` field is missing or not a Hash — it now logs the batch-failure message either way instead of falling through to the generic "failed to process" catch-all.
|
|
41
|
+
- **README's "Request Flow" section corrected** — it previously claimed data is sent to the Coolhand API "asynchronously in a background thread" with "zero performance impact"; it's actually sent inline, after the original response is available, bounded by the 5-second timeout described in 0.5.0's Security section.
|
|
42
|
+
|
|
10
43
|
## [0.5.0] - 2026-07-30
|
|
11
44
|
|
|
12
45
|
### Changed
|
data/README.md
CHANGED
|
@@ -39,14 +39,14 @@ end
|
|
|
39
39
|
# ✅ OpenAI SDK calls
|
|
40
40
|
# ✅ Anthropic API calls
|
|
41
41
|
# ✅ Direct HTTP requests to AI APIs
|
|
42
|
-
# ✅
|
|
42
|
+
# ✅ Any library making AI API calls via Faraday's default adapter
|
|
43
43
|
|
|
44
44
|
# NO code changes needed in your existing services!
|
|
45
45
|
```
|
|
46
46
|
|
|
47
47
|
**✨ Why Automatic Monitoring:**
|
|
48
48
|
- 🚫 **Zero refactoring** - No code changes to existing services
|
|
49
|
-
- 📊 **Complete coverage** - Monitors
|
|
49
|
+
- 📊 **Complete coverage** - Monitors AI libraries using Net::HTTP or Faraday's default adapter automatically
|
|
50
50
|
- 🔒 **Security built-in** - Automatic credential sanitization
|
|
51
51
|
- ⚡ **Performance optimized** - Negligible overhead via async logging
|
|
52
52
|
- 🛡️ **Future-proof** - Automatically captures new AI calls added by your team
|
|
@@ -297,19 +297,25 @@ The filtering is automatic and applies to all monitored API calls and webhook lo
|
|
|
297
297
|
|
|
298
298
|
The monitor works with multiple transport layers and Ruby libraries:
|
|
299
299
|
|
|
300
|
-
**Faraday-based libraries
|
|
300
|
+
**Faraday-based libraries** (via Faraday's default `net_http` adapter — a non-Net::HTTP adapter such as `faraday-typhoeus` is not covered):
|
|
301
301
|
- OpenAI Ruby SDK
|
|
302
302
|
- ruby-anthropic gem (community Anthropic gem)
|
|
303
303
|
- ruby-openai gem
|
|
304
304
|
- LangChain.rb
|
|
305
|
-
- Direct Faraday requests
|
|
306
|
-
- Any other Faraday-based HTTP client
|
|
305
|
+
- Direct Faraday requests using the default adapter
|
|
307
306
|
|
|
308
307
|
**Native HTTP libraries:**
|
|
309
308
|
- Official Anthropic Ruby SDK (using Net::HTTP)
|
|
310
309
|
- GitHub Models SDK / any client using `models.github.ai`
|
|
311
310
|
- Any library using Net::HTTP directly
|
|
312
311
|
|
|
312
|
+
**Other providers monitored out of the box:**
|
|
313
|
+
- Google Gemini (`generativelanguage.googleapis.com`)
|
|
314
|
+
- Google Vertex AI (`aiplatform.googleapis.com`) — see the [Vertex AI batch result logging guide](docs/vertex.md) for async batch jobs
|
|
315
|
+
- AWS Bedrock (OpenAI-compatible endpoint)
|
|
316
|
+
- Cloudflare AI Gateway
|
|
317
|
+
- OpenRouter
|
|
318
|
+
|
|
313
319
|
**Universal Coverage**: Since most Ruby HTTP libraries use Net::HTTP under the hood, Coolhand's single interceptor provides comprehensive monitoring without needing library-specific integrations.
|
|
314
320
|
|
|
315
321
|
## How It Works
|
|
@@ -325,11 +331,11 @@ Coolhand uses a unified Net::HTTP interceptor to capture outgoing requests to co
|
|
|
325
331
|
### Request Flow
|
|
326
332
|
When a request matches configured LLM endpoints:
|
|
327
333
|
|
|
328
|
-
1. The original request executes normally
|
|
334
|
+
1. The original request executes normally
|
|
329
335
|
2. Request and response data (body, headers, status) are captured by the interceptor
|
|
330
336
|
3. For streaming requests, the complete accumulated response is captured (not individual chunks)
|
|
331
|
-
4. Data is sent to the Coolhand API
|
|
332
|
-
5. Your application continues
|
|
337
|
+
4. Data is sent to the Coolhand API inline, after the original response is available (bounded by a 5-second connect/read timeout so a slow or unreachable Coolhand backend can't hang your request indefinitely)
|
|
338
|
+
5. Your application continues once the log is sent (or times out) — the original request/response is never blocked on Coolhand being reachable
|
|
333
339
|
|
|
334
340
|
For non-matching endpoints, requests pass through unchanged.
|
|
335
341
|
|
|
@@ -395,108 +401,14 @@ The monitor handles errors gracefully:
|
|
|
395
401
|
- Invalid API keys will be reported but won't crash your app
|
|
396
402
|
- Network issues are handled with appropriate error messages
|
|
397
403
|
|
|
398
|
-
|
|
399
|
-
## Batch webhook handler (OpenAI)
|
|
400
|
-
|
|
401
|
-
Automatically handle OpenAI batch event logs (batch.completed, batch.failed, batch.expired, batch.cancelled)
|
|
402
|
-
by intercepting webhook requests and enqueuing your batch result processor.
|
|
403
|
-
|
|
404
|
-
Usage:
|
|
405
|
-
- Include the interceptor in your controller:
|
|
406
|
-
include Coolhand::WebhookInterceptor
|
|
407
|
-
- Add the before_action to validate and populate @validator payload:
|
|
408
|
-
before_action :intercept_batch_request, only: :openai
|
|
409
|
-
- Ensure you skip CSRF for the webhook endpoint:
|
|
410
|
-
skip_before_action :verify_authenticity_token
|
|
411
|
-
- Override the webhook_secret method to return your OpenAI webhook secret
|
|
412
|
-
|
|
413
|
-
Minimal example (only key lines shown):
|
|
414
|
-
|
|
415
|
-
```ruby
|
|
416
|
-
# app/controllers/webhooks/batch_api_requests_controller.rb
|
|
417
|
-
# ...existing code...
|
|
418
|
-
include Coolhand::WebhookInterceptor
|
|
419
|
-
|
|
420
|
-
skip_before_action :verify_authenticity_token
|
|
421
|
-
before_action :intercept_batch_request, only: :openai
|
|
422
|
-
|
|
423
|
-
def openai
|
|
424
|
-
event = JSON.parse(@validator.payload)
|
|
425
|
-
case event["type"]
|
|
426
|
-
when "batch.completed", "batch.failed", "batch.expired", "batch.cancelled"
|
|
427
|
-
batch_id = event.dig("data", "id")
|
|
428
|
-
batch_request = BatchApiRequest.find_by(provider: "openai", provider_batch_id: batch_id)
|
|
429
|
-
|
|
430
|
-
if batch_request
|
|
431
|
-
OpenAi::BatchResultProcessor.perform_async(batch_request.id)
|
|
432
|
-
Rails.logger.info("Queued batch result processing for BatchApiRequest #{batch_request.id}")
|
|
433
|
-
else
|
|
434
|
-
Rails.logger.warn("Could not find BatchApiRequest for OpenAI batch ID: #{batch_id}")
|
|
435
|
-
end
|
|
436
|
-
else
|
|
437
|
-
Rails.logger.info("Unhandled OpenAI webhook event type: #{event["type"]}")
|
|
438
|
-
end
|
|
439
|
-
|
|
440
|
-
head :ok
|
|
441
|
-
rescue JSON::ParserError
|
|
442
|
-
head :bad_request
|
|
443
|
-
rescue StandardError => e
|
|
444
|
-
Rails.logger.error("OpenAI webhook error: #{e.message}")
|
|
445
|
-
head :internal_server_error
|
|
446
|
-
end
|
|
447
|
-
|
|
448
|
-
def webhook_secret
|
|
449
|
-
Rails.application.credentials.openai_webhook_secret
|
|
450
|
-
end
|
|
451
|
-
# ...existing code...
|
|
452
|
-
```
|
|
453
|
-
|
|
454
|
-
## Batch webhook handler (Vertex)
|
|
455
|
-
|
|
456
|
-
Automatically handle Vertex batch event logs.
|
|
457
|
-
|
|
458
|
-
Usage:
|
|
459
|
-
- call Coolhand::Vertex::BatchResultProcessor service with batch_info and download batch results
|
|
460
|
-
|
|
461
|
-
Minimal example (only key lines shown):
|
|
462
|
-
|
|
463
|
-
```ruby
|
|
464
|
-
class Vertex::BatchCallbackProcessor < BaseService
|
|
465
|
-
option :batch_request, model: BatchApiRequest
|
|
466
|
-
option :batch_info
|
|
467
|
-
|
|
468
|
-
def call
|
|
469
|
-
case batch_info["state"]
|
|
470
|
-
when "JOB_STATE_PENDING"
|
|
471
|
-
nil
|
|
472
|
-
when "JOB_STATE_RUNNING", "JOB_STATE_QUEUED"
|
|
473
|
-
batch_request.update!(status: "processing")
|
|
474
|
-
|
|
475
|
-
Coolhand::Vertex::BatchResultProcessor.new(batch_info:).call
|
|
476
|
-
when "JOB_STATE_SUCCEEDED"
|
|
477
|
-
output_file_id = batch_info["outputInfo"]["gcsOutputDirectory"]
|
|
478
|
-
results = download_batch_results(output_file_id)
|
|
479
|
-
results.each { |batch_item| process_batch_result(batch_item) }
|
|
480
|
-
|
|
481
|
-
batch_request.update!(status: "completed", completed_at: Time.current, output_file_id:)
|
|
482
|
-
|
|
483
|
-
Coolhand::Vertex::BatchResultProcessor.new(batch_info:).call(results)
|
|
484
|
-
|
|
485
|
-
# Clean up GCS files after successful processing
|
|
486
|
-
cleanup_gcs_files(output_file_id)
|
|
487
|
-
when "JOB_STATE_FAILED"
|
|
488
|
-
handle_failed_batch(batch_info["error"]["message"])
|
|
489
|
-
end
|
|
490
|
-
end
|
|
491
|
-
end
|
|
492
|
-
```
|
|
493
|
-
|
|
494
404
|
## Documentation
|
|
495
405
|
|
|
496
406
|
- **[Configuration](docs/configuration.md)** — Self-hosted deployments, base_url rules, debug mode, custom intercept addresses
|
|
497
407
|
- **[Feedback API](docs/feedback.md)** — Full field reference, matching strategies, sentiment values
|
|
498
408
|
- **[Anthropic Integration](docs/anthropic.md)** — Official and community Anthropic Ruby gems, streaming, dual gem handling, and troubleshooting
|
|
499
409
|
- **[ElevenLabs Integration](docs/elevenlabs.md)** — Webhook capture, feedback submission, and Rails integration
|
|
410
|
+
- **[OpenAI Batch Webhook Handler](docs/openai.md)** — Handle OpenAI batch job completion events via webhook interception
|
|
411
|
+
- **[Google Vertex AI Batch Result Logging](docs/vertex.md)** — Log completed Vertex AI batch prediction job results
|
|
500
412
|
|
|
501
413
|
## Security
|
|
502
414
|
|
data/docs/openai.md
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# OpenAI Batch Webhook Handler
|
|
2
|
+
|
|
3
|
+
Automatically handle OpenAI batch event logs (`batch.completed`, `batch.failed`, `batch.expired`, `batch.cancelled`) by intercepting webhook requests and logging completed batch results to Coolhand.
|
|
4
|
+
|
|
5
|
+
For monitoring regular (non-batch) OpenAI API calls, see the [OpenAI client example](../README.md#with-openai-ruby-client) in the main README — no extra setup is required beyond `Coolhand.configure`.
|
|
6
|
+
|
|
7
|
+
Requires Rails — `Coolhand::WebhookInterceptor` and `Coolhand::OpenAi::BatchResultProcessor` log via `Rails.logger` internally. `config.capture = false` and `Coolhand.without_capture` do not suppress these logs: unlike the passive Net::HTTP interceptor, `intercept_batch_request` dispatching to `Coolhand::OpenAi::BatchResultProcessor` is an explicit, deliberate act, so it always sends.
|
|
8
|
+
|
|
9
|
+
## Usage
|
|
10
|
+
|
|
11
|
+
- Add the `openai` gem to your Gemfile — `Coolhand::OpenAi::BatchResultProcessor` needs it to download batch result files.
|
|
12
|
+
- Include the interceptor in your controller: `include Coolhand::WebhookInterceptor`
|
|
13
|
+
- Add the `before_action` to validate and populate the `@validator` payload: `before_action :intercept_batch_request, only: :openai`
|
|
14
|
+
- Ensure you skip CSRF for the webhook endpoint: `skip_before_action :verify_authenticity_token`
|
|
15
|
+
- Override the `webhook_secret` method to return your OpenAI webhook secret
|
|
16
|
+
|
|
17
|
+
`intercept_batch_request` already validates the webhook and, for `batch.completed`/`batch.failed`/`batch.expired`/`batch.cancelled` events, calls `Coolhand::OpenAi::BatchResultProcessor` **synchronously**, inline in the `before_action` — that part requires no code in your controller action. It downloads two JSONL files from OpenAI and sends one request log per batch item, so a large batch can take long enough to trip your webhook endpoint's timeout (and OpenAI's retry behavior); consider that when sizing batches. The example below shows a controller action doing additional, app-specific work on top of that (e.g. updating your own `BatchApiRequest` record and enqueuing your own background job), which is optional.
|
|
18
|
+
|
|
19
|
+
## Minimal example
|
|
20
|
+
|
|
21
|
+
Only the key lines are shown — wire this into your own controller and background job setup.
|
|
22
|
+
|
|
23
|
+
```ruby
|
|
24
|
+
# app/controllers/webhooks/batch_api_requests_controller.rb
|
|
25
|
+
# ...existing code...
|
|
26
|
+
include Coolhand::WebhookInterceptor
|
|
27
|
+
|
|
28
|
+
skip_before_action :verify_authenticity_token
|
|
29
|
+
before_action :intercept_batch_request, only: :openai
|
|
30
|
+
|
|
31
|
+
def openai
|
|
32
|
+
event = JSON.parse(@validator.payload)
|
|
33
|
+
case event["type"]
|
|
34
|
+
when "batch.completed", "batch.failed", "batch.expired", "batch.cancelled"
|
|
35
|
+
batch_id = event.dig("data", "id")
|
|
36
|
+
batch_request = BatchApiRequest.find_by(provider: "openai", provider_batch_id: batch_id)
|
|
37
|
+
|
|
38
|
+
if batch_request
|
|
39
|
+
MyApp::BatchResultJob.perform_async(batch_request.id)
|
|
40
|
+
Rails.logger.info("Queued batch result processing for BatchApiRequest #{batch_request.id}")
|
|
41
|
+
else
|
|
42
|
+
Rails.logger.warn("Could not find BatchApiRequest for OpenAI batch ID: #{batch_id}")
|
|
43
|
+
end
|
|
44
|
+
else
|
|
45
|
+
Rails.logger.info("Unhandled OpenAI webhook event type: #{event["type"]}")
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
head :ok
|
|
49
|
+
rescue JSON::ParserError
|
|
50
|
+
head :bad_request
|
|
51
|
+
rescue StandardError => e
|
|
52
|
+
Rails.logger.error("OpenAI webhook error: #{e.message}")
|
|
53
|
+
head :internal_server_error
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def webhook_secret
|
|
57
|
+
Rails.application.credentials.openai_webhook_secret
|
|
58
|
+
end
|
|
59
|
+
# ...existing code...
|
|
60
|
+
```
|
data/docs/vertex.md
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# Google Vertex AI Batch Result Logging
|
|
2
|
+
|
|
3
|
+
Log completed Google Vertex AI batch prediction job results the same way regular synchronous calls are logged. Unlike the [OpenAI batch handler](openai.md), this is not webhook-driven — you call `Coolhand::Vertex::BatchResultProcessor` directly from your own batch-job callback/polling code.
|
|
4
|
+
|
|
5
|
+
For monitoring regular (non-batch) Vertex AI calls, no extra setup is required beyond `Coolhand.configure` — `aiplatform.googleapis.com` is intercepted by default.
|
|
6
|
+
|
|
7
|
+
Requires Rails — `Coolhand::Vertex::BatchResultProcessor` logs via `Rails.logger` internally. `config.capture = false` and `Coolhand.without_capture` do not suppress these logs: unlike the passive Net::HTTP interceptor, calling this processor is an explicit, deliberate act, so it always sends.
|
|
8
|
+
|
|
9
|
+
## Usage
|
|
10
|
+
|
|
11
|
+
```ruby
|
|
12
|
+
require "coolhand/vertex/batch_result_processor"
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
- Call the `Coolhand::Vertex::BatchResultProcessor` service with `batch_info` and the downloaded batch results.
|
|
16
|
+
- Optionally pass `model:` to populate the logged `model` field. It falls back to a `"model"` key in `batch_info` if present, and is omitted from the payload entirely when neither is available. If the value (from either source) looks like a Vertex model resource path (`publishers/<publisher>/models/<id>` or `projects/<project>/locations/<location>/models/<id>`), it's normalized down to just the trailing id/version before being logged; any other string is sent as-is.
|
|
17
|
+
- `batch_info["name"]` must be the exact Vertex job resource name (`projects/<project>/locations/<location>/batchPredictionJobs/<job-id>`), and `batch_info["startTime"]`/`["endTime"]` must be valid ISO 8601 timestamps — a batch with a missing or malformed resource name or timestamps is skipped entirely (logged as an error) rather than sent with a broken URL.
|
|
18
|
+
- If `endTime` precedes `startTime` (e.g. clock skew), the batch is still logged — `duration_ms` is clamped to `0` and a warning is logged — rather than dropping the results.
|
|
19
|
+
- Each element of the `batch_results` array passed to `.call` must be a `Hash` with a `"request"` and/or `"response"` key (the request/response body to log for that item — either key alone is accepted). Any other shape is treated as malformed and skipped, with the skip counted in a `Rails.logger.warn` summary rather than raised.
|
|
20
|
+
|
|
21
|
+
## Minimal example
|
|
22
|
+
|
|
23
|
+
Only the key lines are shown — wire this into your own batch callback service.
|
|
24
|
+
|
|
25
|
+
```ruby
|
|
26
|
+
class Vertex::BatchCallbackProcessor < BaseService
|
|
27
|
+
option :batch_request, model: BatchApiRequest
|
|
28
|
+
option :batch_info
|
|
29
|
+
|
|
30
|
+
def call
|
|
31
|
+
case batch_info["state"]
|
|
32
|
+
when "JOB_STATE_PENDING"
|
|
33
|
+
nil
|
|
34
|
+
when "JOB_STATE_RUNNING", "JOB_STATE_QUEUED"
|
|
35
|
+
batch_request.update!(status: "processing")
|
|
36
|
+
|
|
37
|
+
Coolhand::Vertex::BatchResultProcessor.new(batch_info:).call
|
|
38
|
+
when "JOB_STATE_SUCCEEDED"
|
|
39
|
+
output_file_id = batch_info["outputInfo"]["gcsOutputDirectory"]
|
|
40
|
+
results = download_batch_results(output_file_id)
|
|
41
|
+
results.each { |batch_item| process_batch_result(batch_item) }
|
|
42
|
+
|
|
43
|
+
batch_request.update!(status: "completed", completed_at: Time.current, output_file_id:)
|
|
44
|
+
|
|
45
|
+
Coolhand::Vertex::BatchResultProcessor.new(batch_info:, model: batch_request.llm_model).call(results)
|
|
46
|
+
|
|
47
|
+
# Clean up GCS files after successful processing
|
|
48
|
+
cleanup_gcs_files(output_file_id)
|
|
49
|
+
when "JOB_STATE_FAILED"
|
|
50
|
+
handle_failed_batch(batch_info["error"]["message"])
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
```
|
data/lib/coolhand/api_service.rb
CHANGED
|
@@ -92,7 +92,13 @@ module Coolhand
|
|
|
92
92
|
request.body = json_body.force_encoding("UTF-8")
|
|
93
93
|
|
|
94
94
|
begin
|
|
95
|
-
|
|
95
|
+
# This request goes through the same patched Net::HTTP as the host
|
|
96
|
+
# app's real LLM calls. Without this, a base_url/intercept_addresses
|
|
97
|
+
# configuration that also matches Coolhand's own API host would
|
|
98
|
+
# re-intercept this log-shipping call, generating a second log
|
|
99
|
+
# request that itself gets intercepted, and so on — unbounded
|
|
100
|
+
# request amplification.
|
|
101
|
+
response = Coolhand.without_capture { http.request(request) }
|
|
96
102
|
|
|
97
103
|
if response.is_a?(Net::HTTPSuccess)
|
|
98
104
|
result = JSON.parse(response.body, symbolize_names: true)
|
|
@@ -7,10 +7,11 @@ module Coolhand
|
|
|
7
7
|
|
|
8
8
|
# Matches any header whose *name* signals sensitive content, regardless of
|
|
9
9
|
# provider — covers known keys (x-api-key, x-goog-api-key, openai-api-key),
|
|
10
|
-
# AWS SigV4 session tokens (x-amz-security-token),
|
|
11
|
-
#
|
|
12
|
-
#
|
|
13
|
-
|
|
10
|
+
# AWS SigV4 session tokens (x-amz-security-token), session/CSRF cookies
|
|
11
|
+
# (cookie, set-cookie), and future/unknown providers using a
|
|
12
|
+
# similarly-named header. Shared with LoggerService so the two logging
|
|
13
|
+
# paths (interceptor + webhook forwarding) stay consistent.
|
|
14
|
+
SENSITIVE_HEADER_PATTERN = /key|token|secret|signature|authorization|cookie/i
|
|
14
15
|
|
|
15
16
|
def sanitize_headers(headers)
|
|
16
17
|
return {} if headers.nil?
|
|
@@ -69,15 +70,22 @@ module Coolhand
|
|
|
69
70
|
end
|
|
70
71
|
end
|
|
71
72
|
|
|
73
|
+
# Matches query param *names* that signal sensitive content, the same way
|
|
74
|
+
# SENSITIVE_HEADER_PATTERN does for headers — covers exact params this
|
|
75
|
+
# gem already knew about (key/token/secret) as well as presigned-URL
|
|
76
|
+
# credential params used by AWS (X-Amz-Signature, X-Amz-Credential,
|
|
77
|
+
# X-Amz-Security-Token) and Google Cloud (X-Goog-Signature,
|
|
78
|
+
# X-Goog-Credential) storage APIs.
|
|
79
|
+
SENSITIVE_QUERY_PARAM_PATTERN = /key|token|secret|sig|credential|password|auth/i
|
|
80
|
+
|
|
72
81
|
def sanitize_url(url)
|
|
73
82
|
uri = URI.parse(url)
|
|
74
83
|
return url unless uri.query
|
|
75
84
|
|
|
76
|
-
sensitive = %w[key api_key apikey token access_token secret]
|
|
77
85
|
params = URI.decode_www_form(uri.query)
|
|
78
86
|
redacted = false
|
|
79
87
|
params.map! do |n, v|
|
|
80
|
-
if
|
|
88
|
+
if n.match?(SENSITIVE_QUERY_PARAM_PATTERN)
|
|
81
89
|
redacted = true
|
|
82
90
|
[n, "[REDACTED]"]
|
|
83
91
|
else
|
|
@@ -96,23 +104,25 @@ module Coolhand
|
|
|
96
104
|
end
|
|
97
105
|
|
|
98
106
|
def send_complete_request_log(request_id:, method:, url:, request_headers:, request_body:, response_headers:,
|
|
99
|
-
response_body:, status_code:, start_time:, end_time:, duration_ms:, is_streaming:)
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
is_streaming: is_streaming
|
|
114
|
-
}
|
|
107
|
+
response_body:, status_code:, start_time:, end_time:, duration_ms:, is_streaming:, source_api: nil, model: nil)
|
|
108
|
+
raw_request = {
|
|
109
|
+
id: request_id,
|
|
110
|
+
timestamp: start_time.iso8601,
|
|
111
|
+
method: method.to_s.downcase,
|
|
112
|
+
url: sanitize_url(url),
|
|
113
|
+
headers: sanitize_headers(request_headers),
|
|
114
|
+
request_body: request_body,
|
|
115
|
+
response_headers: sanitize_headers(response_headers),
|
|
116
|
+
response_body: response_body,
|
|
117
|
+
status_code: status_code,
|
|
118
|
+
duration_ms: duration_ms,
|
|
119
|
+
completed_at: end_time.iso8601,
|
|
120
|
+
is_streaming: is_streaming
|
|
115
121
|
}
|
|
122
|
+
raw_request[:source_api] = source_api if Coolhand.required_field?(source_api)
|
|
123
|
+
raw_request[:model] = model if Coolhand.required_field?(model)
|
|
124
|
+
|
|
125
|
+
request_data = { raw_request: raw_request }
|
|
116
126
|
|
|
117
127
|
api_service = Coolhand::ApiService.new
|
|
118
128
|
api_service.send_llm_request_log(request_data)
|
|
@@ -6,11 +6,11 @@ require "uri"
|
|
|
6
6
|
module Coolhand
|
|
7
7
|
# Handles all configuration settings for the gem.
|
|
8
8
|
class Configuration
|
|
9
|
-
DEFAULT_EXCLUDE_API_PATTERNS = YAML.
|
|
9
|
+
DEFAULT_EXCLUDE_API_PATTERNS = YAML.safe_load_file(
|
|
10
10
|
File.join(__dir__, "default_exclude_api_patterns.yml")
|
|
11
11
|
).freeze
|
|
12
12
|
|
|
13
|
-
DEFAULT_INTERCEPT_ADDRESSES = YAML.
|
|
13
|
+
DEFAULT_INTERCEPT_ADDRESSES = YAML.safe_load_file(
|
|
14
14
|
File.join(__dir__, "default_intercept_addresses.yml")
|
|
15
15
|
).freeze
|
|
16
16
|
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "stringio"
|
|
4
|
+
|
|
3
5
|
module Coolhand
|
|
4
6
|
module NetHttpInterceptor
|
|
5
7
|
include BaseInterceptor
|
|
@@ -7,7 +9,12 @@ module Coolhand
|
|
|
7
9
|
# Response streaming interceptor nested under NetHttpInterceptor
|
|
8
10
|
module ResponseInterceptor
|
|
9
11
|
def read_body(dest = nil, &block)
|
|
10
|
-
|
|
12
|
+
# Only buffer while a #request call below is actively capturing —
|
|
13
|
+
# otherwise every block-form read_body in the process (including
|
|
14
|
+
# ones on responses this gem never intercepted) accumulates into a
|
|
15
|
+
# thread-local that's never freed, which is an unbounded memory
|
|
16
|
+
# leak on any thread that streams large non-LLM responses.
|
|
17
|
+
return super unless block && Thread.current[:coolhand_capturing_stream]
|
|
11
18
|
|
|
12
19
|
super do |chunk|
|
|
13
20
|
Thread.current[:coolhand_stream_buffer] ||= +""
|
|
@@ -49,8 +56,16 @@ module Coolhand
|
|
|
49
56
|
return super unless should_capture?
|
|
50
57
|
|
|
51
58
|
# Capture body before setting the guard — if this raises we skip logging cleanly
|
|
52
|
-
# and the guard is never set, so there is no leak.
|
|
53
|
-
|
|
59
|
+
# and the guard is never set, so there is no leak. A failure here (e.g. an
|
|
60
|
+
# already-consumed body_stream) must never prevent the real request below
|
|
61
|
+
# from being attempted — this gem must never be the reason the host
|
|
62
|
+
# app's actual LLM call doesn't happen.
|
|
63
|
+
captured_body = begin
|
|
64
|
+
capture_request_body(req, body)
|
|
65
|
+
rescue StandardError => e
|
|
66
|
+
Coolhand.log "❌ Error capturing request body: #{e.message}"
|
|
67
|
+
nil
|
|
68
|
+
end
|
|
54
69
|
|
|
55
70
|
active[self] = true
|
|
56
71
|
start_time = Time.now
|
|
@@ -59,7 +74,14 @@ module Coolhand
|
|
|
59
74
|
status_code = nil
|
|
60
75
|
response_body = nil
|
|
61
76
|
|
|
77
|
+
# Save/restore rather than just nil-ing: a request made from inside
|
|
78
|
+
# this request's own streaming block (nested interception) would
|
|
79
|
+
# otherwise clobber this request's in-progress buffer with its own
|
|
80
|
+
# chunks, mixing one request's content into another's log.
|
|
81
|
+
previous_stream_buffer = Thread.current[:coolhand_stream_buffer]
|
|
82
|
+
previous_capturing_stream = Thread.current[:coolhand_capturing_stream]
|
|
62
83
|
Thread.current[:coolhand_stream_buffer] = nil
|
|
84
|
+
Thread.current[:coolhand_capturing_stream] = true
|
|
63
85
|
|
|
64
86
|
begin
|
|
65
87
|
response = super
|
|
@@ -73,7 +95,8 @@ module Coolhand
|
|
|
73
95
|
raise
|
|
74
96
|
ensure
|
|
75
97
|
active.delete(self)
|
|
76
|
-
Thread.current[:coolhand_stream_buffer] =
|
|
98
|
+
Thread.current[:coolhand_stream_buffer] = previous_stream_buffer
|
|
99
|
+
Thread.current[:coolhand_capturing_stream] = previous_capturing_stream
|
|
77
100
|
end_time = Time.now
|
|
78
101
|
duration_ms = ((end_time - start_time) * 1000).round(2)
|
|
79
102
|
|
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "openssl"
|
|
4
|
+
|
|
3
5
|
module Coolhand
|
|
4
6
|
module OpenAi
|
|
5
7
|
class WebhookValidator
|
|
6
|
-
attr_reader :request, :errors, :payload
|
|
8
|
+
attr_reader :request, :errors, :payload
|
|
7
9
|
|
|
8
10
|
def initialize(request, webhook_secret)
|
|
9
11
|
@request = request
|
|
@@ -16,7 +18,7 @@ module Coolhand
|
|
|
16
18
|
@payload = request.raw_post || request.body.read
|
|
17
19
|
|
|
18
20
|
return false unless payload_valid?
|
|
19
|
-
return validate_in_non_production_env unless webhook_secret
|
|
21
|
+
return validate_in_non_production_env unless Coolhand.required_field?(webhook_secret)
|
|
20
22
|
|
|
21
23
|
secret_bytes = extract_secret_bytes
|
|
22
24
|
webhook_signature, webhook_timestamp, webhook_id = extract_webhook_headers
|
|
@@ -32,6 +34,8 @@ module Coolhand
|
|
|
32
34
|
|
|
33
35
|
private
|
|
34
36
|
|
|
37
|
+
attr_reader :webhook_secret
|
|
38
|
+
|
|
35
39
|
def payload_valid?
|
|
36
40
|
return true if @payload
|
|
37
41
|
|
data/lib/coolhand/version.rb
CHANGED
|
@@ -1,12 +1,27 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require_relative "../../coolhand"
|
|
4
|
+
|
|
3
5
|
module Coolhand
|
|
4
6
|
module Vertex
|
|
5
7
|
class BatchResultProcessor
|
|
6
|
-
|
|
8
|
+
# Global endpoint, not the job's region-specific one — matches the
|
|
9
|
+
# "aiplatform.googleapis.com" entry in default_intercept_addresses.yml
|
|
10
|
+
# so backend URL-shape classification stays consistent with the rest
|
|
11
|
+
# of the gem's Vertex traffic. The URL this produces always contains
|
|
12
|
+
# "/batchPredictionJobs/", which is also the default client-side
|
|
13
|
+
# exclude_api_patterns entry — that's harmless here since this class
|
|
14
|
+
# sends via BaseInterceptor/ApiService directly and never goes through
|
|
15
|
+
# NetHttpInterceptor's intercept/exclude filtering.
|
|
16
|
+
VERTEX_API_BASE_URL = "https://aiplatform.googleapis.com/v1/"
|
|
17
|
+
SOURCE_API = "vertex"
|
|
18
|
+
VALID_NAME_PATTERN = %r{\Aprojects/[^/?#\s]+/locations/[^/?#\s]+/batchPredictionJobs/[^/?#\s]+\z}
|
|
19
|
+
|
|
20
|
+
attr_reader :batch_info, :model
|
|
7
21
|
|
|
8
|
-
def initialize(batch_info:)
|
|
22
|
+
def initialize(batch_info:, model: nil)
|
|
9
23
|
@batch_info = batch_info
|
|
24
|
+
@model = model
|
|
10
25
|
end
|
|
11
26
|
|
|
12
27
|
def call(batch_results = [])
|
|
@@ -16,7 +31,7 @@ module Coolhand
|
|
|
16
31
|
when "JOB_STATE_PENDING", "JOB_STATE_RUNNING", "JOB_STATE_QUEUED"
|
|
17
32
|
Rails.logger.info("[Interceptor] Vertex batch #{batch_info} still processing")
|
|
18
33
|
when "JOB_STATE_SUCCEEDED"
|
|
19
|
-
|
|
34
|
+
process_completed_batch(batch_results)
|
|
20
35
|
when "JOB_STATE_FAILED"
|
|
21
36
|
handle_failed_batch
|
|
22
37
|
else
|
|
@@ -28,56 +43,98 @@ module Coolhand
|
|
|
28
43
|
|
|
29
44
|
private
|
|
30
45
|
|
|
31
|
-
def process_completed_batch(
|
|
32
|
-
|
|
46
|
+
def process_completed_batch(batch_results)
|
|
47
|
+
name = batch_info["name"].to_s.sub(%r{\A/+}, "")
|
|
48
|
+
unless name.match?(VALID_NAME_PATTERN)
|
|
49
|
+
Rails.logger.error("[Interceptor] Vertex batch #{batch_info['displayName']} has a missing or " \
|
|
50
|
+
"invalid job resource name (#{batch_info['name'].inspect}); skipping request log")
|
|
51
|
+
return
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
begin
|
|
55
|
+
start_time = Time.iso8601(batch_info["startTime"])
|
|
56
|
+
end_time = Time.iso8601(batch_info["endTime"])
|
|
57
|
+
rescue TypeError, ArgumentError
|
|
58
|
+
Rails.logger.error("[Interceptor] Vertex batch #{batch_info['displayName']} has a missing or " \
|
|
59
|
+
"invalid startTime/endTime; skipping request log")
|
|
60
|
+
return
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
if end_time < start_time
|
|
64
|
+
Rails.logger.warn("[Interceptor] Vertex batch #{batch_info['displayName']} has an endTime before " \
|
|
65
|
+
"its startTime; sending request log(s) with duration_ms clamped to 0")
|
|
66
|
+
end_time = start_time
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
duration_ms = ((end_time - start_time) * 1000).to_i
|
|
70
|
+
url = "#{VERTEX_API_BASE_URL}#{name}"
|
|
71
|
+
resolved = resolved_model
|
|
72
|
+
|
|
73
|
+
sent = batch_results.count do |batch_item|
|
|
74
|
+
send_item_log(batch_item, url: url, model: resolved, start_time: start_time, end_time: end_time,
|
|
75
|
+
duration_ms: duration_ms)
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# "Sent" here means dispatched to BaseInterceptor without a local
|
|
79
|
+
# error (e.g. a malformed batch_item) — BaseInterceptor swallows any
|
|
80
|
+
# downstream delivery failure itself, so this can't confirm the
|
|
81
|
+
# Coolhand API actually received the log.
|
|
82
|
+
if sent == batch_results.size
|
|
83
|
+
Rails.logger.info("[Interceptor] Dispatched #{sent} result(s) for Vertex batch " \
|
|
84
|
+
"#{batch_info['displayName']} for logging")
|
|
85
|
+
else
|
|
86
|
+
Rails.logger.warn("[Interceptor] Dispatched #{sent}/#{batch_results.size} result(s) for Vertex " \
|
|
87
|
+
"batch #{batch_info['displayName']} for logging — " \
|
|
88
|
+
"#{batch_results.size - sent} item(s) were malformed and skipped")
|
|
89
|
+
end
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
def send_item_log(batch_item, url:, model:, start_time:, end_time:, duration_ms:)
|
|
93
|
+
unless batch_item.is_a?(Hash) && (batch_item.key?("request") || batch_item.key?("response"))
|
|
94
|
+
Rails.logger.error("[Interceptor] Vertex batch #{batch_info['displayName']} has a malformed result " \
|
|
95
|
+
"item (#{batch_item.class}); skipping request log")
|
|
96
|
+
return false
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
BaseInterceptor.send_complete_request_log(
|
|
100
|
+
request_id: SecureRandom.hex(16),
|
|
33
101
|
method: "POST",
|
|
34
|
-
url:
|
|
102
|
+
url: url,
|
|
103
|
+
source_api: SOURCE_API,
|
|
104
|
+
model: model,
|
|
105
|
+
request_headers: {},
|
|
35
106
|
request_body: batch_item["request"],
|
|
107
|
+
response_headers: {},
|
|
36
108
|
response_body: batch_item["response"],
|
|
37
109
|
status_code: 200,
|
|
38
|
-
start_time:
|
|
39
|
-
end_time:
|
|
40
|
-
|
|
41
|
-
|
|
110
|
+
start_time: start_time,
|
|
111
|
+
end_time: end_time,
|
|
112
|
+
duration_ms: duration_ms,
|
|
113
|
+
is_streaming: false
|
|
114
|
+
)
|
|
115
|
+
true
|
|
42
116
|
rescue StandardError => e
|
|
43
117
|
Rails.logger.error("[Interceptor] Failed to send request log: #{e.message}")
|
|
118
|
+
false
|
|
44
119
|
end
|
|
45
120
|
|
|
46
|
-
def
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
end_time = Time.iso8601(end_time)
|
|
50
|
-
duration_ms = ((end_time - start_time) * 1000).to_i
|
|
121
|
+
def resolved_model
|
|
122
|
+
candidate = [model, batch_info["model"]].find { |c| Coolhand.required_field?(c) }&.to_s&.strip
|
|
123
|
+
return if candidate.nil?
|
|
51
124
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
request_body: request_body,
|
|
60
|
-
response_headers: {},
|
|
61
|
-
response_body: response_body,
|
|
62
|
-
status_code: status_code,
|
|
63
|
-
duration_ms: duration_ms,
|
|
64
|
-
completed_at: end_time,
|
|
65
|
-
is_streaming: false
|
|
66
|
-
}
|
|
67
|
-
}
|
|
68
|
-
|
|
69
|
-
api_service = Coolhand::ApiService.new
|
|
70
|
-
api_service.send_llm_request_log(request_data)
|
|
71
|
-
|
|
72
|
-
Coolhand.log "📤 Sent complete request/response log for #{request_id} (duration: #{duration_ms}ms)"
|
|
73
|
-
rescue StandardError => e
|
|
74
|
-
Coolhand.log "❌ Error sending complete request log: #{e.message}"
|
|
125
|
+
# Only normalize genuine Vertex model resource paths (e.g.
|
|
126
|
+
# "publishers/google/models/gemini-2.0-flash" or
|
|
127
|
+
# "projects/P/locations/L/models/M@1") down to their bare id. A
|
|
128
|
+
# caller-supplied `model:` might itself be a provider-qualified slug
|
|
129
|
+
# that happens to contain a slash (e.g. "meta-llama/Llama-3") —
|
|
130
|
+
# leave anything that isn't a resource path untouched.
|
|
131
|
+
candidate.match?(%r{\A(publishers|projects)/}) ? candidate.split("/").last : candidate
|
|
75
132
|
end
|
|
76
133
|
|
|
77
134
|
# TODO: implement API to handle failed batch results and display errors on dashboard page
|
|
78
135
|
def handle_failed_batch
|
|
79
|
-
|
|
80
|
-
|
|
136
|
+
message = batch_info["error"].is_a?(Hash) ? batch_info["error"]["message"] : batch_info["error"]
|
|
137
|
+
Rails.logger.error("[Interceptor] Vertex batch for #{batch_info['displayName']} failed: #{message}")
|
|
81
138
|
end
|
|
82
139
|
end
|
|
83
140
|
end
|
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require_relative "open_ai/webhook_validator"
|
|
4
|
+
require_relative "open_ai/batch_result_processor"
|
|
5
|
+
|
|
3
6
|
module Coolhand
|
|
4
7
|
module WebhookInterceptor
|
|
5
8
|
def intercept_batch_request
|
|
@@ -14,10 +17,18 @@ module Coolhand
|
|
|
14
17
|
end
|
|
15
18
|
|
|
16
19
|
payload = JSON.parse(@validator.payload)
|
|
20
|
+
raise TypeError, "webhook payload must be a JSON object, got #{payload.class}" unless payload.is_a?(Hash)
|
|
17
21
|
|
|
18
22
|
process_event(payload)
|
|
19
23
|
rescue StandardError => e
|
|
24
|
+
# Fail closed: any error here (malformed payload, a bug in
|
|
25
|
+
# process_event/BatchResultProcessor, etc.) must still halt the
|
|
26
|
+
# before_action chain. Falling through without calling `head` would
|
|
27
|
+
# let the controller action run for a request whose webhook
|
|
28
|
+
# signature was never confirmed valid.
|
|
20
29
|
Rails.logger.error("[Interceptor] Failed to intercept batch request: #{e.message}")
|
|
30
|
+
head :unauthorized
|
|
31
|
+
false
|
|
21
32
|
end
|
|
22
33
|
|
|
23
34
|
def webhook_secret
|
data/lib/coolhand.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: coolhand
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.5.
|
|
4
|
+
version: 0.5.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Michael Carroll
|
|
@@ -9,7 +9,7 @@ authors:
|
|
|
9
9
|
autorequire:
|
|
10
10
|
bindir: exe
|
|
11
11
|
cert_chain: []
|
|
12
|
-
date: 2026-
|
|
12
|
+
date: 2026-08-02 00:00:00.000000000 Z
|
|
13
13
|
dependencies:
|
|
14
14
|
- !ruby/object:Gem::Dependency
|
|
15
15
|
name: base64
|
|
@@ -26,23 +26,20 @@ dependencies:
|
|
|
26
26
|
- !ruby/object:Gem::Version
|
|
27
27
|
version: '0.2'
|
|
28
28
|
description: Automatically intercept and log LLM requests from Ruby applications.
|
|
29
|
-
Supports OpenAI, official Anthropic gem, ruby-anthropic gem,
|
|
30
|
-
|
|
31
|
-
|
|
29
|
+
Supports OpenAI, official Anthropic gem, ruby-anthropic gem, Google Gemini, Google
|
|
30
|
+
Vertex AI, AWS Bedrock, OpenRouter, and any other library using Net::HTTP (directly,
|
|
31
|
+
or via Faraday's default adapter). Features a single unified Net::HTTP interceptor,
|
|
32
|
+
streaming support, thread-safe operation, and automatic duplicate request prevention.
|
|
32
33
|
email:
|
|
33
34
|
- mc@coolhandlabs.com
|
|
34
35
|
executables: []
|
|
35
36
|
extensions: []
|
|
36
37
|
extra_rdoc_files: []
|
|
37
38
|
files:
|
|
38
|
-
- ".claude/skills/loop-review/SKILL.md"
|
|
39
|
-
- ".claude/skills/prep-release/SKILL.md"
|
|
40
|
-
- ".idea/coolhand-ruby.iml"
|
|
41
39
|
- ".rspec"
|
|
42
40
|
- ".rubocop.yml"
|
|
43
41
|
- ".simplecov"
|
|
44
42
|
- CHANGELOG.md
|
|
45
|
-
- CLAUDE.md
|
|
46
43
|
- LICENSE
|
|
47
44
|
- README.md
|
|
48
45
|
- Rakefile
|
|
@@ -51,6 +48,8 @@ files:
|
|
|
51
48
|
- docs/configuration.md
|
|
52
49
|
- docs/elevenlabs.md
|
|
53
50
|
- docs/feedback.md
|
|
51
|
+
- docs/openai.md
|
|
52
|
+
- docs/vertex.md
|
|
54
53
|
- lib/coolhand.rb
|
|
55
54
|
- lib/coolhand/api_service.rb
|
|
56
55
|
- lib/coolhand/base_interceptor.rb
|
|
@@ -1,112 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: loop-review
|
|
3
|
-
description: |
|
|
4
|
-
Iteratively runs code review against the current diff, applies fixes, and
|
|
5
|
-
re-reviews until a round comes back clean (or a safety cap is hit). Use
|
|
6
|
-
when the user types /loop-review, asks to "loop the review", "review
|
|
7
|
-
until clean", "keep reviewing and fixing until nothing's left", or wants
|
|
8
|
-
a self-healing code review cycle instead of a single one-shot pass.
|
|
9
|
-
user_invocable: true
|
|
10
|
-
version: 0.3.0
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
# Loop Review
|
|
14
|
-
|
|
15
|
-
This skill runs `/code-review` repeatedly against the working diff, applies
|
|
16
|
-
fixes between rounds, and re-reviews until a round finds nothing new or a
|
|
17
|
-
safety cap is reached. Use it after a merge, a large diff, or whenever a
|
|
18
|
-
single review pass isn't enough to converge on a clean state.
|
|
19
|
-
|
|
20
|
-
## Scope
|
|
21
|
-
|
|
22
|
-
Default scope is the diff between the current branch and its merge-base
|
|
23
|
-
with the repo's base branch (`git diff $(git merge-base origin/main
|
|
24
|
-
HEAD)...HEAD`, or `origin/main` substituted with whatever base branch
|
|
25
|
-
applies). If `$ARGUMENTS` names a path or a narrower scope, review only
|
|
26
|
-
that instead of the full diff.
|
|
27
|
-
|
|
28
|
-
This skill is deliberately diff-scoped. For a whole-package audit before a
|
|
29
|
-
release (full-codebase security red-team, docs review, everything since
|
|
30
|
-
the last tag) use `/prep-release` instead.
|
|
31
|
-
|
|
32
|
-
`$ARGUMENTS` may also contain:
|
|
33
|
-
- An effort level to pass through to `/code-review` (`low`/`medium`/
|
|
34
|
-
`high`/`high→max`/`ultra`). Default: `medium`.
|
|
35
|
-
- A round cap override, e.g. `--max-rounds 3`. Default: 5.
|
|
36
|
-
|
|
37
|
-
## The round loop
|
|
38
|
-
|
|
39
|
-
1. Invoke `/code-review <effort> --fix` (via the `Skill` tool) against the
|
|
40
|
-
current scope.
|
|
41
|
-
2. **Dry round (0 findings) → converged.** Stop and move to verification.
|
|
42
|
-
3. **Findings found and fixed** → do not declare victory yet. Run another
|
|
43
|
-
round to confirm the fixes didn't introduce a regression and that
|
|
44
|
-
nothing was missed.
|
|
45
|
-
4. **No-progress detection**: if two consecutive rounds return the same
|
|
46
|
-
non-empty set of findings, `--fix` isn't resolving them mechanically
|
|
47
|
-
(likely a design/architecture call that needs a human). Stop looping,
|
|
48
|
-
list the stuck findings, and hand them to the user instead of retrying
|
|
49
|
-
forever.
|
|
50
|
-
5. **Safety cap**: if the round cap is reached without converging or
|
|
51
|
-
getting stuck, stop and report the remaining findings — don't loop
|
|
52
|
-
silently past the cap.
|
|
53
|
-
|
|
54
|
-
Each round's fixes should stay reviewable: don't squash multiple rounds
|
|
55
|
-
into one silent edit. Note per-round changes in the final summary so the
|
|
56
|
-
user can inspect them with `git diff`.
|
|
57
|
-
|
|
58
|
-
## Review criteria
|
|
59
|
-
|
|
60
|
-
Beyond whatever `/code-review` already checks for correctness bugs and
|
|
61
|
-
reuse/simplification/efficiency, every round in a Ruby gem repo like this
|
|
62
|
-
one should also flag:
|
|
63
|
-
|
|
64
|
-
- **Ruby idiom / DRY**: semantic, expressive naming; no duplicated logic
|
|
65
|
-
that should be extracted into a shared method; idiomatic use of
|
|
66
|
-
Ruby/Enumerable over manual loops where it reads better; no needless
|
|
67
|
-
boilerplate.
|
|
68
|
-
- **Gem publishing discipline**: don't break public interfaces unless
|
|
69
|
-
necessary.
|
|
70
|
-
- If a break is necessary, it must come with: a `CHANGELOG.md` entry in
|
|
71
|
-
Keep a Changelog format (this repo already follows that format — match
|
|
72
|
-
the style of existing entries, e.g. plain-English migration notes like
|
|
73
|
-
the `0.5.0` entry) and a version bump in `lib/coolhand/version.rb` that
|
|
74
|
-
matches SemVer (patch = fix, minor = backward-compatible addition,
|
|
75
|
-
minor = breaking change while pre-1.0, consistent with this repo's own
|
|
76
|
-
versioning history).
|
|
77
|
-
- Check the optional-provider-dependency rule from this repo's
|
|
78
|
-
`CLAUDE.md`: provider SDK `require`s (`openai`, `anthropic`,
|
|
79
|
-
`google-generativeai`, etc.) must stay scoped to the file that uses
|
|
80
|
-
them and lazy-loaded, never added to `lib/coolhand.rb` or the gemspec
|
|
81
|
-
as a hard dependency.
|
|
82
|
-
- **Security**: injection risks, unsafe deserialization, secrets or
|
|
83
|
-
credentials logged or committed, unvalidated input crossing a trust
|
|
84
|
-
boundary, and anything touching how API keys/tokens are handled,
|
|
85
|
-
stored, or transmitted.
|
|
86
|
-
|
|
87
|
-
## Post-loop verification
|
|
88
|
-
|
|
89
|
-
Once the loop converges (or stops early per the rules above), run the
|
|
90
|
-
project's lint/test command and include the result in the final summary.
|
|
91
|
-
For this repo that's `bundle exec rake` (runs `rspec` then `rubocop`, per
|
|
92
|
-
the `Rakefile`). If a fix round changed something outside this repo's
|
|
93
|
-
usual toolchain, discover the right command instead of assuming.
|
|
94
|
-
|
|
95
|
-
## Rationalizations to resist
|
|
96
|
-
|
|
97
|
-
- *"The first round already looked clean, I don't need a confirming
|
|
98
|
-
round."* A fix round can introduce its own regression. Always re-review
|
|
99
|
-
after applying fixes before declaring convergence.
|
|
100
|
-
- *"Rubocop passed, so the review is done."* Lint passing is not the same
|
|
101
|
-
as the review being clean — lint doesn't check the criteria above
|
|
102
|
-
(interface breakage, changelog/version discipline, security). Run both.
|
|
103
|
-
- *"This finding keeps coming back, I'll just keep re-running --fix and
|
|
104
|
-
it'll eventually take."* If the same non-empty finding set repeats
|
|
105
|
-
across two rounds, `--fix` isn't going to resolve it. Stop and surface
|
|
106
|
-
it — looping past that point just burns rounds for no gain.
|
|
107
|
-
|
|
108
|
-
## Safety
|
|
109
|
-
|
|
110
|
-
- Never force-push or amend existing commits as part of this loop.
|
|
111
|
-
- The skill only edits the working tree; committing and pushing stays with
|
|
112
|
-
the user.
|
|
@@ -1,160 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: prep-release
|
|
3
|
-
description: |
|
|
4
|
-
Prepares this gem for a release: runs the full test suite, updates and
|
|
5
|
-
cleans up the docs (README, CHANGELOG, docs/*.md) to reflect every change
|
|
6
|
-
since the last tag, bumps the version if that hasn't already been done,
|
|
7
|
-
and red-teams the whole package for security - not just this release's
|
|
8
|
-
diff. Never tags or pushes. Use when the user types /prep-release, asks
|
|
9
|
-
to "prep a release", "get ready to cut a release", "release checklist",
|
|
10
|
-
or wants a pre-release audit before tagging/publishing a new version.
|
|
11
|
-
user_invocable: true
|
|
12
|
-
version: 0.2.0
|
|
13
|
-
---
|
|
14
|
-
|
|
15
|
-
# Prep Release
|
|
16
|
-
|
|
17
|
-
Three phases, run in order. This is a whole-package audit, not a diff
|
|
18
|
-
review — do not scope any phase to just what changed since the last
|
|
19
|
-
commit. For an iterative diff-scoped review during normal development, use
|
|
20
|
-
`/loop-review` instead; this skill is for the release boundary.
|
|
21
|
-
|
|
22
|
-
## Phase 1: Run all tests
|
|
23
|
-
|
|
24
|
-
Run `bundle exec rake` (`rspec` then `rubocop`, per the `Rakefile`). If a
|
|
25
|
-
fix round changed something outside this repo's usual toolchain, discover
|
|
26
|
-
the right command instead of assuming. All specs must pass and RuboCop
|
|
27
|
-
must report zero offenses before continuing — a release doesn't ship on a
|
|
28
|
-
red build. If either fails, stop here and report the failures; fixing
|
|
29
|
-
genuine bugs takes priority over Phase 2/3 work.
|
|
30
|
-
|
|
31
|
-
Then judge coverage on quality, not just the SimpleCov percentage the rake
|
|
32
|
-
run reports (written to `coverage/`):
|
|
33
|
-
|
|
34
|
-
1. **Find the gaps.** List files/lines SimpleCov marks uncovered. Weight
|
|
35
|
-
by risk: an uncovered error-handling branch or security check
|
|
36
|
-
(signature/header validation) matters more than an uncovered
|
|
37
|
-
`attr_reader`.
|
|
38
|
-
2. **Audit existing tests for meaningfulness, not just count.** Flag tests
|
|
39
|
-
that only assert a stub returns what it was stubbed to return without
|
|
40
|
-
exercising real conditional logic in the subject under test; missing
|
|
41
|
-
negative/error-path cases (invalid input, malformed provider responses,
|
|
42
|
-
network failure); missing domain edge cases (empty batch results,
|
|
43
|
-
duplicate-request prevention, concurrent access, streaming vs
|
|
44
|
-
non-streaming shapes).
|
|
45
|
-
3. **Recommend, don't pad.** Propose specific specs for the highest-risk
|
|
46
|
-
gaps, named by `file:describe/context`. Don't add tests purely to move
|
|
47
|
-
the percentage — a test with no failure mode it would catch adds
|
|
48
|
-
maintenance cost without adding signal.
|
|
49
|
-
|
|
50
|
-
## Phase 2: Update and clean the docs
|
|
51
|
-
|
|
52
|
-
1. Find the last release tag: `git describe --tags --abbrev=0`.
|
|
53
|
-
2. Diff everything since that tag: `git log <last-tag>..HEAD --oneline` and
|
|
54
|
-
`git diff <last-tag>..HEAD -- lib/` to see every behavioral change, not
|
|
55
|
-
just the most recent commit.
|
|
56
|
-
3. For each change, check it's reflected in:
|
|
57
|
-
- `CHANGELOG.md` — every notable change since the last tag needs an
|
|
58
|
-
entry under `[Unreleased]` (or a new version heading), in Keep a
|
|
59
|
-
Changelog format matching this repo's existing entries (see past
|
|
60
|
-
entries for style — plain-English migration notes for anything
|
|
61
|
-
behavior-affecting).
|
|
62
|
-
- `README.md` / `docs/*.md` — any new config option, public method, or
|
|
63
|
-
behavior change needs the relevant section updated. Follow this
|
|
64
|
-
repo's docs philosophy from `CLAUDE.md`: the README stays a scannable
|
|
65
|
-
landing page (basic config/feedback snippets only); anything needing
|
|
66
|
-
more than one code block belongs in `docs/`.
|
|
67
|
-
4. **Clean, don't just append.** Look for docs that are now stale,
|
|
68
|
-
contradictory, or redundant given the accumulated changes since the
|
|
69
|
-
last tag — consolidate/rewrite rather than layering a new paragraph on
|
|
70
|
-
top of an outdated one. Remove docs for anything removed from the gem.
|
|
71
|
-
5. **Bump the version if it hasn't already been done.** Check whether
|
|
72
|
-
`lib/coolhand/version.rb` was already bumped for the changes
|
|
73
|
-
accumulated since the last tag (e.g. by an earlier commit on this
|
|
74
|
-
branch) — if so, leave it. If not, determine the SemVer bump this
|
|
75
|
-
repo's convention implies (patch = fix, minor = backward-compatible
|
|
76
|
-
addition or breaking change while pre-1.0), write it to
|
|
77
|
-
`lib/coolhand/version.rb`, turn the `[Unreleased]` CHANGELOG heading
|
|
78
|
-
into `## [X.Y.Z] - <today's date>`, and run `bundle install` so
|
|
79
|
-
`Gemfile.lock`'s `coolhand (X.Y.Z)` line matches. State the version and
|
|
80
|
-
bump rationale in the wrap-up summary so the user can override it if
|
|
81
|
-
they'd have picked differently — don't ask before writing it, since
|
|
82
|
-
this is a mechanical, reversible edit gated by Phase 1's green build.
|
|
83
|
-
|
|
84
|
-
## Phase 3: Red-team the whole package
|
|
85
|
-
|
|
86
|
-
Adversarially review the entire `lib/` tree (not just this release's
|
|
87
|
-
diff) for security issues. This gem intercepts outgoing LLM API traffic
|
|
88
|
-
and logs it to Coolhand, so hunt specifically for:
|
|
89
|
-
|
|
90
|
-
- **Credential/secret leakage**: does any interceptor, logger, or error
|
|
91
|
-
handler write an API key, bearer token, or provider auth header value
|
|
92
|
-
into a log line, exception message, or the payload sent to Coolhand?
|
|
93
|
-
Check every header-sanitization path actually strips what it claims to
|
|
94
|
-
(e.g. `WebhookValidator`, provider header redaction) rather than
|
|
95
|
-
sanitizing a differently-cased or differently-named header.
|
|
96
|
-
- **Webhook/signature validation**: can `WebhookValidator#valid?` (or
|
|
97
|
-
equivalent) be bypassed — timing-unsafe comparison instead of a
|
|
98
|
-
constant-time compare, an environment where an empty/missing signature
|
|
99
|
-
is treated as valid, or a fallback path meant for development that's
|
|
100
|
-
reachable in production.
|
|
101
|
-
- **SSRF / address matching**: the default and configurable intercept
|
|
102
|
-
address lists — can a crafted URL (redirect, unicode homograph,
|
|
103
|
-
userinfo trick, subdomain confusion) match or evade the intended
|
|
104
|
-
host-matching logic in a way that intercepts (or fails to intercept)
|
|
105
|
-
the wrong destination?
|
|
106
|
-
- **ReDoS**: any regex built from configurable or user-influenced input
|
|
107
|
-
(intercept patterns, header names) — check for catastrophic backtracking
|
|
108
|
-
shapes (nested quantifiers, overlapping alternation).
|
|
109
|
-
- **Thread safety**: this gem documents thread-safe operation and
|
|
110
|
-
duplicate-request prevention — look for unsynchronized shared mutable
|
|
111
|
-
state (class-level `@@` vars, memoized `@client` on a shared instance)
|
|
112
|
-
that a concurrent request could race on.
|
|
113
|
-
- **Unsafe deserialization**: any `JSON.parse` without checking for
|
|
114
|
-
`Marshal.load`/`YAML.load` (unsafe) usage, and any parsing of
|
|
115
|
-
webhook/batch-result payloads that trusts attacker-controlled shape
|
|
116
|
-
without validation.
|
|
117
|
-
- **Fail-open vs fail-closed**: when Coolhand's API is unreachable, rate
|
|
118
|
-
limited, or returns malformed data, does the gem fail open in a way that
|
|
119
|
-
silently drops security-relevant logging, or fail in a way that breaks
|
|
120
|
-
the host application's actual LLM call (the interceptor must never break
|
|
121
|
-
the underlying request)?
|
|
122
|
-
|
|
123
|
-
For each finding, report file, line, a concrete failure scenario, and
|
|
124
|
-
severity. Apply safe, mechanical, low-risk fixes directly (e.g. a missing
|
|
125
|
-
header-redaction pattern, a missing timeout). Flag but do not silently
|
|
126
|
-
apply anything that's a behavior/architecture decision (e.g. changing a
|
|
127
|
-
fail-open security default, adding replay protection, moving synchronous
|
|
128
|
-
work to a background thread) — surface these to the user for a decision,
|
|
129
|
-
the same "hand it to a human" rule `/loop-review` uses for stuck findings.
|
|
130
|
-
|
|
131
|
-
## Wrap-up
|
|
132
|
-
|
|
133
|
-
Report one consolidated summary: test/lint result, coverage-quality gaps
|
|
134
|
-
plus recommended specs, docs updated, the version (bumped or already
|
|
135
|
-
current, and why), and security findings split into fixed vs.
|
|
136
|
-
flagged-for-decision.
|
|
137
|
-
|
|
138
|
-
## Safety
|
|
139
|
-
|
|
140
|
-
- Bumping `lib/coolhand/version.rb`, finalizing the CHANGELOG heading, and
|
|
141
|
-
running `bundle install` for the lockfile are all in scope and don't need
|
|
142
|
-
a stop-and-ask — they're mechanical, reversible, and gated on Phase 1
|
|
143
|
-
already being green.
|
|
144
|
-
- Never create or push a git tag, never push commits, and never run
|
|
145
|
-
`rake release`, `gem push`, or anything else that publishes the gem or
|
|
146
|
-
touches the remote. Tagging and publishing are the user's action once
|
|
147
|
-
they've reviewed this skill's report, not something this skill does.
|
|
148
|
-
|
|
149
|
-
## Rationalizations to resist
|
|
150
|
-
|
|
151
|
-
- *"The diff since the last tag is small, I'll skip the red-team."* Small
|
|
152
|
-
diffs can still sit on top of latent issues in code nobody's touched
|
|
153
|
-
recently — that's exactly what "whole package, not just the diff" means.
|
|
154
|
-
- *"Tests pass, so coverage is fine."* Passing tests and meaningful
|
|
155
|
-
coverage are different questions. A red build blocks release; a green
|
|
156
|
-
build with hollow tests doesn't guarantee anything.
|
|
157
|
-
- *"Docs are close enough, I'll skip the cleanup pass."* Accumulated
|
|
158
|
-
changes since the last tag are exactly when docs drift from behavior —
|
|
159
|
-
this phase exists because per-PR doc updates miss the cross-cutting
|
|
160
|
-
view.
|
data/.idea/coolhand-ruby.iml
DELETED
data/CLAUDE.md
DELETED
|
@@ -1,47 +0,0 @@
|
|
|
1
|
-
# Development Guidelines
|
|
2
|
-
|
|
3
|
-
## Optional Provider Dependencies
|
|
4
|
-
|
|
5
|
-
Coolhand supports multiple LLM providers (OpenAI, Anthropic, Google Gemini, etc.). These provider gems should **never** be required at gem load time, as clients may not use all providers and shouldn't be forced to install unnecessary dependencies.
|
|
6
|
-
|
|
7
|
-
**Rule**: Any require for provider SDKs (openai, anthropic, google-generativeai, etc.) must be:
|
|
8
|
-
1. Placed in the file where it's actually used (not in the main coolhand.rb)
|
|
9
|
-
2. Only executed when that provider's functionality is accessed
|
|
10
|
-
3. Not declared as a hard dependency in coolhand-ruby.gemspec
|
|
11
|
-
|
|
12
|
-
Example pattern:
|
|
13
|
-
```ruby
|
|
14
|
-
# ❌ DON'T: In lib/coolhand.rb (loads unconditionally)
|
|
15
|
-
require "openai"
|
|
16
|
-
|
|
17
|
-
# ✅ DO: In lib/coolhand/open_ai/batch_result_processor.rb (only when needed)
|
|
18
|
-
require "openai"
|
|
19
|
-
|
|
20
|
-
module Coolhand
|
|
21
|
-
module OpenAi
|
|
22
|
-
class BatchResultProcessor
|
|
23
|
-
def client
|
|
24
|
-
@client ||= OpenAI::Client.new
|
|
25
|
-
end
|
|
26
|
-
end
|
|
27
|
-
end
|
|
28
|
-
end
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
This ensures:
|
|
32
|
-
- Gem loads cleanly regardless of what providers are installed
|
|
33
|
-
- Apps using path gems (local development) don't break from missing optional dependencies
|
|
34
|
-
- Users only need gems for providers they actually use
|
|
35
|
-
|
|
36
|
-
## README and docs philosophy
|
|
37
|
-
|
|
38
|
-
The README is a landing page — install, quick start, what it supports, where to go next. Keep it scannable. When in doubt, link rather than expand.
|
|
39
|
-
|
|
40
|
-
**Three rules:**
|
|
41
|
-
- **Config**: the basic `Coolhand.configure` snippet belongs in the README. Anything requiring more than one code block (self-hosted `base_url`, custom intercept addresses) goes in `docs/configuration.md`.
|
|
42
|
-
- **Feedback**: the basic `create_feedback` snippet belongs in the README. The full field table, matching strategies, and sentiment conversion details go in `docs/feedback.md`.
|
|
43
|
-
- **Integrations**: each integration gets its own `docs/<name>.md` file. The README links to them from the Documentation section.
|
|
44
|
-
|
|
45
|
-
**Align with coolhand-node.** When adding a section that exists in the Node README, match its structure and tone. The two READMEs should feel like siblings.
|
|
46
|
-
|
|
47
|
-
**Discoverability (SEO / AEO).** Write headings, the package description, and the supported-libraries list with search engines and AI agents in mind: use full provider/framework names (e.g. "OpenAI", "Anthropic", "Google Gemini", "Cohere") rather than abbreviations. The goal is that searches for "Ruby LLM monitoring", "Anthropic Ruby logging", or "OpenAI Ruby observability" surface this gem.
|