coolhand 0.4.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3f66ea8b94e39769d66543c75987c2108397f89c93dc53748753b396ca76d921
4
- data.tar.gz: 560b9ec89c71b31095b5f579bb270b295a79b87a7edbec495179f6f1b764f77e
3
+ metadata.gz: daed2b7ac4fe321602eb90f673cb385bd930b9ff503bf7e5d7851cbf06422617
4
+ data.tar.gz: 5d01a6386d142cdb31c490574191ac3867dabd30670c2a832c2aa18c7150c872
5
5
  SHA512:
6
- metadata.gz: ebd806bb3744d3fd16673a98f7746b144b468a158ec855b32a4ba142cf110ff4a78b03ea1b95106b8f29d011425bbe0ec44b8f853acb87b9e4037cf87843991f
7
- data.tar.gz: 3a8078f5dbe7f3a5c7a0892b391a7085d54b5a6aa1db386075e34470c89802ca0bd6bb5dc5654dffa7fcb2595da3df7b2210a52568595c5c77929e435d1a13b8
6
+ metadata.gz: a0681a3419c92e13fbae7a1b81c6d81b7a1cf2f088571dd44b4d2694c0ae5bf5c4c4312d7bff53ccc740833290bc931b932c1179244f004090192b0baaeb19f0
7
+ data.tar.gz: 9efd5e33528e0caeecc770c7286085320f9e4218f01fcf592ca82a27c96df71cf5ae52712a5a9aaeaf0702774be450bb7c3b7c71b0df8e57f56bd038f54f1b3d
@@ -0,0 +1,112 @@
1
+ ---
2
+ name: loop-review
3
+ description: |
4
+ Iteratively runs code review against the current diff, applies fixes, and
5
+ re-reviews until a round comes back clean (or a safety cap is hit). Use
6
+ when the user types /loop-review, asks to "loop the review", "review
7
+ until clean", "keep reviewing and fixing until nothing's left", or wants
8
+ a self-healing code review cycle instead of a single one-shot pass.
9
+ user_invocable: true
10
+ version: 0.3.0
11
+ ---
12
+
13
+ # Loop Review
14
+
15
+ This skill runs `/code-review` repeatedly against the working diff, applies
16
+ fixes between rounds, and re-reviews until a round finds nothing new or a
17
+ safety cap is reached. Use it after a merge, a large diff, or whenever a
18
+ single review pass isn't enough to converge on a clean state.
19
+
20
+ ## Scope
21
+
22
+ Default scope is the diff between the current branch and its merge-base
23
+ with the repo's base branch (`git diff $(git merge-base origin/main
24
+ HEAD)...HEAD`, or `origin/main` substituted with whatever base branch
25
+ applies). If `$ARGUMENTS` names a path or a narrower scope, review only
26
+ that instead of the full diff.
27
+
28
+ This skill is deliberately diff-scoped. For a whole-package audit before a
29
+ release (full-codebase security red-team, docs review, everything since
30
+ the last tag) use `/prep-release` instead.
31
+
32
+ `$ARGUMENTS` may also contain:
33
+ - An effort level to pass through to `/code-review` (`low`/`medium`/
34
+ `high`/`high→max`/`ultra`). Default: `medium`.
35
+ - A round cap override, e.g. `--max-rounds 3`. Default: 5.
36
+
37
+ ## The round loop
38
+
39
+ 1. Invoke `/code-review <effort> --fix` (via the `Skill` tool) against the
40
+ current scope.
41
+ 2. **Dry round (0 findings) → converged.** Stop and move to verification.
42
+ 3. **Findings found and fixed** → do not declare victory yet. Run another
43
+ round to confirm the fixes didn't introduce a regression and that
44
+ nothing was missed.
45
+ 4. **No-progress detection**: if two consecutive rounds return the same
46
+ non-empty set of findings, `--fix` isn't resolving them mechanically
47
+ (likely a design/architecture call that needs a human). Stop looping,
48
+ list the stuck findings, and hand them to the user instead of retrying
49
+ forever.
50
+ 5. **Safety cap**: if the round cap is reached without converging or
51
+ getting stuck, stop and report the remaining findings — don't loop
52
+ silently past the cap.
53
+
54
+ Each round's fixes should stay reviewable: don't squash multiple rounds
55
+ into one silent edit. Note per-round changes in the final summary so the
56
+ user can inspect them with `git diff`.
57
+
58
+ ## Review criteria
59
+
60
+ Beyond whatever `/code-review` already checks for correctness bugs and
61
+ reuse/simplification/efficiency, every round in a Ruby gem repo like this
62
+ one should also flag:
63
+
64
+ - **Ruby idiom / DRY**: semantic, expressive naming; no duplicated logic
65
+ that should be extracted into a shared method; idiomatic use of
66
+ Ruby/Enumerable over manual loops where it reads better; no needless
67
+ boilerplate.
68
+ - **Gem publishing discipline**: don't break public interfaces unless
69
+ necessary.
70
+ - If a break is necessary, it must come with: a `CHANGELOG.md` entry in
71
+ Keep a Changelog format (this repo already follows that format — match
72
+ the style of existing entries, e.g. plain-English migration notes like
73
+ the `0.5.0` entry) and a version bump in `lib/coolhand/version.rb` that
74
+ matches SemVer (patch = fix, minor = backward-compatible addition,
75
+ minor = breaking change while pre-1.0, consistent with this repo's own
76
+ versioning history).
77
+ - Check the optional-provider-dependency rule from this repo's
78
+ `CLAUDE.md`: provider SDK `require`s (`openai`, `anthropic`,
79
+ `google-generativeai`, etc.) must stay scoped to the file that uses
80
+ them and lazy-loaded, never added to `lib/coolhand.rb` or the gemspec
81
+ as a hard dependency.
82
+ - **Security**: injection risks, unsafe deserialization, secrets or
83
+ credentials logged or committed, unvalidated input crossing a trust
84
+ boundary, and anything touching how API keys/tokens are handled,
85
+ stored, or transmitted.
86
+
87
+ ## Post-loop verification
88
+
89
+ Once the loop converges (or stops early per the rules above), run the
90
+ project's lint/test command and include the result in the final summary.
91
+ For this repo that's `bundle exec rake` (runs `rspec` then `rubocop`, per
92
+ the `Rakefile`). If a fix round changed something outside this repo's
93
+ usual toolchain, discover the right command instead of assuming.
94
+
95
+ ## Rationalizations to resist
96
+
97
+ - *"The first round already looked clean, I don't need a confirming
98
+ round."* A fix round can introduce its own regression. Always re-review
99
+ after applying fixes before declaring convergence.
100
+ - *"Rubocop passed, so the review is done."* Lint passing is not the same
101
+ as the review being clean — lint doesn't check the criteria above
102
+ (interface breakage, changelog/version discipline, security). Run both.
103
+ - *"This finding keeps coming back, I'll just keep re-running --fix and
104
+ it'll eventually take."* If the same non-empty finding set repeats
105
+ across two rounds, `--fix` isn't going to resolve it. Stop and surface
106
+ it — looping past that point just burns rounds for no gain.
107
+
108
+ ## Safety
109
+
110
+ - Never force-push or amend existing commits as part of this loop.
111
+ - The skill only edits the working tree; committing and pushing stays with
112
+ the user.
@@ -0,0 +1,160 @@
1
+ ---
2
+ name: prep-release
3
+ description: |
4
+ Prepares this gem for a release: runs the full test suite, updates and
5
+ cleans up the docs (README, CHANGELOG, docs/*.md) to reflect every change
6
+ since the last tag, bumps the version if that hasn't already been done,
7
+ and red-teams the whole package for security - not just this release's
8
+ diff. Never tags or pushes. Use when the user types /prep-release, asks
9
+ to "prep a release", "get ready to cut a release", "release checklist",
10
+ or wants a pre-release audit before tagging/publishing a new version.
11
+ user_invocable: true
12
+ version: 0.2.0
13
+ ---
14
+
15
+ # Prep Release
16
+
17
+ Three phases, run in order. This is a whole-package audit, not a diff
18
+ review — do not scope any phase to just what changed since the last
19
+ commit. For an iterative diff-scoped review during normal development, use
20
+ `/loop-review` instead; this skill is for the release boundary.
21
+
22
+ ## Phase 1: Run all tests
23
+
24
+ Run `bundle exec rake` (`rspec` then `rubocop`, per the `Rakefile`). If a
25
+ fix round changed something outside this repo's usual toolchain, discover
26
+ the right command instead of assuming. All specs must pass and RuboCop
27
+ must report zero offenses before continuing — a release doesn't ship on a
28
+ red build. If either fails, stop here and report the failures; fixing
29
+ genuine bugs takes priority over Phase 2/3 work.
30
+
31
+ Then judge coverage on quality, not just the SimpleCov percentage the rake
32
+ run reports (written to `coverage/`):
33
+
34
+ 1. **Find the gaps.** List files/lines SimpleCov marks uncovered. Weight
35
+ by risk: an uncovered error-handling branch or security check
36
+ (signature/header validation) matters more than an uncovered
37
+ `attr_reader`.
38
+ 2. **Audit existing tests for meaningfulness, not just count.** Flag tests
39
+ that only assert a stub returns what it was stubbed to return without
40
+ exercising real conditional logic in the subject under test; missing
41
+ negative/error-path cases (invalid input, malformed provider responses,
42
+ network failure); missing domain edge cases (empty batch results,
43
+ duplicate-request prevention, concurrent access, streaming vs
44
+ non-streaming shapes).
45
+ 3. **Recommend, don't pad.** Propose specific specs for the highest-risk
46
+ gaps, named by `file:describe/context`. Don't add tests purely to move
47
+ the percentage — a test with no failure mode it would catch adds
48
+ maintenance cost without adding signal.
49
+
50
+ ## Phase 2: Update and clean the docs
51
+
52
+ 1. Find the last release tag: `git describe --tags --abbrev=0`.
53
+ 2. Diff everything since that tag: `git log <last-tag>..HEAD --oneline` and
54
+ `git diff <last-tag>..HEAD -- lib/` to see every behavioral change, not
55
+ just the most recent commit.
56
+ 3. For each change, check it's reflected in:
57
+ - `CHANGELOG.md` — every notable change since the last tag needs an
58
+ entry under `[Unreleased]` (or a new version heading), in Keep a
59
+ Changelog format matching this repo's existing entries (see past
60
+ entries for style — plain-English migration notes for anything
61
+ behavior-affecting).
62
+ - `README.md` / `docs/*.md` — any new config option, public method, or
63
+ behavior change needs the relevant section updated. Follow this
64
+ repo's docs philosophy from `CLAUDE.md`: the README stays a scannable
65
+ landing page (basic config/feedback snippets only); anything needing
66
+ more than one code block belongs in `docs/`.
67
+ 4. **Clean, don't just append.** Look for docs that are now stale,
68
+ contradictory, or redundant given the accumulated changes since the
69
+ last tag — consolidate/rewrite rather than layering a new paragraph on
70
+ top of an outdated one. Remove docs for anything removed from the gem.
71
+ 5. **Bump the version if it hasn't already been done.** Check whether
72
+ `lib/coolhand/version.rb` was already bumped for the changes
73
+ accumulated since the last tag (e.g. by an earlier commit on this
74
+ branch) — if so, leave it. If not, determine the SemVer bump this
75
+ repo's convention implies (patch = fix, minor = backward-compatible
76
+ addition or breaking change while pre-1.0), write it to
77
+ `lib/coolhand/version.rb`, turn the `[Unreleased]` CHANGELOG heading
78
+ into `## [X.Y.Z] - <today's date>`, and run `bundle install` so
79
+ `Gemfile.lock`'s `coolhand (X.Y.Z)` line matches. State the version and
80
+ bump rationale in the wrap-up summary so the user can override it if
81
+ they'd have picked differently — don't ask before writing it, since
82
+ this is a mechanical, reversible edit gated by Phase 1's green build.
83
+
84
+ ## Phase 3: Red-team the whole package
85
+
86
+ Adversarially review the entire `lib/` tree (not just this release's
87
+ diff) for security issues. This gem intercepts outgoing LLM API traffic
88
+ and logs it to Coolhand, so hunt specifically for:
89
+
90
+ - **Credential/secret leakage**: does any interceptor, logger, or error
91
+ handler write an API key, bearer token, or provider auth header value
92
+ into a log line, exception message, or the payload sent to Coolhand?
93
+ Check every header-sanitization path actually strips what it claims to
94
+ (e.g. `WebhookValidator`, provider header redaction) rather than
95
+ sanitizing a differently-cased or differently-named header.
96
+ - **Webhook/signature validation**: can `WebhookValidator#valid?` (or
97
+ equivalent) be bypassed — timing-unsafe comparison instead of a
98
+ constant-time compare, an environment where an empty/missing signature
99
+ is treated as valid, or a fallback path meant for development that's
100
+ reachable in production.
101
+ - **SSRF / address matching**: the default and configurable intercept
102
+ address lists — can a crafted URL (redirect, unicode homograph,
103
+ userinfo trick, subdomain confusion) match or evade the intended
104
+ host-matching logic in a way that intercepts (or fails to intercept)
105
+ the wrong destination?
106
+ - **ReDoS**: any regex built from configurable or user-influenced input
107
+ (intercept patterns, header names) — check for catastrophic backtracking
108
+ shapes (nested quantifiers, overlapping alternation).
109
+ - **Thread safety**: this gem documents thread-safe operation and
110
+ duplicate-request prevention — look for unsynchronized shared mutable
111
+ state (class-level `@@` vars, memoized `@client` on a shared instance)
112
+ that a concurrent request could race on.
113
+ - **Unsafe deserialization**: any `JSON.parse` without checking for
114
+ `Marshal.load`/`YAML.load` (unsafe) usage, and any parsing of
115
+ webhook/batch-result payloads that trusts attacker-controlled shape
116
+ without validation.
117
+ - **Fail-open vs fail-closed**: when Coolhand's API is unreachable, rate
118
+ limited, or returns malformed data, does the gem fail open in a way that
119
+ silently drops security-relevant logging, or fail in a way that breaks
120
+ the host application's actual LLM call (the interceptor must never break
121
+ the underlying request)?
122
+
123
+ For each finding, report file, line, a concrete failure scenario, and
124
+ severity. Apply safe, mechanical, low-risk fixes directly (e.g. a missing
125
+ header-redaction pattern, a missing timeout). Flag but do not silently
126
+ apply anything that's a behavior/architecture decision (e.g. changing a
127
+ fail-open security default, adding replay protection, moving synchronous
128
+ work to a background thread) — surface these to the user for a decision,
129
+ the same "hand it to a human" rule `/loop-review` uses for stuck findings.
130
+
131
+ ## Wrap-up
132
+
133
+ Report one consolidated summary: test/lint result, coverage-quality gaps
134
+ plus recommended specs, docs updated, the version (bumped or already
135
+ current, and why), and security findings split into fixed vs.
136
+ flagged-for-decision.
137
+
138
+ ## Safety
139
+
140
+ - Bumping `lib/coolhand/version.rb`, finalizing the CHANGELOG heading, and
141
+ running `bundle install` for the lockfile are all in scope and don't need
142
+ a stop-and-ask — they're mechanical, reversible, and gated on Phase 1
143
+ already being green.
144
+ - Never create or push a git tag, never push commits, and never run
145
+ `rake release`, `gem push`, or anything else that publishes the gem or
146
+ touches the remote. Tagging and publishing are the user's action once
147
+ they've reviewed this skill's report, not something this skill does.
148
+
149
+ ## Rationalizations to resist
150
+
151
+ - *"The diff since the last tag is small, I'll skip the red-team."* Small
152
+ diffs can still sit on top of latent issues in code nobody's touched
153
+ recently — that's exactly what "whole package, not just the diff" means.
154
+ - *"Tests pass, so coverage is fine."* Passing tests and meaningful
155
+ coverage are different questions. A red build blocks release; a green
156
+ build with hollow tests doesn't guarantee anything.
157
+ - *"Docs are close enough, I'll skip the cleanup pass."* Accumulated
158
+ changes since the last tag are exactly when docs drift from behavior —
159
+ this phase exists because per-PR doc updates miss the cross-cutting
160
+ view.
data/CHANGELOG.md CHANGED
@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.5.0] - 2026-07-30
11
+
12
+ ### Changed
13
+ - **`llm_request_log_id` and `workload_id` in feedback API responses are now hashid strings, not raw integers** — the Coolhand API now returns these as hashids, matching every other external-facing identifier on the record (they previously leaked the raw integer foreign key). This gem never typed or coerced these fields (plain hashes throughout), so no code changes are required here, but if your application stores or compares `result[:llm_request_log_id]` or `result[:workload_id]` as an integer, update it to treat the value as an opaque string identifier instead. The `create_feedback`/`update_feedback` input fields (`llm_request_log_id`, `workload_hashid`) are unaffected — they still accept either a raw integer or a hashid string.
14
+ - **`id` in feedback API responses has actually been a hashid string for some time** — flagging here since it's the same category of field; no gem-level change needed since this was never typed.
15
+ - `BaseInterceptor.sanitize_headers` and `LoggerService#sanitize_headers` (used for the main request/response logging path and webhook forwarding, respectively) now share the same sensitive-header pattern instead of each maintaining its own list, so both paths redact consistently.
16
+
17
+ ### Security
18
+ - **AWS Bedrock SigV4 session tokens (`X-Amz-Security-Token`) are now redacted before logging.** The interceptor's header sanitizer used a hardcoded list of known API-key header names (`api-key`, `x-api-key`, `x-goog-api-key`, `openai-api-key`) instead of a general pattern, so this header — present on requests signed with temporary/STS credentials, the common case for Bedrock — was forwarded to the Coolhand backend and printed in `debug_mode` unredacted. The sanitizer now redacts any header whose name matches `key`, `token`, `secret`, `signature`, or `authorization`, closing this and similar gaps for any current or future provider header.
19
+ - **`debug_mode`'s "skipping capture" log line no longer prints the raw URL.** When a request matched `exclude_api_patterns` while `debug_mode` was on, the log line bypassed the usual URL sanitizer, so a Gemini/Vertex `?key=...` query-param API key could be printed in full. It now goes through the same URL sanitizer as every other log line.
20
+ - **Outbound requests to the Coolhand backend now set a 5-second connect/read timeout.** Previously this call had no explicit timeout and ran inline on the same thread as the intercepted LLM request, so a slow or unreachable Coolhand endpoint (including a self-hosted `base_url`) could add Ruby's ~60s Net::HTTP default (up to ~120s total) of latency to real LLM calls made by the host app.
21
+
22
+ ### Removed
23
+ - Deleted unused `BaseInterceptor` methods with no callers anywhere in the gem: `extract_response_data`, `extract_usage_metadata`, `clean_request_headers`, `clean_response_headers`. The latter two duplicated `sanitize_headers` with a narrower, case-sensitive header list and were never wired into any interceptor — dead code carrying its own security debt.
24
+
25
+ ### Dependencies
26
+ - Bumped `faraday` from 2.14.2 to 2.14.3 (#73).
27
+
10
28
  ## [0.4.0] - 2026-06-22
11
29
 
12
30
  ### Added
data/CLAUDE.md CHANGED
@@ -32,3 +32,16 @@ This ensures:
32
32
  - Gem loads cleanly regardless of what providers are installed
33
33
  - Apps using path gems (local development) don't break from missing optional dependencies
34
34
  - Users only need gems for providers they actually use
35
+
36
+ ## README and docs philosophy
37
+
38
+ The README is a landing page — install, quick start, what it supports, where to go next. Keep it scannable. When in doubt, link rather than expand.
39
+
40
+ **Three rules:**
41
+ - **Config**: the basic `Coolhand.configure` snippet belongs in the README. Anything requiring more than one code block (self-hosted `base_url`, custom intercept addresses) goes in `docs/configuration.md`.
42
+ - **Feedback**: the basic `create_feedback` snippet belongs in the README. The full field table, matching strategies, and sentiment conversion details go in `docs/feedback.md`.
43
+ - **Integrations**: each integration gets its own `docs/<name>.md` file. The README links to them from the Documentation section.
44
+
45
+ **Align with coolhand-node.** When adding a section that exists in the Node README, match its structure and tone. The two READMEs should feel like siblings.
46
+
47
+ **Discoverability (SEO / AEO).** Write headings, the package description, and the supported-libraries list with search engines and AI agents in mind: use full provider/framework names (e.g. "OpenAI", "Anthropic", "Google Gemini", "Cohere") rather than abbreviations. The goal is that searches for "Ruby LLM monitoring", "Anthropic Ruby logging", or "OpenAI Ruby observability" surface this gem.
data/README.md CHANGED
@@ -2,7 +2,9 @@
2
2
 
3
3
  [![Gem Version](https://badge.fury.io/rb/coolhand.svg)](https://badge.fury.io/rb/coolhand)
4
4
 
5
- Monitor and log LLM API calls from multiple providers (OpenAI, Anthropic, Google AI, Cohere, and more) to the Coolhand analytics platform.
5
+ Monitor and log LLM API calls OpenAI, Anthropic, Google Gemini, Cohere, and any Faraday-based or Net::HTTP client — to the Coolhand analytics platform. Supports Ruby LLM monitoring, request logging, and feedback collection.
6
+
7
+ > **Scope**: Coolhand intercepts outgoing HTTP requests to configured LLM API endpoints only. It does not read, scan, or transmit your source code files.
6
8
 
7
9
  ## Installation
8
10
 
@@ -51,24 +53,7 @@ end
51
53
 
52
54
  ## Self-Hosted Deployments
53
55
 
54
- For compliance, data-residency, or cost reasons you can run your own Coolhand-compatible endpoint and point the SDK at it via `config.base_url`:
55
-
56
- ```ruby
57
- Coolhand.configure do |config|
58
- config.api_key = ENV['COOLHAND_API_KEY']
59
- config.base_url = ENV['COOLHAND_BASE_URL'] # e.g. "https://coolhand.internal.example.com/api"
60
- end
61
- ```
62
-
63
- When `base_url` is unset the SDK defaults to `https://coolhandlabs.com/api` and behaviour is unchanged.
64
-
65
- **Accepted values:**
66
- - Any `https://` URL — required for production use
67
- - `http://localhost` or `http://127.0.0.1` — accepted for local development only
68
-
69
- **Trailing slashes** are stripped automatically, so `"https://example.com/api/"` and `"https://example.com/api"` are equivalent.
70
-
71
- The SDK raises `Coolhand::Error` at configure time if `base_url` is set to a plain `http://` URL pointing at a non-localhost host.
56
+ Point the SDK at your own Coolhand-compatible endpoint via `config.base_url` for compliance or data-residency requirements. See [Self-Hosted Deployments →](docs/configuration.md).
72
57
 
73
58
  ## Feedback API
74
59
 
@@ -80,10 +65,10 @@ Collect feedback on LLM responses to improve model performance.
80
65
  require 'coolhand'
81
66
 
82
67
  # Create feedback for an LLM response
83
- feedback_service = Coolhand::FeedbackService.new(Coolhand.configuration)
68
+ feedback_service = Coolhand::FeedbackService.new
84
69
 
85
70
  feedback = feedback_service.create_feedback(
86
- llm_request_log_id: 123,
71
+ llm_request_log_id: 'abc123def456', # hashid from a prior response; a raw integer FK also still works
87
72
  llm_provider_unique_id: 'req_xxxxxxx',
88
73
  client_unique_id: 'workorder-chat-456',
89
74
  creator_unique_id: 'user-789',
@@ -94,22 +79,7 @@ feedback = feedback_service.create_feedback(
94
79
  )
95
80
  ```
96
81
 
97
- **Field Guide:** All fields are optional, but here's how to get the best results:
98
-
99
- ### Matching Fields
100
- - **`llm_request_log_id`** 🎯 *Exact Match* - ID from the Coolhand API response when the original LLM request was logged. Provides exact matching.
101
- - **`llm_provider_unique_id`** 🎯 *Exact Match* - The x-request-id from the LLM API response (e.g., "req_xxxxxxx")
102
- - **`original_output`** 🔍 *Fuzzy Match* - The original LLM response text. Provides fuzzy matching but isn't 100% reliable.
103
- - **`client_unique_id`** 🔗 *Your Internal Matcher* - Connect to an identifier from your system for internal matching
104
-
105
- ### Quality Data
106
- - **`revised_output`** ⭐ *Best Signal* - End user revision of the LLM response. The highest value data for improving quality scores.
107
- - **`explanation`** 💬 *Medium Signal* - End user explanation of why the response was good or bad. Valuable qualitative data.
108
- - **`sentiment`** 🎭 *Preferred* - String sentiment: `'like'`, `'dislike'`, or `'neutral'`. Takes precedence over `like` if both are provided. The gem automatically converts `like` to `sentiment` before sending.
109
- - **`like`** 👍 *Low Signal (Deprecated)* - Boolean: `true` = like, `false` = dislike. Use `sentiment` instead. Conversion: `true` → `"like"`, `false` → `"dislike"`.
110
- - **`workload_hashid`** 🔗 *Workload Association* - Hashid of a workload to associate this feedback with.
111
- - **`creator_unique_id`** 👤 *User Tracking* - Unique ID to match feedback to the end user who created it
112
- - **`creator_type`** 🧑‍🤝‍🤖 *Creator Type* - What kind of creator submitted the feedback: `'human'`, `'agent'`, or `'unknown'`.
82
+ For a full field reference and matching strategy, see [Feedback API →](docs/feedback.md).
113
83
 
114
84
  ## Rails Integration
115
85
 
@@ -138,7 +108,7 @@ end
138
108
  ```ruby
139
109
  class ChatController < ApplicationController
140
110
  def create_feedback
141
- feedback_service = Coolhand::FeedbackService.new(Coolhand.configuration)
111
+ feedback_service = Coolhand::FeedbackService.new
142
112
 
143
113
  feedback = feedback_service.create_feedback(
144
114
  llm_request_log_id: params[:log_id],
@@ -163,7 +133,7 @@ end
163
133
  ```ruby
164
134
  class FeedbackCollectionJob < ApplicationJob
165
135
  def perform(feedback_data)
166
- feedback_service = Coolhand::FeedbackService.new(Coolhand.configuration)
136
+ feedback_service = Coolhand::FeedbackService.new
167
137
 
168
138
  feedback_service.create_feedback(
169
139
  llm_provider_unique_id: feedback_data[:request_id],
@@ -184,6 +154,7 @@ end
184
154
  |--------|------|---------|-------------|
185
155
  | `api_key` | String | `nil` | Your Coolhand API key. If absent, intercepted requests are skipped with a warning log rather than raising at boot time |
186
156
  | `enabled` | Boolean | `true` | Set to `false` to disable all patching and validation (e.g. `Rails.env.production?`) |
157
+ | `capture` | Boolean | `true` | Whether to capture and forward intercepted requests. Set to `false` to monitor without forwarding, then use [`Coolhand.with_capture`](#selective-capture) to re-enable selectively |
187
158
  | `silent` | Boolean | `false` | Whether to suppress console output |
188
159
  | `intercept_addresses` | Array | `["api.openai.com", "api.anthropic.com"]` | Array of API endpoint strings to monitor |
189
160
 
@@ -217,6 +188,38 @@ puts response.dig("choices", 0, "message", "content")
217
188
 
218
189
  📖 **[Complete Anthropic Integration Guide →](docs/anthropic.md)** - Supports both official and community gems with automatic detection
219
190
 
191
+ ### Selective Capture
192
+
193
+ Use `Coolhand.with_capture` and `Coolhand.without_capture` to override the global `capture` setting for a specific block of code, without changing your configuration.
194
+
195
+ **Capture a specific call when capture is globally disabled:**
196
+
197
+ ```ruby
198
+ Coolhand.configure do |config|
199
+ config.api_key = 'your_api_key_here'
200
+ config.capture = false # disabled globally
201
+ end
202
+
203
+ Coolhand.with_capture do
204
+ response = openai_client.chat(parameters: {
205
+ model: "gpt-4o-mini",
206
+ messages: [{ role: "user", content: "Hello!" }],
207
+ })
208
+ # This request is captured and forwarded to Coolhand
209
+ end
210
+ ```
211
+
212
+ **Skip capture for a specific call when capture is globally enabled:**
213
+
214
+ ```ruby
215
+ Coolhand.without_capture do
216
+ response = openai_client.chat(parameters: { ... })
217
+ # This request is NOT forwarded to Coolhand
218
+ end
219
+ ```
220
+
221
+ Both methods are thread-safe. Blocks nest correctly — the innermost block takes precedence, and the outer setting is restored when the block exits.
222
+
220
223
  ## Logging Inbound Webhooks
221
224
 
222
225
  For inbound webhooks (like audio transcripts or tool calls), the automatic interceptor won't capture them since they're incoming requests TO your application. In these cases, use the simple `forward_webhook` helper method:
@@ -311,11 +314,11 @@ The monitor works with multiple transport layers and Ruby libraries:
311
314
 
312
315
  ## How It Works
313
316
 
314
- Coolhand uses a unified Net::HTTP interceptor to monitor all HTTP traffic to configured LLM endpoints:
317
+ Coolhand uses a unified Net::HTTP interceptor to capture outgoing requests to configured LLM API endpoints:
315
318
 
316
319
  ### Net::HTTP Interceptor
317
- - Patches Ruby's core `Net::HTTP` library using `Module#prepend`
318
- - Monitors **all** HTTP libraries that use Net::HTTP under the hood (which is most of them)
320
+ - Extends Ruby's `Net::HTTP` using `Module#prepend` (a standard Ruby technique for wrapping library behavior)
321
+ - Covers HTTP libraries that delegate to Net::HTTP under the hood (which is most of them)
319
322
  - Handles both standard requests and streaming responses via `read_body` interception
320
323
  - Thread-safe design using thread-local storage for streaming buffers
321
324
 
@@ -488,10 +491,12 @@ class Vertex::BatchCallbackProcessor < BaseService
488
491
  end
489
492
  ```
490
493
 
491
- ## Integration Guides
494
+ ## Documentation
492
495
 
493
- - **[Anthropic Integration](docs/anthropic.md)** - Complete guide for both official and community Anthropic gems, including streaming, dual gem handling, and troubleshooting
494
- - **[ElevenLabs Integration](docs/elevenlabs.md)** - Complete guide for integrating ElevenLabs Conversational AI with webhook capture and feedback submission
496
+ - **[Configuration](docs/configuration.md)** — Self-hosted deployments, base_url rules, debug mode, custom intercept addresses
497
+ - **[Feedback API](docs/feedback.md)** Full field reference, matching strategies, sentiment values
498
+ - **[Anthropic Integration](docs/anthropic.md)** — Official and community Anthropic Ruby gems, streaming, dual gem handling, and troubleshooting
499
+ - **[ElevenLabs Integration](docs/elevenlabs.md)** — Webhook capture, feedback submission, and Rails integration
495
500
 
496
501
  ## Security
497
502
 
@@ -511,6 +516,10 @@ end
511
516
  - **Contribute?** [Submit a pull request](https://github.com/Coolhand-Labs/coolhand-ruby/pulls)
512
517
  - **Support?** Visit [coolhandlabs.com](https://coolhandlabs.com)
513
518
 
519
+ ## About Coolhand Labs
520
+
521
+ Coolhand Labs builds LLM observability and feedback tooling so teams can monitor, understand, and improve their AI applications. Learn more at [coolhandlabs.com](https://coolhandlabs.com).
522
+
514
523
  ## License
515
524
 
516
525
  Apache-2.0
data/SECURITY.md ADDED
@@ -0,0 +1,20 @@
1
+ # Security Policy
2
+
3
+ ## Supported Versions
4
+
5
+ Only the latest published version of `coolhand` on RubyGems is supported with security fixes.
6
+ Please upgrade to the latest version before reporting an issue.
7
+
8
+ ## Reporting a Vulnerability
9
+
10
+ Please do **not** open a public GitHub issue for security vulnerabilities.
11
+
12
+ Instead, report vulnerabilities privately using one of the following:
13
+
14
+ - [GitHub Security Advisories](https://github.com/Coolhand-Labs/coolhand-ruby/security/advisories/new)
15
+ for this repository (preferred)
16
+ - Email team@coolhandlabs.com
17
+
18
+ Please include a description of the vulnerability, steps to reproduce, and the impact you believe
19
+ it has. We aim to acknowledge reports within 3 business days and to provide a fix or mitigation
20
+ plan within 30 days, depending on severity.
@@ -0,0 +1,59 @@
1
+ # Advanced Configuration
2
+
3
+ ## Self-Hosted Deployments
4
+
5
+ For compliance, data-residency, or cost reasons you can run your own Coolhand-compatible endpoint and point the SDK at it via `config.base_url`:
6
+
7
+ ```ruby
8
+ Coolhand.configure do |config|
9
+ config.api_key = ENV['COOLHAND_API_KEY']
10
+ config.base_url = ENV['COOLHAND_BASE_URL'] # e.g. "https://coolhand.internal.example.com/api"
11
+ end
12
+ ```
13
+
14
+ When `base_url` is unset the SDK defaults to `https://coolhandlabs.com/api` and behaviour is unchanged.
15
+
16
+ **URL validation rules:**
17
+ - Any `https://` URL — required for production use
18
+ - `http://localhost` or `http://127.0.0.1` — accepted for local development only
19
+ - Non-HTTPS remote URLs are rejected: the SDK raises `Coolhand::Error` at configure time if `base_url` is set to a plain `http://` URL pointing at a non-localhost host
20
+
21
+ **Trailing slashes** are stripped automatically, so `"https://example.com/api/"` and `"https://example.com/api"` are equivalent.
22
+
23
+ ---
24
+
25
+ ## Debug Mode
26
+
27
+ `config.debug_mode` is a local-development aid: it prints every prepared payload to the console instead of sending it to the Coolhand API (or your self-hosted `base_url`), so you can inspect exactly what would be logged without an API key or network access.
28
+
29
+ ```ruby
30
+ Coolhand.configure do |config|
31
+ config.debug_mode = Rails.env.development?
32
+ end
33
+ ```
34
+
35
+ **What changes when `debug_mode` is `true`:**
36
+ - No HTTP request is made — `create_log`, `create_feedback`, and `send_llm_request_log` all return `nil` and print the payload via `JSON.pretty_generate` instead.
37
+ - Capture is forced on for every request, including inside a `Coolhand.without_capture` block and when `config.capture = false`. This is intentional — debug mode is meant to show you everything, not respect capture suppression — so don't leave it enabled in an environment where you rely on `without_capture` to keep specific calls out of the logs.
38
+ - Header and URL sanitization (`[REDACTED]` API keys/tokens, redacted `key=`/`token=` query params) still applies to the printed payload, exactly as it would to a real request.
39
+
40
+ Because it forces capture unconditionally and prints full request/response bodies to the console, only enable `debug_mode` in development — never in production or in an environment handling real user data.
41
+
42
+ ## Custom Intercept Addresses
43
+
44
+ By default Coolhand captures requests to a built-in list of LLM API hosts (OpenAI, Anthropic, Google Gemini, ElevenLabs, GitHub Models, and more). To capture a custom endpoint — an internal proxy, a self-hosted model server, or a third-party gateway — override `intercept_addresses`:
45
+
46
+ ```ruby
47
+ Coolhand.configure do |config|
48
+ config.api_key = ENV['COOLHAND_API_KEY']
49
+ config.intercept_addresses = [
50
+ 'my-llm-proxy.internal',
51
+ 'api.openai.com', # include the defaults you still want
52
+ 'api.anthropic.com',
53
+ ]
54
+ end
55
+ ```
56
+
57
+ Setting `intercept_addresses` **replaces** the default list entirely, so include any default hosts you still need.
58
+
59
+ The default list can be found in `Coolhand::Configuration::DEFAULT_INTERCEPT_ADDRESSES`.
data/docs/feedback.md ADDED
@@ -0,0 +1,79 @@
1
+ # Feedback API
2
+
3
+ Collect user feedback on LLM responses to improve your AI outputs. The Feedback API lets you capture sentiment ratings, explanations, and human-corrected outputs.
4
+
5
+ > **Frontend widget:** For browser-based feedback collection, see [coolhand-js](https://github.com/Coolhand-Labs/coolhand-js) — a lightweight JavaScript widget that captures actionable user feedback on any AI output.
6
+
7
+ ## Basic Usage
8
+
9
+ ```ruby
10
+ require 'coolhand'
11
+
12
+ feedback_service = Coolhand::FeedbackService.new
13
+
14
+ # Positive feedback linked by log ID (most reliable)
15
+ feedback_service.create_feedback(
16
+ llm_request_log_id: 'abc123def456', # hashid from a prior response; a raw integer FK also still works
17
+ sentiment: 'like',
18
+ explanation: 'Clear and accurate answer.',
19
+ )
20
+
21
+ # Negative feedback with a human correction
22
+ feedback_service.create_feedback(
23
+ original_output: 'The capital of France is London.',
24
+ sentiment: 'dislike',
25
+ revised_output: 'The capital of France is Paris.',
26
+ explanation: 'Factually wrong.',
27
+ )
28
+ ```
29
+
30
+ ---
31
+
32
+ ## Field Reference
33
+
34
+ All fields are optional. Use at least one **Matching Field** to link feedback to its originating LLM request.
35
+
36
+ ### Matching Fields
37
+
38
+ These fields identify which LLM request the feedback refers to. Use the most specific one available.
39
+
40
+ | Field | Match type | Description |
41
+ |---|---|---|
42
+ | `llm_request_log_id` | Exact | Hashid returned when the original request was logged (a raw integer FK is also still accepted for backward compatibility). Most reliable. |
43
+ | `llm_provider_unique_id` | Exact | The provider's own request ID (e.g. `x-request-id` from Anthropic or OpenAI). |
44
+ | `client_unique_id` | Exact | Your own internal identifier for the request (e.g. a database row ID). |
45
+ | `original_output` | Fuzzy | The raw text the LLM produced. Used for fuzzy matching when no ID is available — less reliable. |
46
+
47
+ ### Quality Signals
48
+
49
+ | Field | Signal strength | Description |
50
+ |---|---|---|
51
+ | `revised_output` | ⭐ Best | The human-corrected version of the LLM output. Highest-value signal for quality improvement. |
52
+ | `explanation` | Medium | Free-text reason the response was good or bad. |
53
+ | `sentiment` | Low–Medium | `"like"`, `"dislike"`, or `"neutral"`. **Preferred** over the deprecated `like` boolean. |
54
+ | `like` | Low (deprecated) | Boolean: `true` = like, `false` = dislike. Auto-converted to `sentiment` before submission. Use `sentiment` instead. |
55
+
56
+ ### Attribution Fields
57
+
58
+ | Field | Description |
59
+ |---|---|
60
+ | `creator_unique_id` | ID of the user providing feedback (for per-user quality tracking). |
61
+ | `creator_type` | Who submitted the feedback: `"human"`, `"agent"`, or `"unknown"`. |
62
+ | `workload_hashid` | Associate feedback with a specific workload in the Coolhand dashboard. |
63
+
64
+ ---
65
+
66
+ ## Sentiment Values
67
+
68
+ | Value | Meaning |
69
+ |---|---|
70
+ | `"like"` | The response was helpful / correct |
71
+ | `"dislike"` | The response was unhelpful / wrong |
72
+ | `"neutral"` | Neither good nor bad (e.g. factual but irrelevant) |
73
+
74
+ The deprecated `like: bool` field is still accepted and automatically converted:
75
+
76
+ | `like` (bool) | Equivalent `sentiment` |
77
+ |---|---|
78
+ | `true` | `"like"` |
79
+ | `false` | `"dislike"` |
@@ -70,6 +70,11 @@ module Coolhand
70
70
  uri = URI.parse(@api_endpoint)
71
71
  http = Net::HTTP.new(uri.host, uri.port)
72
72
  http.use_ssl = (uri.scheme == "https")
73
+ # Bound worst-case latency: this call happens inline in the intercepted
74
+ # request's path, so a slow/unreachable Coolhand backend must not hang
75
+ # the host app's real LLM call for Ruby's ~60s Net::HTTP defaults.
76
+ http.open_timeout = 5
77
+ http.read_timeout = 5
73
78
 
74
79
  request = Net::HTTP::Post.new(uri.request_uri)
75
80
  headers = create_request_options(payload)
@@ -5,93 +5,12 @@ module Coolhand
5
5
  module BaseInterceptor
6
6
  module_function
7
7
 
8
- def extract_response_data(response)
9
- case response
10
- when Hash
11
- response
12
- when Struct
13
- response.to_h
14
- else
15
- # Handle streaming responses - these are often enumerator objects
16
- # that can't be serialized directly
17
- if response.class.name.include?("Stream") || response.respond_to?(:each)
18
- {
19
- response_type: "streaming",
20
- class: response.class.name,
21
- note: "Streaming response - content captured during enumeration"
22
- }
23
- elsif response.respond_to?(:to_h)
24
- begin
25
- response.to_h
26
- rescue StandardError => e
27
- {
28
- serialization_error: e.message,
29
- class: response.class.name,
30
- raw_response: response.to_s
31
- }
32
- end
33
- else
34
- # Extract content and token usage information
35
- response_data = {}
36
-
37
- # Get content
38
- response_data[:content] = response.content if response.respond_to?(:content)
39
-
40
- # Extract token usage information
41
- response_data[:usage] = extract_usage_metadata(response.usage) if response.respond_to?(:usage)
42
-
43
- # Extract model information
44
- response_data[:model] = response.model if response.respond_to?(:model)
45
-
46
- # Extract role information
47
- response_data[:role] = response.role if response.respond_to?(:role)
48
-
49
- # Extract ID if available
50
- response_data[:id] = response.id if response.respond_to?(:id)
51
-
52
- # Extract stop reason if available
53
- response_data[:stop_reason] = response.stop_reason if response.respond_to?(:stop_reason)
54
-
55
- # Add class info for debugging
56
- response_data[:class] = response.class.name
57
-
58
- response_data.empty? ? { raw_response: response.to_s, class: response.class.name } : response_data
59
- end
60
- end
61
- end
62
-
63
- def extract_usage_metadata(usage)
64
- if usage.respond_to?(:to_h)
65
- usage.to_h
66
- elsif usage.is_a?(Hash)
67
- usage
68
- else
69
- # Extract individual usage fields
70
- usage_data = {}
71
- usage_data[:input_tokens] = usage.input_tokens if usage.respond_to?(:input_tokens)
72
- usage_data[:output_tokens] = usage.output_tokens if usage.respond_to?(:output_tokens)
73
- usage_data[:total_tokens] = usage_data[:input_tokens].to_i + usage_data[:output_tokens].to_i
74
- usage_data
75
- end
76
- end
77
-
78
- def clean_request_headers(headers)
79
- cleaned = headers.dup
80
-
81
- # Remove sensitive headers
82
- cleaned.delete("Authorization")
83
- cleaned.delete("authorization")
84
- cleaned.delete("x-api-key")
85
- cleaned.delete("X-API-Key")
86
-
87
- cleaned
88
- end
89
-
90
- def clean_response_headers(headers)
91
- # Response headers typically don't contain sensitive data
92
- # but we can filter if needed
93
- headers.dup
94
- end
8
+ # Matches any header whose *name* signals sensitive content, regardless of
9
+ # provider — covers known keys (x-api-key, x-goog-api-key, openai-api-key),
10
+ # AWS SigV4 session tokens (x-amz-security-token), and future/unknown
11
+ # providers using a similarly-named header. Shared with LoggerService so
12
+ # the two logging paths (interceptor + webhook forwarding) stay consistent.
13
+ SENSITIVE_HEADER_PATTERN = /key|token|secret|signature|authorization/i
95
14
 
96
15
  def sanitize_headers(headers)
97
16
  return {} if headers.nil?
@@ -122,7 +41,6 @@ module Coolhand
122
41
 
123
42
  sanitized = raw.dup
124
43
 
125
- sanitized_keys = %w[openai-api-key api-key x-api-key x-goog-api-key]
126
44
  sanitized.each do |k, v|
127
45
  next if v.nil?
128
46
 
@@ -134,7 +52,7 @@ module Coolhand
134
52
  else
135
53
  "[REDACTED]"
136
54
  end
137
- elsif sanitized_keys.include?(key_down)
55
+ elsif key_down.match?(SENSITIVE_HEADER_PATTERN)
138
56
  sanitized[k] = "[REDACTED]"
139
57
  end
140
58
  end
@@ -73,8 +73,9 @@ module Coolhand
73
73
  # Convert Rails HTTP_ prefix headers
74
74
  clean_key = key.to_s.gsub(/^HTTP_/, "").tr("_", "-").downcase
75
75
 
76
- # Redact sensitive headers
77
- clean_value = clean_key.match?(/key|token|secret|authorization|signature/i) ? "[REDACTED]" : value.to_s
76
+ # Redact sensitive headers (shared with BaseInterceptor so both logging
77
+ # paths treat the same header names as sensitive)
78
+ clean_value = clean_key.match?(BaseInterceptor::SENSITIVE_HEADER_PATTERN) ? "[REDACTED]" : value.to_s
78
79
  clean_headers[clean_key] = clean_value
79
80
  end
80
81
  clean_headers
@@ -141,7 +141,7 @@ module Coolhand
141
141
 
142
142
  matched = patterns.find { |pattern| url.include?(pattern) }
143
143
  if matched && Coolhand.configuration.debug_mode
144
- Coolhand.log "🚫 Skipping capture for #{url} (matched exclude_api_pattern: \"#{matched}\")"
144
+ Coolhand.log "🚫 Skipping capture for #{sanitize_url(url)} (matched exclude_api_pattern: \"#{matched}\")"
145
145
  end
146
146
  !!matched
147
147
  end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Coolhand
4
- VERSION = "0.4.0"
4
+ VERSION = "0.5.0"
5
5
  end
metadata CHANGED
@@ -1,14 +1,15 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: coolhand
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.4.0
4
+ version: 0.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Michael Carroll
8
8
  - Yaroslav Malyk
9
+ autorequire:
9
10
  bindir: exe
10
11
  cert_chain: []
11
- date: 2026-06-22 00:00:00.000000000 Z
12
+ date: 2026-07-31 00:00:00.000000000 Z
12
13
  dependencies:
13
14
  - !ruby/object:Gem::Dependency
14
15
  name: base64
@@ -34,6 +35,8 @@ executables: []
34
35
  extensions: []
35
36
  extra_rdoc_files: []
36
37
  files:
38
+ - ".claude/skills/loop-review/SKILL.md"
39
+ - ".claude/skills/prep-release/SKILL.md"
37
40
  - ".idea/coolhand-ruby.iml"
38
41
  - ".rspec"
39
42
  - ".rubocop.yml"
@@ -43,8 +46,11 @@ files:
43
46
  - LICENSE
44
47
  - README.md
45
48
  - Rakefile
49
+ - SECURITY.md
46
50
  - docs/anthropic.md
51
+ - docs/configuration.md
47
52
  - docs/elevenlabs.md
53
+ - docs/feedback.md
48
54
  - lib/coolhand.rb
49
55
  - lib/coolhand/api_service.rb
50
56
  - lib/coolhand/base_interceptor.rb
@@ -70,6 +76,7 @@ metadata:
70
76
  source_code_uri: https://github.com/Coolhand-Labs/coolhand-ruby
71
77
  changelog_uri: https://github.com/Coolhand-Labs/coolhand-ruby/blob/main/CHANGELOG.md
72
78
  rubygems_mfa_required: 'true'
79
+ post_install_message:
73
80
  rdoc_options: []
74
81
  require_paths:
75
82
  - lib
@@ -84,7 +91,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
84
91
  - !ruby/object:Gem::Version
85
92
  version: '0'
86
93
  requirements: []
87
- rubygems_version: 3.6.2
94
+ rubygems_version: 3.5.22
95
+ signing_key:
88
96
  specification_version: 4
89
97
  summary: Monitor and log LLM API calls from OpenAI, Anthropic, and other providers
90
98
  to Coolhand analytics.