coolhand 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/skills/loop-review/SKILL.md +112 -0
- data/.claude/skills/prep-release/SKILL.md +160 -0
- data/CHANGELOG.md +33 -0
- data/CLAUDE.md +13 -0
- data/README.md +58 -45
- data/SECURITY.md +20 -0
- data/docs/configuration.md +59 -0
- data/docs/feedback.md +79 -0
- data/lib/coolhand/api_service.rb +18 -0
- data/lib/coolhand/base_interceptor.rb +7 -89
- data/lib/coolhand/configuration.rb +2 -7
- data/lib/coolhand/default_intercept_addresses.yml +4 -0
- data/lib/coolhand/logger_service.rb +3 -2
- data/lib/coolhand/net_http_interceptor.rb +1 -1
- data/lib/coolhand/version.rb +1 -1
- data/lib/coolhand.rb +9 -5
- metadata +11 -4
- data/coolhand-ruby.gemspec +0 -46
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: daed2b7ac4fe321602eb90f673cb385bd930b9ff503bf7e5d7851cbf06422617
|
|
4
|
+
data.tar.gz: 5d01a6386d142cdb31c490574191ac3867dabd30670c2a832c2aa18c7150c872
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a0681a3419c92e13fbae7a1b81c6d81b7a1cf2f088571dd44b4d2694c0ae5bf5c4c4312d7bff53ccc740833290bc931b932c1179244f004090192b0baaeb19f0
|
|
7
|
+
data.tar.gz: 9efd5e33528e0caeecc770c7286085320f9e4218f01fcf592ca82a27c96df71cf5ae52712a5a9aaeaf0702774be450bb7c3b7c71b0df8e57f56bd038f54f1b3d
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: loop-review
|
|
3
|
+
description: |
|
|
4
|
+
Iteratively runs code review against the current diff, applies fixes, and
|
|
5
|
+
re-reviews until a round comes back clean (or a safety cap is hit). Use
|
|
6
|
+
when the user types /loop-review, asks to "loop the review", "review
|
|
7
|
+
until clean", "keep reviewing and fixing until nothing's left", or wants
|
|
8
|
+
a self-healing code review cycle instead of a single one-shot pass.
|
|
9
|
+
user_invocable: true
|
|
10
|
+
version: 0.3.0
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
# Loop Review
|
|
14
|
+
|
|
15
|
+
This skill runs `/code-review` repeatedly against the working diff, applies
|
|
16
|
+
fixes between rounds, and re-reviews until a round finds nothing new or a
|
|
17
|
+
safety cap is reached. Use it after a merge, a large diff, or whenever a
|
|
18
|
+
single review pass isn't enough to converge on a clean state.
|
|
19
|
+
|
|
20
|
+
## Scope
|
|
21
|
+
|
|
22
|
+
Default scope is the diff between the current branch and its merge-base
|
|
23
|
+
with the repo's base branch (`git diff $(git merge-base origin/main
|
|
24
|
+
HEAD)...HEAD`, or `origin/main` substituted with whatever base branch
|
|
25
|
+
applies). If `$ARGUMENTS` names a path or a narrower scope, review only
|
|
26
|
+
that instead of the full diff.
|
|
27
|
+
|
|
28
|
+
This skill is deliberately diff-scoped. For a whole-package audit before a
|
|
29
|
+
release (full-codebase security red-team, docs review, everything since
|
|
30
|
+
the last tag) use `/prep-release` instead.
|
|
31
|
+
|
|
32
|
+
`$ARGUMENTS` may also contain:
|
|
33
|
+
- An effort level to pass through to `/code-review` (`low`/`medium`/
|
|
34
|
+
`high`/`high→max`/`ultra`). Default: `medium`.
|
|
35
|
+
- A round cap override, e.g. `--max-rounds 3`. Default: 5.
|
|
36
|
+
|
|
37
|
+
## The round loop
|
|
38
|
+
|
|
39
|
+
1. Invoke `/code-review <effort> --fix` (via the `Skill` tool) against the
|
|
40
|
+
current scope.
|
|
41
|
+
2. **Dry round (0 findings) → converged.** Stop and move to verification.
|
|
42
|
+
3. **Findings found and fixed** → do not declare victory yet. Run another
|
|
43
|
+
round to confirm the fixes didn't introduce a regression and that
|
|
44
|
+
nothing was missed.
|
|
45
|
+
4. **No-progress detection**: if two consecutive rounds return the same
|
|
46
|
+
non-empty set of findings, `--fix` isn't resolving them mechanically
|
|
47
|
+
(likely a design/architecture call that needs a human). Stop looping,
|
|
48
|
+
list the stuck findings, and hand them to the user instead of retrying
|
|
49
|
+
forever.
|
|
50
|
+
5. **Safety cap**: if the round cap is reached without converging or
|
|
51
|
+
getting stuck, stop and report the remaining findings — don't loop
|
|
52
|
+
silently past the cap.
|
|
53
|
+
|
|
54
|
+
Each round's fixes should stay reviewable: don't squash multiple rounds
|
|
55
|
+
into one silent edit. Note per-round changes in the final summary so the
|
|
56
|
+
user can inspect them with `git diff`.
|
|
57
|
+
|
|
58
|
+
## Review criteria
|
|
59
|
+
|
|
60
|
+
Beyond whatever `/code-review` already checks for correctness bugs and
|
|
61
|
+
reuse/simplification/efficiency, every round in a Ruby gem repo like this
|
|
62
|
+
one should also flag:
|
|
63
|
+
|
|
64
|
+
- **Ruby idiom / DRY**: semantic, expressive naming; no duplicated logic
|
|
65
|
+
that should be extracted into a shared method; idiomatic use of
|
|
66
|
+
Ruby/Enumerable over manual loops where it reads better; no needless
|
|
67
|
+
boilerplate.
|
|
68
|
+
- **Gem publishing discipline**: don't break public interfaces unless
|
|
69
|
+
necessary.
|
|
70
|
+
- If a break is necessary, it must come with: a `CHANGELOG.md` entry in
|
|
71
|
+
Keep a Changelog format (this repo already follows that format — match
|
|
72
|
+
the style of existing entries, e.g. plain-English migration notes like
|
|
73
|
+
the `0.5.0` entry) and a version bump in `lib/coolhand/version.rb` that
|
|
74
|
+
matches SemVer (patch = fix, minor = backward-compatible addition,
|
|
75
|
+
minor = breaking change while pre-1.0, consistent with this repo's own
|
|
76
|
+
versioning history).
|
|
77
|
+
- Check the optional-provider-dependency rule from this repo's
|
|
78
|
+
`CLAUDE.md`: provider SDK `require`s (`openai`, `anthropic`,
|
|
79
|
+
`google-generativeai`, etc.) must stay scoped to the file that uses
|
|
80
|
+
them and lazy-loaded, never added to `lib/coolhand.rb` or the gemspec
|
|
81
|
+
as a hard dependency.
|
|
82
|
+
- **Security**: injection risks, unsafe deserialization, secrets or
|
|
83
|
+
credentials logged or committed, unvalidated input crossing a trust
|
|
84
|
+
boundary, and anything touching how API keys/tokens are handled,
|
|
85
|
+
stored, or transmitted.
|
|
86
|
+
|
|
87
|
+
## Post-loop verification
|
|
88
|
+
|
|
89
|
+
Once the loop converges (or stops early per the rules above), run the
|
|
90
|
+
project's lint/test command and include the result in the final summary.
|
|
91
|
+
For this repo that's `bundle exec rake` (runs `rspec` then `rubocop`, per
|
|
92
|
+
the `Rakefile`). If a fix round changed something outside this repo's
|
|
93
|
+
usual toolchain, discover the right command instead of assuming.
|
|
94
|
+
|
|
95
|
+
## Rationalizations to resist
|
|
96
|
+
|
|
97
|
+
- *"The first round already looked clean, I don't need a confirming
|
|
98
|
+
round."* A fix round can introduce its own regression. Always re-review
|
|
99
|
+
after applying fixes before declaring convergence.
|
|
100
|
+
- *"Rubocop passed, so the review is done."* Lint passing is not the same
|
|
101
|
+
as the review being clean — lint doesn't check the criteria above
|
|
102
|
+
(interface breakage, changelog/version discipline, security). Run both.
|
|
103
|
+
- *"This finding keeps coming back, I'll just keep re-running --fix and
|
|
104
|
+
it'll eventually take."* If the same non-empty finding set repeats
|
|
105
|
+
across two rounds, `--fix` isn't going to resolve it. Stop and surface
|
|
106
|
+
it — looping past that point just burns rounds for no gain.
|
|
107
|
+
|
|
108
|
+
## Safety
|
|
109
|
+
|
|
110
|
+
- Never force-push or amend existing commits as part of this loop.
|
|
111
|
+
- The skill only edits the working tree; committing and pushing stays with
|
|
112
|
+
the user.
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: prep-release
|
|
3
|
+
description: |
|
|
4
|
+
Prepares this gem for a release: runs the full test suite, updates and
|
|
5
|
+
cleans up the docs (README, CHANGELOG, docs/*.md) to reflect every change
|
|
6
|
+
since the last tag, bumps the version if that hasn't already been done,
|
|
7
|
+
and red-teams the whole package for security - not just this release's
|
|
8
|
+
diff. Never tags or pushes. Use when the user types /prep-release, asks
|
|
9
|
+
to "prep a release", "get ready to cut a release", "release checklist",
|
|
10
|
+
or wants a pre-release audit before tagging/publishing a new version.
|
|
11
|
+
user_invocable: true
|
|
12
|
+
version: 0.2.0
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
# Prep Release
|
|
16
|
+
|
|
17
|
+
Three phases, run in order. This is a whole-package audit, not a diff
|
|
18
|
+
review — do not scope any phase to just what changed since the last
|
|
19
|
+
commit. For an iterative diff-scoped review during normal development, use
|
|
20
|
+
`/loop-review` instead; this skill is for the release boundary.
|
|
21
|
+
|
|
22
|
+
## Phase 1: Run all tests
|
|
23
|
+
|
|
24
|
+
Run `bundle exec rake` (`rspec` then `rubocop`, per the `Rakefile`). If a
|
|
25
|
+
fix round changed something outside this repo's usual toolchain, discover
|
|
26
|
+
the right command instead of assuming. All specs must pass and RuboCop
|
|
27
|
+
must report zero offenses before continuing — a release doesn't ship on a
|
|
28
|
+
red build. If either fails, stop here and report the failures; fixing
|
|
29
|
+
genuine bugs takes priority over Phase 2/3 work.
|
|
30
|
+
|
|
31
|
+
Then judge coverage on quality, not just the SimpleCov percentage the rake
|
|
32
|
+
run reports (written to `coverage/`):
|
|
33
|
+
|
|
34
|
+
1. **Find the gaps.** List files/lines SimpleCov marks uncovered. Weight
|
|
35
|
+
by risk: an uncovered error-handling branch or security check
|
|
36
|
+
(signature/header validation) matters more than an uncovered
|
|
37
|
+
`attr_reader`.
|
|
38
|
+
2. **Audit existing tests for meaningfulness, not just count.** Flag tests
|
|
39
|
+
that only assert a stub returns what it was stubbed to return without
|
|
40
|
+
exercising real conditional logic in the subject under test; missing
|
|
41
|
+
negative/error-path cases (invalid input, malformed provider responses,
|
|
42
|
+
network failure); missing domain edge cases (empty batch results,
|
|
43
|
+
duplicate-request prevention, concurrent access, streaming vs
|
|
44
|
+
non-streaming shapes).
|
|
45
|
+
3. **Recommend, don't pad.** Propose specific specs for the highest-risk
|
|
46
|
+
gaps, named by `file:describe/context`. Don't add tests purely to move
|
|
47
|
+
the percentage — a test with no failure mode it would catch adds
|
|
48
|
+
maintenance cost without adding signal.
|
|
49
|
+
|
|
50
|
+
## Phase 2: Update and clean the docs
|
|
51
|
+
|
|
52
|
+
1. Find the last release tag: `git describe --tags --abbrev=0`.
|
|
53
|
+
2. Diff everything since that tag: `git log <last-tag>..HEAD --oneline` and
|
|
54
|
+
`git diff <last-tag>..HEAD -- lib/` to see every behavioral change, not
|
|
55
|
+
just the most recent commit.
|
|
56
|
+
3. For each change, check it's reflected in:
|
|
57
|
+
- `CHANGELOG.md` — every notable change since the last tag needs an
|
|
58
|
+
entry under `[Unreleased]` (or a new version heading), in Keep a
|
|
59
|
+
Changelog format matching this repo's existing entries (see past
|
|
60
|
+
entries for style — plain-English migration notes for anything
|
|
61
|
+
behavior-affecting).
|
|
62
|
+
- `README.md` / `docs/*.md` — any new config option, public method, or
|
|
63
|
+
behavior change needs the relevant section updated. Follow this
|
|
64
|
+
repo's docs philosophy from `CLAUDE.md`: the README stays a scannable
|
|
65
|
+
landing page (basic config/feedback snippets only); anything needing
|
|
66
|
+
more than one code block belongs in `docs/`.
|
|
67
|
+
4. **Clean, don't just append.** Look for docs that are now stale,
|
|
68
|
+
contradictory, or redundant given the accumulated changes since the
|
|
69
|
+
last tag — consolidate/rewrite rather than layering a new paragraph on
|
|
70
|
+
top of an outdated one. Remove docs for anything removed from the gem.
|
|
71
|
+
5. **Bump the version if it hasn't already been done.** Check whether
|
|
72
|
+
`lib/coolhand/version.rb` was already bumped for the changes
|
|
73
|
+
accumulated since the last tag (e.g. by an earlier commit on this
|
|
74
|
+
branch) — if so, leave it. If not, determine the SemVer bump this
|
|
75
|
+
repo's convention implies (patch = fix, minor = backward-compatible
|
|
76
|
+
addition or breaking change while pre-1.0), write it to
|
|
77
|
+
`lib/coolhand/version.rb`, turn the `[Unreleased]` CHANGELOG heading
|
|
78
|
+
into `## [X.Y.Z] - <today's date>`, and run `bundle install` so
|
|
79
|
+
`Gemfile.lock`'s `coolhand (X.Y.Z)` line matches. State the version and
|
|
80
|
+
bump rationale in the wrap-up summary so the user can override it if
|
|
81
|
+
they'd have picked differently — don't ask before writing it, since
|
|
82
|
+
this is a mechanical, reversible edit gated by Phase 1's green build.
|
|
83
|
+
|
|
84
|
+
## Phase 3: Red-team the whole package
|
|
85
|
+
|
|
86
|
+
Adversarially review the entire `lib/` tree (not just this release's
|
|
87
|
+
diff) for security issues. This gem intercepts outgoing LLM API traffic
|
|
88
|
+
and logs it to Coolhand, so hunt specifically for:
|
|
89
|
+
|
|
90
|
+
- **Credential/secret leakage**: does any interceptor, logger, or error
|
|
91
|
+
handler write an API key, bearer token, or provider auth header value
|
|
92
|
+
into a log line, exception message, or the payload sent to Coolhand?
|
|
93
|
+
Check every header-sanitization path actually strips what it claims to
|
|
94
|
+
(e.g. `WebhookValidator`, provider header redaction) rather than
|
|
95
|
+
sanitizing a differently-cased or differently-named header.
|
|
96
|
+
- **Webhook/signature validation**: can `WebhookValidator#valid?` (or
|
|
97
|
+
equivalent) be bypassed — timing-unsafe comparison instead of a
|
|
98
|
+
constant-time compare, an environment where an empty/missing signature
|
|
99
|
+
is treated as valid, or a fallback path meant for development that's
|
|
100
|
+
reachable in production.
|
|
101
|
+
- **SSRF / address matching**: the default and configurable intercept
|
|
102
|
+
address lists — can a crafted URL (redirect, unicode homograph,
|
|
103
|
+
userinfo trick, subdomain confusion) match or evade the intended
|
|
104
|
+
host-matching logic in a way that intercepts (or fails to intercept)
|
|
105
|
+
the wrong destination?
|
|
106
|
+
- **ReDoS**: any regex built from configurable or user-influenced input
|
|
107
|
+
(intercept patterns, header names) — check for catastrophic backtracking
|
|
108
|
+
shapes (nested quantifiers, overlapping alternation).
|
|
109
|
+
- **Thread safety**: this gem documents thread-safe operation and
|
|
110
|
+
duplicate-request prevention — look for unsynchronized shared mutable
|
|
111
|
+
state (class-level `@@` vars, memoized `@client` on a shared instance)
|
|
112
|
+
that a concurrent request could race on.
|
|
113
|
+
- **Unsafe deserialization**: any `JSON.parse` without checking for
|
|
114
|
+
`Marshal.load`/`YAML.load` (unsafe) usage, and any parsing of
|
|
115
|
+
webhook/batch-result payloads that trusts attacker-controlled shape
|
|
116
|
+
without validation.
|
|
117
|
+
- **Fail-open vs fail-closed**: when Coolhand's API is unreachable, rate
|
|
118
|
+
limited, or returns malformed data, does the gem fail open in a way that
|
|
119
|
+
silently drops security-relevant logging, or fail in a way that breaks
|
|
120
|
+
the host application's actual LLM call (the interceptor must never break
|
|
121
|
+
the underlying request)?
|
|
122
|
+
|
|
123
|
+
For each finding, report file, line, a concrete failure scenario, and
|
|
124
|
+
severity. Apply safe, mechanical, low-risk fixes directly (e.g. a missing
|
|
125
|
+
header-redaction pattern, a missing timeout). Flag but do not silently
|
|
126
|
+
apply anything that's a behavior/architecture decision (e.g. changing a
|
|
127
|
+
fail-open security default, adding replay protection, moving synchronous
|
|
128
|
+
work to a background thread) — surface these to the user for a decision,
|
|
129
|
+
the same "hand it to a human" rule `/loop-review` uses for stuck findings.
|
|
130
|
+
|
|
131
|
+
## Wrap-up
|
|
132
|
+
|
|
133
|
+
Report one consolidated summary: test/lint result, coverage-quality gaps
|
|
134
|
+
plus recommended specs, docs updated, the version (bumped or already
|
|
135
|
+
current, and why), and security findings split into fixed vs.
|
|
136
|
+
flagged-for-decision.
|
|
137
|
+
|
|
138
|
+
## Safety
|
|
139
|
+
|
|
140
|
+
- Bumping `lib/coolhand/version.rb`, finalizing the CHANGELOG heading, and
|
|
141
|
+
running `bundle install` for the lockfile are all in scope and don't need
|
|
142
|
+
a stop-and-ask — they're mechanical, reversible, and gated on Phase 1
|
|
143
|
+
already being green.
|
|
144
|
+
- Never create or push a git tag, never push commits, and never run
|
|
145
|
+
`rake release`, `gem push`, or anything else that publishes the gem or
|
|
146
|
+
touches the remote. Tagging and publishing are the user's action once
|
|
147
|
+
they've reviewed this skill's report, not something this skill does.
|
|
148
|
+
|
|
149
|
+
## Rationalizations to resist
|
|
150
|
+
|
|
151
|
+
- *"The diff since the last tag is small, I'll skip the red-team."* Small
|
|
152
|
+
diffs can still sit on top of latent issues in code nobody's touched
|
|
153
|
+
recently — that's exactly what "whole package, not just the diff" means.
|
|
154
|
+
- *"Tests pass, so coverage is fine."* Passing tests and meaningful
|
|
155
|
+
coverage are different questions. A red build blocks release; a green
|
|
156
|
+
build with hollow tests doesn't guarantee anything.
|
|
157
|
+
- *"Docs are close enough, I'll skip the cleanup pass."* Accumulated
|
|
158
|
+
changes since the last tag are exactly when docs drift from behavior —
|
|
159
|
+
this phase exists because per-PR doc updates miss the cross-cutting
|
|
160
|
+
view.
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,39 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.5.0] - 2026-07-30
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
- **`llm_request_log_id` and `workload_id` in feedback API responses are now hashid strings, not raw integers** — the Coolhand API now returns these as hashids, matching every other external-facing identifier on the record (they previously leaked the raw integer foreign key). This gem never typed or coerced these fields (plain hashes throughout), so no code changes are required here, but if your application stores or compares `result[:llm_request_log_id]` or `result[:workload_id]` as an integer, update it to treat the value as an opaque string identifier instead. The `create_feedback`/`update_feedback` input fields (`llm_request_log_id`, `workload_hashid`) are unaffected — they still accept either a raw integer or a hashid string.
|
|
14
|
+
- **`id` in feedback API responses has actually been a hashid string for some time** — flagging here since it's the same category of field; no gem-level change needed since this was never typed.
|
|
15
|
+
- `BaseInterceptor.sanitize_headers` and `LoggerService#sanitize_headers` (used for the main request/response logging path and webhook forwarding, respectively) now share the same sensitive-header pattern instead of each maintaining its own list, so both paths redact consistently.
|
|
16
|
+
|
|
17
|
+
### Security
|
|
18
|
+
- **AWS Bedrock SigV4 session tokens (`X-Amz-Security-Token`) are now redacted before logging.** The interceptor's header sanitizer used a hardcoded list of known API-key header names (`api-key`, `x-api-key`, `x-goog-api-key`, `openai-api-key`) instead of a general pattern, so this header — present on requests signed with temporary/STS credentials, the common case for Bedrock — was forwarded to the Coolhand backend and printed in `debug_mode` unredacted. The sanitizer now redacts any header whose name matches `key`, `token`, `secret`, `signature`, or `authorization`, closing this and similar gaps for any current or future provider header.
|
|
19
|
+
- **`debug_mode`'s "skipping capture" log line no longer prints the raw URL.** When a request matched `exclude_api_patterns` while `debug_mode` was on, the log line bypassed the usual URL sanitizer, so a Gemini/Vertex `?key=...` query-param API key could be printed in full. It now goes through the same URL sanitizer as every other log line.
|
|
20
|
+
- **Outbound requests to the Coolhand backend now set a 5-second connect/read timeout.** Previously this call had no explicit timeout and ran inline on the same thread as the intercepted LLM request, so a slow or unreachable Coolhand endpoint (including a self-hosted `base_url`) could add Ruby's ~60s Net::HTTP default (up to ~120s total) of latency to real LLM calls made by the host app.
|
|
21
|
+
|
|
22
|
+
### Removed
|
|
23
|
+
- Deleted unused `BaseInterceptor` methods with no callers anywhere in the gem: `extract_response_data`, `extract_usage_metadata`, `clean_request_headers`, `clean_response_headers`. The latter two duplicated `sanitize_headers` with a narrower, case-sensitive header list and were never wired into any interceptor — dead code carrying its own security debt.
|
|
24
|
+
|
|
25
|
+
### Dependencies
|
|
26
|
+
- Bumped `faraday` from 2.14.2 to 2.14.3 (#73).
|
|
27
|
+
|
|
28
|
+
## [0.4.0] - 2026-06-22
|
|
29
|
+
|
|
30
|
+
### Added
|
|
31
|
+
- **More default intercept addresses** — Vertex AI (`aiplatform.googleapis.com`), Cloudflare AI Gateway (`gateway.ai.cloudflare.com`), AWS Bedrock OpenAI-compatible endpoint (`bedrock-runtime`), and OpenRouter (`openrouter.ai`) are now monitored out of the box with no configuration required (#66).
|
|
32
|
+
- **`Configuration#enabled` flag** — Set `config.enabled = false` (e.g. `config.enabled = Rails.env.production?`) to skip all patching and validation globally without restructuring your configure block (#68).
|
|
33
|
+
- **Feedback `creator_type` field** — Pass `creator_type: 'human'`, `'agent'`, or `'unknown'` when submitting feedback to identify who originated the feedback; matches the Coolhand API field (#67).
|
|
34
|
+
|
|
35
|
+
### Changed
|
|
36
|
+
- **Deferred `api_key` validation** — A missing `api_key` no longer raises at `Coolhand.configure` time. Intercepted requests are silently skipped (with a warning log) when the key is absent, so apps that boot without a key in CI or non-production environments no longer crash (#68).
|
|
37
|
+
|
|
38
|
+
### Dependencies
|
|
39
|
+
- Bumped `faraday` from 2.14.1 to 2.14.2 (#63).
|
|
40
|
+
|
|
8
41
|
## [0.3.0] - 2026-05-14
|
|
9
42
|
|
|
10
43
|
### 🚀 Major Changes
|
data/CLAUDE.md
CHANGED
|
@@ -32,3 +32,16 @@ This ensures:
|
|
|
32
32
|
- Gem loads cleanly regardless of what providers are installed
|
|
33
33
|
- Apps using path gems (local development) don't break from missing optional dependencies
|
|
34
34
|
- Users only need gems for providers they actually use
|
|
35
|
+
|
|
36
|
+
## README and docs philosophy
|
|
37
|
+
|
|
38
|
+
The README is a landing page — install, quick start, what it supports, where to go next. Keep it scannable. When in doubt, link rather than expand.
|
|
39
|
+
|
|
40
|
+
**Three rules:**
|
|
41
|
+
- **Config**: the basic `Coolhand.configure` snippet belongs in the README. Anything requiring more than one code block (self-hosted `base_url`, custom intercept addresses) goes in `docs/configuration.md`.
|
|
42
|
+
- **Feedback**: the basic `create_feedback` snippet belongs in the README. The full field table, matching strategies, and sentiment conversion details go in `docs/feedback.md`.
|
|
43
|
+
- **Integrations**: each integration gets its own `docs/<name>.md` file. The README links to them from the Documentation section.
|
|
44
|
+
|
|
45
|
+
**Align with coolhand-node.** When adding a section that exists in the Node README, match its structure and tone. The two READMEs should feel like siblings.
|
|
46
|
+
|
|
47
|
+
**Discoverability (SEO / AEO).** Write headings, the package description, and the supported-libraries list with search engines and AI agents in mind: use full provider/framework names (e.g. "OpenAI", "Anthropic", "Google Gemini", "Cohere") rather than abbreviations. The goal is that searches for "Ruby LLM monitoring", "Anthropic Ruby logging", or "OpenAI Ruby observability" surface this gem.
|
data/README.md
CHANGED
|
@@ -1,6 +1,10 @@
|
|
|
1
1
|
# Coolhand Ruby Monitor
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://badge.fury.io/rb/coolhand)
|
|
4
|
+
|
|
5
|
+
Monitor and log LLM API calls — OpenAI, Anthropic, Google Gemini, Cohere, and any Faraday-based or Net::HTTP client — to the Coolhand analytics platform. Supports Ruby LLM monitoring, request logging, and feedback collection.
|
|
6
|
+
|
|
7
|
+
> **Scope**: Coolhand intercepts outgoing HTTP requests to configured LLM API endpoints only. It does not read, scan, or transmit your source code files.
|
|
4
8
|
|
|
5
9
|
## Installation
|
|
6
10
|
|
|
@@ -49,24 +53,7 @@ end
|
|
|
49
53
|
|
|
50
54
|
## Self-Hosted Deployments
|
|
51
55
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
```ruby
|
|
55
|
-
Coolhand.configure do |config|
|
|
56
|
-
config.api_key = ENV['COOLHAND_API_KEY']
|
|
57
|
-
config.base_url = ENV['COOLHAND_BASE_URL'] # e.g. "https://coolhand.internal.example.com/api"
|
|
58
|
-
end
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
When `base_url` is unset the SDK defaults to `https://coolhandlabs.com/api` and behaviour is unchanged.
|
|
62
|
-
|
|
63
|
-
**Accepted values:**
|
|
64
|
-
- Any `https://` URL — required for production use
|
|
65
|
-
- `http://localhost` or `http://127.0.0.1` — accepted for local development only
|
|
66
|
-
|
|
67
|
-
**Trailing slashes** are stripped automatically, so `"https://example.com/api/"` and `"https://example.com/api"` are equivalent.
|
|
68
|
-
|
|
69
|
-
The SDK raises `Coolhand::Error` at configure time if `base_url` is set to a plain `http://` URL pointing at a non-localhost host.
|
|
56
|
+
Point the SDK at your own Coolhand-compatible endpoint via `config.base_url` for compliance or data-residency requirements. See [Self-Hosted Deployments →](docs/configuration.md).
|
|
70
57
|
|
|
71
58
|
## Feedback API
|
|
72
59
|
|
|
@@ -78,10 +65,10 @@ Collect feedback on LLM responses to improve model performance.
|
|
|
78
65
|
require 'coolhand'
|
|
79
66
|
|
|
80
67
|
# Create feedback for an LLM response
|
|
81
|
-
feedback_service = Coolhand::FeedbackService.new
|
|
68
|
+
feedback_service = Coolhand::FeedbackService.new
|
|
82
69
|
|
|
83
70
|
feedback = feedback_service.create_feedback(
|
|
84
|
-
llm_request_log_id:
|
|
71
|
+
llm_request_log_id: 'abc123def456', # hashid from a prior response; a raw integer FK also still works
|
|
85
72
|
llm_provider_unique_id: 'req_xxxxxxx',
|
|
86
73
|
client_unique_id: 'workorder-chat-456',
|
|
87
74
|
creator_unique_id: 'user-789',
|
|
@@ -92,21 +79,7 @@ feedback = feedback_service.create_feedback(
|
|
|
92
79
|
)
|
|
93
80
|
```
|
|
94
81
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
### Matching Fields
|
|
98
|
-
- **`llm_request_log_id`** 🎯 *Exact Match* - ID from the Coolhand API response when the original LLM request was logged. Provides exact matching.
|
|
99
|
-
- **`llm_provider_unique_id`** 🎯 *Exact Match* - The x-request-id from the LLM API response (e.g., "req_xxxxxxx")
|
|
100
|
-
- **`original_output`** 🔍 *Fuzzy Match* - The original LLM response text. Provides fuzzy matching but isn't 100% reliable.
|
|
101
|
-
- **`client_unique_id`** 🔗 *Your Internal Matcher* - Connect to an identifier from your system for internal matching
|
|
102
|
-
|
|
103
|
-
### Quality Data
|
|
104
|
-
- **`revised_output`** ⭐ *Best Signal* - End user revision of the LLM response. The highest value data for improving quality scores.
|
|
105
|
-
- **`explanation`** 💬 *Medium Signal* - End user explanation of why the response was good or bad. Valuable qualitative data.
|
|
106
|
-
- **`sentiment`** 🎭 *Preferred* - String sentiment: `'like'`, `'dislike'`, or `'neutral'`. Takes precedence over `like` if both are provided. The gem automatically converts `like` to `sentiment` before sending.
|
|
107
|
-
- **`like`** 👍 *Low Signal (Deprecated)* - Boolean: `true` = like, `false` = dislike. Use `sentiment` instead. Conversion: `true` → `"like"`, `false` → `"dislike"`.
|
|
108
|
-
- **`workload_hashid`** 🔗 *Workload Association* - Hashid of a workload to associate this feedback with.
|
|
109
|
-
- **`creator_unique_id`** 👤 *User Tracking* - Unique ID to match feedback to the end user who created it
|
|
82
|
+
For a full field reference and matching strategy, see [Feedback API →](docs/feedback.md).
|
|
110
83
|
|
|
111
84
|
## Rails Integration
|
|
112
85
|
|
|
@@ -135,7 +108,7 @@ end
|
|
|
135
108
|
```ruby
|
|
136
109
|
class ChatController < ApplicationController
|
|
137
110
|
def create_feedback
|
|
138
|
-
feedback_service = Coolhand::FeedbackService.new
|
|
111
|
+
feedback_service = Coolhand::FeedbackService.new
|
|
139
112
|
|
|
140
113
|
feedback = feedback_service.create_feedback(
|
|
141
114
|
llm_request_log_id: params[:log_id],
|
|
@@ -160,7 +133,7 @@ end
|
|
|
160
133
|
```ruby
|
|
161
134
|
class FeedbackCollectionJob < ApplicationJob
|
|
162
135
|
def perform(feedback_data)
|
|
163
|
-
feedback_service = Coolhand::FeedbackService.new
|
|
136
|
+
feedback_service = Coolhand::FeedbackService.new
|
|
164
137
|
|
|
165
138
|
feedback_service.create_feedback(
|
|
166
139
|
llm_provider_unique_id: feedback_data[:request_id],
|
|
@@ -179,7 +152,9 @@ end
|
|
|
179
152
|
|
|
180
153
|
| Option | Type | Default | Description |
|
|
181
154
|
|--------|------|---------|-------------|
|
|
182
|
-
| `api_key` | String |
|
|
155
|
+
| `api_key` | String | `nil` | Your Coolhand API key. If absent, intercepted requests are skipped with a warning log rather than raising at boot time |
|
|
156
|
+
| `enabled` | Boolean | `true` | Set to `false` to disable all patching and validation (e.g. `Rails.env.production?`) |
|
|
157
|
+
| `capture` | Boolean | `true` | Whether to capture and forward intercepted requests. Set to `false` to monitor without forwarding, then use [`Coolhand.with_capture`](#selective-capture) to re-enable selectively |
|
|
183
158
|
| `silent` | Boolean | `false` | Whether to suppress console output |
|
|
184
159
|
| `intercept_addresses` | Array | `["api.openai.com", "api.anthropic.com"]` | Array of API endpoint strings to monitor |
|
|
185
160
|
|
|
@@ -213,6 +188,38 @@ puts response.dig("choices", 0, "message", "content")
|
|
|
213
188
|
|
|
214
189
|
📖 **[Complete Anthropic Integration Guide →](docs/anthropic.md)** - Supports both official and community gems with automatic detection
|
|
215
190
|
|
|
191
|
+
### Selective Capture
|
|
192
|
+
|
|
193
|
+
Use `Coolhand.with_capture` and `Coolhand.without_capture` to override the global `capture` setting for a specific block of code, without changing your configuration.
|
|
194
|
+
|
|
195
|
+
**Capture a specific call when capture is globally disabled:**
|
|
196
|
+
|
|
197
|
+
```ruby
|
|
198
|
+
Coolhand.configure do |config|
|
|
199
|
+
config.api_key = 'your_api_key_here'
|
|
200
|
+
config.capture = false # disabled globally
|
|
201
|
+
end
|
|
202
|
+
|
|
203
|
+
Coolhand.with_capture do
|
|
204
|
+
response = openai_client.chat(parameters: {
|
|
205
|
+
model: "gpt-4o-mini",
|
|
206
|
+
messages: [{ role: "user", content: "Hello!" }],
|
|
207
|
+
})
|
|
208
|
+
# This request is captured and forwarded to Coolhand
|
|
209
|
+
end
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
**Skip capture for a specific call when capture is globally enabled:**
|
|
213
|
+
|
|
214
|
+
```ruby
|
|
215
|
+
Coolhand.without_capture do
|
|
216
|
+
response = openai_client.chat(parameters: { ... })
|
|
217
|
+
# This request is NOT forwarded to Coolhand
|
|
218
|
+
end
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
Both methods are thread-safe. Blocks nest correctly — the innermost block takes precedence, and the outer setting is restored when the block exits.
|
|
222
|
+
|
|
216
223
|
## Logging Inbound Webhooks
|
|
217
224
|
|
|
218
225
|
For inbound webhooks (like audio transcripts or tool calls), the automatic interceptor won't capture them since they're incoming requests TO your application. In these cases, use the simple `forward_webhook` helper method:
|
|
@@ -307,11 +314,11 @@ The monitor works with multiple transport layers and Ruby libraries:
|
|
|
307
314
|
|
|
308
315
|
## How It Works
|
|
309
316
|
|
|
310
|
-
Coolhand uses a unified Net::HTTP interceptor to
|
|
317
|
+
Coolhand uses a unified Net::HTTP interceptor to capture outgoing requests to configured LLM API endpoints:
|
|
311
318
|
|
|
312
319
|
### Net::HTTP Interceptor
|
|
313
|
-
-
|
|
314
|
-
-
|
|
320
|
+
- Extends Ruby's `Net::HTTP` using `Module#prepend` (a standard Ruby technique for wrapping library behavior)
|
|
321
|
+
- Covers HTTP libraries that delegate to Net::HTTP under the hood (which is most of them)
|
|
315
322
|
- Handles both standard requests and streaming responses via `read_body` interception
|
|
316
323
|
- Thread-safe design using thread-local storage for streaming buffers
|
|
317
324
|
|
|
@@ -484,10 +491,12 @@ class Vertex::BatchCallbackProcessor < BaseService
|
|
|
484
491
|
end
|
|
485
492
|
```
|
|
486
493
|
|
|
487
|
-
##
|
|
494
|
+
## Documentation
|
|
488
495
|
|
|
489
|
-
- **[
|
|
490
|
-
- **[
|
|
496
|
+
- **[Configuration](docs/configuration.md)** — Self-hosted deployments, base_url rules, debug mode, custom intercept addresses
|
|
497
|
+
- **[Feedback API](docs/feedback.md)** — Full field reference, matching strategies, sentiment values
|
|
498
|
+
- **[Anthropic Integration](docs/anthropic.md)** — Official and community Anthropic Ruby gems, streaming, dual gem handling, and troubleshooting
|
|
499
|
+
- **[ElevenLabs Integration](docs/elevenlabs.md)** — Webhook capture, feedback submission, and Rails integration
|
|
491
500
|
|
|
492
501
|
## Security
|
|
493
502
|
|
|
@@ -507,6 +516,10 @@ end
|
|
|
507
516
|
- **Contribute?** [Submit a pull request](https://github.com/Coolhand-Labs/coolhand-ruby/pulls)
|
|
508
517
|
- **Support?** Visit [coolhandlabs.com](https://coolhandlabs.com)
|
|
509
518
|
|
|
519
|
+
## About Coolhand Labs
|
|
520
|
+
|
|
521
|
+
Coolhand Labs builds LLM observability and feedback tooling so teams can monitor, understand, and improve their AI applications. Learn more at [coolhandlabs.com](https://coolhandlabs.com).
|
|
522
|
+
|
|
510
523
|
## License
|
|
511
524
|
|
|
512
525
|
Apache-2.0
|
data/SECURITY.md
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# Security Policy
|
|
2
|
+
|
|
3
|
+
## Supported Versions
|
|
4
|
+
|
|
5
|
+
Only the latest published version of `coolhand` on RubyGems is supported with security fixes.
|
|
6
|
+
Please upgrade to the latest version before reporting an issue.
|
|
7
|
+
|
|
8
|
+
## Reporting a Vulnerability
|
|
9
|
+
|
|
10
|
+
Please do **not** open a public GitHub issue for security vulnerabilities.
|
|
11
|
+
|
|
12
|
+
Instead, report vulnerabilities privately using one of the following:
|
|
13
|
+
|
|
14
|
+
- [GitHub Security Advisories](https://github.com/Coolhand-Labs/coolhand-ruby/security/advisories/new)
|
|
15
|
+
for this repository (preferred)
|
|
16
|
+
- Email team@coolhandlabs.com
|
|
17
|
+
|
|
18
|
+
Please include a description of the vulnerability, steps to reproduce, and the impact you believe
|
|
19
|
+
it has. We aim to acknowledge reports within 3 business days and to provide a fix or mitigation
|
|
20
|
+
plan within 30 days, depending on severity.
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Advanced Configuration
|
|
2
|
+
|
|
3
|
+
## Self-Hosted Deployments
|
|
4
|
+
|
|
5
|
+
For compliance, data-residency, or cost reasons you can run your own Coolhand-compatible endpoint and point the SDK at it via `config.base_url`:
|
|
6
|
+
|
|
7
|
+
```ruby
|
|
8
|
+
Coolhand.configure do |config|
|
|
9
|
+
config.api_key = ENV['COOLHAND_API_KEY']
|
|
10
|
+
config.base_url = ENV['COOLHAND_BASE_URL'] # e.g. "https://coolhand.internal.example.com/api"
|
|
11
|
+
end
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
When `base_url` is unset the SDK defaults to `https://coolhandlabs.com/api` and behaviour is unchanged.
|
|
15
|
+
|
|
16
|
+
**URL validation rules:**
|
|
17
|
+
- Any `https://` URL — required for production use
|
|
18
|
+
- `http://localhost` or `http://127.0.0.1` — accepted for local development only
|
|
19
|
+
- Non-HTTPS remote URLs are rejected: the SDK raises `Coolhand::Error` at configure time if `base_url` is set to a plain `http://` URL pointing at a non-localhost host
|
|
20
|
+
|
|
21
|
+
**Trailing slashes** are stripped automatically, so `"https://example.com/api/"` and `"https://example.com/api"` are equivalent.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## Debug Mode
|
|
26
|
+
|
|
27
|
+
`config.debug_mode` is a local-development aid: it prints every prepared payload to the console instead of sending it to the Coolhand API (or your self-hosted `base_url`), so you can inspect exactly what would be logged without an API key or network access.
|
|
28
|
+
|
|
29
|
+
```ruby
|
|
30
|
+
Coolhand.configure do |config|
|
|
31
|
+
config.debug_mode = Rails.env.development?
|
|
32
|
+
end
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
**What changes when `debug_mode` is `true`:**
|
|
36
|
+
- No HTTP request is made — `create_log`, `create_feedback`, and `send_llm_request_log` all return `nil` and print the payload via `JSON.pretty_generate` instead.
|
|
37
|
+
- Capture is forced on for every request, including inside a `Coolhand.without_capture` block and when `config.capture = false`. This is intentional — debug mode is meant to show you everything, not respect capture suppression — so don't leave it enabled in an environment where you rely on `without_capture` to keep specific calls out of the logs.
|
|
38
|
+
- Header and URL sanitization (`[REDACTED]` API keys/tokens, redacted `key=`/`token=` query params) still applies to the printed payload, exactly as it would to a real request.
|
|
39
|
+
|
|
40
|
+
Because it forces capture unconditionally and prints full request/response bodies to the console, only enable `debug_mode` in development — never in production or in an environment handling real user data.
|
|
41
|
+
|
|
42
|
+
## Custom Intercept Addresses
|
|
43
|
+
|
|
44
|
+
By default Coolhand captures requests to a built-in list of LLM API hosts (OpenAI, Anthropic, Google Gemini, ElevenLabs, GitHub Models, and more). To capture a custom endpoint — an internal proxy, a self-hosted model server, or a third-party gateway — override `intercept_addresses`:
|
|
45
|
+
|
|
46
|
+
```ruby
|
|
47
|
+
Coolhand.configure do |config|
|
|
48
|
+
config.api_key = ENV['COOLHAND_API_KEY']
|
|
49
|
+
config.intercept_addresses = [
|
|
50
|
+
'my-llm-proxy.internal',
|
|
51
|
+
'api.openai.com', # include the defaults you still want
|
|
52
|
+
'api.anthropic.com',
|
|
53
|
+
]
|
|
54
|
+
end
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Setting `intercept_addresses` **replaces** the default list entirely, so include any default hosts you still need.
|
|
58
|
+
|
|
59
|
+
The default list can be found in `Coolhand::Configuration::DEFAULT_INTERCEPT_ADDRESSES`.
|
data/docs/feedback.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Feedback API
|
|
2
|
+
|
|
3
|
+
Collect user feedback on LLM responses to improve your AI outputs. The Feedback API lets you capture sentiment ratings, explanations, and human-corrected outputs.
|
|
4
|
+
|
|
5
|
+
> **Frontend widget:** For browser-based feedback collection, see [coolhand-js](https://github.com/Coolhand-Labs/coolhand-js) — a lightweight JavaScript widget that captures actionable user feedback on any AI output.
|
|
6
|
+
|
|
7
|
+
## Basic Usage
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
require 'coolhand'
|
|
11
|
+
|
|
12
|
+
feedback_service = Coolhand::FeedbackService.new
|
|
13
|
+
|
|
14
|
+
# Positive feedback linked by log ID (most reliable)
|
|
15
|
+
feedback_service.create_feedback(
|
|
16
|
+
llm_request_log_id: 'abc123def456', # hashid from a prior response; a raw integer FK also still works
|
|
17
|
+
sentiment: 'like',
|
|
18
|
+
explanation: 'Clear and accurate answer.',
|
|
19
|
+
)
|
|
20
|
+
|
|
21
|
+
# Negative feedback with a human correction
|
|
22
|
+
feedback_service.create_feedback(
|
|
23
|
+
original_output: 'The capital of France is London.',
|
|
24
|
+
sentiment: 'dislike',
|
|
25
|
+
revised_output: 'The capital of France is Paris.',
|
|
26
|
+
explanation: 'Factually wrong.',
|
|
27
|
+
)
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## Field Reference
|
|
33
|
+
|
|
34
|
+
All fields are optional. Use at least one **Matching Field** to link feedback to its originating LLM request.
|
|
35
|
+
|
|
36
|
+
### Matching Fields
|
|
37
|
+
|
|
38
|
+
These fields identify which LLM request the feedback refers to. Use the most specific one available.
|
|
39
|
+
|
|
40
|
+
| Field | Match type | Description |
|
|
41
|
+
|---|---|---|
|
|
42
|
+
| `llm_request_log_id` | Exact | Hashid returned when the original request was logged (a raw integer FK is also still accepted for backward compatibility). Most reliable. |
|
|
43
|
+
| `llm_provider_unique_id` | Exact | The provider's own request ID (e.g. `x-request-id` from Anthropic or OpenAI). |
|
|
44
|
+
| `client_unique_id` | Exact | Your own internal identifier for the request (e.g. a database row ID). |
|
|
45
|
+
| `original_output` | Fuzzy | The raw text the LLM produced. Used for fuzzy matching when no ID is available — less reliable. |
|
|
46
|
+
|
|
47
|
+
### Quality Signals
|
|
48
|
+
|
|
49
|
+
| Field | Signal strength | Description |
|
|
50
|
+
|---|---|---|
|
|
51
|
+
| `revised_output` | ⭐ Best | The human-corrected version of the LLM output. Highest-value signal for quality improvement. |
|
|
52
|
+
| `explanation` | Medium | Free-text reason the response was good or bad. |
|
|
53
|
+
| `sentiment` | Low–Medium | `"like"`, `"dislike"`, or `"neutral"`. **Preferred** over the deprecated `like` boolean. |
|
|
54
|
+
| `like` | Low (deprecated) | Boolean: `true` = like, `false` = dislike. Auto-converted to `sentiment` before submission. Use `sentiment` instead. |
|
|
55
|
+
|
|
56
|
+
### Attribution Fields
|
|
57
|
+
|
|
58
|
+
| Field | Description |
|
|
59
|
+
|---|---|
|
|
60
|
+
| `creator_unique_id` | ID of the user providing feedback (for per-user quality tracking). |
|
|
61
|
+
| `creator_type` | Who submitted the feedback: `"human"`, `"agent"`, or `"unknown"`. |
|
|
62
|
+
| `workload_hashid` | Associate feedback with a specific workload in the Coolhand dashboard. |
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## Sentiment Values
|
|
67
|
+
|
|
68
|
+
| Value | Meaning |
|
|
69
|
+
|---|---|
|
|
70
|
+
| `"like"` | The response was helpful / correct |
|
|
71
|
+
| `"dislike"` | The response was unhelpful / wrong |
|
|
72
|
+
| `"neutral"` | Neither good nor bad (e.g. factual but irrelevant) |
|
|
73
|
+
|
|
74
|
+
The deprecated `like: bool` field is still accepted and automatically converted:
|
|
75
|
+
|
|
76
|
+
| `like` (bool) | Equivalent `sentiment` |
|
|
77
|
+
|---|---|
|
|
78
|
+
| `true` | `"like"` |
|
|
79
|
+
| `false` | `"dislike"` |
|
data/lib/coolhand/api_service.rb
CHANGED
|
@@ -65,9 +65,16 @@ module Coolhand
|
|
|
65
65
|
end
|
|
66
66
|
|
|
67
67
|
def send_request(payload, success_message)
|
|
68
|
+
return nil if missing_api_key?
|
|
69
|
+
|
|
68
70
|
uri = URI.parse(@api_endpoint)
|
|
69
71
|
http = Net::HTTP.new(uri.host, uri.port)
|
|
70
72
|
http.use_ssl = (uri.scheme == "https")
|
|
73
|
+
# Bound worst-case latency: this call happens inline in the intercepted
|
|
74
|
+
# request's path, so a slow/unreachable Coolhand backend must not hang
|
|
75
|
+
# the host app's real LLM call for Ruby's ~60s Net::HTTP defaults.
|
|
76
|
+
http.open_timeout = 5
|
|
77
|
+
http.read_timeout = 5
|
|
71
78
|
|
|
72
79
|
request = Net::HTTP::Post.new(uri.request_uri)
|
|
73
80
|
headers = create_request_options(payload)
|
|
@@ -117,6 +124,8 @@ module Coolhand
|
|
|
117
124
|
end
|
|
118
125
|
|
|
119
126
|
def create_feedback(feedback, collection_method = nil)
|
|
127
|
+
return nil if !debug_mode? && missing_api_key?
|
|
128
|
+
|
|
120
129
|
normalized = normalize_feedback_sentiment(feedback)
|
|
121
130
|
feedback_with_collector = add_collector_to_data(normalized, collection_method)
|
|
122
131
|
|
|
@@ -144,6 +153,8 @@ module Coolhand
|
|
|
144
153
|
end
|
|
145
154
|
|
|
146
155
|
def create_log(captured_data, collection_method = nil)
|
|
156
|
+
return nil if !debug_mode? && missing_api_key?
|
|
157
|
+
|
|
147
158
|
raw_request_with_collector = add_collector_to_data({ raw_request: captured_data }, collection_method)
|
|
148
159
|
|
|
149
160
|
payload = {
|
|
@@ -193,6 +204,13 @@ module Coolhand
|
|
|
193
204
|
|
|
194
205
|
private
|
|
195
206
|
|
|
207
|
+
def missing_api_key?
|
|
208
|
+
return false if Coolhand.required_field?(api_key)
|
|
209
|
+
|
|
210
|
+
Coolhand.log "⚠️ Coolhand: API key is missing — skipping log for this request."
|
|
211
|
+
true
|
|
212
|
+
end
|
|
213
|
+
|
|
196
214
|
# Get all filtered field names as a flat array
|
|
197
215
|
def filtered_field_names
|
|
198
216
|
@filtered_field_names ||= BINARY_DATA_FILTERS.values.flatten.map(&:downcase)
|
|
@@ -5,93 +5,12 @@ module Coolhand
|
|
|
5
5
|
module BaseInterceptor
|
|
6
6
|
module_function
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
else
|
|
15
|
-
# Handle streaming responses - these are often enumerator objects
|
|
16
|
-
# that can't be serialized directly
|
|
17
|
-
if response.class.name.include?("Stream") || response.respond_to?(:each)
|
|
18
|
-
{
|
|
19
|
-
response_type: "streaming",
|
|
20
|
-
class: response.class.name,
|
|
21
|
-
note: "Streaming response - content captured during enumeration"
|
|
22
|
-
}
|
|
23
|
-
elsif response.respond_to?(:to_h)
|
|
24
|
-
begin
|
|
25
|
-
response.to_h
|
|
26
|
-
rescue StandardError => e
|
|
27
|
-
{
|
|
28
|
-
serialization_error: e.message,
|
|
29
|
-
class: response.class.name,
|
|
30
|
-
raw_response: response.to_s
|
|
31
|
-
}
|
|
32
|
-
end
|
|
33
|
-
else
|
|
34
|
-
# Extract content and token usage information
|
|
35
|
-
response_data = {}
|
|
36
|
-
|
|
37
|
-
# Get content
|
|
38
|
-
response_data[:content] = response.content if response.respond_to?(:content)
|
|
39
|
-
|
|
40
|
-
# Extract token usage information
|
|
41
|
-
response_data[:usage] = extract_usage_metadata(response.usage) if response.respond_to?(:usage)
|
|
42
|
-
|
|
43
|
-
# Extract model information
|
|
44
|
-
response_data[:model] = response.model if response.respond_to?(:model)
|
|
45
|
-
|
|
46
|
-
# Extract role information
|
|
47
|
-
response_data[:role] = response.role if response.respond_to?(:role)
|
|
48
|
-
|
|
49
|
-
# Extract ID if available
|
|
50
|
-
response_data[:id] = response.id if response.respond_to?(:id)
|
|
51
|
-
|
|
52
|
-
# Extract stop reason if available
|
|
53
|
-
response_data[:stop_reason] = response.stop_reason if response.respond_to?(:stop_reason)
|
|
54
|
-
|
|
55
|
-
# Add class info for debugging
|
|
56
|
-
response_data[:class] = response.class.name
|
|
57
|
-
|
|
58
|
-
response_data.empty? ? { raw_response: response.to_s, class: response.class.name } : response_data
|
|
59
|
-
end
|
|
60
|
-
end
|
|
61
|
-
end
|
|
62
|
-
|
|
63
|
-
def extract_usage_metadata(usage)
|
|
64
|
-
if usage.respond_to?(:to_h)
|
|
65
|
-
usage.to_h
|
|
66
|
-
elsif usage.is_a?(Hash)
|
|
67
|
-
usage
|
|
68
|
-
else
|
|
69
|
-
# Extract individual usage fields
|
|
70
|
-
usage_data = {}
|
|
71
|
-
usage_data[:input_tokens] = usage.input_tokens if usage.respond_to?(:input_tokens)
|
|
72
|
-
usage_data[:output_tokens] = usage.output_tokens if usage.respond_to?(:output_tokens)
|
|
73
|
-
usage_data[:total_tokens] = usage_data[:input_tokens].to_i + usage_data[:output_tokens].to_i
|
|
74
|
-
usage_data
|
|
75
|
-
end
|
|
76
|
-
end
|
|
77
|
-
|
|
78
|
-
def clean_request_headers(headers)
|
|
79
|
-
cleaned = headers.dup
|
|
80
|
-
|
|
81
|
-
# Remove sensitive headers
|
|
82
|
-
cleaned.delete("Authorization")
|
|
83
|
-
cleaned.delete("authorization")
|
|
84
|
-
cleaned.delete("x-api-key")
|
|
85
|
-
cleaned.delete("X-API-Key")
|
|
86
|
-
|
|
87
|
-
cleaned
|
|
88
|
-
end
|
|
89
|
-
|
|
90
|
-
def clean_response_headers(headers)
|
|
91
|
-
# Response headers typically don't contain sensitive data
|
|
92
|
-
# but we can filter if needed
|
|
93
|
-
headers.dup
|
|
94
|
-
end
|
|
8
|
+
# Matches any header whose *name* signals sensitive content, regardless of
|
|
9
|
+
# provider — covers known keys (x-api-key, x-goog-api-key, openai-api-key),
|
|
10
|
+
# AWS SigV4 session tokens (x-amz-security-token), and future/unknown
|
|
11
|
+
# providers using a similarly-named header. Shared with LoggerService so
|
|
12
|
+
# the two logging paths (interceptor + webhook forwarding) stay consistent.
|
|
13
|
+
SENSITIVE_HEADER_PATTERN = /key|token|secret|signature|authorization/i
|
|
95
14
|
|
|
96
15
|
def sanitize_headers(headers)
|
|
97
16
|
return {} if headers.nil?
|
|
@@ -122,7 +41,6 @@ module Coolhand
|
|
|
122
41
|
|
|
123
42
|
sanitized = raw.dup
|
|
124
43
|
|
|
125
|
-
sanitized_keys = %w[openai-api-key api-key x-api-key x-goog-api-key]
|
|
126
44
|
sanitized.each do |k, v|
|
|
127
45
|
next if v.nil?
|
|
128
46
|
|
|
@@ -134,7 +52,7 @@ module Coolhand
|
|
|
134
52
|
else
|
|
135
53
|
"[REDACTED]"
|
|
136
54
|
end
|
|
137
|
-
elsif
|
|
55
|
+
elsif key_down.match?(SENSITIVE_HEADER_PATTERN)
|
|
138
56
|
sanitized[k] = "[REDACTED]"
|
|
139
57
|
end
|
|
140
58
|
end
|
|
@@ -17,7 +17,7 @@ module Coolhand
|
|
|
17
17
|
BASE_URL_ERROR_MSG = "base_url must use https:// (or http://localhost / http://127.0.0.1 for local dev)"
|
|
18
18
|
LOOPBACK_HOSTS = %w[localhost 127.0.0.1 ::1].freeze
|
|
19
19
|
|
|
20
|
-
attr_accessor :api_key, :environment, :silent, :debug_mode, :capture, :exclude_api_patterns
|
|
20
|
+
attr_accessor :api_key, :environment, :silent, :debug_mode, :capture, :exclude_api_patterns, :enabled
|
|
21
21
|
attr_reader :intercept_addresses, :base_url
|
|
22
22
|
|
|
23
23
|
def initialize
|
|
@@ -30,6 +30,7 @@ module Coolhand
|
|
|
30
30
|
@debug_mode = false
|
|
31
31
|
@capture = true
|
|
32
32
|
@exclude_api_patterns = DEFAULT_EXCLUDE_API_PATTERNS.dup
|
|
33
|
+
@enabled = true
|
|
33
34
|
end
|
|
34
35
|
|
|
35
36
|
# Custom setter that preserves defaults when nil/empty array is provided
|
|
@@ -47,12 +48,6 @@ module Coolhand
|
|
|
47
48
|
end
|
|
48
49
|
|
|
49
50
|
def validate!
|
|
50
|
-
# Validate API Key after configuration
|
|
51
|
-
if api_key.nil?
|
|
52
|
-
Coolhand.log "❌ Coolhand Error: API Key is required. Please set it in the configuration."
|
|
53
|
-
raise Error, "API Key is required"
|
|
54
|
-
end
|
|
55
|
-
|
|
56
51
|
# Validate intercept_addresses after configuration
|
|
57
52
|
if intercept_addresses.nil? || intercept_addresses.empty?
|
|
58
53
|
Coolhand.log "❌ Coolhand Error: Intercept addresses cannot be empty. Please set it in the configuration."
|
|
@@ -73,8 +73,9 @@ module Coolhand
|
|
|
73
73
|
# Convert Rails HTTP_ prefix headers
|
|
74
74
|
clean_key = key.to_s.gsub(/^HTTP_/, "").tr("_", "-").downcase
|
|
75
75
|
|
|
76
|
-
# Redact sensitive headers
|
|
77
|
-
|
|
76
|
+
# Redact sensitive headers (shared with BaseInterceptor so both logging
|
|
77
|
+
# paths treat the same header names as sensitive)
|
|
78
|
+
clean_value = clean_key.match?(BaseInterceptor::SENSITIVE_HEADER_PATTERN) ? "[REDACTED]" : value.to_s
|
|
78
79
|
clean_headers[clean_key] = clean_value
|
|
79
80
|
end
|
|
80
81
|
clean_headers
|
|
@@ -141,7 +141,7 @@ module Coolhand
|
|
|
141
141
|
|
|
142
142
|
matched = patterns.find { |pattern| url.include?(pattern) }
|
|
143
143
|
if matched && Coolhand.configuration.debug_mode
|
|
144
|
-
Coolhand.log "🚫 Skipping capture for #{url} (matched exclude_api_pattern: \"#{matched}\")"
|
|
144
|
+
Coolhand.log "🚫 Skipping capture for #{sanitize_url(url)} (matched exclude_api_pattern: \"#{matched}\")"
|
|
145
145
|
end
|
|
146
146
|
!!matched
|
|
147
147
|
end
|
data/lib/coolhand/version.rb
CHANGED
data/lib/coolhand.rb
CHANGED
|
@@ -45,6 +45,8 @@ module Coolhand
|
|
|
45
45
|
def configure
|
|
46
46
|
yield(configuration)
|
|
47
47
|
|
|
48
|
+
return unless configuration.enabled
|
|
49
|
+
|
|
48
50
|
configuration.validate!
|
|
49
51
|
|
|
50
52
|
NetHttpInterceptor.patch!
|
|
@@ -58,13 +60,15 @@ module Coolhand
|
|
|
58
60
|
return
|
|
59
61
|
end
|
|
60
62
|
|
|
61
|
-
|
|
63
|
+
return yield unless configuration.enabled
|
|
62
64
|
|
|
65
|
+
patched = NetHttpInterceptor.patched?
|
|
63
66
|
NetHttpInterceptor.patch!
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
67
|
+
begin
|
|
68
|
+
yield
|
|
69
|
+
ensure
|
|
70
|
+
NetHttpInterceptor.unpatch! unless patched
|
|
71
|
+
end
|
|
68
72
|
end
|
|
69
73
|
|
|
70
74
|
def without_capture
|
metadata
CHANGED
|
@@ -1,14 +1,15 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: coolhand
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.5.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Michael Carroll
|
|
8
8
|
- Yaroslav Malyk
|
|
9
|
+
autorequire:
|
|
9
10
|
bindir: exe
|
|
10
11
|
cert_chain: []
|
|
11
|
-
date: 2026-
|
|
12
|
+
date: 2026-07-31 00:00:00.000000000 Z
|
|
12
13
|
dependencies:
|
|
13
14
|
- !ruby/object:Gem::Dependency
|
|
14
15
|
name: base64
|
|
@@ -34,6 +35,8 @@ executables: []
|
|
|
34
35
|
extensions: []
|
|
35
36
|
extra_rdoc_files: []
|
|
36
37
|
files:
|
|
38
|
+
- ".claude/skills/loop-review/SKILL.md"
|
|
39
|
+
- ".claude/skills/prep-release/SKILL.md"
|
|
37
40
|
- ".idea/coolhand-ruby.iml"
|
|
38
41
|
- ".rspec"
|
|
39
42
|
- ".rubocop.yml"
|
|
@@ -43,9 +46,11 @@ files:
|
|
|
43
46
|
- LICENSE
|
|
44
47
|
- README.md
|
|
45
48
|
- Rakefile
|
|
46
|
-
-
|
|
49
|
+
- SECURITY.md
|
|
47
50
|
- docs/anthropic.md
|
|
51
|
+
- docs/configuration.md
|
|
48
52
|
- docs/elevenlabs.md
|
|
53
|
+
- docs/feedback.md
|
|
49
54
|
- lib/coolhand.rb
|
|
50
55
|
- lib/coolhand/api_service.rb
|
|
51
56
|
- lib/coolhand/base_interceptor.rb
|
|
@@ -71,6 +76,7 @@ metadata:
|
|
|
71
76
|
source_code_uri: https://github.com/Coolhand-Labs/coolhand-ruby
|
|
72
77
|
changelog_uri: https://github.com/Coolhand-Labs/coolhand-ruby/blob/main/CHANGELOG.md
|
|
73
78
|
rubygems_mfa_required: 'true'
|
|
79
|
+
post_install_message:
|
|
74
80
|
rdoc_options: []
|
|
75
81
|
require_paths:
|
|
76
82
|
- lib
|
|
@@ -85,7 +91,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
85
91
|
- !ruby/object:Gem::Version
|
|
86
92
|
version: '0'
|
|
87
93
|
requirements: []
|
|
88
|
-
rubygems_version: 3.
|
|
94
|
+
rubygems_version: 3.5.22
|
|
95
|
+
signing_key:
|
|
89
96
|
specification_version: 4
|
|
90
97
|
summary: Monitor and log LLM API calls from OpenAI, Anthropic, and other providers
|
|
91
98
|
to Coolhand analytics.
|
data/coolhand-ruby.gemspec
DELETED
|
@@ -1,46 +0,0 @@
|
|
|
1
|
-
# frozen_string_literal: true
|
|
2
|
-
|
|
3
|
-
require_relative "lib/coolhand/version"
|
|
4
|
-
|
|
5
|
-
Gem::Specification.new do |spec|
|
|
6
|
-
spec.name = "coolhand"
|
|
7
|
-
spec.version = Coolhand::VERSION
|
|
8
|
-
spec.authors = ["Michael Carroll", "Yaroslav Malyk"]
|
|
9
|
-
spec.email = ["mc@coolhandlabs.com"]
|
|
10
|
-
|
|
11
|
-
spec.summary = "Monitor and log LLM API calls from OpenAI, Anthropic, and other providers to Coolhand analytics."
|
|
12
|
-
spec.description = "Automatically intercept and log LLM requests from Ruby applications. Supports OpenAI, " \
|
|
13
|
-
"official Anthropic gem, ruby-anthropic gem, and other Faraday-based libraries. Features " \
|
|
14
|
-
"dual interceptor architecture, streaming support, thread-safe operation, and automatic " \
|
|
15
|
-
"duplicate request prevention."
|
|
16
|
-
spec.homepage = "https://coolhandlabs.com/"
|
|
17
|
-
spec.license = "Apache-2.0"
|
|
18
|
-
spec.required_ruby_version = ">= 3.0.0"
|
|
19
|
-
|
|
20
|
-
spec.metadata["allowed_push_host"] = "https://rubygems.org"
|
|
21
|
-
|
|
22
|
-
spec.metadata["homepage_uri"] = spec.homepage
|
|
23
|
-
spec.metadata["source_code_uri"] = "https://github.com/Coolhand-Labs/coolhand-ruby"
|
|
24
|
-
spec.metadata["changelog_uri"] = "https://github.com/Coolhand-Labs/coolhand-ruby/blob/main/CHANGELOG.md"
|
|
25
|
-
|
|
26
|
-
# Specify which files should be added to the gem when it is released.
|
|
27
|
-
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
|
28
|
-
spec.files = Dir.chdir(__dir__) do
|
|
29
|
-
`git ls-files -z`.split("\x0").reject do |f|
|
|
30
|
-
(File.expand_path(f) == __FILE__) ||
|
|
31
|
-
f.start_with?(*%w[bin/ test/ spec/ features/ .git appveyor Gemfile])
|
|
32
|
-
end
|
|
33
|
-
end
|
|
34
|
-
spec.bindir = "exe"
|
|
35
|
-
spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
|
|
36
|
-
spec.require_paths = ["lib"]
|
|
37
|
-
|
|
38
|
-
# Uncomment to register a new dependency of your gem
|
|
39
|
-
# spec.add_dependency "example-gem", "~> 1.0"
|
|
40
|
-
|
|
41
|
-
spec.add_dependency "base64", "~> 0.2"
|
|
42
|
-
|
|
43
|
-
# For more information and examples about making a new gem, check out our
|
|
44
|
-
# guide at: https://bundler.io/guides/creating_gem.html
|
|
45
|
-
spec.metadata["rubygems_mfa_required"] = "true"
|
|
46
|
-
end
|