bulldogger 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 90b1681b42b715c3a8c90836ae9fe8c2378d15fb542b014512ad9cbab6451089
+  data.tar.gz: 9535ac4a1206eb1dd91bef0170abf2d7d877ac4a18257dd47faafb15c4bcfa71
+SHA512:
+  metadata.gz: 042df51e319ebf9d8057d2f9125cffdd8437c6b3898e1ee5d450c33be21bec3d1218321f2256e0ea5cd51d4f9c1c73d9350e2fcfb2a40424ac7deb8d627df473
+  data.tar.gz: 27e433a4e256a55150270aa5448a8fb9ee06d378a1e528236d63b5f558016e04724cf574382ae64b9730f878deb3e3d5b49f3825b9ba4865de0f40ff86f93b1c

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 meganemura
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,262 @@
+# bulldogger
+
+bulldogger writes Ruby test failures as structured evidence for coding agents.
+Each JSON file contains the exception, the backtrace, and captured frame values.
+The failure output gives the absolute path to that file.
+
+Development and measurements used Ruby 4.0.6 and debug 1.11.1.
+Publication of bulldogger 0.1.0 is pending.
+
+## Install
+
+Add both gems to the test group:
+
+```ruby
+gem "bulldogger", group: :test
+gem "debug", group: :test
+```
+
+The `debug` gem supplies `DEBUGGER__.capture_frames`, which gives bulldogger locals from each frame.
+Ruby distributes `debug` as a bundled gem.
+Bundler only exposes it when the application Gemfile includes it.
+
+Without `debug`, bulldogger records locals from the raising frame.
+It records file, line, and label data for the remaining frames.
+
+Add one framework entry point.
+
+For Minitest, add this line to `test_helper.rb`:
+
+```ruby
+require "bulldogger/minitest"
+```
+
+For RSpec, add this line to `spec_helper.rb`:
+
+```ruby
+require "bulldogger/rspec"
+```
+
+Each entry point starts capture and records each failed test.
+It finishes the run index when the suite ends.
+
+## Three approaches
+
+bulldogger provides three ways to collect runtime evidence:
+
+- A failure snapshot is the default. Green tests do not capture data when they raise no exception.
+- `probe` watches named methods during one explicit run.
+- `record` traces all Ruby method calls during one explicit run.
+
+Use a failure snapshot when a failed test already provides an evidence path.
+Use `probe` to inspect one method or to compare behavior before and after a change.
+Use `record` when you must follow the full call sequence.
+
+## Failure output
+
+This output came from:
+
+```sh
+bundle exec ruby -Ilib test/fixtures/minitest_red/red_test.rb
+```
+
+```text
+  2) Error:
+RedTest#test_deep_raise:
+ArgumentError: expected 3 to equal the sum of [1, 2, 3]
+    test/fixtures/minitest_red/app.rb:9:in 'Order.total'
+    test/fixtures/minitest_red/red_test.rb:20:in 'RedTest#test_deep_raise'
+bulldogger evidence: /home/you/project/tmp/bulldogger/run-20260828-173512-6178/002-RedTest-test_deep_raise.json
+```
+
+Open the path after `bulldogger evidence:`.
+The file contains one failure and its captured runtime values.
+
+Each run uses this layout:
+
+```text
+tmp/bulldogger/
+  latest -> run-20260828-173512-6178
+  run-20260828-173512-6178/
+    001-RedTest-test_assertion_failure.json
+    002-RedTest-test_deep_raise.json
+    index.json
+```
+
+The [`bulldogger` skill](skills/bulldogger/SKILL.md) tells an agent how to inspect these files.
+The [evidence schema](docs/evidence-schema.md) defines every field and capture mode.
+
+## Target a method with probe
+
+Wrap the relevant test or operation with a named target:
+
+```ruby
+before_path = Bulldogger.probe("Billing::Invoice#amount") do
+  run_related_test
+end
+```
+
+The evidence summarizes argument and return classes, `nil` values, raised exits, and callers.
+It serializes the first 10 samples by default and counts every call.
+
+Run the probe before and after a change, then compare the two files:
+
+```ruby
+result = Bulldogger.probe_compare(before_path, after_path)
+result.fetch("identical")
+```
+
+An `identical` value of `true` shows that the compared behavior stayed the same.
+The comparison covers call counts, classes, `nil` counts, raised exits, parameters, callers, and normalized samples.
+
+This excerpt came from a generated probe file:
+
+```json
+{
+  "kind": "probe",
+  "targets": ["ProseSample#amount"],
+  "methods": {
+    "ProseSample#amount": {
+      "calls": 3,
+      "raised_exits": 1,
+      "returns": {
+        "classes": {"Integer": 1, "NilClass": 1},
+        "nil_count": 1,
+        "samples": [{"value": "21"}, {"value": "nil"}]
+      },
+      "raised": {"ArgumentError": 1},
+      "callers": {"-e:1:in 'block in <main>'": 3}
+    }
+  },
+  "limits": {"max_samples": 10, "max_value_length": 200}
+}
+```
+
+## Record the call sequence
+
+Wrap one focused operation when you need a full call sequence:
+
+```ruby
+trace_path = Bulldogger.record do
+  run_related_test
+end
+```
+
+The result is a JSONL file with a header and one object for each call, return, or raise event.
+This excerpt came from a generated trace:
+
+```jsonl
+{"schema_version":1,"kind":"record","events":["call","return","raise"],"limits":{"max_value_length":200}}
+{"event":"call","seq":1,"depth":1,"path":"-e","line":1,"method":"ProseTrace#outer","args":{"value":{"value":"3"}}}
+{"event":"return","seq":4,"depth":1,"path":"-e","line":1,"method":"ProseTrace#outer","return":{"value":"6"}}
+{"event":"raise","seq":7,"depth":2,"path":"-e","line":1,"method":"ProseTrace#inner","exception":{"class":"ArgumentError","message":"negative"}}
+{"event":"return","seq":8,"depth":2,"path":"-e","line":1,"method":"ProseTrace#inner","raised":true}
+```
+
+The [trace schema](docs/trace-schema.md) defines the event fields and tested `jq` queries.
+
+JSONL is the primary record format.
+`Bulldogger.trace_to_sqlite(trace_path, db_path)` converts an existing trace when the `sqlite3` gem is available.
+The converter uses a soft require, so bulldogger keeps zero runtime dependencies.
+
+## Cost
+
+`TracePoint(:raise)` observes every raised exception, including exceptions that application code rescues.
+These measurements used Ruby 4.0.6.
+Each condition has three runs, and the table gives each median.
+
+| condition | without bulldogger | with bulldogger | ratio |
+|---|---:|---:|---:|
+| 2,000,000 no-op iterations with no raised exception | 0.0389s | 0.0387s | 0.99x |
+| 10,000 raise and rescue cycles | 0.0052s | 0.4256s | 82.60x |
+| 200 recorded failures with file output | 0.0001s | 0.0252s | 412.85x |
+
+The second condition costs 42.042 microseconds for each exception.
+Repeated runs measured 40 to 42 microseconds and 76x to 83x.
+
+The capture cost has this measured breakdown:
+
+| stage | added cost for each exception |
+|---|---:|
+| Subscribe to `TracePoint(:raise)` | 0.019 microseconds |
+| Call `DEBUGGER__.capture_frames` | 1.411 microseconds |
+| Serialize, redact, and insert into the ring | 24.198 microseconds |
+
+Frame capture has a small cost in this measurement, while later processing accounts for most of the measured cost.
+
+A green suite with no raised exception caused no measurable overhead in this test.
+A green suite can still raise and rescue exceptions, and each exception incurs the capture cost.
+
+### Explicit verb cost
+
+The proportionality harness used one app fixture for both verbs.
+It measured about 1,330 ns per targeted method call for `probe`.
+It measured about 4,500 ns per traced call for `record`.
+
+`probe` cost scales with calls to the targeted method, which the harness names M.
+`record` cost scales with all traced calls, which the harness names N.
+With M/N at 0.25, the fixture measured `probe` at 8.28x and `record` at 111.76x.
+These ratios describe this app fixture, and another app has a different M/N value.
+
+`probe` and `record` are explicit verbs for one focused run before or after a change.
+Do not apply either verb continuously to the full suite.
+
+## Disable bulldogger
+
+Set `BULLDOGGER_DISABLE=1` to disable capture and output for one test process.
+`BULLDOGGER_DISABLED=1` is an alias with the same behavior.
+
+With either switch, startup returns before the `TracePoint(:raise)` subscription.
+It writes no evidence, creates no run directory, and adds no evidence line to a failure.
+The test exit code and failure count stay unchanged.
+Acceptance tests confirm this behavior for Minitest and RSpec.
+
+A rescue-heavy green suite measured 0.99x with bulldogger disabled.
+
+## Environment variables
+
+These environment variables configure a child test process:
+
+| Variable | Accepted value | Default and effect |
+|---|---|---|
+| `BULLDOGGER_DISABLE` | `1` | Capture is enabled by default. `1` disables capture and output. |
+| `BULLDOGGER_DISABLED` | `1` | Alias for `BULLDOGGER_DISABLE`. |
+| `BULLDOGGER_OUTPUT_DIR` | A nonempty path | The default is `tmp/bulldogger`, relative to the working directory. |
+| `BULLDOGGER_FRAME_SOURCE` | `capture_frames` or `degraded` | The default is automatic selection. |
+
+## Secrets and limits
+
+Captured values can contain secrets.
+bulldogger checks each local name before it calls `inspect` on the value.
+A matching local becomes `{"redacted": true, "reason": "name"}` and has no `value` field.
+
+The default patterns match these names without regard to case:
+
+- `password`, `passwd`, and `pass`
+- `secret` and `token`
+- `api_key` and `api-key`
+- the word `key`
+- `credential`, `auth`, `session`, and `cookie`
+
+The patterns favor redaction when a name is ambiguous.
+For example, `/auth/i` also matches `author` and `authorized`.
+Applications can replace `Bulldogger.config.redact_patterns` with their own regular expressions.
+
+bulldogger also checks keys while it renders a Hash.
+A matching key has the string `"[REDACTED]"` as its rendered value.
+
+The defaults keep 20 frames and 50 locals for each frame.
+Each rendered value keeps 200 characters, and each Array or Hash keeps 10 elements.
+The evidence file marks omitted or truncated data.
+
+## Scope of version 0.1
+
+Version 0.1 provides failure snapshots, targeted probes, and explicit full records.
+It writes JSON evidence and JSONL traces.
+The version 0.1 boundary exposes file artifacts and an offline SQLite converter.
+
+The [design decisions](docs/design-decisions.md) explain the three approaches and their measured costs.
+
+## License
+
+MIT. See `LICENSE`.

data/docs/design-decisions.md

@@ -0,0 +1,142 @@
+# Design decisions
+
+These decisions define the failure capture path in bulldogger 0.1.0.
+The measurements used Ruby 4.0.6 and debug 1.11.1.
+
+## Capture only raised exceptions
+
+An early design traced each method call and return.
+Measurements placed its run time at 60x to 106x the baseline.
+That cost applied throughout each test run.
+
+bulldogger subscribes to `TracePoint(:raise)`.
+The subscription produces no capture work until Ruby raises an exception.
+
+## Build no shadow stack
+
+Another design kept each frame `Binding` in a shadow call stack.
+A `TracePoint` updated the stack on every call and return.
+Measurements placed this design at 9x to 18x the baseline.
+
+bulldogger asks `DEBUGGER__.capture_frames` for the stack when Ruby raises.
+This choice removes the continuous call and return work.
+
+## Use one frame capture call
+
+`binding_of_caller` can return a caller binding.
+Repeated calls intended to walk outward returned the same frame during testing.
+This behavior made the captured stack incorrect.
+
+`DEBUGGER__.capture_frames` returns the available frame bindings in one call.
+
+## Serialize during the raise hook
+
+A `DEBUGGER__::FrameInfo` holds a live `Binding`.
+The binding keeps objects reachable from its frame alive.
+
+A measurement retained a frame from a `TracePoint` block.
+The retained binding could still reach the block's captured array.
+Keeping frame objects would let a bounded entry ring retain unbounded object graphs.
+
+bulldogger renders values before the `:raise` hook returns.
+The pending ring then holds Hashes, Strings, numbers, and Boolean values.
+It holds no captured `FrameInfo` or `Binding`.
+
+The full capture path measured 42.042 microseconds for each exception.
+This path captures frames, renders values, applies redaction, and inserts the snapshot into the ring.
+
+## Exclude bulldogger frames
+
+`DEBUGGER__.capture_frames(prefix)` removes frames whose paths start with `prefix`.
+A test passed the application directory and received zero application frames.
+
+bulldogger passes its own `lib` directory as the prefix.
+This value removes bulldogger frames while it retains application frames.
+
+## Keep zero runtime dependencies
+
+The gemspec declares development dependencies and no runtime dependency.
+The core uses the Ruby standard library.
+
+The complete frame source uses `debug/frame_info` when the application bundle exposes it.
+The `debug` gem is bundled with Ruby 4.0.6, and it is not a default gem.
+Under Bundler, the application Gemfile must include `debug`.
+
+When `debug/frame_info` raises `LoadError`, bulldogger uses the degraded frame source.
+That source records raising-frame locals and the remaining frame locations.
+
+This design lets applications choose the `debug` dependency.
+The installation guide includes `gem "debug", group: :test` for complete frame locals.
+
+## Require an explicit Minitest entry point
+
+Minitest discovers files named `minitest/*_plugin.rb` in each active gem.
+A measured plugin file started bulldogger in every Minitest run that activated the gem.
+The test suite did not need a `require "bulldogger"` line to trigger this behavior.
+
+This repository's own tests then ran with bulldogger capture active.
+The automatic behavior conflicted with the explicit integration boundary.
+
+bulldogger provides `require "bulldogger/minitest"` as the Minitest entry point.
+The require line makes capture activation visible in the test setup.
+
+## Keep complete probe counts
+
+The probe records every caller, class, and `nil` occurrence.
+Caller sampling could hide a call site and support a false claim that it did not occur.
+Count sampling could hide a later `nil` value and support the same false claim.
+We rejected both performance proposals because they would create false negative evidence.
+
+The probe serializes only the first 10 values by default.
+It continues the complete class, `nil`, and caller counts after that limit.
+This design keeps the evidence complete for the questions that probe answers.
+
+A raised method also produces a Ruby `:return` event with a `nil` value.
+The Ruby-level rescue counter identifies that raised exit.
+The evidence records `raised_exits` and excludes that event from normal return counts.
+
+## Measure the shipped event work
+
+Early estimates measured a different amount of work.
+Using those estimates would state a low probe cost and a high record cost.
+Measurement of the shipped paths corrected both directions.
+
+The common proportionality harness measured about 1,330 ns per targeted probe call.
+It measured about 4,500 ns per traced record call.
+The record-specific harness measured 40.94x for capture and 58.34x with file output.
+
+## Use proportionality as the performance rule
+
+A probe observes M calls to its named methods.
+A record observes N calls to all Ruby methods in its traced operation.
+The measured probe cost follows M, and the measured record cost follows N.
+
+A fixed cost ratio changes when the workload changes M/N.
+The proportionality rule lets a reader apply the measurement to an application call graph.
+
+## Keep JSONL as the record writer
+
+The record path writes versioned JSONL with one event on each line.
+`trace_to_sqlite` converts a finished file when the `sqlite3` gem is available.
+
+This converter keeps SQLite behind the file boundary.
+It also keeps the core at zero runtime dependencies.
+
+## Account for the coverage blind spot
+
+Ruby `Coverage` does not observe lines that run under a `TracePoint` callback.
+The suite calls the callback logic directly to test those lines.
+
+The coverage gate also keeps a line-specific ledger for callback gaps.
+The measured ledger has zero entries, and its fixed cap is zero.
+The gate fails when a new entry appears.
+
+## Limit dogfooding to the acceptance suite
+
+The acceptance suite runs under bulldogger and keeps its outer capture instance active.
+The dogfood demo produced `capture_frames` evidence with a planted local and a redacted secret.
+
+The unit setup resets the module singleton before each test.
+That reset also stops an outer dogfood instance.
+An instance API would let the subject and the test tool coexist.
+That API is planned after version 0.1.