specproof 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (119) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +10 -0
  3. data/LICENSE.txt +21 -0
  4. data/README.md +283 -0
  5. data/docs/agent-skills.md +60 -0
  6. data/docs/benchmark.md +24 -0
  7. data/docs/constrained-partition-pairs-results.md +44 -0
  8. data/docs/finite-strings-results.md +75 -0
  9. data/docs/implementation-status.md +65 -0
  10. data/docs/input-selection-results.md +123 -0
  11. data/docs/lint-cleanup-plan.md +12 -0
  12. data/docs/method-targets-results.md +68 -0
  13. data/docs/minitest-compatibility.md +33 -0
  14. data/docs/optional-arguments-plan.md +25 -0
  15. data/docs/optional-arguments-results.md +45 -0
  16. data/docs/pairwise-partitions-results.md +53 -0
  17. data/docs/pairwise-results.md +77 -0
  18. data/docs/pairwise-suggestions-results.md +104 -0
  19. data/docs/rbs-implementation-status.md +115 -0
  20. data/docs/rbs-string-literals-results.md +58 -0
  21. data/docs/real-project-pilot.md +84 -0
  22. data/docs/report-format.md +244 -0
  23. data/examples/age_policy/README.md +39 -0
  24. data/examples/age_policy/faulty/lib/age_policy.rb +9 -0
  25. data/examples/age_policy/faulty/test/age_policy_examples_test.rb +17 -0
  26. data/examples/age_policy/faulty/test/age_policy_test.rb +21 -0
  27. data/examples/age_policy/lib/age_policy.rb +8 -0
  28. data/examples/age_policy/run.rb +23 -0
  29. data/examples/age_policy/specification.rb +8 -0
  30. data/examples/age_policy/test/age_policy_test.rb +18 -0
  31. data/examples/constrained_partition_pairs/README.md +40 -0
  32. data/examples/constrained_partition_pairs/lib/account_eligibility.rb +8 -0
  33. data/examples/constrained_partition_pairs/requirements.md +11 -0
  34. data/examples/constrained_partition_pairs/run.rb +22 -0
  35. data/examples/constrained_partition_pairs/specification.rb +17 -0
  36. data/examples/constrained_partition_pairs/test_account_eligibility.rb +34 -0
  37. data/examples/input_selection/README.md +22 -0
  38. data/examples/input_selection/delivery_policy.rb +11 -0
  39. data/examples/input_selection/requirements.md +9 -0
  40. data/examples/input_selection/specification.rb +8 -0
  41. data/examples/input_selection/test_delivery_policy.rb +38 -0
  42. data/examples/method_targets/README.md +30 -0
  43. data/examples/method_targets/lib/delivery_defaults.rb +8 -0
  44. data/examples/method_targets/lib/delivery_options.rb +12 -0
  45. data/examples/method_targets/requirements.md +20 -0
  46. data/examples/method_targets/specification_objects.rb +17 -0
  47. data/examples/method_targets/specification_strings.rb +15 -0
  48. data/examples/pairwise/README.md +46 -0
  49. data/examples/pairwise/constrained_requirements.md +6 -0
  50. data/examples/pairwise/constrained_specification.rb +11 -0
  51. data/examples/pairwise/corrected/lib/delivery_options.rb +11 -0
  52. data/examples/pairwise/corrected/test/delivery_options_test.rb +28 -0
  53. data/examples/pairwise/faulty/lib/delivery_options.rb +11 -0
  54. data/examples/pairwise/faulty/test/gap_delivery_options_test.rb +10 -0
  55. data/examples/pairwise/faulty/test/initial_delivery_options_test.rb +23 -0
  56. data/examples/pairwise/requirements.md +11 -0
  57. data/examples/pairwise/run.rb +36 -0
  58. data/examples/pairwise/specification.rb +9 -0
  59. data/examples/pairwise/three_boolean/lib/boolean_batch.rb +8 -0
  60. data/examples/pairwise/three_boolean/test/boolean_batch_test.rb +28 -0
  61. data/examples/pairwise/three_boolean_specification.rb +10 -0
  62. data/examples/pairwise_partitions/README.md +34 -0
  63. data/examples/pairwise_partitions/lib/age_policy.rb +8 -0
  64. data/examples/pairwise_partitions/requirements.md +8 -0
  65. data/examples/pairwise_partitions/run.rb +22 -0
  66. data/examples/pairwise_partitions/specification.rb +13 -0
  67. data/examples/pairwise_partitions/test_age_policy.rb +29 -0
  68. data/examples/pairwise_suggestions/README.md +48 -0
  69. data/examples/pairwise_suggestions/run.rb +55 -0
  70. data/examples/pairwise_suggestions/three_boolean/lib/suggested_boolean_batch.rb +8 -0
  71. data/examples/pairwise_suggestions/three_boolean/requirements.md +7 -0
  72. data/examples/pairwise_suggestions/three_boolean/specification.rb +11 -0
  73. data/examples/pairwise_suggestions/three_boolean/test/baseline_test.rb +9 -0
  74. data/examples/pairwise_suggestions/three_boolean/test/complete_test.rb +22 -0
  75. data/examples/partition_gap.rb +8 -0
  76. data/examples/rbs_registration/README.md +58 -0
  77. data/examples/rbs_registration/implementations/age_gt/registration.rb +8 -0
  78. data/examples/rbs_registration/implementations/age_gte/registration.rb +8 -0
  79. data/examples/rbs_registration/requirements.md +9 -0
  80. data/examples/rbs_registration/sig/registration.rbs +3 -0
  81. data/examples/rbs_registration/specification.rb +10 -0
  82. data/examples/rbs_registration/test_registration.rb +43 -0
  83. data/examples/rbs_string_literals/README.md +34 -0
  84. data/examples/rbs_string_literals/lib/delivery_options.rb +9 -0
  85. data/examples/rbs_string_literals/requirements.md +11 -0
  86. data/examples/rbs_string_literals/sig/delivery_options.rbs +3 -0
  87. data/examples/rbs_string_literals/specification.rb +7 -0
  88. data/examples/rbs_string_literals/test_delivery_options.rb +46 -0
  89. data/examples/shipping.rb +12 -0
  90. data/exe/specproof +7 -0
  91. data/lib/specproof/checker.rb +144 -0
  92. data/lib/specproof/cli.rb +292 -0
  93. data/lib/specproof/collector.rb +304 -0
  94. data/lib/specproof/combinations.rb +226 -0
  95. data/lib/specproof/composer.rb +97 -0
  96. data/lib/specproof/coverage.rb +229 -0
  97. data/lib/specproof/coverage_report.rb +334 -0
  98. data/lib/specproof/domains.rb +124 -0
  99. data/lib/specproof/loader.rb +285 -0
  100. data/lib/specproof/minitest.rb +443 -0
  101. data/lib/specproof/model.rb +198 -0
  102. data/lib/specproof/partition_space.rb +268 -0
  103. data/lib/specproof/planner.rb +125 -0
  104. data/lib/specproof/report.rb +248 -0
  105. data/lib/specproof/skills.rb +188 -0
  106. data/lib/specproof/sources/rbs.rb +137 -0
  107. data/lib/specproof/sources/rbs_projector.rb +268 -0
  108. data/lib/specproof/suggestion_report.rb +159 -0
  109. data/lib/specproof/suggestions.rb +352 -0
  110. data/lib/specproof/target_name.rb +100 -0
  111. data/lib/specproof/type_constraints.rb +76 -0
  112. data/lib/specproof/version.rb +7 -0
  113. data/lib/specproof.rb +100 -0
  114. data/sig/specproof.rbs +222 -0
  115. data/skills/specproof-specify/SKILL.md +282 -0
  116. data/skills/specproof-specify/references/usage.md +265 -0
  117. data/skills/specproof-test-gaps/SKILL.md +214 -0
  118. data/skills/specproof-test-gaps/references/usage.md +231 -0
  119. metadata +165 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 100497f9c2832ec3daa9906c1e7f80579a134fa82bf107c4ac4b2024b8fbf345
4
+ data.tar.gz: 3cd2f171453545d1d423bd6df3a15d4b98bd14dd073a9cb26d04c824f67cb897
5
+ SHA512:
6
+ metadata.gz: 1bdf315ba69e5ed2a55174c81d2318383b8362ec6129852285cd4798265fae31a8befdddba0ed005f37d4473600b3096b0fb0c81c1a0129fb50f79ef097540e7
7
+ data.tar.gz: 2b4932bf07e4bb785a39bc89a490bca27907f197cd59501d064f3b8062dfeae3691271fc3073fee12c79a55adae4ed0851910fb0ef1081a35726dc64cceea923
data/CHANGELOG.md ADDED
@@ -0,0 +1,10 @@
1
+ ## [Unreleased]
2
+
3
+ ## [0.1.0] - 2026-10-08
4
+
5
+ - First public proof-of-concept release. The public API, specification DSL, and report formats may change, including breaking changes.
6
+ - Declare bounded integer and finite input domains; check partition gaps, overlaps, and incompatible constraints.
7
+ - Plan boundary candidates and feasible pairwise combinations, with deterministic suggestions for missing pairs.
8
+ - Observe supported method calls during Minitest runs and produce human-readable and JSON coverage reports.
9
+ - Support string and Ruby method-object targets, including class/module methods, plus optional RBS input declarations.
10
+ - Include agent skills for writing specifications and reviewing test gaps.
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2026 Lucian Ghinda
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,283 @@
1
+ # SpecProof
2
+
3
+ **Proof of concept:** SpecProof is experimental and under active development.
4
+ The public API, specification DSL, and report formats may change, including
5
+ breaking changes between releases. Pin the gem version when trying it in a project.
6
+
7
+ SpecProof checks a small, explicit contract about a Ruby method's inputs. It can find gaps or overlaps in declared input partitions, propose boundary-focused example values, and report which proposed values an existing Minitest suite actually passed to a method.
8
+
9
+ SpecProof is an aid for reviewing test gaps. It does not prove application behavior, generate tests, infer expected results, or claim that a passing test suite establishes correctness.
10
+
11
+ Use the `SpecProof` namespace in new code. The previous `Specproof` constant remains an alias.
12
+
13
+ ## Install
14
+
15
+ SpecProof requires Ruby 3.3 or newer and has no runtime gem dependencies. This release is tested on CRuby 3.3, 3.4, and 4.0. Add it to your application's Gemfile, pinning the version while the API is experimental:
16
+
17
+ ```ruby
18
+ gem "specproof", "0.1.0"
19
+ ```
20
+
21
+ Runtime Minitest collection is optional and supports Minitest `>= 5.26.2` and `< 5.28.0` with matching package/runtime major and minor versions; keep Minitest in the application's test group. The development bundle selects `~> 5.26.2` because the published 5.27.0 package contains a mismatched runtime version; see [compatibility evidence](docs/minitest-compatibility.md).
22
+
23
+ For development in this checkout, run commands from the gem root with `bundle exec`. To use a development checkout in another application, use a local path dependency:
24
+
25
+ ```ruby
26
+ gem "specproof", path: "/path/to/specproof/gems/specproof"
27
+ ```
28
+
29
+ ## Describe an input contract
30
+
31
+ Create a Ruby file that declares the target method and the input values it accepts:
32
+
33
+ ```ruby
34
+ require "specproof"
35
+
36
+ SpecProof.define "AgePolicy#adult?" do
37
+ input :age, range: 0..120, allow_gaps: false, allow_overlaps: false do
38
+ partition :minor, range: 0...18
39
+ partition :adult, range: 18..120
40
+ end
41
+ end
42
+ ```
43
+
44
+ The declaration is executable Ruby. Use it only for trusted specification files. SpecProof does not resolve or autoload the declared application targets during offline analysis; the declaration file itself can execute any Ruby its author writes. Integer ranges are inclusive at both ends unless written with `...`; finite domains accept `nil`, booleans, integers, symbols, and explicitly listed text values.
45
+
46
+ Targets can be written as canonical strings or as Ruby `Method`/`UnboundMethod`
47
+ objects. For example, `"DeliveryOptions#build"` and
48
+ `DeliveryOptions.instance_method(:build)` normalize to the same target. A
49
+ class or module singleton method uses a dot, as in `"DeliveryOptions.build"`
50
+ or `DeliveryOptions.method(:build)`. Strings are useful for offline
51
+ specifications: normalization does not load or autoload the application owner.
52
+ Method objects are convenient when the application code is already loaded;
53
+ they use Ruby reflection and do not invoke the method. The declaration-time
54
+ method must be a directly defined public method on a named class or module;
55
+ aliases, inherited singleton methods, anonymous owners, and methods bound to
56
+ arbitrary instances are unsupported. See the runnable
57
+ [method targets example](examples/method_targets/README.md) and
58
+ [verification results](docs/method-targets-results.md).
59
+
60
+ Finite strings are exact values, not a general String domain. For example, `values: [nil, 1, "user_123"]` describes three scenario values; `"user_123"` is distinct from `:user_123`. Only exact `String` instances with valid UTF-8 or US-ASCII encoding are accepted, and US-ASCII is canonicalized to UTF-8. Empty strings, whitespace, Unicode, and embedded NUL are valid; invalid bytes, other encodings, and String subclasses are rejected. Text is not trimmed, case-folded, or normalized. SpecProof uses owned snapshots without changing application strings. RBS string literals and finite unions of supported literals provide exact string scenarios; unrestricted `String` remains unsupported, and a selected union containing an unsupported member fails atomically.
61
+
62
+ `allow_gaps: false` requires every value in the input domain to belong to at
63
+ least one partition. `allow_overlaps: false` requires it to belong to at most
64
+ one. Together, every valid value belongs to exactly one partition. These rules
65
+ check the specification's grouping; they do not require testing every value.
66
+ Omitting either option permits that condition, equivalent to setting it to
67
+ `true`. Partition-based pairwise coverage requires both options to be `false`.
68
+
69
+ Existing specifications can still use `exhaustive: true` in place of
70
+ `allow_gaps: false`, and `disjoint: true` in place of `allow_overlaps: false`.
71
+ Their `false` values map to the new options' `true` values. Use only one
72
+ spelling per policy in a declaration. Equivalent spellings preserve contract
73
+ digests and obligation IDs.
74
+
75
+ Run the checker and planner:
76
+
77
+ ```sh
78
+ bundle exec specproof check examples/age_policy/specification.rb
79
+ bundle exec specproof plan examples/age_policy/specification.rb
80
+ bundle exec specproof plan examples/age_policy/specification.rb --format json
81
+ ```
82
+
83
+ `check` reports declared partition gaps and overlaps. `plan` reports candidate values from the `input-basics-v1` criterion, including values at the domain edges and partition boundaries. It does not claim those candidates form a complete test suite. JSON output has a versioned schema and includes source locations and a digest of the declared contract.
84
+
85
+ The age example plans `0`, `1`, `17`, `18`, `119`, and `120`. The shipping example in [examples/shipping.rb](examples/shipping.rb) shows finite domains. [examples/partition_gap.rb](examples/partition_gap.rb) is intentionally incomplete and makes `check` exit with a semantic finding.
86
+
87
+ ## Reuse an explicit RBS signature
88
+
89
+ RBS can supply supported finite scalar input types for explicitly selected
90
+ instance-method and singleton-method targets. Selectors use `Class#method` for
91
+ instance methods and `Class.method` or `Module.method` for singleton methods.
92
+ Add only behavioral bounds and partitions that the
93
+ signature cannot express; an unrestricted `Integer` needs such a refinement.
94
+ For a target selected from RBS, each DSL `input` or `exclude_input` must use a
95
+ name declared in that signature. By default all named signature parameters
96
+ must be analyzed; a deliberately narrower contract accounts for each omitted
97
+ name with an explicit reasoned exclusion, and still analyzes at least one
98
+ input. A mismatch reports `rbs_input_mismatch`; other DSL-only targets can
99
+ still be supplied alongside selected RBS targets. Supported finite RBS types
100
+ remain imported without duplicate DSL inputs. An explicit exclusion can scope
101
+ out a type-only parameter, while an unsupported selected union remains an
102
+ atomic failure.
103
+ The reader supports the verified RBS `4.2.0` parser only. RBS remains an
104
+ optional dependency, so consumers who use this workflow must add it to their
105
+ own bundle:
106
+
107
+ ```ruby
108
+ gem "rbs", "4.2.0", require: false
109
+ ```
110
+
111
+ The core gem has no runtime gem dependencies. See
112
+ [the supported integration limits](docs/rbs-implementation-status.md). For
113
+ the opt-in adapter bundle in this checkout, the [registration example](examples/rbs_registration) combines
114
+ `sig/registration.rbs` with a DSL age refinement. Its independent
115
+ [eligibility rule](examples/rbs_registration/requirements.md) says an applicant
116
+ is eligible exactly when `age >= 18` and consent is true.
117
+
118
+ The compact [finite string example](examples/rbs_string_literals) imports
119
+ `country: ("ro" | "de")` and `expedited: bool` without duplicate DSL inputs.
120
+ Its plan includes the exact country strings. String values remain distinct
121
+ from same-spelling symbols. RBS `String` remains an unrestricted domain and is
122
+ not supported. RBS return types are ignored; expected behavior must come from
123
+ an independent requirement.
124
+
125
+ From this checkout, run the check and plan with the explicit file and target:
126
+
127
+ ```sh
128
+ BUNDLE_GEMFILE=Gemfile.rbs bundle exec ruby -Ilib exe/specproof check examples/rbs_registration/specification.rb --rbs examples/rbs_registration/sig/registration.rbs --target 'Registration#eligible?' --format json
129
+ BUNDLE_GEMFILE=Gemfile.rbs bundle exec ruby -Ilib exe/specproof plan examples/rbs_registration/specification.rb --rbs examples/rbs_registration/sig/registration.rbs --target 'Registration#eligible?' --format json
130
+ ```
131
+
132
+ The plan contains nine marginal obligations: six age candidates and three
133
+ consent values (`nil`, `false`, and `true`). They are not nine input
134
+ combinations, do not specify expected results, and do not prove output
135
+ behavior. The runnable example shows an eight-of-nine report when age 18 is
136
+ omitted, the independent boundary assertion failing against an `age > 18`
137
+ implementation, and a fresh complete report with the corrected `age >= 18`
138
+ implementation. The faulty and corrected versions are separate files and each
139
+ checkpoint runs in its own process. See the example README for exact commands.
140
+
141
+ ## Observe an existing Minitest suite
142
+
143
+ The optional adapter records calls to directly defined public instance methods
144
+ and class/module singleton methods with named required or optional
145
+ positional/keyword parameters. By default, analyze every parameter, including
146
+ optional ones. A deliberately partial input scope must explicitly exclude every
147
+ other named parameter with `exclude_input :name, reason: "..."`, and at least
148
+ one input must remain analyzed. It observes effective bound scalar values at
149
+ method entry after Ruby applies defaults; an omitted argument and an explicitly
150
+ supplied equal value exercise the same obligation. A default that raises before
151
+ entry contributes no entry observation. Rest, keyword-rest, forwarding, and
152
+ explicit block parameters remain unsupported.
153
+
154
+ When only some parameters have finite input domains, you may explicitly exclude
155
+ the remaining named parameters with a reason. The declaration must still account
156
+ for every name in the runtime signature: each parameter is either analyzed or
157
+ excluded. Excluded values are not read by the collector. In particular, an
158
+ excluded object such as a `Hash` can coexist with selected scalar coverage,
159
+ while an object-valued selected input still makes collection incomplete. This
160
+ scope says nothing about excluded values or behavior involving their content.
161
+ See the runnable [input selection example](examples/input_selection/README.md).
162
+
163
+ It resolves the selected owner's canonical method name afresh when collection
164
+ starts, then observes that directly defined method during the original test
165
+ run in the same process and thread. A method object supplied at declaration
166
+ time must still describe that current direct definition; declaration and
167
+ collection both reject changed method identity. Calling an alias of the
168
+ selected definition counts; a subclass override counts only when it calls the
169
+ selected definition through `super`. Aliased target declarations and inherited
170
+ target declarations are unsupported. It does not replay tests, capture return
171
+ values, inspect assertions, infer defaults, or infer expected outcomes. A
172
+ non-primitive effective value for an analyzed input makes collection
173
+ incomplete, even if it comes from an optional default.
174
+
175
+ Add a separate test entrypoint after Minitest and your application code have loaded:
176
+
177
+ ```ruby
178
+ require "minitest/autorun"
179
+ require "age_policy"
180
+ require "specproof/minitest"
181
+
182
+ SpecProof::Minitest.install!(
183
+ specifications: ["examples/age_policy/specification.rb"],
184
+ output: "tmp/specproof-coverage.json",
185
+ fail_on_missing: false
186
+ )
187
+ ```
188
+
189
+ The runnable [age policy example](examples/age_policy) includes a passing implementation and assertions at all six planned values. Its `faulty` variant isolates the common `age > 18` boundary defect: the five non-boundary examples pass, while the planned value `18` remains unobserved. The separate explicit boundary assertion fails against that faulty implementation. Collection remains independent of the test result; a collector integrity failure fails the run, and original Minitest failures are preserved.
190
+
191
+ Reports distinguish exercised, not-observed, and unknown obligations; the `missing` summary lists the unobserved obligations. Missing values are informational unless `fail_on_missing: true` is selected. Reports identify their run, state whether collection completed, and use `obligation_hits` evidence with `replay: false`. JSON values include a type; integer values are decimal strings so large integers remain exact. String values use `{ "kind": "string", "value": "..." }` under the existing schema version 1, so consumers must handle this additive kind explicitly. Offline check/plan and runtime reports identify `scope.completeness_scope: "analyzed_inputs"` and list analyzed and excluded names in `scope.input_scope`. Human reports disclose excluded names and reasons. A matching set of obligation IDs does not establish matching scope: compare a fresh report's digest and input scope. `100%` means all candidates for analyzed inputs were exercised; excluded inputs, behavior, output, and interactions are not covered.
192
+
193
+ ## Scope and limits
194
+
195
+ SpecProof currently focuses on bounded integer ranges and finite primitive values, including exact enumerated text. The opt-in RBS reader supports only explicitly selected scalar signatures and rejects unsupported types rather than guessing. It does not analyze floats, time values, arbitrary objects, stateful or multi-call behavior, or SMT-based constraints. Specifications are trusted Ruby code; do not load declarations from untrusted sources. Collection adds an observation hook to selected application methods during the test run, so inspect the report as supporting evidence rather than as a semantic proof.
196
+
197
+ ## Agent skills
198
+
199
+ The gem includes two optional skills for authoring specifications and reviewing test gaps. Export them only to an explicit client directory:
200
+
201
+ ```sh
202
+ bundle exec specproof skills export --to .agents/skills
203
+ ```
204
+
205
+ The exporter reports the files it writes and refuses conflicting destination content. See [docs/agent-skills.md](docs/agent-skills.md) for their purpose and use.
206
+
207
+ Try “Use specproof-specify to check whether these age partitions cover their declared range” or “Use specproof-test-gaps to find missing boundary inputs and add tests justified by the supplied eligibility rules.” See the [behavioral evaluation record](docs/evaluations/agent-skills-v0.1.md) for observed outcomes and limits.
208
+
209
+ The [optional-argument pilot](docs/optional-arguments-results.md) shows adoption on PStoreLogger and the distinction between a complete scalar scenario report and incomplete full-suite collection on object payloads.
210
+
211
+ The [RBS implementation results](docs/rbs-implementation-status.md) include
212
+ the three-Ruby matrix, [six guided agent scenarios](docs/evaluations/rbs-agent-skills-v0.1.md),
213
+ and [unchanged-signature BranchProof adoption](docs/pilots/rbs-v0.1/README.md).
214
+
215
+ The [input-selection results](docs/input-selection-results.md) show the unchanged
216
+ PStoreLogger suite collecting scoped file-ID coverage with Hash payloads,
217
+ plus agent exercises that preserve excluded-input scope and failing assertions.
218
+
219
+ The [finite-string results](docs/finite-strings-results.md) extend that pilot to
220
+ the documented `"user_123"` case, showing its missing obligation and a separate
221
+ requirement-backed assertion that exercises it.
222
+
223
+ ## Pairwise finite input combinations
224
+
225
+ `input-basics-v1` remains the default scalar criterion. When a requirement
226
+ defines a finite joint scenario, declare its inputs in one `combinations`
227
+ group and explicitly choose `all_allowed: true` or list requirement-backed
228
+ `forbid` clauses. The `pairwise-finite-v1` criterion plans each pair of values
229
+ that occurs together in at least one allowed full assignment. It does not
230
+ assert expected outputs or establish that inputs are behaviorally independent.
231
+ Each analyzed input in a runtime call must still be valid before that call can
232
+ earn pairwise evidence.
233
+
234
+ The runnable [pairwise example](examples/pairwise/README.md) documents two
235
+ independent DeliveryOptions requirements and shows the faulty interaction
236
+ implementation passing three calls while missing the `expedited: true`,
237
+ `notify: true` pair. A requirement-backed assertion exposes the defect; a
238
+ separate corrected implementation then covers all four pair obligations. The
239
+ example also shows a constrained plan with three required and one excluded
240
+ pair, and a three-boolean plan with 12 pair obligations versus eight exhaustive
241
+ tuples.
242
+
243
+ Use `specproof suggest` when an intact pairwise coverage report has missing
244
+ pairs. It replans the current declarations, checks that the report matches
245
+ their exact typed obligations and recorded scope, then prints allowed full
246
+ group assignments that cover the missing pairs. Suggestions are candidates
247
+ for new tests: they are not executed, carry no expected results, and do not
248
+ fingerprint implementation or test files. Rerun the intended suite after
249
+ source or test changes. The [suggestion example](examples/pairwise_suggestions/README.md)
250
+ shows a three-of-four DeliveryOptions baseline, the `11` candidate, a separate
251
+ requirement-backed failing assertion, and a fresh corrected run. It also shows
252
+ one suggested three-boolean assignment covering several missing pairs, with
253
+ the other method inputs left for the caller to supply when outside the group.
254
+
255
+ ## Pairwise coverage over integer partitions
256
+
257
+ `pairwise-partitions-v1` pairs explicitly complete and non-overlapping integer
258
+ categories with other finite inputs. It supports `all_allowed: true` or
259
+ requirement-backed `forbid` assignments that select whole integer
260
+ partitions with `in_partition(:name)` and exact finite values. Each clause
261
+ forbids matching full assignments; required and excluded pairs are derived
262
+ from whether an allowed full completion exists. An excluded pair can therefore
263
+ be impossible indirectly through a restriction on a third input. Integer
264
+ restrictions must select a whole partition; raw integer values and partial
265
+ range predicates are not supported. Finite inputs continue to use exact values.
266
+ Category coverage is not boundary-value coverage and does not assert equivalent
267
+ behavior throughout a category. The runnable [unrestricted partition
268
+ example](examples/pairwise_partitions/README.md) shows a missing minor/`"de"`
269
+ category pair. The [constrained example](examples/constrained_partition_pairs/README.md)
270
+ shows requirement-backed restrictions, an excluded category pair, and allowed
271
+ assignment suggestions. See the [initial partition results](docs/pairwise-partitions-results.md)
272
+ and [constrained partition results](docs/constrained-partition-pairs-results.md)
273
+ for verification evidence.
274
+
275
+ ## Development
276
+
277
+ Run the test and lint tasks with `bundle exec rake`. The default task runs Minitest and RuboCop. The CI matrix covers Ruby 3.3, 3.4, and 4.0.
278
+
279
+ To reproduce the small baseline and instrumented timing comparison, see [docs/benchmark.md](docs/benchmark.md).
280
+
281
+ ## License
282
+
283
+ SpecProof is released under the MIT License. See [LICENSE.txt](LICENSE.txt).
@@ -0,0 +1,60 @@
1
+ # Agent skills
2
+
3
+ SpecProof ships two optional Agent Skills alongside the Ruby gem. Export them
4
+ into an application's own skill directory with an explicit destination:
5
+
6
+ ```sh
7
+ specproof skills export --to .agents/skills
8
+ ```
9
+
10
+ The command copies `specproof-specify/` and `specproof-test-gaps/` into that
11
+ directory. Repeating the command leaves identical files alone. If a packaged
12
+ file would overwrite different content, export reports a conflict before
13
+ writing any skill files. Other files in the destination, including local files
14
+ inside either skill folder, are kept. An interrupted filesystem operation can
15
+ still leave a partial copy; rerun the command after correcting the reported
16
+ filesystem issue.
17
+
18
+ The skills also document the opt-in RBS workflow: inspect explicitly selected
19
+ signatures, reuse supported finite input types, and add only requirement-backed
20
+ behavioral refinements. They support deliberate partial input scope through
21
+ one named, reasoned exclusion for each out-of-scope parameter; default scope
22
+ remains every named parameter, with at least one analyzed input required. The
23
+ gap-review skill requires fresh runtime evidence and compares the report's
24
+ analyzed/excluded input scope as well as its digest, integrity, test results,
25
+ and actual test command. A complete percentage covers analyzed input
26
+ obligations only, and human reports disclose exclusions. RBS supplies no output
27
+ oracle. The runnable [input selection example](../examples/input_selection/README.md)
28
+ shows a Hash payload excluded while a boolean option remains analyzed. Earlier evaluation results
29
+ in this document's linked record describe the workflows and artifacts named
30
+ there; they do not establish agent performance for this newer RBS workflow.
31
+
32
+ For Codex repository discovery, place the exported folders under
33
+ `.agents/skills` in the repository. Codex searches that directory from the
34
+ current working directory through its repository ancestors. Open a new Codex
35
+ session in the project and ask for one of the named skills to confirm it is
36
+ available. This discovery check is separate from the export result: export
37
+ confirms that files were copied, while the client session confirms that the
38
+ client found and loaded them. See the [Codex skills guide](https://learn.chatgpt.com/docs/build-skills)
39
+ and the [Agent Skills format](https://agentskills.io/specification).
40
+
41
+ Keep skill use opt-in. The exporter writes only to the path supplied by the
42
+ caller; it does not select a home directory, edit global configuration, alter
43
+ `AGENTS.md`, or access the network. The packaged skills and their local
44
+ references are self-contained, so the exported folder can be reviewed and
45
+ removed as ordinary project files.
46
+
47
+ The current RBS workflow was exercised in six fresh, isolated native Luna
48
+ tasks using skills exported from an installed gem. See the [RBS evaluation
49
+ record](evaluations/rbs-agent-skills-v0.1.md) for actual outcomes, retained
50
+ commands and reports, and the limits of these guided scenarios.
51
+
52
+ The export and one specification workflow were exercised in Codex CLI
53
+ `0.160.0` with `gpt-6-luna`, using `.agents/skills` in an isolated project and
54
+ the installed gem rather than this checkout. A separate fresh formatting task
55
+ did not load either SpecProof skill or create specifications. See the
56
+ [evaluation record](evaluations/agent-skills-v0.1.md) for retained artifacts,
57
+ baseline observations, repeated attempts, and limitations. These checks are
58
+ bounded observations of agent behavior, not a guarantee for every agent or
59
+ client. Other clients can receive the same exported folders through their
60
+ documented manual skill installation workflow; they have not been tested here.
data/docs/benchmark.md ADDED
@@ -0,0 +1,24 @@
1
+ # Small-suite collection timing
2
+
3
+ This benchmark compares equivalent Minitest subprocesses that call `AgePolicy#adult?` three or six times. The test methods and assertions are generated once for each suite size and run unchanged in baseline and instrumented modes. The instrumented mode installs the optional adapter and verifies that report hit totals match the direct call counter.
4
+
5
+ Run it from `gems/specproof` with a Minitest 5.27.x installation whose activated gem version and `Minitest::VERSION` both identify the supported release:
6
+
7
+ ```sh
8
+ ruby script/benchmark_age_policy.rb
9
+ ```
10
+
11
+ The script runs five repetitions per mode by default. Pass a positive iteration count as the first argument for a shorter smoke run.
12
+
13
+ This recorded run used Ruby 4.0.1 on arm64 macOS, Minitest gem `5.27.0`, runtime `Minitest::VERSION` `5.27.0`, and SpecProof `0.1.0`. The available RubyGems 5.27.0 artifact on this machine had an embedded runtime version of `5.26.2`, so the measurement used an unmodified local rebuild of the `v5.27.0` source tag to meet the adapter's package/runtime version check. No Minitest code was patched for the run.
14
+
15
+ Each timing includes Ruby process startup, Bundler setup, Minitest startup, the test suite, and report writing. All five repetitions made exactly the number of calls shown; instrumented hit totals matched those counts.
16
+
17
+ | Suite size | Mode | Calls per run | Wall time, repetitions 1–5 (seconds) |
18
+ | --- | --- | ---: | --- |
19
+ | 3 tests | Baseline | 3 | 0.229863, 0.231492, 0.218486, 0.235095, 0.229194 |
20
+ | 3 tests | Instrumented | 3 | 0.262548, 0.254388, 0.261317, 0.280937, 0.239743 |
21
+ | 6 tests | Baseline | 6 | 0.236327, 0.230261, 0.241569, 0.230310, 0.220956 |
22
+ | 6 tests | Instrumented | 6 | 0.241652, 0.250458, 0.237547, 0.246748, 0.239821 |
23
+
24
+ These measurements describe only this tiny fixture and subprocess setup. Process startup dominates much of the elapsed time, so the results do not estimate overhead for a larger application suite.
@@ -0,0 +1,44 @@
1
+ # Constrained partition pairs: verification
2
+
3
+ The [example](../examples/constrained_partition_pairs/README.md) declares
4
+ `forbid age: in_partition(:minor), account: "business"`. Filtering complete
5
+ category assignments leaves six allowed tuples, eleven required pairs, and
6
+ one excluded pair. Its three baseline calls cover nine pairs. Two suggested
7
+ calls, with independent requirement-based assertions, bring coverage to eleven
8
+ of eleven.
9
+
10
+ Luna subagents implemented the core, integration tests and documentation.
11
+ Separate specification and code reviews found no remaining code issues.
12
+ Review corrected missing restriction provenance and clarified historical
13
+ documentation. Native argument validation retains exact finite types,
14
+ including `nil` and `false`; integer restrictions require explicit selectors.
15
+
16
+ Verification on CRuby 3.3.6, 3.4.7 and 4.0.1 passed on each version:
17
+
18
+ | Suite | Tests | Assertions |
19
+ | --- | ---: | ---: |
20
+ | Core | 303 | 1,579 |
21
+ | Optional RBS | 55 | 509 |
22
+
23
+ RBS signature validation, RuboCop (144 files), syntax checks, packaged skill
24
+ checks and gem building passed. A final reuse of the memoized allowed
25
+ assignments was checked with 50 focused tests and 240 assertions on Ruby 4.
26
+
27
+ A separate exhaustive oracle checked all 256 forbidden subsets of an
28
+ eight-tuple category space: two integer inputs sharing partition names and
29
+ a finite input containing `nil` and `false`. Allowed assignments, required
30
+ pairs, excluded pairs and acceptance of 8,192 native tuples matched independent
31
+ expected results, including the entirely forbidden space.
32
+
33
+ A Luna agent followed the updated test-gap skill against a scratch copy of
34
+ the example. It obtained the two legal suggestions from the 9/11 baseline,
35
+ used the separate requirements to justify expected outputs, and verified the
36
+ complete run: two tests, five assertions and 11/11 feasible pairs exercised.
37
+ The excluded minor/business pair remained outside the denominator. This was
38
+ a source-checkout guidance exercise, not a fresh installed-gem evaluation.
39
+
40
+ The implementation reuses existing coverage accounting, report validation and
41
+ suggestion selection. No dependencies were added. Restrictions select whole
42
+ integer partitions or exact finite values; arbitrary predicates and partial
43
+ integer restrictions remain unsupported. Existing enumeration limits apply
44
+ before restrictions reduce the space.
@@ -0,0 +1,75 @@
1
+ # Finite string values
2
+
3
+ Implemented on `implement/finite-strings`, based on `main` at `f63dcaa`.
4
+ Core, tests, documentation and skills are committed as `8195cae`.
5
+
6
+ ```ruby
7
+ SpecProof.define "PStoreLogger#save" do
8
+ input :file_id, values: [nil, 1, "user_123"]
9
+ exclude_input :data, reason: "Payload contents are outside these file-ID scenarios"
10
+ end
11
+ ```
12
+
13
+ Finite domains and partitions now accept exact valid UTF-8 or US-ASCII String
14
+ values, including empty strings and Unicode text. US-ASCII is stored as UTF-8;
15
+ there is no trimming, case folding or Unicode normalization. Same-spelling
16
+ strings and symbols are distinct. Declaration values and runtime observations
17
+ are copied and frozen without changing application-owned values. Collection
18
+ uses the value at method entry even when the method later mutates it.
19
+
20
+ JSON adds the typed value `{ "kind": "string", "value": "user_123" }` under
21
+ schema 1 and the existing `input-basics-v1` criterion. Existing nonstring IDs
22
+ and digests remain unchanged. Consumers must handle the new value kind.
23
+
24
+ ## Changes and simplifications
25
+
26
+ - `lib/specproof/domains.rb` centralizes supported-value validation and text
27
+ snapshots. The existing finite-domain representation, checker and planner
28
+ handle strings without a parallel string-domain hierarchy.
29
+ - Model, coverage, collector, report and source-neutral type constraints reuse
30
+ those rules; `sig/specproof.rbs` describes the expanded public value union.
31
+ - Focused tests cover encodings, mutation, mixed symbol/string values,
32
+ partitions, typed reports, runtime integrity and legacy identities.
33
+ - README, report documentation and both packaged agent skills explain exact
34
+ scenario scope and preserve independent assertion requirements.
35
+
36
+ No new dependency, CLI option, adequacy criterion or RBS extraction feature was
37
+ added. RBS String types and string literals remain unsupported atomically,
38
+ including mixed unions.
39
+
40
+ ## Verification
41
+
42
+ Each CRuby version (3.3.6, 3.4.7, 4.0.1) passes **178 core tests / 885
43
+ assertions** and **46 optional RBS tests / 433 assertions**, with zero failures,
44
+ errors or skips. RBS signatures validate on all three. The locked default
45
+ Ruby 4 bundle passes `rake`, including RuboCop's 98-file check. `bin/check`
46
+ passes syntax, package and skill checks (2 tests / 76 assertions).
47
+
48
+ Independent Luna specification and quality reviews approved the change.
49
+ [Matrix records](pilots/finite-strings/matrix/) retain actual commands, outputs,
50
+ statuses and library hashes. [Agent exercises](evaluations/finite-strings-skills.md)
51
+ retain an old-guidance baseline and actual installed-package application runs.
52
+
53
+ ## Real-project value
54
+
55
+ The [PStoreLogger pilot](pilots/finite-strings/README.md) preserves its original
56
+ five source/test files. Its unchanged suite passes 7 tests / 23 assertions but
57
+ exercises only 2/3 declared values: `"user_123"` is missing. Strict missing mode
58
+ correctly exits 1 while the report preserves `tests_passed: true`.
59
+
60
+ A separate test uses its README's custom-file-ID and collision rules to check
61
+ `user_123.pstore`, `user_123_1.pstore`, and the stored payloads. The resulting
62
+ suite passes 8 tests / 27 assertions and exercises 3/3 with intact collection.
63
+ The original application and tests are unchanged.
64
+ An independent evidence review confirmed the five source-file hashes, all 17
65
+ runtime library hashes, matching scope/digest/obligation IDs, and distinct
66
+ baseline/final run IDs.
67
+
68
+ ## Limits
69
+
70
+ This covers explicitly enumerated scenarios, not the entire String domain or
71
+ whole-method correctness. Other encodings, invalid text, String subclasses,
72
+ patterns, length constraints, RBS strings, outputs and interaction coverage are
73
+ outside this milestone. Outside-domain witness count remains bounded, but a
74
+ retained string witness contains the full text, so there is no byte-size bound.
75
+ The agent exercises are small guided tasks, not a general reliability estimate.
@@ -0,0 +1,65 @@
1
+ # First usable release implementation
2
+
3
+ The [approved plan](../../../docs/superpowers/plans/2026-10-02-first-usable-release.md) is implemented inside this gem on branch `implement/first-usable-release` and recorded in the implementation commit containing this document. The gem remains unpublished. Verification below records local execution on 2026-10-02; GitHub Actions has been configured but has not been run remotely.
4
+
5
+ ## Scope and coordination
6
+
7
+ - Core: explicit scalar input declarations, exact partition checks, `input-basics-v1`, deterministic human/JSON reports.
8
+ - Runtime: bounded streaming input evidence for selected serial Ruby method calls; optional Minitest integration for >= 5.26.2 and < 5.28.0 with matching package/runtime minor versions.
9
+ - Agent interface: two packaged skills, explicit offline export, isolated behavioral evaluations.
10
+ - Existing development RuboCop/Rake tooling is retained; no new runtime dependencies.
11
+ - Public namespace: `SpecProof`; preserve scaffold compatibility through `Specproof`.
12
+
13
+ ## Delivered work
14
+
15
+ - Tasks 1–4: immutable scalar domains, trusted Ruby declaration loading, exact partition checks, shared `input-basics-v1` planning, human and JSON CLI reports.
16
+ - Task 5: bounded TracePoint collection for directly owned public instance methods with named positional/keyword inputs, including optional parameters. Unsupported or duplicate resolved bindings, redefinition, and changed constant paths cannot produce complete evidence.
17
+ - Tasks 6–7: streaming counters, whole-tuple input validation, positive versus unknown evidence, optional serial Minitest adapter with fresh run IDs and atomic reports, optional strict missing-obligation failure.
18
+ - Task 8: [executable boundary workflow](../examples/age_policy/README.md), shipping and gap examples, [report contract](report-format.md), packaging, API signatures, compatibility checks, and [timing measurements](benchmark.md).
19
+ - Task 9: two packaged skills, conflict-preserving explicit export, and [retained agent evaluations](evaluations/agent-skills-v0.1.md).
20
+
21
+ Luna subagents implemented independent slices and reviewed specification conformance, runtime correctness, code quality, and evaluation outcomes. The implementation uses one shared obligation model rather than duplicating boundary logic in runtime coverage.
22
+
23
+ ## Acceptance evidence
24
+
25
+ | Plan criteria | Evidence |
26
+ | --- | --- |
27
+ | A01–A04 | Loader/checker/planner tests cover unresolved and autoloaded target names, exact gap/overlap witnesses, large domains without enumeration, small-domain reference comparisons, and finite-value identity. |
28
+ | A05–A06, A16 | Planner/coverage share obligations and IDs. CLI/report tests cover deterministic schema-versioned JSON, typed values, provenance, errors, and exit codes. |
29
+ | A07–A11 | Collector/coverage/subprocess tests cover scope, method identity, invalid tuples, bounded retained evidence, interruption, changed definitions/constants, original failures/exceptions, strict mode, and unknown evidence. |
30
+ | A12 | Shipped runner and workflow tests show passing tests with an age-18 gap, a meaningful assertion failing against the boundary bug, and the corrected implementation exercising all six unchanged obligations. |
31
+ | A13–A14 | Local suites pass on CRuby 3.3.6, 3.4.7, and 4.0.1. Actual gem installation and CLI export work outside the checkout. Remote matrix execution remains unobserved. |
32
+ | A15 | Packaged skill/reference validation, installed-gem export, idempotence, conflict preservation, symlink rejection, and paths containing spaces are tested. |
33
+ | A17–A19 | All six scenarios passed two fresh attempts with the final skills (12 accepted outcomes), independently audited. Baselines, discarded attempts, final files, hashes, and real Codex client positive/negative discovery runs are retained. Small fixtures do not establish general agent reliability or a causal improvement rate. |
34
+
35
+ ## First implementation verification (commit fc876dd)
36
+
37
+ - `bin/test`: **111 runs, 562 assertions, zero failures, errors, or skips**, separately on Ruby 3.3.6, 3.4.7, and 4.0.1.
38
+ - Default `bundle exec rake` on Ruby 4.0.1: full suite and RuboCop pass; **81 files**, zero offenses.
39
+ - `bin/check` from outside the checkout: syntax checks, skill/package tests (**2 runs, 66 assertions**), and gem build with its library present pass.
40
+ - RBS 4.2.0 `rbs -I sig validate` passes. This validates declarations, not implementation type checking.
41
+ - Five benchmark repetitions per mode preserve unchanged test bodies and exact 3/6 call counts; report hits match. No real-application overhead claim is made.
42
+
43
+ ### Minitest validation environment
44
+
45
+ The locally available gem labeled Minitest `5.27.0` contained runtime version `5.26.2`. The adapter correctly rejects that mismatch. Validation used unmodified library files from the official `v5.27.0` source tag, packaged into an isolated temporary GEM_HOME. Neither global gems nor the runtime version constant were patched. Source archive SHA-256: `766c98a1d07df624320f69a013bd11994506eec930ce70964f87c7d2ee8a97c7`.
46
+
47
+ That initial verification required both activated package and runtime versions to identify Minitest 5.27.x. The subsequent [compatibility change](minitest-compatibility.md) supports correctly labeled published 5.26.2 and removes the need for a source rebuild in the development bundle. Mismatched package/runtime minor versions remain rejected. Offline checking/planning and skill export do not need Minitest.
48
+
49
+ ## Installation and real-project pilot follow-up
50
+
51
+ The normal development bundle now resolves published Minitest 5.26.2. Full suites pass with **115 tests and 573 assertions** on Ruby 3.3.6, 3.4.7, and 4.0.1. Ruby 3.3/3.4 checks used standard local Bundler installation from the checksum-verified published package, with unrelated ambient Ruby 4 gem paths cleared. No custom dependency source rebuild or version-constant patch was needed.
52
+
53
+ The [real-project pilot](real-project-pilot.md) applied the updated exported skills to QualityGate's existing configuration tests. One default-key input gap was closed, moving scoped exercise from 7/8 to 8/8 with unchanged IDs/digest and a fresh report. All 143 copied source files remain unchanged. Two other real candidate methods were rejected because of optional keyword parameters and class-method syntax; three successful integrations were not demonstrated.
54
+
55
+ Five baseline/instrumented timing pairs preserve the original 45-test, 342-assertion focused suite. The measured median subprocess difference was about 38 ms (12.5%); this is not a general application overhead claim. Source snapshots, package checksum, exact skills, test additions, reports, raw timings, and rejected probes are retained.
56
+
57
+ ## Limits and release follow-up
58
+
59
+ The [optional-argument follow-up](optional-arguments-results.md) records support for effective scalar values after Ruby applies defaults and a scoped PStoreLogger adoption pilot. Reports explicitly state that argument presence is not observed. Earlier pilot rejections and archived skill bytes describe their original implementation cuts.
60
+
61
+ This release checks explicit scalar input specifications and measures exercised input candidates; it does not prove implementation behavior or assertion quality. Declarations are trusted executable Ruby. Collection requires stable constant bindings and selected definitions in a serial same-process/thread window. Transient constant changes restored before final inspection are not detected.
62
+
63
+ Sorbet/RBS importers, output contracts, inferred partitions, state machines, combinatorial criteria, SMT, parallel collection, and MCP remain deferred. API signatures do not constitute an RBS specification importer.
64
+
65
+ The initial model evaluations preceded the final additive constant-binding guards and example-runner convenience changes; deterministic tests and review cover those changes. Their original skill bytes remain archived. Later live skill setup instructions reflect Minitest 5.26.2 compatibility and were freshly used in the focused real-project pilot. Remote CI, broader project adoption, and publishing remain release follow-up work. No gem was published and no branch was pushed.