specproof 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +10 -0
- data/LICENSE.txt +21 -0
- data/README.md +283 -0
- data/docs/agent-skills.md +60 -0
- data/docs/benchmark.md +24 -0
- data/docs/constrained-partition-pairs-results.md +44 -0
- data/docs/finite-strings-results.md +75 -0
- data/docs/implementation-status.md +65 -0
- data/docs/input-selection-results.md +123 -0
- data/docs/lint-cleanup-plan.md +12 -0
- data/docs/method-targets-results.md +68 -0
- data/docs/minitest-compatibility.md +33 -0
- data/docs/optional-arguments-plan.md +25 -0
- data/docs/optional-arguments-results.md +45 -0
- data/docs/pairwise-partitions-results.md +53 -0
- data/docs/pairwise-results.md +77 -0
- data/docs/pairwise-suggestions-results.md +104 -0
- data/docs/rbs-implementation-status.md +115 -0
- data/docs/rbs-string-literals-results.md +58 -0
- data/docs/real-project-pilot.md +84 -0
- data/docs/report-format.md +244 -0
- data/examples/age_policy/README.md +39 -0
- data/examples/age_policy/faulty/lib/age_policy.rb +9 -0
- data/examples/age_policy/faulty/test/age_policy_examples_test.rb +17 -0
- data/examples/age_policy/faulty/test/age_policy_test.rb +21 -0
- data/examples/age_policy/lib/age_policy.rb +8 -0
- data/examples/age_policy/run.rb +23 -0
- data/examples/age_policy/specification.rb +8 -0
- data/examples/age_policy/test/age_policy_test.rb +18 -0
- data/examples/constrained_partition_pairs/README.md +40 -0
- data/examples/constrained_partition_pairs/lib/account_eligibility.rb +8 -0
- data/examples/constrained_partition_pairs/requirements.md +11 -0
- data/examples/constrained_partition_pairs/run.rb +22 -0
- data/examples/constrained_partition_pairs/specification.rb +17 -0
- data/examples/constrained_partition_pairs/test_account_eligibility.rb +34 -0
- data/examples/input_selection/README.md +22 -0
- data/examples/input_selection/delivery_policy.rb +11 -0
- data/examples/input_selection/requirements.md +9 -0
- data/examples/input_selection/specification.rb +8 -0
- data/examples/input_selection/test_delivery_policy.rb +38 -0
- data/examples/method_targets/README.md +30 -0
- data/examples/method_targets/lib/delivery_defaults.rb +8 -0
- data/examples/method_targets/lib/delivery_options.rb +12 -0
- data/examples/method_targets/requirements.md +20 -0
- data/examples/method_targets/specification_objects.rb +17 -0
- data/examples/method_targets/specification_strings.rb +15 -0
- data/examples/pairwise/README.md +46 -0
- data/examples/pairwise/constrained_requirements.md +6 -0
- data/examples/pairwise/constrained_specification.rb +11 -0
- data/examples/pairwise/corrected/lib/delivery_options.rb +11 -0
- data/examples/pairwise/corrected/test/delivery_options_test.rb +28 -0
- data/examples/pairwise/faulty/lib/delivery_options.rb +11 -0
- data/examples/pairwise/faulty/test/gap_delivery_options_test.rb +10 -0
- data/examples/pairwise/faulty/test/initial_delivery_options_test.rb +23 -0
- data/examples/pairwise/requirements.md +11 -0
- data/examples/pairwise/run.rb +36 -0
- data/examples/pairwise/specification.rb +9 -0
- data/examples/pairwise/three_boolean/lib/boolean_batch.rb +8 -0
- data/examples/pairwise/three_boolean/test/boolean_batch_test.rb +28 -0
- data/examples/pairwise/three_boolean_specification.rb +10 -0
- data/examples/pairwise_partitions/README.md +34 -0
- data/examples/pairwise_partitions/lib/age_policy.rb +8 -0
- data/examples/pairwise_partitions/requirements.md +8 -0
- data/examples/pairwise_partitions/run.rb +22 -0
- data/examples/pairwise_partitions/specification.rb +13 -0
- data/examples/pairwise_partitions/test_age_policy.rb +29 -0
- data/examples/pairwise_suggestions/README.md +48 -0
- data/examples/pairwise_suggestions/run.rb +55 -0
- data/examples/pairwise_suggestions/three_boolean/lib/suggested_boolean_batch.rb +8 -0
- data/examples/pairwise_suggestions/three_boolean/requirements.md +7 -0
- data/examples/pairwise_suggestions/three_boolean/specification.rb +11 -0
- data/examples/pairwise_suggestions/three_boolean/test/baseline_test.rb +9 -0
- data/examples/pairwise_suggestions/three_boolean/test/complete_test.rb +22 -0
- data/examples/partition_gap.rb +8 -0
- data/examples/rbs_registration/README.md +58 -0
- data/examples/rbs_registration/implementations/age_gt/registration.rb +8 -0
- data/examples/rbs_registration/implementations/age_gte/registration.rb +8 -0
- data/examples/rbs_registration/requirements.md +9 -0
- data/examples/rbs_registration/sig/registration.rbs +3 -0
- data/examples/rbs_registration/specification.rb +10 -0
- data/examples/rbs_registration/test_registration.rb +43 -0
- data/examples/rbs_string_literals/README.md +34 -0
- data/examples/rbs_string_literals/lib/delivery_options.rb +9 -0
- data/examples/rbs_string_literals/requirements.md +11 -0
- data/examples/rbs_string_literals/sig/delivery_options.rbs +3 -0
- data/examples/rbs_string_literals/specification.rb +7 -0
- data/examples/rbs_string_literals/test_delivery_options.rb +46 -0
- data/examples/shipping.rb +12 -0
- data/exe/specproof +7 -0
- data/lib/specproof/checker.rb +144 -0
- data/lib/specproof/cli.rb +292 -0
- data/lib/specproof/collector.rb +304 -0
- data/lib/specproof/combinations.rb +226 -0
- data/lib/specproof/composer.rb +97 -0
- data/lib/specproof/coverage.rb +229 -0
- data/lib/specproof/coverage_report.rb +334 -0
- data/lib/specproof/domains.rb +124 -0
- data/lib/specproof/loader.rb +285 -0
- data/lib/specproof/minitest.rb +443 -0
- data/lib/specproof/model.rb +198 -0
- data/lib/specproof/partition_space.rb +268 -0
- data/lib/specproof/planner.rb +125 -0
- data/lib/specproof/report.rb +248 -0
- data/lib/specproof/skills.rb +188 -0
- data/lib/specproof/sources/rbs.rb +137 -0
- data/lib/specproof/sources/rbs_projector.rb +268 -0
- data/lib/specproof/suggestion_report.rb +159 -0
- data/lib/specproof/suggestions.rb +352 -0
- data/lib/specproof/target_name.rb +100 -0
- data/lib/specproof/type_constraints.rb +76 -0
- data/lib/specproof/version.rb +7 -0
- data/lib/specproof.rb +100 -0
- data/sig/specproof.rbs +222 -0
- data/skills/specproof-specify/SKILL.md +282 -0
- data/skills/specproof-specify/references/usage.md +265 -0
- data/skills/specproof-test-gaps/SKILL.md +214 -0
- data/skills/specproof-test-gaps/references/usage.md +231 -0
- metadata +165 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 100497f9c2832ec3daa9906c1e7f80579a134fa82bf107c4ac4b2024b8fbf345
|
|
4
|
+
data.tar.gz: 3cd2f171453545d1d423bd6df3a15d4b98bd14dd073a9cb26d04c824f67cb897
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 1bdf315ba69e5ed2a55174c81d2318383b8362ec6129852285cd4798265fae31a8befdddba0ed005f37d4473600b3096b0fb0c81c1a0129fb50f79ef097540e7
|
|
7
|
+
data.tar.gz: 2b4932bf07e4bb785a39bc89a490bca27907f197cd59501d064f3b8062dfeae3691271fc3073fee12c79a55adae4ed0851910fb0ef1081a35726dc64cceea923
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
## [Unreleased]
|
|
2
|
+
|
|
3
|
+
## [0.1.0] - 2026-10-08
|
|
4
|
+
|
|
5
|
+
- First public proof-of-concept release. The public API, specification DSL, and report formats may change, including breaking changes.
|
|
6
|
+
- Declare bounded integer and finite input domains; check partition gaps, overlaps, and incompatible constraints.
|
|
7
|
+
- Plan boundary candidates and feasible pairwise combinations, with deterministic suggestions for missing pairs.
|
|
8
|
+
- Observe supported method calls during Minitest runs and produce human-readable and JSON coverage reports.
|
|
9
|
+
- Support string and Ruby method-object targets, including class/module methods, plus optional RBS input declarations.
|
|
10
|
+
- Include agent skills for writing specifications and reviewing test gaps.
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Lucian Ghinda
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,283 @@
|
|
|
1
|
+
# SpecProof
|
|
2
|
+
|
|
3
|
+
**Proof of concept:** SpecProof is experimental and under active development.
|
|
4
|
+
The public API, specification DSL, and report formats may change, including
|
|
5
|
+
breaking changes between releases. Pin the gem version when trying it in a project.
|
|
6
|
+
|
|
7
|
+
SpecProof checks a small, explicit contract about a Ruby method's inputs. It can find gaps or overlaps in declared input partitions, propose boundary-focused example values, and report which proposed values an existing Minitest suite actually passed to a method.
|
|
8
|
+
|
|
9
|
+
SpecProof is an aid for reviewing test gaps. It does not prove application behavior, generate tests, infer expected results, or claim that a passing test suite establishes correctness.
|
|
10
|
+
|
|
11
|
+
Use the `SpecProof` namespace in new code. The previous `Specproof` constant remains an alias.
|
|
12
|
+
|
|
13
|
+
## Install
|
|
14
|
+
|
|
15
|
+
SpecProof requires Ruby 3.3 or newer and has no runtime gem dependencies. This release is tested on CRuby 3.3, 3.4, and 4.0. Add it to your application's Gemfile, pinning the version while the API is experimental:
|
|
16
|
+
|
|
17
|
+
```ruby
|
|
18
|
+
gem "specproof", "0.1.0"
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Runtime Minitest collection is optional and supports Minitest `>= 5.26.2` and `< 5.28.0` with matching package/runtime major and minor versions; keep Minitest in the application's test group. The development bundle selects `~> 5.26.2` because the published 5.27.0 package contains a mismatched runtime version; see [compatibility evidence](docs/minitest-compatibility.md).
|
|
22
|
+
|
|
23
|
+
For development in this checkout, run commands from the gem root with `bundle exec`. To use a development checkout in another application, use a local path dependency:
|
|
24
|
+
|
|
25
|
+
```ruby
|
|
26
|
+
gem "specproof", path: "/path/to/specproof/gems/specproof"
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
## Describe an input contract
|
|
30
|
+
|
|
31
|
+
Create a Ruby file that declares the target method and the input values it accepts:
|
|
32
|
+
|
|
33
|
+
```ruby
|
|
34
|
+
require "specproof"
|
|
35
|
+
|
|
36
|
+
SpecProof.define "AgePolicy#adult?" do
|
|
37
|
+
input :age, range: 0..120, allow_gaps: false, allow_overlaps: false do
|
|
38
|
+
partition :minor, range: 0...18
|
|
39
|
+
partition :adult, range: 18..120
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
The declaration is executable Ruby. Use it only for trusted specification files. SpecProof does not resolve or autoload the declared application targets during offline analysis; the declaration file itself can execute any Ruby its author writes. Integer ranges are inclusive at both ends unless written with `...`; finite domains accept `nil`, booleans, integers, symbols, and explicitly listed text values.
|
|
45
|
+
|
|
46
|
+
Targets can be written as canonical strings or as Ruby `Method`/`UnboundMethod`
|
|
47
|
+
objects. For example, `"DeliveryOptions#build"` and
|
|
48
|
+
`DeliveryOptions.instance_method(:build)` normalize to the same target. A
|
|
49
|
+
class or module singleton method uses a dot, as in `"DeliveryOptions.build"`
|
|
50
|
+
or `DeliveryOptions.method(:build)`. Strings are useful for offline
|
|
51
|
+
specifications: normalization does not load or autoload the application owner.
|
|
52
|
+
Method objects are convenient when the application code is already loaded;
|
|
53
|
+
they use Ruby reflection and do not invoke the method. The declaration-time
|
|
54
|
+
method must be a directly defined public method on a named class or module;
|
|
55
|
+
aliases, inherited singleton methods, anonymous owners, and methods bound to
|
|
56
|
+
arbitrary instances are unsupported. See the runnable
|
|
57
|
+
[method targets example](examples/method_targets/README.md) and
|
|
58
|
+
[verification results](docs/method-targets-results.md).
|
|
59
|
+
|
|
60
|
+
Finite strings are exact values, not a general String domain. For example, `values: [nil, 1, "user_123"]` describes three scenario values; `"user_123"` is distinct from `:user_123`. Only exact `String` instances with valid UTF-8 or US-ASCII encoding are accepted, and US-ASCII is canonicalized to UTF-8. Empty strings, whitespace, Unicode, and embedded NUL are valid; invalid bytes, other encodings, and String subclasses are rejected. Text is not trimmed, case-folded, or normalized. SpecProof uses owned snapshots without changing application strings. RBS string literals and finite unions of supported literals provide exact string scenarios; unrestricted `String` remains unsupported, and a selected union containing an unsupported member fails atomically.
|
|
61
|
+
|
|
62
|
+
`allow_gaps: false` requires every value in the input domain to belong to at
|
|
63
|
+
least one partition. `allow_overlaps: false` requires it to belong to at most
|
|
64
|
+
one. Together, every valid value belongs to exactly one partition. These rules
|
|
65
|
+
check the specification's grouping; they do not require testing every value.
|
|
66
|
+
Omitting either option permits that condition, equivalent to setting it to
|
|
67
|
+
`true`. Partition-based pairwise coverage requires both options to be `false`.
|
|
68
|
+
|
|
69
|
+
Existing specifications can still use `exhaustive: true` in place of
|
|
70
|
+
`allow_gaps: false`, and `disjoint: true` in place of `allow_overlaps: false`.
|
|
71
|
+
Their `false` values map to the new options' `true` values. Use only one
|
|
72
|
+
spelling per policy in a declaration. Equivalent spellings preserve contract
|
|
73
|
+
digests and obligation IDs.
|
|
74
|
+
|
|
75
|
+
Run the checker and planner:
|
|
76
|
+
|
|
77
|
+
```sh
|
|
78
|
+
bundle exec specproof check examples/age_policy/specification.rb
|
|
79
|
+
bundle exec specproof plan examples/age_policy/specification.rb
|
|
80
|
+
bundle exec specproof plan examples/age_policy/specification.rb --format json
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
`check` reports declared partition gaps and overlaps. `plan` reports candidate values from the `input-basics-v1` criterion, including values at the domain edges and partition boundaries. It does not claim those candidates form a complete test suite. JSON output has a versioned schema and includes source locations and a digest of the declared contract.
|
|
84
|
+
|
|
85
|
+
The age example plans `0`, `1`, `17`, `18`, `119`, and `120`. The shipping example in [examples/shipping.rb](examples/shipping.rb) shows finite domains. [examples/partition_gap.rb](examples/partition_gap.rb) is intentionally incomplete and makes `check` exit with a semantic finding.
|
|
86
|
+
|
|
87
|
+
## Reuse an explicit RBS signature
|
|
88
|
+
|
|
89
|
+
RBS can supply supported finite scalar input types for explicitly selected
|
|
90
|
+
instance-method and singleton-method targets. Selectors use `Class#method` for
|
|
91
|
+
instance methods and `Class.method` or `Module.method` for singleton methods.
|
|
92
|
+
Add only behavioral bounds and partitions that the
|
|
93
|
+
signature cannot express; an unrestricted `Integer` needs such a refinement.
|
|
94
|
+
For a target selected from RBS, each DSL `input` or `exclude_input` must use a
|
|
95
|
+
name declared in that signature. By default all named signature parameters
|
|
96
|
+
must be analyzed; a deliberately narrower contract accounts for each omitted
|
|
97
|
+
name with an explicit reasoned exclusion, and still analyzes at least one
|
|
98
|
+
input. A mismatch reports `rbs_input_mismatch`; other DSL-only targets can
|
|
99
|
+
still be supplied alongside selected RBS targets. Supported finite RBS types
|
|
100
|
+
remain imported without duplicate DSL inputs. An explicit exclusion can scope
|
|
101
|
+
out a type-only parameter, while an unsupported selected union remains an
|
|
102
|
+
atomic failure.
|
|
103
|
+
The reader supports the verified RBS `4.2.0` parser only. RBS remains an
|
|
104
|
+
optional dependency, so consumers who use this workflow must add it to their
|
|
105
|
+
own bundle:
|
|
106
|
+
|
|
107
|
+
```ruby
|
|
108
|
+
gem "rbs", "4.2.0", require: false
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
The core gem has no runtime gem dependencies. See
|
|
112
|
+
[the supported integration limits](docs/rbs-implementation-status.md). For
|
|
113
|
+
the opt-in adapter bundle in this checkout, the [registration example](examples/rbs_registration) combines
|
|
114
|
+
`sig/registration.rbs` with a DSL age refinement. Its independent
|
|
115
|
+
[eligibility rule](examples/rbs_registration/requirements.md) says an applicant
|
|
116
|
+
is eligible exactly when `age >= 18` and consent is true.
|
|
117
|
+
|
|
118
|
+
The compact [finite string example](examples/rbs_string_literals) imports
|
|
119
|
+
`country: ("ro" | "de")` and `expedited: bool` without duplicate DSL inputs.
|
|
120
|
+
Its plan includes the exact country strings. String values remain distinct
|
|
121
|
+
from same-spelling symbols. RBS `String` remains an unrestricted domain and is
|
|
122
|
+
not supported. RBS return types are ignored; expected behavior must come from
|
|
123
|
+
an independent requirement.
|
|
124
|
+
|
|
125
|
+
From this checkout, run the check and plan with the explicit file and target:
|
|
126
|
+
|
|
127
|
+
```sh
|
|
128
|
+
BUNDLE_GEMFILE=Gemfile.rbs bundle exec ruby -Ilib exe/specproof check examples/rbs_registration/specification.rb --rbs examples/rbs_registration/sig/registration.rbs --target 'Registration#eligible?' --format json
|
|
129
|
+
BUNDLE_GEMFILE=Gemfile.rbs bundle exec ruby -Ilib exe/specproof plan examples/rbs_registration/specification.rb --rbs examples/rbs_registration/sig/registration.rbs --target 'Registration#eligible?' --format json
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
The plan contains nine marginal obligations: six age candidates and three
|
|
133
|
+
consent values (`nil`, `false`, and `true`). They are not nine input
|
|
134
|
+
combinations, do not specify expected results, and do not prove output
|
|
135
|
+
behavior. The runnable example shows an eight-of-nine report when age 18 is
|
|
136
|
+
omitted, the independent boundary assertion failing against an `age > 18`
|
|
137
|
+
implementation, and a fresh complete report with the corrected `age >= 18`
|
|
138
|
+
implementation. The faulty and corrected versions are separate files and each
|
|
139
|
+
checkpoint runs in its own process. See the example README for exact commands.
|
|
140
|
+
|
|
141
|
+
## Observe an existing Minitest suite
|
|
142
|
+
|
|
143
|
+
The optional adapter records calls to directly defined public instance methods
|
|
144
|
+
and class/module singleton methods with named required or optional
|
|
145
|
+
positional/keyword parameters. By default, analyze every parameter, including
|
|
146
|
+
optional ones. A deliberately partial input scope must explicitly exclude every
|
|
147
|
+
other named parameter with `exclude_input :name, reason: "..."`, and at least
|
|
148
|
+
one input must remain analyzed. It observes effective bound scalar values at
|
|
149
|
+
method entry after Ruby applies defaults; an omitted argument and an explicitly
|
|
150
|
+
supplied equal value exercise the same obligation. A default that raises before
|
|
151
|
+
entry contributes no entry observation. Rest, keyword-rest, forwarding, and
|
|
152
|
+
explicit block parameters remain unsupported.
|
|
153
|
+
|
|
154
|
+
When only some parameters have finite input domains, you may explicitly exclude
|
|
155
|
+
the remaining named parameters with a reason. The declaration must still account
|
|
156
|
+
for every name in the runtime signature: each parameter is either analyzed or
|
|
157
|
+
excluded. Excluded values are not read by the collector. In particular, an
|
|
158
|
+
excluded object such as a `Hash` can coexist with selected scalar coverage,
|
|
159
|
+
while an object-valued selected input still makes collection incomplete. This
|
|
160
|
+
scope says nothing about excluded values or behavior involving their content.
|
|
161
|
+
See the runnable [input selection example](examples/input_selection/README.md).
|
|
162
|
+
|
|
163
|
+
It resolves the selected owner's canonical method name afresh when collection
|
|
164
|
+
starts, then observes that directly defined method during the original test
|
|
165
|
+
run in the same process and thread. A method object supplied at declaration
|
|
166
|
+
time must still describe that current direct definition; declaration and
|
|
167
|
+
collection both reject changed method identity. Calling an alias of the
|
|
168
|
+
selected definition counts; a subclass override counts only when it calls the
|
|
169
|
+
selected definition through `super`. Aliased target declarations and inherited
|
|
170
|
+
target declarations are unsupported. It does not replay tests, capture return
|
|
171
|
+
values, inspect assertions, infer defaults, or infer expected outcomes. A
|
|
172
|
+
non-primitive effective value for an analyzed input makes collection
|
|
173
|
+
incomplete, even if it comes from an optional default.
|
|
174
|
+
|
|
175
|
+
Add a separate test entrypoint after Minitest and your application code have loaded:
|
|
176
|
+
|
|
177
|
+
```ruby
|
|
178
|
+
require "minitest/autorun"
|
|
179
|
+
require "age_policy"
|
|
180
|
+
require "specproof/minitest"
|
|
181
|
+
|
|
182
|
+
SpecProof::Minitest.install!(
|
|
183
|
+
specifications: ["examples/age_policy/specification.rb"],
|
|
184
|
+
output: "tmp/specproof-coverage.json",
|
|
185
|
+
fail_on_missing: false
|
|
186
|
+
)
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
The runnable [age policy example](examples/age_policy) includes a passing implementation and assertions at all six planned values. Its `faulty` variant isolates the common `age > 18` boundary defect: the five non-boundary examples pass, while the planned value `18` remains unobserved. The separate explicit boundary assertion fails against that faulty implementation. Collection remains independent of the test result; a collector integrity failure fails the run, and original Minitest failures are preserved.
|
|
190
|
+
|
|
191
|
+
Reports distinguish exercised, not-observed, and unknown obligations; the `missing` summary lists the unobserved obligations. Missing values are informational unless `fail_on_missing: true` is selected. Reports identify their run, state whether collection completed, and use `obligation_hits` evidence with `replay: false`. JSON values include a type; integer values are decimal strings so large integers remain exact. String values use `{ "kind": "string", "value": "..." }` under the existing schema version 1, so consumers must handle this additive kind explicitly. Offline check/plan and runtime reports identify `scope.completeness_scope: "analyzed_inputs"` and list analyzed and excluded names in `scope.input_scope`. Human reports disclose excluded names and reasons. A matching set of obligation IDs does not establish matching scope: compare a fresh report's digest and input scope. `100%` means all candidates for analyzed inputs were exercised; excluded inputs, behavior, output, and interactions are not covered.
|
|
192
|
+
|
|
193
|
+
## Scope and limits
|
|
194
|
+
|
|
195
|
+
SpecProof currently focuses on bounded integer ranges and finite primitive values, including exact enumerated text. The opt-in RBS reader supports only explicitly selected scalar signatures and rejects unsupported types rather than guessing. It does not analyze floats, time values, arbitrary objects, stateful or multi-call behavior, or SMT-based constraints. Specifications are trusted Ruby code; do not load declarations from untrusted sources. Collection adds an observation hook to selected application methods during the test run, so inspect the report as supporting evidence rather than as a semantic proof.
|
|
196
|
+
|
|
197
|
+
## Agent skills
|
|
198
|
+
|
|
199
|
+
The gem includes two optional skills for authoring specifications and reviewing test gaps. Export them only to an explicit client directory:
|
|
200
|
+
|
|
201
|
+
```sh
|
|
202
|
+
bundle exec specproof skills export --to .agents/skills
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
The exporter reports the files it writes and refuses conflicting destination content. See [docs/agent-skills.md](docs/agent-skills.md) for their purpose and use.
|
|
206
|
+
|
|
207
|
+
Try “Use specproof-specify to check whether these age partitions cover their declared range” or “Use specproof-test-gaps to find missing boundary inputs and add tests justified by the supplied eligibility rules.” See the [behavioral evaluation record](docs/evaluations/agent-skills-v0.1.md) for observed outcomes and limits.
|
|
208
|
+
|
|
209
|
+
The [optional-argument pilot](docs/optional-arguments-results.md) shows adoption on PStoreLogger and the distinction between a complete scalar scenario report and incomplete full-suite collection on object payloads.
|
|
210
|
+
|
|
211
|
+
The [RBS implementation results](docs/rbs-implementation-status.md) include
|
|
212
|
+
the three-Ruby matrix, [six guided agent scenarios](docs/evaluations/rbs-agent-skills-v0.1.md),
|
|
213
|
+
and [unchanged-signature BranchProof adoption](docs/pilots/rbs-v0.1/README.md).
|
|
214
|
+
|
|
215
|
+
The [input-selection results](docs/input-selection-results.md) show the unchanged
|
|
216
|
+
PStoreLogger suite collecting scoped file-ID coverage with Hash payloads,
|
|
217
|
+
plus agent exercises that preserve excluded-input scope and failing assertions.
|
|
218
|
+
|
|
219
|
+
The [finite-string results](docs/finite-strings-results.md) extend that pilot to
|
|
220
|
+
the documented `"user_123"` case, showing its missing obligation and a separate
|
|
221
|
+
requirement-backed assertion that exercises it.
|
|
222
|
+
|
|
223
|
+
## Pairwise finite input combinations
|
|
224
|
+
|
|
225
|
+
`input-basics-v1` remains the default scalar criterion. When a requirement
|
|
226
|
+
defines a finite joint scenario, declare its inputs in one `combinations`
|
|
227
|
+
group and explicitly choose `all_allowed: true` or list requirement-backed
|
|
228
|
+
`forbid` clauses. The `pairwise-finite-v1` criterion plans each pair of values
|
|
229
|
+
that occurs together in at least one allowed full assignment. It does not
|
|
230
|
+
assert expected outputs or establish that inputs are behaviorally independent.
|
|
231
|
+
Each analyzed input in a runtime call must still be valid before that call can
|
|
232
|
+
earn pairwise evidence.
|
|
233
|
+
|
|
234
|
+
The runnable [pairwise example](examples/pairwise/README.md) documents two
|
|
235
|
+
independent DeliveryOptions requirements and shows the faulty interaction
|
|
236
|
+
implementation passing three calls while missing the `expedited: true`,
|
|
237
|
+
`notify: true` pair. A requirement-backed assertion exposes the defect; a
|
|
238
|
+
separate corrected implementation then covers all four pair obligations. The
|
|
239
|
+
example also shows a constrained plan with three required and one excluded
|
|
240
|
+
pair, and a three-boolean plan with 12 pair obligations versus eight exhaustive
|
|
241
|
+
tuples.
|
|
242
|
+
|
|
243
|
+
Use `specproof suggest` when an intact pairwise coverage report has missing
|
|
244
|
+
pairs. It replans the current declarations, checks that the report matches
|
|
245
|
+
their exact typed obligations and recorded scope, then prints allowed full
|
|
246
|
+
group assignments that cover the missing pairs. Suggestions are candidates
|
|
247
|
+
for new tests: they are not executed, carry no expected results, and do not
|
|
248
|
+
fingerprint implementation or test files. Rerun the intended suite after
|
|
249
|
+
source or test changes. The [suggestion example](examples/pairwise_suggestions/README.md)
|
|
250
|
+
shows a three-of-four DeliveryOptions baseline, the `11` candidate, a separate
|
|
251
|
+
requirement-backed failing assertion, and a fresh corrected run. It also shows
|
|
252
|
+
one suggested three-boolean assignment covering several missing pairs, with
|
|
253
|
+
the other method inputs left for the caller to supply when outside the group.
|
|
254
|
+
|
|
255
|
+
## Pairwise coverage over integer partitions
|
|
256
|
+
|
|
257
|
+
`pairwise-partitions-v1` pairs explicitly complete and non-overlapping integer
|
|
258
|
+
categories with other finite inputs. It supports `all_allowed: true` or
|
|
259
|
+
requirement-backed `forbid` assignments that select whole integer
|
|
260
|
+
partitions with `in_partition(:name)` and exact finite values. Each clause
|
|
261
|
+
forbids matching full assignments; required and excluded pairs are derived
|
|
262
|
+
from whether an allowed full completion exists. An excluded pair can therefore
|
|
263
|
+
be impossible indirectly through a restriction on a third input. Integer
|
|
264
|
+
restrictions must select a whole partition; raw integer values and partial
|
|
265
|
+
range predicates are not supported. Finite inputs continue to use exact values.
|
|
266
|
+
Category coverage is not boundary-value coverage and does not assert equivalent
|
|
267
|
+
behavior throughout a category. The runnable [unrestricted partition
|
|
268
|
+
example](examples/pairwise_partitions/README.md) shows a missing minor/`"de"`
|
|
269
|
+
category pair. The [constrained example](examples/constrained_partition_pairs/README.md)
|
|
270
|
+
shows requirement-backed restrictions, an excluded category pair, and allowed
|
|
271
|
+
assignment suggestions. See the [initial partition results](docs/pairwise-partitions-results.md)
|
|
272
|
+
and [constrained partition results](docs/constrained-partition-pairs-results.md)
|
|
273
|
+
for verification evidence.
|
|
274
|
+
|
|
275
|
+
## Development
|
|
276
|
+
|
|
277
|
+
Run the test and lint tasks with `bundle exec rake`. The default task runs Minitest and RuboCop. The CI matrix covers Ruby 3.3, 3.4, and 4.0.
|
|
278
|
+
|
|
279
|
+
To reproduce the small baseline and instrumented timing comparison, see [docs/benchmark.md](docs/benchmark.md).
|
|
280
|
+
|
|
281
|
+
## License
|
|
282
|
+
|
|
283
|
+
SpecProof is released under the MIT License. See [LICENSE.txt](LICENSE.txt).
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Agent skills
|
|
2
|
+
|
|
3
|
+
SpecProof ships two optional Agent Skills alongside the Ruby gem. Export them
|
|
4
|
+
into an application's own skill directory with an explicit destination:
|
|
5
|
+
|
|
6
|
+
```sh
|
|
7
|
+
specproof skills export --to .agents/skills
|
|
8
|
+
```
|
|
9
|
+
|
|
10
|
+
The command copies `specproof-specify/` and `specproof-test-gaps/` into that
|
|
11
|
+
directory. Repeating the command leaves identical files alone. If a packaged
|
|
12
|
+
file would overwrite different content, export reports a conflict before
|
|
13
|
+
writing any skill files. Other files in the destination, including local files
|
|
14
|
+
inside either skill folder, are kept. An interrupted filesystem operation can
|
|
15
|
+
still leave a partial copy; rerun the command after correcting the reported
|
|
16
|
+
filesystem issue.
|
|
17
|
+
|
|
18
|
+
The skills also document the opt-in RBS workflow: inspect explicitly selected
|
|
19
|
+
signatures, reuse supported finite input types, and add only requirement-backed
|
|
20
|
+
behavioral refinements. They support deliberate partial input scope through
|
|
21
|
+
one named, reasoned exclusion for each out-of-scope parameter; default scope
|
|
22
|
+
remains every named parameter, with at least one analyzed input required. The
|
|
23
|
+
gap-review skill requires fresh runtime evidence and compares the report's
|
|
24
|
+
analyzed/excluded input scope as well as its digest, integrity, test results,
|
|
25
|
+
and actual test command. A complete percentage covers analyzed input
|
|
26
|
+
obligations only, and human reports disclose exclusions. RBS supplies no output
|
|
27
|
+
oracle. The runnable [input selection example](../examples/input_selection/README.md)
|
|
28
|
+
shows a Hash payload excluded while a boolean option remains analyzed. Earlier evaluation results
|
|
29
|
+
in this document's linked record describe the workflows and artifacts named
|
|
30
|
+
there; they do not establish agent performance for this newer RBS workflow.
|
|
31
|
+
|
|
32
|
+
For Codex repository discovery, place the exported folders under
|
|
33
|
+
`.agents/skills` in the repository. Codex searches that directory from the
|
|
34
|
+
current working directory through its repository ancestors. Open a new Codex
|
|
35
|
+
session in the project and ask for one of the named skills to confirm it is
|
|
36
|
+
available. This discovery check is separate from the export result: export
|
|
37
|
+
confirms that files were copied, while the client session confirms that the
|
|
38
|
+
client found and loaded them. See the [Codex skills guide](https://learn.chatgpt.com/docs/build-skills)
|
|
39
|
+
and the [Agent Skills format](https://agentskills.io/specification).
|
|
40
|
+
|
|
41
|
+
Keep skill use opt-in. The exporter writes only to the path supplied by the
|
|
42
|
+
caller; it does not select a home directory, edit global configuration, alter
|
|
43
|
+
`AGENTS.md`, or access the network. The packaged skills and their local
|
|
44
|
+
references are self-contained, so the exported folder can be reviewed and
|
|
45
|
+
removed as ordinary project files.
|
|
46
|
+
|
|
47
|
+
The current RBS workflow was exercised in six fresh, isolated native Luna
|
|
48
|
+
tasks using skills exported from an installed gem. See the [RBS evaluation
|
|
49
|
+
record](evaluations/rbs-agent-skills-v0.1.md) for actual outcomes, retained
|
|
50
|
+
commands and reports, and the limits of these guided scenarios.
|
|
51
|
+
|
|
52
|
+
The export and one specification workflow were exercised in Codex CLI
|
|
53
|
+
`0.160.0` with `gpt-6-luna`, using `.agents/skills` in an isolated project and
|
|
54
|
+
the installed gem rather than this checkout. A separate fresh formatting task
|
|
55
|
+
did not load either SpecProof skill or create specifications. See the
|
|
56
|
+
[evaluation record](evaluations/agent-skills-v0.1.md) for retained artifacts,
|
|
57
|
+
baseline observations, repeated attempts, and limitations. These checks are
|
|
58
|
+
bounded observations of agent behavior, not a guarantee for every agent or
|
|
59
|
+
client. Other clients can receive the same exported folders through their
|
|
60
|
+
documented manual skill installation workflow; they have not been tested here.
|
data/docs/benchmark.md
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Small-suite collection timing
|
|
2
|
+
|
|
3
|
+
This benchmark compares equivalent Minitest subprocesses that call `AgePolicy#adult?` three or six times. The test methods and assertions are generated once for each suite size and run unchanged in baseline and instrumented modes. The instrumented mode installs the optional adapter and verifies that report hit totals match the direct call counter.
|
|
4
|
+
|
|
5
|
+
Run it from `gems/specproof` with a Minitest 5.27.x installation whose activated gem version and `Minitest::VERSION` both identify the supported release:
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
ruby script/benchmark_age_policy.rb
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
The script runs five repetitions per mode by default. Pass a positive iteration count as the first argument for a shorter smoke run.
|
|
12
|
+
|
|
13
|
+
This recorded run used Ruby 4.0.1 on arm64 macOS, Minitest gem `5.27.0`, runtime `Minitest::VERSION` `5.27.0`, and SpecProof `0.1.0`. The available RubyGems 5.27.0 artifact on this machine had an embedded runtime version of `5.26.2`, so the measurement used an unmodified local rebuild of the `v5.27.0` source tag to meet the adapter's package/runtime version check. No Minitest code was patched for the run.
|
|
14
|
+
|
|
15
|
+
Each timing includes Ruby process startup, Bundler setup, Minitest startup, the test suite, and report writing. All five repetitions made exactly the number of calls shown; instrumented hit totals matched those counts.
|
|
16
|
+
|
|
17
|
+
| Suite size | Mode | Calls per run | Wall time, repetitions 1–5 (seconds) |
|
|
18
|
+
| --- | --- | ---: | --- |
|
|
19
|
+
| 3 tests | Baseline | 3 | 0.229863, 0.231492, 0.218486, 0.235095, 0.229194 |
|
|
20
|
+
| 3 tests | Instrumented | 3 | 0.262548, 0.254388, 0.261317, 0.280937, 0.239743 |
|
|
21
|
+
| 6 tests | Baseline | 6 | 0.236327, 0.230261, 0.241569, 0.230310, 0.220956 |
|
|
22
|
+
| 6 tests | Instrumented | 6 | 0.241652, 0.250458, 0.237547, 0.246748, 0.239821 |
|
|
23
|
+
|
|
24
|
+
These measurements describe only this tiny fixture and subprocess setup. Process startup dominates much of the elapsed time, so the results do not estimate overhead for a larger application suite.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Constrained partition pairs: verification
|
|
2
|
+
|
|
3
|
+
The [example](../examples/constrained_partition_pairs/README.md) declares
|
|
4
|
+
`forbid age: in_partition(:minor), account: "business"`. Filtering complete
|
|
5
|
+
category assignments leaves six allowed tuples, eleven required pairs, and
|
|
6
|
+
one excluded pair. Its three baseline calls cover nine pairs. Two suggested
|
|
7
|
+
calls, with independent requirement-based assertions, bring coverage to eleven
|
|
8
|
+
of eleven.
|
|
9
|
+
|
|
10
|
+
Luna subagents implemented the core, integration tests and documentation.
|
|
11
|
+
Separate specification and code reviews found no remaining code issues.
|
|
12
|
+
Review corrected missing restriction provenance and clarified historical
|
|
13
|
+
documentation. Native argument validation retains exact finite types,
|
|
14
|
+
including `nil` and `false`; integer restrictions require explicit selectors.
|
|
15
|
+
|
|
16
|
+
Verification on CRuby 3.3.6, 3.4.7 and 4.0.1 passed on each version:
|
|
17
|
+
|
|
18
|
+
| Suite | Tests | Assertions |
|
|
19
|
+
| --- | ---: | ---: |
|
|
20
|
+
| Core | 303 | 1,579 |
|
|
21
|
+
| Optional RBS | 55 | 509 |
|
|
22
|
+
|
|
23
|
+
RBS signature validation, RuboCop (144 files), syntax checks, packaged skill
|
|
24
|
+
checks and gem building passed. A final reuse of the memoized allowed
|
|
25
|
+
assignments was checked with 50 focused tests and 240 assertions on Ruby 4.
|
|
26
|
+
|
|
27
|
+
A separate exhaustive oracle checked all 256 forbidden subsets of an
|
|
28
|
+
eight-tuple category space: two integer inputs sharing partition names and
|
|
29
|
+
a finite input containing `nil` and `false`. Allowed assignments, required
|
|
30
|
+
pairs, excluded pairs and acceptance of 8,192 native tuples matched independent
|
|
31
|
+
expected results, including the entirely forbidden space.
|
|
32
|
+
|
|
33
|
+
A Luna agent followed the updated test-gap skill against a scratch copy of
|
|
34
|
+
the example. It obtained the two legal suggestions from the 9/11 baseline,
|
|
35
|
+
used the separate requirements to justify expected outputs, and verified the
|
|
36
|
+
complete run: two tests, five assertions and 11/11 feasible pairs exercised.
|
|
37
|
+
The excluded minor/business pair remained outside the denominator. This was
|
|
38
|
+
a source-checkout guidance exercise, not a fresh installed-gem evaluation.
|
|
39
|
+
|
|
40
|
+
The implementation reuses existing coverage accounting, report validation and
|
|
41
|
+
suggestion selection. No dependencies were added. Restrictions select whole
|
|
42
|
+
integer partitions or exact finite values; arbitrary predicates and partial
|
|
43
|
+
integer restrictions remain unsupported. Existing enumeration limits apply
|
|
44
|
+
before restrictions reduce the space.
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# Finite string values
|
|
2
|
+
|
|
3
|
+
Implemented on `implement/finite-strings`, based on `main` at `f63dcaa`.
|
|
4
|
+
Core, tests, documentation and skills are committed as `8195cae`.
|
|
5
|
+
|
|
6
|
+
```ruby
|
|
7
|
+
SpecProof.define "PStoreLogger#save" do
|
|
8
|
+
input :file_id, values: [nil, 1, "user_123"]
|
|
9
|
+
exclude_input :data, reason: "Payload contents are outside these file-ID scenarios"
|
|
10
|
+
end
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Finite domains and partitions now accept exact valid UTF-8 or US-ASCII String
|
|
14
|
+
values, including empty strings and Unicode text. US-ASCII is stored as UTF-8;
|
|
15
|
+
there is no trimming, case folding or Unicode normalization. Same-spelling
|
|
16
|
+
strings and symbols are distinct. Declaration values and runtime observations
|
|
17
|
+
are copied and frozen without changing application-owned values. Collection
|
|
18
|
+
uses the value at method entry even when the method later mutates it.
|
|
19
|
+
|
|
20
|
+
JSON adds the typed value `{ "kind": "string", "value": "user_123" }` under
|
|
21
|
+
schema 1 and the existing `input-basics-v1` criterion. Existing nonstring IDs
|
|
22
|
+
and digests remain unchanged. Consumers must handle the new value kind.
|
|
23
|
+
|
|
24
|
+
## Changes and simplifications
|
|
25
|
+
|
|
26
|
+
- `lib/specproof/domains.rb` centralizes supported-value validation and text
|
|
27
|
+
snapshots. The existing finite-domain representation, checker and planner
|
|
28
|
+
handle strings without a parallel string-domain hierarchy.
|
|
29
|
+
- Model, coverage, collector, report and source-neutral type constraints reuse
|
|
30
|
+
those rules; `sig/specproof.rbs` describes the expanded public value union.
|
|
31
|
+
- Focused tests cover encodings, mutation, mixed symbol/string values,
|
|
32
|
+
partitions, typed reports, runtime integrity and legacy identities.
|
|
33
|
+
- README, report documentation and both packaged agent skills explain exact
|
|
34
|
+
scenario scope and preserve independent assertion requirements.
|
|
35
|
+
|
|
36
|
+
No new dependency, CLI option, adequacy criterion or RBS extraction feature was
|
|
37
|
+
added. RBS String types and string literals remain unsupported atomically,
|
|
38
|
+
including mixed unions.
|
|
39
|
+
|
|
40
|
+
## Verification
|
|
41
|
+
|
|
42
|
+
Each CRuby version (3.3.6, 3.4.7, 4.0.1) passes **178 core tests / 885
|
|
43
|
+
assertions** and **46 optional RBS tests / 433 assertions**, with zero failures,
|
|
44
|
+
errors or skips. RBS signatures validate on all three. The locked default
|
|
45
|
+
Ruby 4 bundle passes `rake`, including RuboCop's 98-file check. `bin/check`
|
|
46
|
+
passes syntax, package and skill checks (2 tests / 76 assertions).
|
|
47
|
+
|
|
48
|
+
Independent Luna specification and quality reviews approved the change.
|
|
49
|
+
[Matrix records](pilots/finite-strings/matrix/) retain actual commands, outputs,
|
|
50
|
+
statuses and library hashes. [Agent exercises](evaluations/finite-strings-skills.md)
|
|
51
|
+
retain an old-guidance baseline and actual installed-package application runs.
|
|
52
|
+
|
|
53
|
+
## Real-project value
|
|
54
|
+
|
|
55
|
+
The [PStoreLogger pilot](pilots/finite-strings/README.md) preserves its original
|
|
56
|
+
five source/test files. Its unchanged suite passes 7 tests / 23 assertions but
|
|
57
|
+
exercises only 2/3 declared values: `"user_123"` is missing. Strict missing mode
|
|
58
|
+
correctly exits 1 while the report preserves `tests_passed: true`.
|
|
59
|
+
|
|
60
|
+
A separate test uses its README's custom-file-ID and collision rules to check
|
|
61
|
+
`user_123.pstore`, `user_123_1.pstore`, and the stored payloads. The resulting
|
|
62
|
+
suite passes 8 tests / 27 assertions and exercises 3/3 with intact collection.
|
|
63
|
+
The original application and tests are unchanged.
|
|
64
|
+
An independent evidence review confirmed the five source-file hashes, all 17
|
|
65
|
+
runtime library hashes, matching scope/digest/obligation IDs, and distinct
|
|
66
|
+
baseline/final run IDs.
|
|
67
|
+
|
|
68
|
+
## Limits
|
|
69
|
+
|
|
70
|
+
This covers explicitly enumerated scenarios, not the entire String domain or
|
|
71
|
+
whole-method correctness. Other encodings, invalid text, String subclasses,
|
|
72
|
+
patterns, length constraints, RBS strings, outputs and interaction coverage are
|
|
73
|
+
outside this milestone. Outside-domain witness count remains bounded, but a
|
|
74
|
+
retained string witness contains the full text, so there is no byte-size bound.
|
|
75
|
+
The agent exercises are small guided tasks, not a general reliability estimate.
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# First usable release implementation
|
|
2
|
+
|
|
3
|
+
The [approved plan](../../../docs/superpowers/plans/2026-10-02-first-usable-release.md) is implemented inside this gem on branch `implement/first-usable-release` and recorded in the implementation commit containing this document. The gem remains unpublished. Verification below records local execution on 2026-10-02; GitHub Actions has been configured but has not been run remotely.
|
|
4
|
+
|
|
5
|
+
## Scope and coordination
|
|
6
|
+
|
|
7
|
+
- Core: explicit scalar input declarations, exact partition checks, `input-basics-v1`, deterministic human/JSON reports.
|
|
8
|
+
- Runtime: bounded streaming input evidence for selected serial Ruby method calls; optional Minitest integration for >= 5.26.2 and < 5.28.0 with matching package/runtime minor versions.
|
|
9
|
+
- Agent interface: two packaged skills, explicit offline export, isolated behavioral evaluations.
|
|
10
|
+
- Existing development RuboCop/Rake tooling is retained; no new runtime dependencies.
|
|
11
|
+
- Public namespace: `SpecProof`; preserve scaffold compatibility through `Specproof`.
|
|
12
|
+
|
|
13
|
+
## Delivered work
|
|
14
|
+
|
|
15
|
+
- Tasks 1–4: immutable scalar domains, trusted Ruby declaration loading, exact partition checks, shared `input-basics-v1` planning, human and JSON CLI reports.
|
|
16
|
+
- Task 5: bounded TracePoint collection for directly owned public instance methods with named positional/keyword inputs, including optional parameters. Unsupported or duplicate resolved bindings, redefinition, and changed constant paths cannot produce complete evidence.
|
|
17
|
+
- Tasks 6–7: streaming counters, whole-tuple input validation, positive versus unknown evidence, optional serial Minitest adapter with fresh run IDs and atomic reports, optional strict missing-obligation failure.
|
|
18
|
+
- Task 8: [executable boundary workflow](../examples/age_policy/README.md), shipping and gap examples, [report contract](report-format.md), packaging, API signatures, compatibility checks, and [timing measurements](benchmark.md).
|
|
19
|
+
- Task 9: two packaged skills, conflict-preserving explicit export, and [retained agent evaluations](evaluations/agent-skills-v0.1.md).
|
|
20
|
+
|
|
21
|
+
Luna subagents implemented independent slices and reviewed specification conformance, runtime correctness, code quality, and evaluation outcomes. The implementation uses one shared obligation model rather than duplicating boundary logic in runtime coverage.
|
|
22
|
+
|
|
23
|
+
## Acceptance evidence
|
|
24
|
+
|
|
25
|
+
| Plan criteria | Evidence |
|
|
26
|
+
| --- | --- |
|
|
27
|
+
| A01–A04 | Loader/checker/planner tests cover unresolved and autoloaded target names, exact gap/overlap witnesses, large domains without enumeration, small-domain reference comparisons, and finite-value identity. |
|
|
28
|
+
| A05–A06, A16 | Planner/coverage share obligations and IDs. CLI/report tests cover deterministic schema-versioned JSON, typed values, provenance, errors, and exit codes. |
|
|
29
|
+
| A07–A11 | Collector/coverage/subprocess tests cover scope, method identity, invalid tuples, bounded retained evidence, interruption, changed definitions/constants, original failures/exceptions, strict mode, and unknown evidence. |
|
|
30
|
+
| A12 | Shipped runner and workflow tests show passing tests with an age-18 gap, a meaningful assertion failing against the boundary bug, and the corrected implementation exercising all six unchanged obligations. |
|
|
31
|
+
| A13–A14 | Local suites pass on CRuby 3.3.6, 3.4.7, and 4.0.1. Actual gem installation and CLI export work outside the checkout. Remote matrix execution remains unobserved. |
|
|
32
|
+
| A15 | Packaged skill/reference validation, installed-gem export, idempotence, conflict preservation, symlink rejection, and paths containing spaces are tested. |
|
|
33
|
+
| A17–A19 | All six scenarios passed two fresh attempts with the final skills (12 accepted outcomes), independently audited. Baselines, discarded attempts, final files, hashes, and real Codex client positive/negative discovery runs are retained. Small fixtures do not establish general agent reliability or a causal improvement rate. |
|
|
34
|
+
|
|
35
|
+
## First implementation verification (commit fc876dd)
|
|
36
|
+
|
|
37
|
+
- `bin/test`: **111 runs, 562 assertions, zero failures, errors, or skips**, separately on Ruby 3.3.6, 3.4.7, and 4.0.1.
|
|
38
|
+
- Default `bundle exec rake` on Ruby 4.0.1: full suite and RuboCop pass; **81 files**, zero offenses.
|
|
39
|
+
- `bin/check` from outside the checkout: syntax checks, skill/package tests (**2 runs, 66 assertions**), and gem build with its library present pass.
|
|
40
|
+
- RBS 4.2.0 `rbs -I sig validate` passes. This validates declarations, not implementation type checking.
|
|
41
|
+
- Five benchmark repetitions per mode preserve unchanged test bodies and exact 3/6 call counts; report hits match. No real-application overhead claim is made.
|
|
42
|
+
|
|
43
|
+
### Minitest validation environment
|
|
44
|
+
|
|
45
|
+
The locally available gem labeled Minitest `5.27.0` contained runtime version `5.26.2`. The adapter correctly rejects that mismatch. Validation used unmodified library files from the official `v5.27.0` source tag, packaged into an isolated temporary GEM_HOME. Neither global gems nor the runtime version constant were patched. Source archive SHA-256: `766c98a1d07df624320f69a013bd11994506eec930ce70964f87c7d2ee8a97c7`.
|
|
46
|
+
|
|
47
|
+
That initial verification required both activated package and runtime versions to identify Minitest 5.27.x. The subsequent [compatibility change](minitest-compatibility.md) supports correctly labeled published 5.26.2 and removes the need for a source rebuild in the development bundle. Mismatched package/runtime minor versions remain rejected. Offline checking/planning and skill export do not need Minitest.
|
|
48
|
+
|
|
49
|
+
## Installation and real-project pilot follow-up
|
|
50
|
+
|
|
51
|
+
The normal development bundle now resolves published Minitest 5.26.2. Full suites pass with **115 tests and 573 assertions** on Ruby 3.3.6, 3.4.7, and 4.0.1. Ruby 3.3/3.4 checks used standard local Bundler installation from the checksum-verified published package, with unrelated ambient Ruby 4 gem paths cleared. No custom dependency source rebuild or version-constant patch was needed.
|
|
52
|
+
|
|
53
|
+
The [real-project pilot](real-project-pilot.md) applied the updated exported skills to QualityGate's existing configuration tests. One default-key input gap was closed, moving scoped exercise from 7/8 to 8/8 with unchanged IDs/digest and a fresh report. All 143 copied source files remain unchanged. Two other real candidate methods were rejected because of optional keyword parameters and class-method syntax; three successful integrations were not demonstrated.
|
|
54
|
+
|
|
55
|
+
Five baseline/instrumented timing pairs preserve the original 45-test, 342-assertion focused suite. The measured median subprocess difference was about 38 ms (12.5%); this is not a general application overhead claim. Source snapshots, package checksum, exact skills, test additions, reports, raw timings, and rejected probes are retained.
|
|
56
|
+
|
|
57
|
+
## Limits and release follow-up
|
|
58
|
+
|
|
59
|
+
The [optional-argument follow-up](optional-arguments-results.md) records support for effective scalar values after Ruby applies defaults and a scoped PStoreLogger adoption pilot. Reports explicitly state that argument presence is not observed. Earlier pilot rejections and archived skill bytes describe their original implementation cuts.
|
|
60
|
+
|
|
61
|
+
This release checks explicit scalar input specifications and measures exercised input candidates; it does not prove implementation behavior or assertion quality. Declarations are trusted executable Ruby. Collection requires stable constant bindings and selected definitions in a serial same-process/thread window. Transient constant changes restored before final inspection are not detected.
|
|
62
|
+
|
|
63
|
+
Sorbet/RBS importers, output contracts, inferred partitions, state machines, combinatorial criteria, SMT, parallel collection, and MCP remain deferred. API signatures do not constitute an RBS specification importer.
|
|
64
|
+
|
|
65
|
+
The initial model evaluations preceded the final additive constant-binding guards and example-runner convenience changes; deterministic tests and review cover those changes. Their original skill bytes remain archived. Later live skill setup instructions reflect Minitest 5.26.2 compatibility and were freshly used in the focused real-project pilot. Remote CI, broader project adoption, and publishing remain release follow-up work. No gem was published and no branch was pushed.
|