lawspec 0.22.0 → 0.23.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +258 -2
- package/README.md +3 -3
- package/bin/lawspec.mjs +81 -12
- package/build.json +165 -101
- package/compatibility.json +8 -0
- package/core.wasm +2 -2
- package/doctor.mjs +22 -1
- package/examples/native-payments/csharp/lawspec.json +51 -0
- package/examples/native-payments/csharp/src/Domain/Payments.cs +19 -0
- package/examples/native-payments/csharp/tests/Native/PaymentGenerators.cs +14 -0
- package/examples/native-payments/fsharp/lawspec.json +167 -0
- package/examples/native-payments/fsharp/src/Native/Types/PaymentsDomain.fs +23 -0
- package/examples/native-payments/fsharp/tests/Native/PaymentGenerators.fs +11 -0
- package/examples/specs/collections.lawspec +44 -0
- package/examples/specs/composition.lawspec +110 -0
- package/examples/specs/contracts.lawspec +124 -0
- package/examples/specs/generation.lawspec +11 -0
- package/examples/specs/records.lawspec +191 -0
- package/examples/specs/records_inner.lawspec +10 -0
- package/examples/specs/sequences.lawspec +86 -0
- package/index.d.ts +36 -1
- package/package.json +2 -2
- package/templates.mjs +45 -3
- package/test-command.mjs +28 -3
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,261 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.23.0] - 2026-10-12
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- F# and C# targets (`--target fsharp`, `--target csharp`) on .NET 10, with
|
|
14
|
+
xunit.v3 on Microsoft.Testing.Platform and `dotnet test`. F# gets a runtime
|
|
15
|
+
written for it: records and unions in LawSpec's order, exact integers,
|
|
16
|
+
persistent collections, `Result` for typed failures, abilities as
|
|
17
|
+
interfaces, `Task` for async, typed actors and supervisors, single-use
|
|
18
|
+
session ends, mailboxes, a virtual clock, remote evaluation and the secure
|
|
19
|
+
network on the shared wire format, and post-quantum crypto from .NET with
|
|
20
|
+
BouncyCastle where .NET lacks it. Properties run on FsCheck 3.4.0, driven
|
|
21
|
+
from the xUnit facts: LawSpec's refinement-directed generators are FsCheck
|
|
22
|
+
generators of shrink trees, so shrinks stay inside refinements, and FsCheck
|
|
23
|
+
runs, sizes, shrinks and replays (`LAWSPEC_SEED`) the cases; generator
|
|
24
|
+
bindings return FsCheck Arbitraries. Models, linearizability, scenarios and
|
|
25
|
+
the whole harness run around it. C# runs over it: the program's model is a generated F#
|
|
26
|
+
project, and adapters, entries, the embedding API and tests are C#, with
|
|
27
|
+
`LawSpec.CSharp` as the compatibility layer. Definitions whose values all
|
|
28
|
+
have native types run typed, over the program's own F# types with checked
|
|
29
|
+
.NET integers, and each spec handler is a native class with `Result`
|
|
30
|
+
try-forms and its state to read or restore (typed handlers keep it native).
|
|
31
|
+
Native bindings, `lawspec test`, `lawspec doctor`, the native payments
|
|
32
|
+
example and `lawspec-dev bench kafka` cover both targets. Every acceptance
|
|
33
|
+
suite passes on both, with its mutants rejected, and the tutorial lessons
|
|
34
|
+
have F# and C# implementations. Text in a Set or a KeyVal orders by code
|
|
35
|
+
point on .NET as everywhere: F#'s own string order (by UTF-16 unit) is
|
|
36
|
+
never observed, as items, keys, entries or a comparison. See the F# and C#
|
|
37
|
+
setup guides.
|
|
38
|
+
- The collections suite checks that text keys and items order by code point,
|
|
39
|
+
with a character outside the Basic Multilingual Plane beside a fullwidth
|
|
40
|
+
letter, on every target.
|
|
41
|
+
- `lawspec evidence` says which tested laws ran against stand-ins. It reads
|
|
42
|
+
an environment attestation, a versioned JSON document any tool may write
|
|
43
|
+
for one run: the environment, the run's id and times, and for each
|
|
44
|
+
dependency its endpoint, whether it was reached, and whether it was the
|
|
45
|
+
real dependency or a stand-in (with the stand-in's identity, version,
|
|
46
|
+
declared conformance level and optional history digest). A dependency is
|
|
47
|
+
matched by name to an ability whose production handler a law runs under
|
|
48
|
+
(`unit::Name` or `unit::ability::Name`) or to an adapter or bound native
|
|
49
|
+
function the law calls. Each such law whose tests ran gets a line, and
|
|
50
|
+
`--json` the fields `environment`, `environmentRun` and `against`; an
|
|
51
|
+
`ENVIRONMENT` section lists the dependencies. It is reporting only: no
|
|
52
|
+
status changes and no law means anything different.
|
|
53
|
+
- `lawspec evidence --attestation <file>` and `lawspec test --attestation
|
|
54
|
+
<file>`, or `attestation` in `lawspec.json`, name the attestation. `lawspec
|
|
55
|
+
test` keeps it with the run's results in `.lawspec/results`, so a later
|
|
56
|
+
`lawspec evidence` shows the attested run each law last passed in; a law it
|
|
57
|
+
touches runs again when its dependencies are wired differently.
|
|
58
|
+
- The compiler API takes an `attestation` in any request: the result's
|
|
59
|
+
`evidence` carries the fields above and `attestation` lists each dependency
|
|
60
|
+
with the obligations it touched. A malformed attestation, or an unknown
|
|
61
|
+
format or version, is a diagnostic with code `attestation`.
|
|
62
|
+
- Checked definitions and stateful spec handlers run as typed native code on
|
|
63
|
+
all eight targets wherever every value they touch has a typed form
|
|
64
|
+
(Bool, Unit, the bounded integers, Integer, Text, Bytes, List, Maybe,
|
|
65
|
+
Either, KeyVal, Set and data built from them); anything else runs on the
|
|
66
|
+
dynamic values, as before. A definition that performs an operation of
|
|
67
|
+
another ability, or calls one that does, runs typed too: it takes the
|
|
68
|
+
handler context first and performs through the handler in scope, so
|
|
69
|
+
LawTwin's Kafka client handlers (`kafkaProducer`, `kafkaConsumer`,
|
|
70
|
+
`kafkaTransactions`) run typed, and 829 of the contract's 865 definitions
|
|
71
|
+
do (only the IntSize and UIntPtr helpers stay dynamic). Python represents
|
|
72
|
+
data as tuples, lists as shared cons cells that know their length and
|
|
73
|
+
KeyVal and Set as persistent AVL trees, and turns self tail calls into
|
|
74
|
+
loops; JavaScript and TypeScript use classes with generated TypeScript
|
|
75
|
+
types, and a number for every integer that fits a safe integer, a BigInt
|
|
76
|
+
only beyond; Haskell uses strict data types, `Data.Map.Strict` and
|
|
77
|
+
`Data.Set`, and threads a handler's state through an `MVar`. Definitions that `fail with E`, `raise` and `prelude.attempt`, field
|
|
78
|
+
access and record update, bounded arithmetic, recursion on integers and
|
|
79
|
+
the sequence helpers run typed, and so does a spec handler of several
|
|
80
|
+
abilities with a computed start; a typed failure is the same panic or
|
|
81
|
+
exception the dynamic code raises, so each catches the other's.
|
|
82
|
+
`lawspec-dev bench <name> --typed` lists what stays dynamic, and why. Lists are
|
|
83
|
+
persistent cons lists that share their tails, KeyVal and Set persistent AVL
|
|
84
|
+
trees in the portable order, Integer stays in a machine word until it
|
|
85
|
+
outgrows it, data with fields is shared by reference, and a definition's
|
|
86
|
+
self tail calls become a loop. A handler's state is never copied: an
|
|
87
|
+
operation costs what it changes. Values convert directly between the typed
|
|
88
|
+
forms and each target's native types at a handler's interface, and through
|
|
89
|
+
the dynamic value where typed and dynamic code meet.
|
|
90
|
+
- Embedding entry points: a stateful spec handler can be made outside any law
|
|
91
|
+
and called from native code, `Handler::start()` in Rust,
|
|
92
|
+
`Start<Handler>()` in Go, the no-argument constructor in Java and Kotlin,
|
|
93
|
+
`Handler()` in Python, `new Handler()` in JavaScript and TypeScript and
|
|
94
|
+
the handler's function over a symbol context in Haskell. Its operations apply one at a time, so it can
|
|
95
|
+
be shared between threads. Each operation that fails with `E` also has a
|
|
96
|
+
form that returns the failure as a value: `try_op` gives `Result<T, E>` in
|
|
97
|
+
Rust, `TryOp` gives `(T, error)` with a `*LawSpecRaised[E]` in Go, `tryOp`
|
|
98
|
+
an `Outcome<T, E>` in Java and Kotlin, `Right` or `Left` in Python,
|
|
99
|
+
JavaScript and TypeScript, and `Either E T` in Haskell. A handler's state
|
|
100
|
+
can be read and replaced (`state`, `set_state` and their typed forms;
|
|
101
|
+
`<h>Snapshot` and `<h>Restore` in Haskell), and the state's type is
|
|
102
|
+
public. The new guide "Embed a generated model" shows how.
|
|
103
|
+
- `lawspec-dev bench <name>` runs a benchmark of generated code on all eight
|
|
104
|
+
targets. `bench/kafka` drives LawTwin's Kafka reference model (a
|
|
105
|
+
copy under its MIT licence, re-expressed with failing operations, one
|
|
106
|
+
handler for three abilities, fields and bounded arithmetic) through its
|
|
107
|
+
`cluster` handler: a million produces over 100 partitions, a consumer
|
|
108
|
+
group of ten fetching and committing with a rebalance, and ten thousand
|
|
109
|
+
transactions. Every definition of the model and the handler run typed.
|
|
110
|
+
It runs the whole workload at 0.53 (Rust), 1.03 (Go), 0.78 (Java) and 0.93
|
|
111
|
+
(Kotlin) million operations a second, holding about 190 to 270 MiB for the
|
|
112
|
+
million records, where 0.21.0's generated Rust managed 9 operations a
|
|
113
|
+
second, and 1.5 GiB, at two thousand. Python runs it at 118 thousand
|
|
114
|
+
operations a second (it ran 1,100 on the dynamic values, at twenty
|
|
115
|
+
thousand), JavaScript at 942 thousand and TypeScript at 964 thousand
|
|
116
|
+
(7,300 before) and Haskell at 1.83 million (57 before, at two thousand).
|
|
117
|
+
`lawspec-dev bench kafka --deterministic` generates the contract on each
|
|
118
|
+
target with and without a cache, cold and warm, and by `lawspec-core`, and
|
|
119
|
+
fails unless every run gives the same bytes.
|
|
120
|
+
- Fields: `b.balance` reads a field of a single-constructor value, and
|
|
121
|
+
`b with balance = v, owner = o` copies it with fields replaced.
|
|
122
|
+
- Handlers of several abilities: `handler reference for Broker, Admin with
|
|
123
|
+
state s :: S start e is ... end` answers every operation of each from one
|
|
124
|
+
state. A law that runs it for one of its abilities runs it for the others
|
|
125
|
+
it uses, made once per case.
|
|
126
|
+
- Sets of handlers: `handlers reference is cluster, client end`. Every law of
|
|
127
|
+
an ability a set covers runs under all of its handlers at once,
|
|
128
|
+
`` Broker: law [reference] ``, and so does every law that uses one of its
|
|
129
|
+
abilities.
|
|
130
|
+
- `ability Broker with Cluster is ... end`: native handlers made with a
|
|
131
|
+
resource. Each case acquires one `Cluster` and makes every such native
|
|
132
|
+
handler with it, so native handlers of one case share it.
|
|
133
|
+
- Ability operations may fail: `produce :: Batch -> Int64 fails with
|
|
134
|
+
KafkaError`. Spec handler clauses `raise`; native handlers raise the
|
|
135
|
+
runtime's `Fail`. `prelude.attempt` works in definitions.
|
|
136
|
+
- Handler clauses: a start value may call definitions and the prelude
|
|
137
|
+
(`start prelude.keyValOf []`); a tail `match` or `if` may update the state
|
|
138
|
+
in its branches; `op by f` makes a clause of a definition that gives
|
|
139
|
+
`Pair result next`.
|
|
140
|
+
- Sequences: `++`, `prelude.slice`, `take`, `drop`, `at`, `range`, `octets`,
|
|
141
|
+
`bytesOf`, `utf8`, `textOf` and `join` over `Text`, `Bytes` and `List`.
|
|
142
|
+
- Bounded arithmetic: `prelude.wrapping`, `prelude.saturating` and
|
|
143
|
+
`prelude.bounded` put an integer result into the bounded type the context
|
|
144
|
+
wants.
|
|
145
|
+
- Recursion on integers: a recursive call may move an integer argument
|
|
146
|
+
strictly toward a bound.
|
|
147
|
+
- The totality checker knows what guards say about calls, fields and lengths,
|
|
148
|
+
that a call lies in its result type's range, that lengths are not negative,
|
|
149
|
+
and the bounds of `prelude.rem` and `prelude.quot`.
|
|
150
|
+
- A text refined by a regex, directly or through a checked definition, is
|
|
151
|
+
generated from texts the regex matches instead of filtered from any text,
|
|
152
|
+
also in laws that mix it with record or list inputs.
|
|
153
|
+
- On Python, a unit named like another unit's package (`shop.tax` beside
|
|
154
|
+
`shop.tax.rates`) is the package's `__init__.py`.
|
|
155
|
+
- New acceptance suites `records`, `sequences` and `composition`.
|
|
156
|
+
- Erlang, Elixir and Gleam run the language gaps on their dynamic path:
|
|
157
|
+
handlers of several abilities and sets of handlers, operations that `fail
|
|
158
|
+
with`, natives made with a resource, computed starts and state updated in
|
|
159
|
+
branches, field access and record update, the sequence, `Bytes` and `Text`
|
|
160
|
+
operations, bounded arithmetic, and regex-shaped generation of refined text
|
|
161
|
+
fields. A handler of several abilities gives one native interface per
|
|
162
|
+
ability over its one state; each operation that fails has a try form
|
|
163
|
+
(`{ok, V}` or `{error, E}`, a `Result` in Gleam), and a spec handler with
|
|
164
|
+
state has `<handler>_state` and `<handler>_set_state`. The `records`,
|
|
165
|
+
`sequences`, `composition` and `contracts` suites run on all eleven
|
|
166
|
+
targets, and a `beam-embedding` suite checks the embedding forms.
|
|
167
|
+
- Vetting on Rice's Tax's review page: `make assess` records every pending
|
|
168
|
+
row, `make tax` serves the page on which each comment, page, decision and
|
|
169
|
+
reference is judged beside the code or page it explains, and `make audit`
|
|
170
|
+
says what is left. `lawspec-dev tax` runs the Rice's Tax checkout named by
|
|
171
|
+
`RICESTAX_HOME`; see the new how-to guide, Vet the canonical material.
|
|
172
|
+
|
|
173
|
+
### Changed
|
|
174
|
+
|
|
175
|
+
- `canon.yaml` names the canon version its vetting files were written with,
|
|
176
|
+
0.2.1, which both `make canon` and Rice's Tax require. Code reshaped in
|
|
177
|
+
0.20.1 for canon's Haskell grammar has its original forms back, since canon
|
|
178
|
+
0.2.1 reads them.
|
|
179
|
+
- Definitions that are the same up to their names, such as an import's
|
|
180
|
+
copies, share one generated body: Rust aliases the copies' functions, and
|
|
181
|
+
Java and Go delegate to it. Go packages hold only the bodies they reach,
|
|
182
|
+
and the native entry points of typed definitions convert their values
|
|
183
|
+
directly. LawTwin's Kafka contract generates 23,606 lines of
|
|
184
|
+
`lawspec_definitions.rs` instead of 83,157, 27,439 lines of Java
|
|
185
|
+
definitions instead of 80,444, 27,309 of Kotlin and Java instead of 87,537,
|
|
186
|
+
and 37,900 lines of Go definitions over its four packages instead of
|
|
187
|
+
113,462.
|
|
188
|
+
- Generated Rust builds its data schema once: `lawspec_schema::schema()`
|
|
189
|
+
returns `&'static ls::Schema`.
|
|
190
|
+
- `lawspec-dev ci` runs its steps in parallel: `--jobs N` at a time
|
|
191
|
+
(`LAWSPEC_CI_JOBS`, half the processors by default), and `--jobs 1` runs
|
|
192
|
+
them one at a time as before. Each step declares the steps it follows and
|
|
193
|
+
what it holds (its target's `.integration` environment, its acceptance
|
|
194
|
+
project, the repository's `.stack-work`, Stack's snapshot database, the
|
|
195
|
+
runtime crate's Cargo target, a timing class for real-time suites), so
|
|
196
|
+
parallel steps never share what one of them changes, and every step still
|
|
197
|
+
runs after a failure. A line is printed as each step finishes, a terminal
|
|
198
|
+
shows the running steps, the summary names failures in step order, and
|
|
199
|
+
Ctrl-C stops every running step's process group. Cached steps are skipped
|
|
200
|
+
exactly as before. See Contribute to LawSpec.
|
|
201
|
+
- The npm package's `prepack` replaces `README.md` and `CHANGELOG.md` in
|
|
202
|
+
`npm/` by rename, and leaves identical copies alone, so concurrent packs
|
|
203
|
+
never read a half-written copy.
|
|
204
|
+
- `package-smoke` allows half an hour for exporting the examples, and
|
|
205
|
+
`lawspec-dev ci` runs it alone.
|
|
206
|
+
|
|
207
|
+
### Fixed
|
|
208
|
+
|
|
209
|
+
- The predicates npm test makes its scratch project under `.artifacts`, so a
|
|
210
|
+
`generate --check` running beside it in parallel CI no longer sees it.
|
|
211
|
+
- `tools/bootstrap-integration.mjs` installs the crypto packages the crypto
|
|
212
|
+
and distribution suites need (`@noble/post-quantum` from the JavaScript and
|
|
213
|
+
TypeScript locks, `cryptography` for Python), so a new `.integration`
|
|
214
|
+
passes them.
|
|
215
|
+
|
|
216
|
+
- The acceptance harness strips ANSI escape sequences before it decides
|
|
217
|
+
whether a mutant failed to compile, so a coloured toolchain (Gleam under
|
|
218
|
+
`FORCE_COLOR`) no longer hides a compile failure behind its colour codes.
|
|
219
|
+
- Generation is deterministic: the on-disk cache is keyed by the build of
|
|
220
|
+
the compiler that wrote it (its executable's path, size and time) as well
|
|
221
|
+
as its version, so a rebuilt compiler no longer reads another build's
|
|
222
|
+
entries.
|
|
223
|
+
- Generated names are stable under unrelated edits: the typed functions a
|
|
224
|
+
handler's clauses call, and its start, are named from the definitions and
|
|
225
|
+
the handler (with a suffix from the digest of what they name where one is
|
|
226
|
+
needed), not from their position among the others.
|
|
227
|
+
- Generated code builds without warnings, or with targeted allow attributes,
|
|
228
|
+
on every target, also as a dependency of another project.
|
|
229
|
+
- A TypeScript type with several type arguments no longer has a trailing
|
|
230
|
+
comma.
|
|
231
|
+
- A spec handler clause that fails no longer poisons the handler on Rust:
|
|
232
|
+
later operations see the state as the failure left it.
|
|
233
|
+
- A field's refinement may call a checked definition of its unit
|
|
234
|
+
(`label :: (v :: Text where validName v)`).
|
|
235
|
+
- A record update or a rebuild proves a field's regex refinement, which the
|
|
236
|
+
copied field already met; text tests and the sequence helpers are facts
|
|
237
|
+
the checker knows.
|
|
238
|
+
- A definition that says it `fails with E` but never raises may be a
|
|
239
|
+
handler clause (`op by f`).
|
|
240
|
+
- A spec handler named like its ability (`counter` for `Counter`) is
|
|
241
|
+
`CounterSpec` where the two would share a name.
|
|
242
|
+
- A record whose field a regex constrains has boundary witnesses that match
|
|
243
|
+
the regex.
|
|
244
|
+
- Bounded arithmetic used by an import's definitions resolves in the
|
|
245
|
+
importing unit, packages included.
|
|
246
|
+
- Checking scales with a program's size: LawTwin's import of GitHub's whole
|
|
247
|
+
REST description (2,108 types, 1,232 operations, 87 laws) checks in
|
|
248
|
+
minutes, where it did not finish in 30. Field access is typed without
|
|
249
|
+
building its match, registries, indexes, audits and unit elaboration are
|
|
250
|
+
made once per program, and sets and maps replace quadratic list scans.
|
|
251
|
+
`lawspec-dev bench compile` times a scaled model. Field access and record
|
|
252
|
+
update are Core's own projection and update, read directly by validation,
|
|
253
|
+
evaluation and the totality audits, instead of a match binding every field
|
|
254
|
+
of the record: the scaled model of 100, 200 and 400 types checks in 22.5,
|
|
255
|
+
46.9 and 129.9 seconds instead of 33.8, 95.5 and 340.2 (unoptimized
|
|
256
|
+
builds).
|
|
257
|
+
- Java and Kotlin generation accepts a field named `record`, `sealed`,
|
|
258
|
+
`permits` or `var`, which Java restricts only as type names.
|
|
259
|
+
- Prelude helpers such as `prelude.isRight` work in an ability's laws and in
|
|
260
|
+
handler clauses.
|
|
261
|
+
- A numeric literal in a `match` or `if` branch, or a list, takes its type
|
|
262
|
+
from the other branches or items.
|
|
263
|
+
|
|
9
264
|
## [0.22.0] - 2026-10-10
|
|
10
265
|
|
|
11
266
|
### Added
|
|
@@ -372,7 +627,7 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
372
627
|
requirements the tests verify), `canonical_decisions.yaml` (the design
|
|
373
628
|
decisions, each cited as `ref:KEY`) and `canonical_exemptions.yaml` sit at
|
|
374
629
|
the root, and `canonical_vetting/` holds every comment, page, decision and
|
|
375
|
-
reference as pending with owner review deferred until
|
|
630
|
+
reference as pending with owner review deferred until 1.0.
|
|
376
631
|
- `make canon` and `lawspec-dev canon` run canon over the repository, and the
|
|
377
632
|
complete check runs it as its `canon` step. Set `CANON_HOME` to a canon
|
|
378
633
|
checkout built with `stack build`; without one the step is skipped with a
|
|
@@ -1280,7 +1535,8 @@ A maintenance release. The language, generated code and API are unchanged from
|
|
|
1280
1535
|
machine-sized bindings reject an architecture mismatch. Rust adapters take
|
|
1281
1536
|
owned values.
|
|
1282
1537
|
|
|
1283
|
-
[Unreleased]: https://github.com/brain-fuel/lawspec/compare/v0.
|
|
1538
|
+
[Unreleased]: https://github.com/brain-fuel/lawspec/compare/v0.23.0...HEAD
|
|
1539
|
+
[0.23.0]: https://github.com/brain-fuel/lawspec/compare/v0.22.0...v0.23.0
|
|
1284
1540
|
[0.22.0]: https://github.com/brain-fuel/lawspec/compare/v0.21.0...v0.22.0
|
|
1285
1541
|
[0.21.0]: https://github.com/brain-fuel/lawspec/compare/v0.20.1...v0.21.0
|
|
1286
1542
|
[0.20.1]: https://github.com/brain-fuel/lawspec/compare/v0.20.0...v0.20.1
|
package/README.md
CHANGED
|
@@ -30,7 +30,7 @@ Node 22+ and the selected target's build tools are required. No Haskell
|
|
|
30
30
|
toolchain is needed. The reference platforms are macOS and Linux.
|
|
31
31
|
|
|
32
32
|
```sh
|
|
33
|
-
npm install --save-dev lawspec@0.
|
|
33
|
+
npm install --save-dev lawspec@0.23.0
|
|
34
34
|
npx lawspec --version
|
|
35
35
|
```
|
|
36
36
|
|
|
@@ -40,8 +40,8 @@ LawSpec can create `package.json` and its test script:
|
|
|
40
40
|
```sh
|
|
41
41
|
mkdir lawspec-example
|
|
42
42
|
cd lawspec-example
|
|
43
|
-
npm exec --package=lawspec@0.
|
|
44
|
-
npm install --save-dev lawspec@0.
|
|
43
|
+
npm exec --package=lawspec@0.23.0 -- lawspec init --target javascript
|
|
44
|
+
npm install --save-dev lawspec@0.23.0
|
|
45
45
|
npx lawspec check
|
|
46
46
|
npx lawspec explain 'example.atoi_codec::itoa and then atoi yields a'
|
|
47
47
|
npx lawspec doctor
|
package/bin/lawspec.mjs
CHANGED
|
@@ -20,14 +20,14 @@ import {
|
|
|
20
20
|
atomicWrite,
|
|
21
21
|
safePath,
|
|
22
22
|
} from "../files.mjs";
|
|
23
|
-
const VERSION = "0.
|
|
23
|
+
const VERSION = "0.23.0";
|
|
24
24
|
const args = process.argv.slice(2);
|
|
25
25
|
const verb = args.shift();
|
|
26
26
|
const options = {};
|
|
27
27
|
const positional = [];
|
|
28
28
|
for (let i = 0; i < args.length; i++) {
|
|
29
29
|
const arg = args[i];
|
|
30
|
-
if (["--target", "--project", "--config", "--output", "--machine-bits", "--example", "--seed", "--report"].includes(arg)) {
|
|
30
|
+
if (["--target", "--project", "--config", "--output", "--machine-bits", "--example", "--seed", "--report", "--attestation"].includes(arg)) {
|
|
31
31
|
if (!args[i + 1] || args[i + 1].startsWith("--"))
|
|
32
32
|
throw new Error(`Missing value for ${arg}`);
|
|
33
33
|
options[arg.slice(2)] = args[++i];
|
|
@@ -272,19 +272,49 @@ function explainExamples(law) {
|
|
|
272
272
|
ex.expectations.map(e => ` expect ${showAssertion(e)}`).join("\n")
|
|
273
273
|
).join("\n");
|
|
274
274
|
}
|
|
275
|
-
// What lawspec test last recorded for each law, across targets
|
|
275
|
+
// What lawspec test last recorded for each law, across targets, and the
|
|
276
|
+
// environment attestations its runs were given, by run id. A law passed in
|
|
277
|
+
// an attested run names that run; across targets, its latest passing run.
|
|
276
278
|
async function lastRuns() {
|
|
277
279
|
const folder = path.join(configRoot, ".lawspec", "results");
|
|
278
280
|
const runs = new Map();
|
|
281
|
+
const attestations = new Map();
|
|
279
282
|
for (const name of (await readOptionalDirectory(folder)) ?? []) {
|
|
280
283
|
if (!name.endsWith(".json")) continue;
|
|
281
284
|
const recorded = JSON.parse((await readOptional(path.join(folder, name))) ?? "{}");
|
|
285
|
+
for (const [id, attestation] of Object.entries(recorded.attestations ?? {})) attestations.set(id, attestation);
|
|
282
286
|
for (const [law, run] of Object.entries(recorded.laws ?? {})) {
|
|
283
287
|
const known = runs.get(law) ?? {};
|
|
284
|
-
|
|
288
|
+
const latest = (known.passed ?? "") > (run.passed ?? "") ? known : run;
|
|
289
|
+
runs.set(law, { flaky: known.flaky || run.flaky, adequacy: run.adequacy ?? known.adequacy,
|
|
290
|
+
passed: latest.passed, environment: latest.environment });
|
|
285
291
|
}
|
|
286
292
|
}
|
|
287
|
-
return runs;
|
|
293
|
+
return { runs, attestations };
|
|
294
|
+
}
|
|
295
|
+
// An environment attestation: a versioned JSON document any tool may write,
|
|
296
|
+
// recording for one run where each dependency was wired and whether it was
|
|
297
|
+
// the real dependency or a stand-in. --attestation names one, or lawspec.json
|
|
298
|
+
// does; the compiler checks it. It is reporting only and changes no evidence.
|
|
299
|
+
async function readAttestation(config) {
|
|
300
|
+
const file = options.attestation !== undefined ? path.resolve(options.attestation)
|
|
301
|
+
: config.attestation !== undefined ? path.resolve(configRoot, config.attestation) : null;
|
|
302
|
+
if (file === null) return null;
|
|
303
|
+
const text = await readOptional(file);
|
|
304
|
+
if (text === null) throw new Error(`No environment attestation at ${file}`);
|
|
305
|
+
try {
|
|
306
|
+
return JSON.parse(text);
|
|
307
|
+
} catch (error) {
|
|
308
|
+
throw new Error(`${file}: the environment attestation is not JSON: ${error.message}`);
|
|
309
|
+
}
|
|
310
|
+
}
|
|
311
|
+
// One attested dependency, as lawspec evidence shows it.
|
|
312
|
+
function dependencyText(d) {
|
|
313
|
+
const level = (c) => /^[0-9]+$/.test(c) ? `conformance level ${c}` : `conformance ${c}`;
|
|
314
|
+
return `${d.dependency}: ` + (d.standIn
|
|
315
|
+
? `stand-in ${d.standIn.identity} ${d.standIn.version} (${level(d.standIn.conformance)}` +
|
|
316
|
+
(d.standIn.historyDigest ? `, history ${d.standIn.historyDigest}` : "") + ")"
|
|
317
|
+
: "real") + ` at ${d.endpoint}` + (d.reached ? "" : " (not reached when recorded)");
|
|
288
318
|
}
|
|
289
319
|
// One law's harness, as lawspec evidence shows it.
|
|
290
320
|
function harnessText(item) {
|
|
@@ -358,12 +388,18 @@ async function runTests(compiler, input, selected, roots, config) {
|
|
|
358
388
|
const summaries = [];
|
|
359
389
|
const junit = [];
|
|
360
390
|
const report = options.report === undefined ? null : parseReport(options.report);
|
|
391
|
+
const attestation = await readAttestation(config);
|
|
361
392
|
for (const [i, target] of selected.entries()) {
|
|
362
393
|
const root = roots[i];
|
|
363
394
|
const planned = diagnostics(await compiler.planGeneration({
|
|
364
395
|
...input, target: target.language, sourceDir: target.sourceDir, testDir: target.testDir,
|
|
365
396
|
nativeBindings: target.nativeBindings, minify: options.minify === true,
|
|
397
|
+
...(attestation ? { attestation } : {}),
|
|
366
398
|
}));
|
|
399
|
+
// The laws the attestation touches, each with how its dependencies
|
|
400
|
+
// were wired (the run's id and times aside).
|
|
401
|
+
const attested = new Map((planned.evidence ?? []).filter((item) => item.against)
|
|
402
|
+
.map((item) => [item.declaration, JSON.stringify([item.environment, item.against])]));
|
|
367
403
|
const pending = await planWrites(root, planned.files);
|
|
368
404
|
if (pending.changes.length)
|
|
369
405
|
throw new Error(`${target.language}: generated files are out of date; run lawspec generate${options.minify ? " --minify" : ""}`);
|
|
@@ -376,6 +412,7 @@ async function runTests(compiler, input, selected, roots, config) {
|
|
|
376
412
|
// Recorded values are spec data: a changed recording runs its laws again.
|
|
377
413
|
environment: await environmentDigest(root, report_) + (await recordedDigest(recordedFolder)),
|
|
378
414
|
project: await projectDigest(root, generated),
|
|
415
|
+
attested,
|
|
379
416
|
});
|
|
380
417
|
const identity = projectKey(root);
|
|
381
418
|
const resultsFile = path.join(configRoot, ".lawspec", "results", `${target.language}-${identity}.json`);
|
|
@@ -476,6 +513,8 @@ async function runTests(compiler, input, selected, roots, config) {
|
|
|
476
513
|
const label = entry.label;
|
|
477
514
|
const observed = statistics.filter((s) => s.law === label);
|
|
478
515
|
laws[entry.law] = { key: keys.get(entry.law), seed: Number(seedOf.get(entry.law) ?? seed), passed: new Date().toISOString(),
|
|
516
|
+
// A law the attestation touches names the run it passed in.
|
|
517
|
+
...(attestation && attested.has(entry.law) ? { environment: attestation.run.id } : {}),
|
|
479
518
|
...(observed.some((s) => s.outcome === "flaky") ? { flaky: true } : {}),
|
|
480
519
|
...(observed.some((s) => s.cover || s.labels || s.classes) ? { adequacy: observed.filter((s) => s.cover || s.labels || s.classes)
|
|
481
520
|
.map(({ test, cases, cover, classes, labels }) => ({ test, cases, cover, classes, labels })) } : {}) };
|
|
@@ -492,13 +531,19 @@ async function runTests(compiler, input, selected, roots, config) {
|
|
|
492
531
|
await writeFile(databaseFile, JSON.stringify(database, null, 2) + "\n");
|
|
493
532
|
await mkdir(path.dirname(resultsFile), { recursive: true });
|
|
494
533
|
await writeFile(path.join(path.dirname(resultsFile), ".gitignore"), "*\n");
|
|
495
|
-
|
|
534
|
+
// Each attestation a recorded law ran under is kept with the results.
|
|
535
|
+
const attestations = Object.fromEntries(Object.entries({ ...(previous.attestations ?? {}),
|
|
536
|
+
...(attestation ? { [attestation.run.id]: attestation } : {}) })
|
|
537
|
+
.filter(([id]) => Object.values(laws).some((run) => run.environment === id)));
|
|
538
|
+
await writeFile(resultsFile, JSON.stringify({ version: 1, laws,
|
|
539
|
+
...(Object.keys(attestations).length ? { attestations } : {}) }, null, 2) + "\n");
|
|
496
540
|
const flaky = statistics.filter((s) => s.outcome === "flaky").map((s) => s.law);
|
|
497
541
|
const unmet = statistics.flatMap((s) => (s.cover ?? []).filter((c) => !c.met).map((c) => `${s.law}: cover ${c.required}% "${c.label}" (${c.observed}%)`));
|
|
498
542
|
const benchmarks = statistics.filter((s) => s.benchmark);
|
|
499
543
|
// How each parallel unit's tests actually ran at the same time.
|
|
500
544
|
const parallelism = statistics.filter((s) => s.parallel).map(({ parallel, mode, workers }) => ({ unit: parallel, mode, workers }));
|
|
501
545
|
summaries.push({ target: target.language, root: target.root, seed: Number(seed), ran: stale.map((e) => e.law),
|
|
546
|
+
...(attestation ? { environment: attestation.environment } : {}),
|
|
502
547
|
unchanged: chosen.length - stale.length, ok: !failed,
|
|
503
548
|
...(chosen.length !== planned.tests.length ? { deselected: planned.tests.length - chosen.length } : {}),
|
|
504
549
|
...(unrun.length ? { unrun: unrun.map((e) => e.law) } : {}),
|
|
@@ -514,7 +559,7 @@ async function runTests(compiler, input, selected, roots, config) {
|
|
|
514
559
|
await writeFile(path.resolve(report.path), mergeJunit(junit));
|
|
515
560
|
}
|
|
516
561
|
output(options.json ? summaries : summaries.map((s) =>
|
|
517
|
-
`${s.target}${summaries.filter(other => other.target === s.target).length > 1 ? ` (${s.root})` : ""}: ${s.ran.length ? `ran ${s.ran.length} law(s) with seed ${s.seed}` : "nothing to run"}` +
|
|
562
|
+
`${s.target}${summaries.filter(other => other.target === s.target).length > 1 ? ` (${s.root})` : ""}: ${s.ran.length ? `ran ${s.ran.length} law(s) with seed ${s.seed}${s.environment ? ` in environment ${s.environment}` : ""}` : "nothing to run"}` +
|
|
518
563
|
`, ${s.unchanged} unchanged since their last passing run.${s.ok ? "" : " FAILED"}` +
|
|
519
564
|
(s.deselected ? `\n${s.deselected} law(s) not selected by --tag or --exclude-tag.` : "") +
|
|
520
565
|
(s.unrun ? `\nNo tests ran for ${s.unrun.join(", ")}; the runner matched none of their tests.` : "") +
|
|
@@ -615,7 +660,7 @@ async function main() {
|
|
|
615
660
|
}
|
|
616
661
|
if (!verb || ["help", "--help", "-h"].includes(verb)) {
|
|
617
662
|
output(
|
|
618
|
-
"LawSpec " + VERSION + "\nUsage: lawspec init --target <language> [--project <directory>] [--minify]\n lawspec check | doctor | explain <unit>::<law> | generate\n lawspec evidence [<unit> | <unit>::<declaration>]\n lawspec test [--target <language>] [--fresh] [--seed <n>] [--update-recorded] [--tag <t>] [--exclude-tag <t>] [--report junit=<path>] [--coverage] [--benchmarks]\n lawspec package [--project <package directory>]\n lawspec examples [--example payments] [--target <language>] [--output <directory>]\nOptions: --config <path>, --target <language>, --machine-bits <32|64>, --json, --no-cache\nGeneration: --dry-run, --check, --minify\nTargets: " +
|
|
663
|
+
"LawSpec " + VERSION + "\nUsage: lawspec init --target <language> [--project <directory>] [--minify]\n lawspec check | doctor | explain <unit>::<law> | generate\n lawspec evidence [<unit> | <unit>::<declaration>] [--attestation <file>]\n lawspec test [--target <language>] [--fresh] [--seed <n>] [--attestation <file>] [--update-recorded] [--tag <t>] [--exclude-tag <t>] [--report junit=<path>] [--coverage] [--benchmarks]\n lawspec package [--project <package directory>]\n lawspec examples [--example payments] [--target <language>] [--output <directory>]\nOptions: --config <path>, --target <language>, --machine-bits <32|64>, --json, --no-cache\nGeneration: --dry-run, --check, --minify\nTargets: " +
|
|
619
664
|
targets.join(", "),
|
|
620
665
|
);
|
|
621
666
|
return;
|
|
@@ -662,6 +707,8 @@ async function main() {
|
|
|
662
707
|
if ((options.fresh || options.seed !== undefined || options.tag || options["exclude-tag"] ||
|
|
663
708
|
options.report !== undefined || options.coverage || options["update-recorded"] || options.benchmarks) && verb !== "test")
|
|
664
709
|
throw new Error("--fresh, --seed, --update-recorded, --tag, --exclude-tag, --report, --coverage and --benchmarks apply to test");
|
|
710
|
+
if (options.attestation !== undefined && !["evidence", "test"].includes(verb))
|
|
711
|
+
throw new Error("--attestation applies to evidence and test");
|
|
665
712
|
if (options.seed !== undefined && !/^[0-9]+$/.test(options.seed))
|
|
666
713
|
throw new Error("--seed must be a whole number");
|
|
667
714
|
if (verb === "init") return init();
|
|
@@ -704,6 +751,8 @@ async function main() {
|
|
|
704
751
|
}
|
|
705
752
|
if (config.machineBits !== undefined && ![32, 64].includes(config.machineBits)) throw new Error("machineBits must be 32 or 64");
|
|
706
753
|
if (config.cache !== undefined && typeof config.cache !== "boolean") throw new Error("cache must be true or false");
|
|
754
|
+
if (config.attestation !== undefined && typeof config.attestation !== "string")
|
|
755
|
+
throw new Error("attestation must be the path of an environment attestation");
|
|
707
756
|
const cache = await cacheDirectory(config);
|
|
708
757
|
const input = {
|
|
709
758
|
...(cache === undefined ? {} : { cacheDirectory: cache }),
|
|
@@ -752,8 +801,23 @@ async function main() {
|
|
|
752
801
|
return;
|
|
753
802
|
}
|
|
754
803
|
if (verb === "evidence") {
|
|
755
|
-
|
|
804
|
+
// An attestation named here, or in lawspec.json, says where the tests
|
|
805
|
+
// ran; otherwise each law shows the attested run it last passed in.
|
|
806
|
+
const named = await readAttestation(config);
|
|
807
|
+
const result = diagnostics(await compiler.check({ ...input, ...(named ? { attestation: named } : {}) }));
|
|
756
808
|
let evidence = result.evidence;
|
|
809
|
+
const { runs, attestations } = await lastRuns();
|
|
810
|
+
const environments = named ? [result.attestation] : [];
|
|
811
|
+
if (!named) for (const [id, attestation] of attestations) {
|
|
812
|
+
const laws = new Set([...runs].filter(([, run]) => run.environment === id).map(([law]) => law));
|
|
813
|
+
if (!laws.size) continue;
|
|
814
|
+
const attested = diagnostics(await compiler.check({ ...input, attestation }));
|
|
815
|
+
const fields = new Map(attested.evidence.filter((item) => item.against && laws.has(item.declaration))
|
|
816
|
+
.map(({ declaration, environment, environmentRun, against }) => [declaration, { environment, environmentRun, against }]));
|
|
817
|
+
evidence = evidence.map((item) => item.stage === "law" && fields.has(item.declaration) ? { ...item, ...fields.get(item.declaration) } : item);
|
|
818
|
+
environments.push({ ...attested.attestation, dependencies: attested.attestation.dependencies
|
|
819
|
+
.map((d) => ({ ...d, obligations: d.obligations.filter((law) => laws.has(law)) })) });
|
|
820
|
+
}
|
|
757
821
|
// Native bindings are configured per target; their obligations are added.
|
|
758
822
|
for (const target of selected) {
|
|
759
823
|
if (target.nativeBindings === undefined) continue;
|
|
@@ -767,7 +831,6 @@ async function main() {
|
|
|
767
831
|
if (positional[0] && !evidence.length) throw new Error("No matching obligation");
|
|
768
832
|
// The last run's outcome (lawspec test): a law that failed, then passed
|
|
769
833
|
// on a retry, is flaky; what its generated cases covered is its adequacy.
|
|
770
|
-
const runs = await lastRuns();
|
|
771
834
|
evidence = evidence.map((item) => {
|
|
772
835
|
const run = item.stage === "law" ? runs.get(item.declaration) : undefined;
|
|
773
836
|
if (!run) return item;
|
|
@@ -785,12 +848,18 @@ async function main() {
|
|
|
785
848
|
if (!items.length) return null;
|
|
786
849
|
return `${label} (${items.length})\n` + items.map((item) =>
|
|
787
850
|
` ${item.stage} ${obligationName(item)}${item.target ? ` [${item.target}]` : ""}` +
|
|
788
|
-
(item.claim ? `: ${item.claim.text}` : "") + `\n ${item.reason}`
|
|
851
|
+
(item.claim ? `: ${item.claim.text}` : "") + `\n ${item.reason}` +
|
|
852
|
+
(item.against ? `\n ran in environment ${item.environment} (run ${item.environmentRun.id}) against ` +
|
|
853
|
+
item.against.map(dependencyText).join("; ") : "")).join("\n");
|
|
789
854
|
})
|
|
790
855
|
.filter((section) => section !== null),
|
|
791
856
|
// How each law is discharged: the harness plane, apart from what
|
|
792
857
|
// the laws say.
|
|
793
|
-
...(harnessed.length ? [`HARNESS (${harnessed.length})\n` + harnessed.map(harnessText).join("\n")] : [])
|
|
858
|
+
...(harnessed.length ? [`HARNESS (${harnessed.length})\n` + harnessed.map(harnessText).join("\n")] : []),
|
|
859
|
+
// Where the tests ran: reporting only, so no status above changes.
|
|
860
|
+
...environments.map((e) => `ENVIRONMENT ${e.environment} (run ${e.run.id}, recorded ${e.run.recordedAt})\n` +
|
|
861
|
+
e.dependencies.map((d) => ` ${dependencyText(d)}; ` + (d.obligations.length
|
|
862
|
+
? `${d.obligations.length} tested obligation(s)` : "no tested obligation relies on it")).join("\n"))]
|
|
794
863
|
.join("\n\n"),
|
|
795
864
|
);
|
|
796
865
|
return;
|