verity 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 56eba3ab988df08937ded0d1a64257aef3e6b0377e89c205cb1142555cfe6022
+  data.tar.gz: 767c78b325581dbb233805be573efcf77a1ef6842d1ecb79c23e1036f0d18dd5
+SHA512:
+  metadata.gz: 9d1afd58fb7e9eead47231b4c08ef719fcdbba846ed760fbc84238baceaca870c76bb160e678f1a3ff81c3ed40ed4fcc2460f7709bbe7fda911008b1704f26b3
+  data.tar.gz: 99325207bf1dbf549b2ee760d5e2528f8b2ae23e98930c3fe67123c18cf577fb3e8b572ef42ebcb8784f4acf13239516e6b45d2e5c9c468d68be5c6c893f4a83

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tad Thorley
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,227 @@
+# Verity
+
+Metadata-first Ruby tests: each case is a structured record (tags, timeouts, resource hints) backed by a SQLite manifest queue. The CLI loads discovery files, syncs them into the manifest, and runs tests — either on a single worker or across parallel forked processes that claim tests atomically from the queue.
+
+## Requirements
+
+- Ruby **≥ 3.3**
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "verity"
+```
+
+Or install from the repository root:
+
+```bash
+gem build verity.gemspec && gem install verity-*.gem
+```
+
+## Running
+
+From a checkout, after dependencies are available:
+
+```bash
+./bin/verity
+# or
+bundle exec verity
+```
+
+Positional arguments are treated as file paths or globs — only those files are loaded instead of the configured `test_globs`:
+
+```bash
+verity verity/models/user_test.rb
+verity verity/models/*_test.rb verity/lib/auth_test.rb
+```
+
+Each argument is resolved with `File.expand_path`, so relative paths work from any directory.
+
+Use `--workers` / `-w` to run tests in parallel across forked processes:
+
+```bash
+verity -w 4                 # exactly 4 workers
+verity -w cpus              # one worker per CPU (Etc.nprocessors)
+verity --workers 2 verity/  # combine with positional args
+```
+
+Exit status is **0** if every claimed test passes, **1** otherwise (`exit` in `bin/verity` mirrors that). **2** is used for invalid CLI options or an invalid `--reporter` / `-r` value.
+
+```bash
+verity --reporter dots
+verity -r null
+verity -r ./reporters/mine.rb:MyReporter
+```
+
+There is no `--version` flag. The current version is available programmatically as `Verity::VERSION`.
+
+Built-in names are the same as for `Verity.build_reporter` (case-insensitive): `colored`, `colored_dots`, `documentation`, `doc`, `dots`, `null`, `none`, `silent`. Custom reporters: `path/to/file.rb:ClassName` (class must `include Verity::Reporter`); the file is `load`ed, then `ClassName.new` is called with no arguments.
+
+`ColoredDotsReporter` (the default) prints green **.** / red **F** / yellow **E** when stdout is a TTY. Set `NO_COLOR` in the environment to disable; set `FORCE_COLOR` or `VERITY_FORCE_COLOR` to `"1"`, `"true"`, or `"yes"` (case-insensitive) to force color when not a TTY.
+
+## Configuration
+
+Use `Verity.configure` before `Verity.run` (or ensure defaults match your layout):
+
+```ruby
+Verity.configure do |c|
+  c.manifest_path = "verity/manifest.db"   # default; path relative to cwd (ignored by git); or ":memory:" for single-process only
+  c.test_globs = ["verity/**/*_test.rb"]   # default; set to your Verity discovery globs
+  # c.worker_count = :cpus                 # default; or a positive Integer, or "cpus" / :cpu / "cpu"
+  # c.reporter = Verity::Reporters::ColoredDotsReporter.new($stdout)  # default
+end
+```
+
+- **`test_globs`** — array of patterns passed to `Dir.glob`; merged and de-duplicated for **`test_files`**.
+- **`manifest_path`** — SQLite database path (default **`verity/manifest.db`**), or `":memory:"` for an in-memory DB (only with **`worker_count` 1**).
+- **`worker_count`** — number of parallel worker processes (`Integer` or decimal string), or **`:cpus`** / **`:cpu`** / **`"cpus"`** / **`"cpu"`** to use `Etc.nprocessors` (minimum **1**). Resolved at run time via **`Configuration#resolved_worker_count`**. Parallel runs need a **file** manifest (not `":memory:"`) and **`Kernel#fork`**.
+- **`reporter`** — object that includes `Verity::Reporter` (default: `Verity::Reporters::ColoredDotsReporter` on `$stdout`). See **Custom reporters** below.
+
+`Verity.run(worker_id: 0)` loads all `test_files`, migrates the manifest, replaces the `tests` table from the registry, then runs the manifest-driven runner for that worker.
+
+`Verity.load_discovery!` only clears the registry and loads `test_files` (useful if you build your own harness). For each file it precomputes **fingerprints** with **Prism**: the hash covers the **block body** only (description and metadata changes do not change identity). **`Test#file`** and **`Test#line`** remain the **`test` call** location. If you `load` a file outside that path (no plan installed), fingerprints fall back to a line-based slug.
+
+### Custom reporters
+
+Implement {Verity::Reporter} and assign it on configuration. `Verity.run` and `Runner.new` (no `reporter:` keyword) use `Verity.configuration.reporter`. Built-ins live under `Verity::Reporters`:
+
+| Class | Purpose |
+|-------|---------|
+| `ColoredDotsReporter` | Default — green/red/yellow dots with ANSI color (TTY-aware) |
+| `DotsReporter` | Plain `.` / `F` / `E` dots, no color |
+| `DocumentationReporter` | Prints group titles and test descriptions (outline style) |
+| `NullReporter` | Discards all output (used internally for parallel child workers) |
+| `TestReporter` | In-memory recorder for testing integrations (see below) |
+| `CompositeReporter` | Delegates to multiple reporters |
+| `ParallelSummaryReporter` | Emits the multi-worker summary block after parallel runs |
+
+```ruby
+class MyReporter
+  include Verity::Reporter
+
+  def on_run_start(total:, worker_id:)
+    # total: expected number of examples for this worker (nil if unknown)
+  end
+
+  def on_test_complete(result:, worker_id:)
+    # See Verity::Runner::Result: :test, :status (:pass | :fail | :error), :error
+  end
+
+  def on_run_finish(summary:, worker_id:)
+    # summary: :total, :passed, :failed, :errored, :skipped, :focus
+  end
+
+  # Optional: after Verity.run with worker_count > 1 (parent process only)
+  def on_parallel_complete(counts:, problem_rows:)
+  end
+end
+
+Verity.configure do |c|
+  c.reporter = MyReporter.new
+end
+```
+
+For a one-off run without changing global config, pass `Verity::Runner.new(reporter: MyReporter.new)`.
+
+### TestReporter
+
+`Verity::Reporters::TestReporter` records every callback in memory (no I/O), useful for testing integrations against the reporter protocol. It exposes four readers:
+
+| Reader | Stores |
+|--------|--------|
+| `run_starts` | `[{ total:, worker_id: }, ...]` |
+| `test_completes` | `[{ status:, worker_id: }, ...]` |
+| `run_finishes` | `[{ summary:, worker_id: }, ...]` |
+| `parallel_finishes` | `[{ counts:, problem_rows: }, ...]` |
+
+```ruby
+reporter = Verity::Reporters::TestReporter.new
+Verity.configure { |c| c.reporter = reporter }
+Verity.run
+reporter.test_completes.count { _1[:status] == :pass }
+```
+
+## Grouping
+
+Nest tests under titled sections with **`group`**. Each `test` registers with a **`group_path`** (array of titles) used for output and tooling; fingerprints and execution order are unchanged.
+
+```ruby
+group "Authentication", tags: [:integration] do
+  group "sessions", tags: [:focus] do
+    test "creates a session" do
+      # ...
+    end
+  end
+end
+
+group "WIP", tags: [:skip] do
+  test "not scheduled yet" do
+  end
+end
+```
+
+Tags on a **`group`** apply to **every nested `test`** (and inner groups): they are stored on each test as **`inherited_group_tags`** (outer groups first) and merged with the test’s own **`tags:`** for **`Verity.skipped?`**, **`Verity.focus_tag?`**, and **`Verity.effective_tags`**. **`:skip`** on any ancestor (or on the test) skips the example; **`:focus`** follows the same suite-wide rules as test-level **`:focus`**.
+
+**`Verity::Reporters::DocumentationReporter`** prints new group titles when the path changes (indented like an outline). Dot reporters do not show groups. Custom reporters can read **`result.test.group_path`** and **`result.test.inherited_group_tags`**.
+
+The group stack is cleared before each discovery file is loaded so a stray unclosed `group` in one file does not affect the next.
+
+## Tags
+
+- **`tags: [:skip]`** — The example is **not** enqueued in the manifest and does **not** run. It still appears in **`Verity::Registry.all`**. The summary line includes **`N skipped`** when `N > 0`. String `"skip"` in tags is treated the same (normalized with `to_sym`). A **`group`** may use **`tags: [:skip]`**; that applies to all nested tests (see **Grouping**).
+- **`tags: [:focus]`** — If **any** non-skipped registered test has **`:focus`** (including via an enclosing **`group`**), **only** tests that have **`:focus`** in their effective tags are runnable (manifest + direct **`Runner#run`**). If every non-skipped test is focused, the filter does nothing (same as “all focused”). **`Skip` wins:** a test with both **`skip`** and **`focus`** is skipped. When focus narrows the suite, the summary ends with **`(focus)`**.
+
+## `Verity::Test` fields
+
+Each registered test is a `Data.define` struct with 11 fields:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `fingerprint` | `String` | Stable identity hash derived from the block body via Prism AST |
+| `description` | `String` | Human-readable name passed to `test "..."` |
+| `tags` | `Array<Symbol>` | Tags declared on the test itself (e.g. `[:unit, :focus]`) |
+| `timeout` | `Float`, `nil` | Optional per-test timeout in seconds |
+| `requires` | `Array` | Declared dependency hints (e.g. `[:active_record]`) |
+| `resources` | `Hash` | Extra keyword args from `test` (e.g. `{ tables: [:users] }`) |
+| `file` | `String` | Absolute path of the file containing the `test` call |
+| `line` | `Integer` | Line number of the `test` call |
+| `fn` | `Proc` | The test body block |
+| `group_path` | `Array<String>` | Nested `group` titles at registration time (outer first) |
+| `inherited_group_tags` | `Array<Symbol>` | Tags accumulated from enclosing `group` blocks (outer first) |
+
+## Repository layout (this project)
+
+| Directory | Role |
+|-----------|------|
+| `test/` | Minitest for Verity internals |
+| `spec/` | RSpec examples |
+| `verity/` | Verity DSL files (default discovery glob targets `verity/**/*_test.rb`) |
+| `lib/` | Gem implementation |
+
+### Triple suite: compare, convert, and cross-check behavior
+
+Integration scenarios are spelled out three ways on purpose:
+
+| Layer | Paths | Audience |
+|-------|-------|----------|
+| **Dogfood DSL** | `verity/<topic>_test.rb` | Readers learning Verity (`test`, `group`, built-in assertions) |
+| **Minitest** | `test/<topic>_test.rb` | Readers used to `@test`/assert style and class-based suites |
+| **RSpec** | `spec/verity/<topic>_spec.rb` | Readers used to `describe`/`it` matchers |
+
+Matching files share the **same basename** (`foo_test.rb` ↔ `foo_spec.rb`). Scenario titles are aligned so you can open two panes side by side when porting assertions or onboarding a team. Keeping all three suites green is deliberate **redundant proof** — the SQLite manifest and runner stay honest under different loaders and assertions.
+
+Example triplet:
+
+- [`verity/assertions_test.rb`](verity/assertions_test.rb)
+- [`test/assertions_test.rb`](test/assertions_test.rb)
+- [`spec/verity/assertions_spec.rb`](spec/verity/assertions_spec.rb)
+
+## Design notes
+
+See [verity-notes.md](verity-notes.md) for schema, fingerprints, and planned execution model.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

data/bin/benchmark

@@ -0,0 +1,99 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "optparse"
+require "rbconfig"
+require "bundler/setup"
+
+root = File.expand_path("..", __dir__)
+Dir.chdir(root) or abort("benchmark: cannot chdir to #{root}")
+
+ruby = RbConfig.ruby
+opts = { iterations: 1 }
+OptionParser.new do |o|
+  o.banner = "Usage: benchmark [options]"
+  o.separator ""
+  o.separator "Runs the same three suites as bin/test_all and prints wall-clock timings."
+  o.separator ""
+  o.on("-n", "--iterations N", Integer, "Repeat each suite N times (default: 1)") do |n|
+    opts[:iterations] = n
+  end
+  o.on("-h", "--help", "Show this help") do
+    puts o
+    exit 0
+  end
+end.parse!
+
+bench_iterations = opts[:iterations]
+abort "benchmark: iterations must be >= 1" if bench_iterations < 1
+
+def fmt_sec(s)
+  format("%0.3f", s)
+end
+
+def summarize(samples)
+  return { min: samples.last, max: samples.last, mean: samples.last } if samples.size == 1
+
+  {
+    min: samples.min,
+    max: samples.max,
+    mean: samples.sum / samples.size
+  }
+end
+
+def run_suite(iterations)
+  times = []
+  iterations.times do |i|
+    t0 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    ok = yield
+    t1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    abort "benchmark: suite failed (iteration #{i + 1})" unless ok
+
+    times << (t1 - t0)
+  end
+  times
+end
+
+verity_bin = File.join(root, "bin", "verity")
+results = []
+
+results << [
+  "Dogfood (bin/verity)",
+  summarize(run_suite(bench_iterations) do
+    system(ruby, verity_bin)
+  end)
+]
+
+minitest_files = Dir.glob("test/**/*_test.rb").sort
+abort "benchmark: no files matched test/**/*_test.rb" if minitest_files.empty?
+
+results << [
+  "Minitest (test/**/*_test.rb, #{minitest_files.size} files)",
+  summarize(run_suite(bench_iterations) do
+    minitest_files.all? do |path|
+      system(ruby, "-Ilib:test", path)
+    end
+  end)
+]
+
+results << [
+  "RSpec (spec/)",
+  summarize(run_suite(bench_iterations) do
+    system(ruby, "-S", "rspec", "--format", "progress")
+  end)
+]
+
+puts
+puts "Benchmark: #{bench_iterations} iteration(s) per suite, monotonic wall clock (seconds)"
+puts "-" * 72
+results.each do |label, stat|
+  line = format("%-48s %7ss", label, fmt_sec(stat[:mean]))
+  line += format("  (min #{fmt_sec(stat[:min])} .. max #{fmt_sec(stat[:max])})") if bench_iterations > 1
+
+  puts line
+end
+
+total_mean = results.sum { |_, s| s[:mean] }
+puts "-" * 72
+puts format("%-48s %7ss", "Total (sum of suite means)", fmt_sec(total_mean))
+puts

data/bin/test_all

@@ -0,0 +1,43 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "rbconfig"
+require "bundler/setup"
+
+root = File.expand_path("..", __dir__)
+Dir.chdir(root) or abort("test_all: cannot chdir to #{root}")
+
+ruby = RbConfig.ruby
+failed = false
+
+def banner(title)
+  sep = "=" * 60
+  puts "\n#{sep}\n#{title}\n#{sep}\n"
+end
+
+verity_bin = File.join(root, "bin", "verity")
+
+banner "1/3 Dogfood — bin/verity (verity/**/*_test.rb)"
+failed = true unless system(ruby, verity_bin)
+
+banner "2/3 Minitest — test/**/*_test.rb"
+minitest_files = Dir.glob("test/**/*_test.rb").sort
+if minitest_files.empty?
+  warn "test_all: no files matched test/**/*_test.rb"
+else
+  minitest_files.each do |path|
+    puts "\n--- #{path} ---"
+    failed = true unless system(ruby, "-Ilib:test", path)
+  end
+end
+
+banner "3/3 RSpec — spec/"
+failed = true unless system(ruby, "-S", "rspec", "--format", "documentation")
+
+if failed
+  warn "\ntest_all: one or more suites failed"
+  exit 1
+end
+
+puts "\ntest_all: all suites passed"
+exit 0

data/bin/verity

@@ -0,0 +1,82 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "optparse"
+
+$LOAD_PATH.unshift File.expand_path("../lib", __dir__)
+require "verity"
+
+begin
+  OptionParser.new do |o|
+    o.banner = "Usage: verity [options] [file-or-glob ...]"
+    o.separator ""
+    o.separator "If file-or-glob arguments are given, only those paths are loaded."
+    o.separator "Use path/to/file.rb:LINE to run the test on that line or every test in the group opened there."
+    o.separator "Otherwise the configured default is used (see Verity::Configuration#test_globs)."
+    o.separator ""
+    o.on(
+      "-r",
+      "--reporter NAME",
+      "Output reporter: colored (default), colored_dots, dots, documentation, doc, null, none, silent;",
+      "or path/to/reporter.rb:ClassName for a custom class (include Verity::Reporter)."
+    ) do |name|
+      Verity.configure { |c| c.reporter = Verity.build_reporter(name) }
+    end
+    o.on(
+      "-w",
+      "--workers COUNT",
+      "Parallel worker processes: a positive integer or cpus (one worker per CPU via Etc.nprocessors)."
+    ) do |v|
+      Verity.configure { |c| c.worker_count = v.strip }
+    end
+    o.on("--order MODE", "Test order: random (default; use --seed to fix RNG) or fingerprint (sorted).") do |v|
+      mode = v.to_s.strip.downcase
+      unless %w[fingerprint random].include?(mode)
+        warn "verity: --order must be fingerprint or random (got #{v.inspect})"
+        exit 2
+      end
+      Verity.configure { |c| c.test_order = mode.to_sym }
+    end
+    o.on("--seed INTEGER", Integer, "RNG seed for shuffled order (printed only when auto-chosen).") do |n|
+      Verity.configure { |c| c.shuffle_seed = n }
+    end
+    o.on("-h", "--help", "Show this help") do
+      puts o
+      exit 0
+    end
+  end.parse!
+
+  if ARGV.any?
+    globs = []
+    filters = []
+    ARGV.each do |raw|
+      if (m = raw.match(/\A(.+):(\d+)\z/))
+        path_part = m[1]
+        line_part = Integer(m[2], 10)
+        abs = File.expand_path(path_part)
+        unless File.file?(abs)
+          warn "verity: #{raw.inspect} — line filter requires an existing file (got #{abs.inspect})"
+          exit 2
+        end
+        globs << abs
+        filters << [abs, line_part]
+      else
+        globs << File.expand_path(raw)
+      end
+    end
+    Verity.configure do |c|
+      c.test_globs = globs.uniq
+      c.location_filters = filters unless filters.empty?
+    end
+  end
+rescue OptionParser::InvalidOption, OptionParser::MissingArgument => e
+  warn "verity: #{e.message}"
+  exit 2
+end
+
+begin
+  exit(Verity.run ? 0 : 1)
+rescue ArgumentError => e
+  warn "verity: #{e.message}"
+  exit 2
+end