smartest 0.1.0 → 0.2.0.alpha2

data/SMARTEST_DESIGN.md

@@ -50,7 +50,10 @@ class WebFixture < Smartest::Fixture
   end
 end
 
-use_fixture WebFixture
+around_suite do |suite|
+  use_fixture WebFixture
+  suite.run
+end
 ```
 
 The core decision is:
@@ -294,7 +297,10 @@ context.instance_exec(**fixtures, &block)
 
 This keeps the top-level DSL small.
 
-Only `test`, `fixture`, `use_fixture`, and `use_matcher` need to be globally available when using `smartest/autorun`.
+Only `test`, `around_suite`, and `around_test` should be globally available when
+using `smartest/autorun`. `use_fixture` and `use_matcher` are available only
+inside hook execution contexts. `skip` and `pending` are available only inside
+test bodies and `around_test` hook execution contexts.
 
 ## Core architecture
 
@@ -543,8 +549,11 @@ class AdminFixture < Smartest::Fixture
   end
 end
 
-use_fixture UserFixture
-use_fixture AdminFixture
+around_suite do |suite|
+  use_fixture UserFixture
+  use_fixture AdminFixture
+  suite.run
+end
 ```
 
 Error:
@@ -683,6 +692,8 @@ end
 Fixture blocks can call private methods because they execute with `instance_exec`.
 
 Fixture classes may optionally delegate missing methods to the execution context.
+They should not delegate `skip` or `pending`; those are test-body and
+`around_test` controls, not fixture APIs.
 
 This is useful for integration helpers.
 
@@ -823,28 +834,38 @@ Care must be taken not to run twice if both CLI and autorun are used.
 
 ## Exit status
 
-- all tests passed: `0`
+- all tests passed, skipped, or pending as expected: `0`
 - any test failed: `1`
+- pending test unexpectedly passed: `1`
 - configuration/load error: `1`
 - interrupted: re-raise or exit non-zero
 
 ## Metadata
 
-`test` should accept metadata:
+`test` accepts metadata and stores it on `TestCase`, but metadata does not drive
+runner behavior in the MVP:
 
 ```ruby
-test("name", skip: true) do
-end
-
 test("name", tags: [:db]) do
 end
 ```
 
-MVP can store metadata without implementing all behavior.
+Runtime skipping and pending behavior are method calls inside the test body or
+`around_test`, not metadata:
+
+```ruby
+test("name") do
+  skip "reason" if runtime_condition?
+end
+
+test("name") do
+  pending "reason" if expected_to_fail?
+  expect(actual).to eq(expected)
+end
+```
 
 Useful metadata later:
 
-- `skip: true`
 - `only: true`
 - `tags: [:db]`
 - `timeout: 5`
@@ -853,7 +874,70 @@ Useful metadata later:
 
 Hooks are separate from fixtures.
 
-Potential API:
+Supported suite API:
+
+```ruby
+around_suite do |suite|
+  Async do
+    suite.run
+  end
+end
+```
+
+`around_suite` wraps the full suite body, including all tests and suite fixture
+cleanup. The hook receives a run target and must call `suite.run` exactly once.
+Multiple hooks compose in registration order, with the first hook as the
+outermost wrapper.
+
+Supported per-test API:
+
+```ruby
+around_test do |test|
+  SomeAutoCloseResource.new do
+    test.run
+  end
+end
+```
+
+When `around_test` is written directly in a test file, Smartest treats it as
+file-scoped by snapshotting the file's current around-test hooks into each
+`TestCase` at test registration time. Hooks defined later in the file apply only
+to later tests.
+
+`around_test` can also be registered inside `around_suite`. In that case the hook
+is suite-wide and applies when `suite.run` executes:
+
+```ruby
+around_suite do |suite|
+  around_test do |test|
+    TestServer.run do
+      test.run
+    end
+  end
+
+  suite.run
+end
+```
+
+`use_fixture` and `use_matcher` are not top-level DSL methods. They are available
+inside `around_suite` and `around_test` contexts:
+
+```ruby
+around_test do |test|
+  use_fixture LocalFixture
+  use_matcher LocalMatcher
+  test.run
+end
+```
+
+For `around_test`, those registrations are test-run local and must happen before
+`test.run`.
+
+Fixture classes registered from `around_test` must not define `suite_fixture`.
+Suite-scoped fixtures need suite-level cache and cleanup ownership, so classes
+with suite-scoped fixtures must be registered from `around_suite`.
+
+Potential simpler per-test API:
 
 ```ruby
 before do
@@ -889,10 +973,29 @@ fixture cleanup
 
 This needs a final decision later.
 
-For MVP, hooks can be omitted.
-
 Fixture cleanup already handles resource-specific teardown.
 
+### Around-test parallelism note
+
+The file-scoped `around_test` design is intended to remain compatible with
+future parallel execution if it is hardened in these directions:
+
+- keep registration protected by a `Mutex` when tests may be loaded from
+  multiple threads
+- use copy-on-write arrays for `around_test` hooks by file
+- snapshot the current file-local hook stack into each `TestCase` at registration
+  time
+- treat `TestCase` as immutable after registration
+- expose `use_fixture` and `use_matcher` through a per-run `TestRun` object, not
+  by mutating the global suite registry during test execution
+- snapshot the suite's fixture classes and matcher modules before a test starts
+- keep test-run fixture and matcher additions local to that run
+
+That shape lets multiple tests execute concurrently by reading immutable
+`TestCase` state and using per-test fixture/matcher registries. The current
+implementation can be simpler, but it should not rely on refinements or global
+method rewriting for file-local behavior.
+
 ## Scoping
 
 Fixtures are test-scoped by default.
@@ -1073,6 +1176,11 @@ require "smartest/autorun"
 Dir[File.join(__dir__, "fixtures", "**", "*.rb")].sort.each do |fixture_file|
   require fixture_file
 end
+
+around_suite do |suite|
+  use_fixture AppFixture
+  suite.run
+end
 ```
 
 ```ruby
@@ -1098,18 +1206,30 @@ end
 # smartest/example_test.rb
 require "test_helper"
 
-use_fixture AppFixture
-
 test("GET /health") do |client:|
   expect(client.get("/health").status).to eq(200)
 end
 ```
 
+```ruby
+test("PDF export") do |browser:|
+  skip "firefox is not supported" if browser.firefox?
+
+  export_pdf(browser)
+end
+
+test("PDF export over BiDi") do |browser:|
+  pending "Not supported by WebDriver BiDi yet" if browser.bidi?
+
+  export_pdf(browser)
+  expect(browser.downloads).not_to be_empty
+end
+```
+
 ## Future ideas
 
 Possible future features:
 
-- `skip`
 - `only`
 - tags
 - custom reporters
@@ -1117,7 +1237,6 @@ Possible future features:
 - richer matchers
 - block expectations
 - `raise_error`
-- hooks
 - file-scoped fixtures
 - parallel execution
 - watch mode

data/lib/smartest/dsl.rb

@@ -3,24 +3,31 @@
 module Smartest
   module DSL
     def test(name, **metadata, &block)
+      location = caller_locations(1, 1).first
+
       Smartest.suite.tests.add(
         TestCase.new(
           name: name,
           metadata: metadata,
           block: block,
-          location: caller_locations(1, 1).first
+          location: location,
+          around_test_hooks: Smartest.suite.around_test_hooks_for(location)
         )
       )
     end
 
-    def use_fixture(klass)
-      Smartest.suite.fixture_classes.add(klass)
+    def around_suite(&block)
+      raise ArgumentError, "around_suite block is required" unless block
+
+      Smartest.suite.around_suite_hooks << block
     end
 
-    def use_matcher(matcher_module)
-      Smartest.suite.matcher_modules.add(matcher_module)
+    def around_test(&block)
+      raise ArgumentError, "around_test block is required" unless block
+
+      Smartest.suite.add_around_test_hook(caller_locations(1, 1).first, block)
     end
 
-    private :test, :use_fixture, :use_matcher
+    private :test, :around_suite, :around_test
   end
 end

data/lib/smartest/errors.rb

@@ -1,6 +1,15 @@
 # frozen_string_literal: true
 
 module Smartest
+  module StatusReason
+    DEFAULT_REASON = "No reason given"
+
+    def self.normalize(reason)
+      text = reason.to_s
+      text.empty? ? DEFAULT_REASON : text
+    end
+  end
+
   class Error < StandardError; end
 
   class FixtureNotFoundError < Error
@@ -48,5 +57,36 @@ module Smartest
 
   class InvalidFixtureParameterError < Error; end
 
+  class AroundSuiteRunError < Error; end
+
+  class AroundTestFixtureScopeError < Error
+    def initialize(fixture_class, fixture_names)
+      class_name = fixture_class.name || fixture_class.inspect
+      names = fixture_names.map { |fixture_name| ":#{fixture_name}" }.join(", ")
+
+      super(
+        "#{class_name} cannot be registered from around_test because it defines suite-scoped fixtures: #{names}. " \
+        "Register fixture classes with suite_fixture from around_suite instead."
+      )
+    end
+  end
+
+  class AroundTestRunError < Error; end
+
   class AssertionFailed < Error; end
+
+  class Skipped < Error
+    attr_reader :reason
+
+    def initialize(reason = nil)
+      @reason = StatusReason.normalize(reason)
+      super(@reason)
+    end
+  end
+
+  class PendingPassedError < AssertionFailed
+    def initialize(reason = nil)
+      super("expected pending test to fail, but it passed: #{StatusReason.normalize(reason)}")
+    end
+  end
 end

data/lib/smartest/execution_context.rb

@@ -4,5 +4,19 @@ module Smartest
   class ExecutionContext
     include Expectations
     include Matchers
+
+    def initialize(run_state: TestRunState.new)
+      @run_state = run_state
+    end
+
+    private
+
+    def skip(reason = nil)
+      raise Skipped, reason
+    end
+
+    def pending(reason = nil)
+      @run_state.pending(reason)
+    end
   end
 end

data/lib/smartest/fixture.rb

@@ -2,6 +2,8 @@
 
 module Smartest
   class Fixture
+    RESERVED_CONTEXT_METHODS = %i[skip pending].freeze
+
     class << self
       def fixture(name, scope: :test, &block)
         define_fixture(
@@ -64,6 +66,8 @@ module Smartest
     end
 
     def method_missing(method_name, *args, &block)
+      return super if RESERVED_CONTEXT_METHODS.include?(method_name)
+
       if @context.respond_to?(method_name, true)
         @context.__send__(method_name, *args, &block)
       else
@@ -72,6 +76,8 @@ module Smartest
     end
 
     def respond_to_missing?(method_name, include_private = false)
+      return super if RESERVED_CONTEXT_METHODS.include?(method_name)
+
       @context.respond_to?(method_name, true) || super
     end
   end

data/lib/smartest/hook_contexts.rb

@@ -0,0 +1,60 @@
+# frozen_string_literal: true
+
+module Smartest
+  class AroundSuiteContext
+    def initialize(suite)
+      @suite = suite
+    end
+
+    def call(hook, suite_run)
+      @suite.around_suite_hook do
+        instance_exec(suite_run, &hook)
+      end
+    end
+
+    private
+
+    def use_fixture(klass)
+      @suite.fixture_classes.add(klass)
+    end
+
+    def use_matcher(matcher_module)
+      @suite.matcher_modules.add(matcher_module)
+    end
+
+    def around_test(&block)
+      raise ArgumentError, "around_test block is required" unless block
+
+      @suite.add_around_test_hook(caller_locations(1, 1).first, block)
+    end
+  end
+
+  class AroundTestContext
+    def initialize(test_run, run_state:)
+      @test_run = test_run
+      @run_state = run_state
+    end
+
+    def call(hook, run_target = @test_run)
+      instance_exec(run_target, &hook)
+    end
+
+    private
+
+    def use_fixture(klass)
+      @test_run.add_fixture_class(klass)
+    end
+
+    def use_matcher(matcher_module)
+      @test_run.add_matcher_module(matcher_module)
+    end
+
+    def skip(reason = nil)
+      raise Skipped, reason
+    end
+
+    def pending(reason = nil)
+      @run_state.pending(reason)
+    end
+  end
+end

data/lib/smartest/init_generator.rb

@@ -18,7 +18,10 @@ module Smartest
           require matcher_file
         end
 
-        use_matcher PredicateMatcher
+        around_suite do |suite|
+          use_matcher PredicateMatcher
+          suite.run
+        end
       RUBY
       "smartest/matchers/predicate_matcher.rb" => <<~RUBY,
         # frozen_string_literal: true

data/lib/smartest/reporter.rb

@@ -4,6 +4,8 @@ module Smartest
   class Reporter
     PASS_MARK = "\u2713"
     FAIL_MARK = "\u2717"
+    SKIP_MARK = "-"
+    PENDING_MARK = "*"
 
     def initialize(io = $stdout)
       @io = io
@@ -15,18 +17,26 @@ module Smartest
     end
 
     def record(result)
-      mark = result.passed? ? PASS_MARK : FAIL_MARK
-      @io.puts "#{mark} #{result.test_case.name}"
+      @io.puts record_line(result)
     end
 
-    def finish(results, suite_cleanup_errors: [])
+    def finish(results, suite_cleanup_errors: [], suite_errors: [])
       failures = results.select(&:failed?)
+      skipped = results.select(&:skipped?)
+      pending = results.select(&:pending?)
 
       report_failures(failures) if failures.any?
+      report_suite_errors(suite_errors) if suite_errors.any?
       report_suite_cleanup_errors(suite_cleanup_errors) if suite_cleanup_errors.any?
 
       @io.puts
       summary = "#{results.count} #{results.count == 1 ? 'test' : 'tests'}, #{results.count(&:passed?)} passed, #{failures.count} failed"
+      summary = "#{summary}, #{skipped.count} skipped" if skipped.any?
+      summary = "#{summary}, #{pending.count} pending" if pending.any?
+      if suite_errors.any?
+        suite_label = suite_errors.count == 1 ? "suite failure" : "suite failures"
+        summary = "#{summary}, #{suite_errors.count} #{suite_label}"
+      end
       if suite_cleanup_errors.any?
         cleanup_label = suite_cleanup_errors.count == 1 ? "suite cleanup" : "suite cleanups"
         summary = "#{summary}, #{suite_cleanup_errors.count} #{cleanup_label} failed"
@@ -36,6 +46,21 @@ module Smartest
 
     private
 
+    def record_line(result)
+      case result.status
+      when :passed
+        "#{PASS_MARK} #{result.test_case.name}"
+      when :failed
+        "#{FAIL_MARK} #{result.test_case.name}"
+      when :skipped
+        "#{SKIP_MARK} #{result.test_case.name} (skipped: #{result.reason})"
+      when :pending
+        "#{PENDING_MARK} #{result.test_case.name} (pending: #{result.reason})"
+      else
+        "#{FAIL_MARK} #{result.test_case.name}"
+      end
+    end
+
     def report_failures(failures)
       @io.puts
       @io.puts "Failures:"
@@ -50,6 +75,18 @@ module Smartest
       end
     end
 
+    def report_suite_errors(errors)
+      @io.puts
+      @io.puts "Suite failures:"
+      @io.puts
+
+      errors.each_with_index do |error, index|
+        @io.puts "#{index + 1}) suite"
+        report_error(error)
+        @io.puts
+      end
+    end
+
     def report_suite_cleanup_errors(errors)
       @io.puts
       @io.puts "Suite cleanup failures:"

data/lib/smartest/runner.rb

@@ -11,10 +11,33 @@ module Smartest
     def run
       results = []
       suite_cleanup_errors = []
+      suite_errors = []
       @suite_fixture_set = nil
 
       @reporter.start(@tests.count)
 
+      begin
+        run_around_suite_hooks(@suite.around_suite_hooks.dup) do
+          run_tests(results, suite_cleanup_errors)
+        end
+      rescue Exception => error
+        raise if Smartest.fatal_exception?(error)
+
+        suite_errors << error
+      end
+
+      @reporter.finish(
+        results,
+        suite_cleanup_errors: suite_cleanup_errors,
+        suite_errors: suite_errors
+      )
+
+      results.any?(&:failed?) || suite_cleanup_errors.any? || suite_errors.any? ? 1 : 0
+    end
+
+    private
+
+    def run_tests(results, suite_cleanup_errors)
       begin
         @tests.each do |test_case|
           result = run_one(test_case)
@@ -22,38 +45,63 @@ module Smartest
           @reporter.record(result)
         end
       ensure
-        suite_cleanup_errors = @suite_fixture_set.run_cleanups if @suite_fixture_set
+        suite_cleanup_errors.concat(@suite_fixture_set.run_cleanups) if @suite_fixture_set
         @suite_fixture_set = nil
       end
+    end
 
-      @reporter.finish(results, suite_cleanup_errors: suite_cleanup_errors)
+    def run_around_suite_hooks(hooks, index = 0, &block)
+      return yield if index >= hooks.length
 
-      results.any?(&:failed?) || suite_cleanup_errors.any? ? 1 : 0
-    end
+      hook = hooks[index]
+      suite_run = SuiteRun.new do
+        run_around_suite_hooks(hooks, index + 1, &block)
+      end
 
-    private
+      AroundSuiteContext.new(@suite).call(hook, suite_run)
+      raise AroundSuiteRunError, "around_suite hook did not call suite.run" unless suite_run.ran?
+
+      suite_run.result
+    end
 
     def run_one(test_case)
       started_at = now
-      context = build_context
-      fixture_set = nil
       error = nil
+      skipped = nil
       cleanup_errors = []
+      run_state = TestRunState.new
+      test_run = TestRun.new(
+        fixture_classes: @suite.fixture_classes,
+        matcher_modules: @suite.matcher_modules
+      ) do |fixture_classes:, matcher_modules:|
+        run_test_body(test_case, fixture_classes, matcher_modules, run_state, cleanup_errors)
+      end
 
       begin
-        fixture_set = FixtureSet.new(@suite.fixture_classes, context: context, parent: suite_fixture_set)
-        fixtures = fixture_set.resolve_keywords(test_case.fixture_names)
-        context.instance_exec(**fixtures, &test_case.block)
+        run_around_test_hooks(@suite.around_test_hooks + test_case.around_test_hooks, test_run, run_state)
+      rescue Skipped => skipped_error
+        skipped = skipped_error
       rescue Exception => rescued_error
         raise if Smartest.fatal_exception?(rescued_error)
 
         error = rescued_error
-      ensure
-        cleanup_errors = fixture_set.run_cleanups if fixture_set
       end
 
       duration = now - started_at
 
+      return TestResult.failed(test_case: test_case, error: nil, duration: duration, cleanup_errors: cleanup_errors) if skipped && cleanup_errors.any?
+      return TestResult.skipped(test_case: test_case, reason: skipped.reason, duration: duration) if skipped
+
+      if run_state.pending?
+        if error && !around_test_protocol_error?(error)
+          return TestResult.failed(test_case: test_case, error: nil, duration: duration, cleanup_errors: cleanup_errors) if cleanup_errors.any?
+
+          return TestResult.pending(test_case: test_case, reason: run_state.pending_reason, duration: duration)
+        end
+
+        error ||= PendingPassedError.new(run_state.pending_reason)
+      end
+
       if error || cleanup_errors.any?
         TestResult.failed(
           test_case: test_case,
@@ -66,6 +114,36 @@ module Smartest
       end
     end
 
+    def run_test_body(test_case, fixture_classes, matcher_modules, run_state, cleanup_errors)
+      context = build_context(matcher_modules, run_state)
+      fixture_set = nil
+
+      begin
+        fixture_set = FixtureSet.new(fixture_classes, context: context, parent: suite_fixture_set)
+        fixtures = fixture_set.resolve_keywords(test_case.fixture_names)
+        context.instance_exec(**fixtures, &test_case.block)
+      ensure
+        cleanup_errors.concat(fixture_set.run_cleanups) if fixture_set
+      end
+    end
+
+    def run_around_test_hooks(hooks, test_run, run_state, index = 0)
+      return test_run.run if index >= hooks.length
+
+      hook = hooks[index]
+      next_run = TestRun.new(
+        fixture_classes: [],
+        matcher_modules: []
+      ) do |**_keywords|
+        run_around_test_hooks(hooks, test_run, run_state, index + 1)
+      end
+
+      AroundTestContext.new(test_run, run_state: run_state).call(hook, next_run)
+      raise AroundTestRunError, "around_test hook did not call test.run" unless next_run.ran?
+
+      next_run.result
+    end
+
     def suite_fixture_set
       @suite_fixture_set ||= FixtureSet.new(
         @suite.fixture_classes,
@@ -74,12 +152,16 @@ module Smartest
       )
     end
 
-    def build_context
-      ExecutionContext.new.tap do |context|
-        @suite.matcher_modules.each { |matcher_module| context.extend(matcher_module) }
+    def build_context(matcher_modules = @suite.matcher_modules, run_state = TestRunState.new)
+      ExecutionContext.new(run_state: run_state).tap do |context|
+        matcher_modules.each { |matcher_module| context.extend(matcher_module) }
       end
     end
 
+    def around_test_protocol_error?(error)
+      error.is_a?(AroundTestRunError)
+    end
+
     def now
       Process.clock_gettime(Process::CLOCK_MONOTONIC)
     end