pbt 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0a6f9001f79b18e4cb3699704355bb25cfae69720c9179f63e468617909ef813
-  data.tar.gz: d384534a54f566e83680b54bfbf42aebc0137ae1ec5dccc937c27f672bfd9abb
+  metadata.gz: 02cfb8777d4c14fdb39421dc64c3e4d3e23ec57f1e933cccaac0ef67b8e103b0
+  data.tar.gz: 6c30345dec4c52854f52c5a28c2894fb0d41b58d36ada5337f983323440ca9c5
 SHA512:
-  metadata.gz: 87827a542745ba10b1f086bf40221280499ed9844f9503f11b43a499badb0ab020ccd214e0b5d0bcb13e0efbe95a3d29fee2a758fd5a47d9d58b356400d48dd3
-  data.tar.gz: 3947e569a6f48802ca1490b4e2c6763f38e40d08528e0aed8185ec45b418efce41f7596b641da93efea2f7c553a16e3b132a54fe9b4d73d886456b5921d0869e
+  metadata.gz: a152aa19cd0f919b86216fe54457b50d7953aa96c93a302d0044ec7535177efbfa14104a20bd13e8fd2ad9aa9b7fc5cb10d6c1fbcf4e9ec9a0117c24b8bc2140
+  data.tar.gz: 9ba28731ba906fab360a1df6030d2f4c4d34a591ec70339f8d81fb36d2db427d6ca046849a005fff7ecfcfc4fa684869455f597e38acecd0dbdef2b5a3613904

data/.standard.yml

@@ -1,3 +1,3 @@
 # For available configuration options, see:
 #   https://github.com/standardrb/standard
-ruby_version: 3.3
+ruby_version: 3.1

data/CHANGELOG.md

@@ -1,5 +1,20 @@
 ## [Unreleased]
 
-## [0.1.0] - 2024-01-27
+## [0.1.0] - 2024-04-13
 
-- Initial release
+- Implement basic primitive arbitraries
+- Implement composite arbitraries
+- Support shrinking
+- Support multiple concurrency methods
+    - Ractor
+    - Process
+    - Thread
+    - None (Run tests sequentially)
+- Documentation
+    - Add better examples
+    - Arbitrary usage
+    - Configuration
+ 
+## [0.0.1] - 2024-01-27
+
+- Initial release (Proof of concept)

data/README.md

@@ -1,73 +1,318 @@
 # Property-Based Testing in Ruby
 
-A property-based testing tool for Ruby, utilizing Ractor for parallelizing test cases.
+[![Gem Version](https://badge.fury.io/rb/pbt.svg)](https://rubygems.org/gems/pbt)
+[![Build Status](https://github.com/ohbarye/pbt/actions/workflows/main.yml/badge.svg)](https://github.com/ohbarye/pbt/actions/workflows/main.yml)
+[![RubyDoc](https://img.shields.io/badge/%F0%9F%93%9ARubyDoc-documentation-informational.svg)](https://www.rubydoc.info/gems/pbt)
+
+An experimental property-based testing tool for Ruby that allows you to run test cases in parallel.
+
+PBT stands for Property-Based Testing.
+
+## What's Property-Based Testing?
+
+Property-Based Testing is a testing methodology that focuses on the properties a system should always satisfy, rather than checking individual examples. Instead of writing tests for predefined inputs and outputs, PBT allows you to specify the general characteristics that your code should adhere to and then automatically generates a wide range of inputs to verify these properties.
+
+The key benefits of property-based testing include the ability to cover more edge cases and the potential to discover bugs that traditional example-based tests might miss. It's particularly useful for identifying unexpected behaviors in your code by testing it against a vast set of inputs, including those you might not have considered.
+
+For a more in-depth understanding of Property-Based Testing, please refer to external resources.
+
+- Original ideas
+  - [Property-based testing of privileged programs](https://ieeexplore.ieee.org/document/367311) (1994)
+  - [Property-based testing: a new approach to testing for assurance](https://dl.acm.org/doi/abs/10.1145/263244.263267) (1997)
+  - [QuickCheck: a lightweight tool for random testing of Haskell programs](https://dl.acm.org/doi/10.1145/351240.351266) (2000)
+- Rather new introductory resources
+  - Fred Hebert's book [Property-Based Testing With PropEr, Erlang and Elixir](https://propertesting.com/).
+  - [fast-check - Why Property-Based?](https://fast-check.dev/docs/introduction/why-property-based/)
 
 ## Installation
 
-Install the gem and add to the application's Gemfile by executing:
+Add this line to your application's Gemfile and run `bundle install`.
 
-```shell
-$ bundle add pbt
+```ruby
+gem 'pbt'
 ```
 
-If bundler is not being used to manage dependencies, install the gem by executing:
+If you want to use multi-processes or multi-threads (other than Ractor) as workers to run tests, install the [parallel](https://github.com/grosser/parallel) gem.
 
-```shell
-$ gem install pbt
+```ruby
+gem 'parallel'
+```
+
+Off course you can install with `gem intstall pbt`.
+
+## Basic Usage
+
+### Simple property
+
+```ruby
+# Let's say you have a method that returns just a multiplicative inverse.
+def multiplicative_inverse(number)
+  Rational(1, number)
+end
+
+Pbt.assert do
+  # The given block is executed 100 times with different random numbers.
+  # Besides, the block runs in parallel by Ractor.
+  Pbt.property(Pbt.integer) do |number|
+    result = multiplicative_inverse(number)
+    raise "Result should be the multiplicative inverse of the number" if result * number != 1
+  end
+end
+
+# If the function has a bug, the test fails with a counterexample.
+# For example, the multiplicative_inverse method doesn't work for 0 regardless of the behavior is intended or not.
+#
+# Pbt::PropertyFailure:
+#   Property failed after 23 test(s)
+#   { seed: 11001296583699917659214176011685741769 }
+#   Counterexample: 0
+#   Shrunk 3 time(s)
+#   Got ZeroDivisionError: divided by 0
+```
+
+### Explain The Snippet
+
+The above snippet is very simple but contains the basic components.
+
+#### Runner
+
+`Pbt.assert` is the runner. The runner interprets and executes the given property. `Pbt.assert` takes a property and runs it multiple times. If the property fails, it tries to shrink the input that caused the failure.
+
+#### Property
+
+The snippet above declared a property by calling `Pbt.property`. The property describes the following:
+
+1. What the user wants to evaluate. This corresponds to the block (let's call this `predicate`) enclosed by `do` `end`
+2. How to generate inputs for the predicate — using `Arbitrary`
+
+The `predicate` block is a function that directly asserts, taking values generated by `Arbitrary` as input.
+
+#### Arbitrary
+
+Arbitrary generates random values. It is also responsible for shrinking those values if asked to shrink a failed value as input.
+
+Here, we used only one type of arbitrary, `Pbt.integer`. There are many other built-in arbitraries, and you can create a variety of inputs by combining existing ones.
+
+#### Shrink
+
+In PBT, If a test fails, it attempts to shrink the case that caused the failure into a form that is easier for humans to understand.
+In other words, instead of stopping the test itself the first time it fails and reporting the failed value, it tries to find the minimal value that causes the error.
+
+When there is a test that fails when given an even number, a counterexample of `2` is simpler and easier to understand than `432743417662`.
+
+### Arbitrary
+
+There are many built-in arbitraries in `Pbt`. You can use them to generate random values for your tests. Here are some representative arbitraries.
+
+#### Primitives
+
+```ruby
+rng = Random.new(
+
+Pbt.integer.generate(rng) # => 42
+Pbt.integer(min: -1, max: 8).generate(rng) # => Integer between -1 and 8
+
+Pbt.symbol.generate(rng) # => :atq
+
+Pbt.ascii_char.generate(rng) # => "a"
+Pbt.ascii_string.generate(rng) # => "aagjZfao"
+
+Pbt.boolean.generate(rng) # => true or false
+Pbt.constant(42).generate(rng) # => 42 always
+```
+
+#### Composites
+
+```ruby
+rng = Random.new
+
+Pbt.array(Pbt.integer).generate(rng) # => [121, -13141, 9825]
+Pbt.array(Pbt.integer, max: 1, empty: true).generate(rng) # => [] or [42] etc.
+
+Pbt.tuple(Pbt.symbol, Pbt.integer).generate(rng) # => [:atq, 42]
+
+Pbt.fixed_hash(x: Pbt.symbol, y: Pbt.integer).generate(rng) # => {x: :atq, y: 42}
+Pbt.hash(Pbt.symbol, Pbt.integer).generate(rng) # => {atq: 121, ygab: -1142}
+
+Pbt.one_of(:a, 1, 0.1).generate(rng) # => :a or 1 or 0.1
+````
+
+See [ArbitraryMethods](https://github.com/ohbarye/pbt/blob/main/lib/pbt/arbitrary/arbitrary_methods.rb) module for more details.
+
+## Configuration
+
+You can configure `Pbt` by calling `Pbt.configure` before running tests.
+
+```ruby
+Pbt.configure do |config|
+  # Whether to print verbose output. Default is `false`.
+  config.verbose = 100
+
+  # The concurrency method to use. :ractor`, `:thread`, `:process` and `:none` are supported. Default is `:ractor`.
+  config.worker = :ractor
+
+  # The number of runs to perform. Default is `100`.
+  config.num_runs = 100
+
+  # The seed to use for random number generation.
+  # It's useful to reproduce failed test with the seed you'd pick up from failure messages. Default is a random seed.
+  config.seed = 42
+
+  # Whether to report exceptions in threads.
+  # It's useful to suppress error logs on Ractor that reports many errors. Default is `false`.
+  config.thread_report_on_exception = false
+end
+```
+
+Or, you can pass the configuration to `Pbt.assert` as an argument.
+
+```ruby
+Pbt.assert(num_runs: 100, seed: 42) do
+  # ...
+end
+```
+
+## Concurrent methods
+
+One of the key features of `Pbt` is its ability to rapidly execute test cases in parallel or concurrently, using a large number of values (by default, `100`) generated by `Arbitrary`.
+
+For concurrent processing, you can specify any of the three workers—`:ractor`, `:process`, or `:thread`—using the `worker` option. Alternatively, choose `:none` for serial execution.
+
+`Pbt` supports 3 concurrency methods and 1 sequential one. You can choose one of them by setting the `worker` option.
+
+### Ractor
+
+```ruby
+Pbt.assert(worker: :ractor) do
+  Pbt.property(Pbt.integer) do |n|
+    # ...
+  end
+end
 ```
 
-## Usage
+#### Limitation
+
+Please note that Ractor support is an experimental feature of this gem. Due to Ractor's limitations, you may encounter some issues when using it.
+
+For example, you cannot access anything out of block.
 
 ```ruby
-# Let's say you have a method that returns just even numbers.
-def twice(number)
-  number * 2
+a = 1
+
+Pbt.assert(worker: :ractor) do
+  Pbt.property(Pbt.integer) do |n|
+    # You cannot access `a` here because this block is executed in a Ractor and it doesn't allow implicit sharing of objects.
+    a + n # => Ractor::RemoteError (can not share object between ractors)
+  end
 end
+```
 
-RSpec.describe Pbt do
-  it "works" do
-    # The given block is executed 100 times with different random numbers.
-    # Besides, the block runs in parallel by Ractor.
-    Pbt.forall(Pbt::Generator.integer) do |number|
-      result = twice(number)
-      raise "Result should be even number" if result % 2 != 0
+You cannot use any methods provided by test frameworks like `expect` or `assert` because they are not available in a Ractor.
+
+```ruby
+it do
+  Pbt.assert(worker: :ractor) do
+    Pbt.property(Pbt.integer) do |n|
+      # This is not possible because `self` if a Ractor here.
+      expect(n).to be_an(Integer) # => Ractor::RemoteError (cause by NoMethodError for `expect` or `be_an`)
     end
-    
-    # If the function has a bug, the test fails with a counterexample.
-    # Pbt::CaseFailure:
-    #   RuntimeError:
-    #   Failed on:
-    #            0.5
+  end
+end
+```
+
+### Process
+
+```ruby
+Pbt.assert(worker: :process) do
+  Pbt.property(Pbt.integer) do |n|
+    # ...
+  end
+end
+```
+
+### Thread
+
+```ruby
+Pbt.assert(worker: :thread) do
+  Pbt.property(Pbt.integer) do |n|
+    # ...
+  end
+end
+```
+
+### None
+
+```ruby
+Pbt.assert(worker: :none) do
+  Pbt.property(Pbt.integer) do |n|
+    # ...
   end
 end
 ```
 
 ## TODOs
 
-- [ ] More generators
-  - https://proper-testing.github.io/apidocs/
-- [ ] Enable to combine generators
-  - e.g. `Pbt::Generator.list(Pbt::Generator.integer)`
-- [ ] More sophisticated syntax for property-based testing
-  - e.g. `forall(integer) { |number| ... }` (Omit `Pbt` module)
-- [ ] Support for shrinking
-- [ ] Allow to use assertions
-  - It's hard to pass assertions like `expect`, `assert` to a Ractor?
+Once this project finishes the following, we will release v1.0.0.
+
+- [x] Implement basic primitive arbitraries
+- [x] Implement composite arbitraries
+- [x] Support shrinking
+- [x] Support multiple concurrency methods
+  - [x] Ractor
+  - [x] Process
+  - [x] Thread
+  - [x] None (Run tests sequentially)
+- [x] Documentation
+  - [x] Add better examples
+  - [x] Arbitrary usage
+  - [x] Configuration
+- [ ] Rich report like verbose mode
+- [ ] Allow to use expectations and matchers provided by test framework in Ractor if possible.
+  - It'd be so hard to pass assertions like `expect`, `assert` to a Ractor.
+- [ ] Benchmark
+- [ ] More parallelism or faster execution if possible
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+### Setup
+
+```shell
+bin/setup
+bundle exec rake # Run tests and lint at once
+```
+
+### Test
+
+```shell
+bundle exec rspec
+```
+
+### Lint
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+```shell
+bundle exec rake standard:fix
+```
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/pbt. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/pbt/blob/master/CODE_OF_CONDUCT.md).
+Bug reports and pull requests are welcome on GitHub at https://github.com/ohbarye/pbt. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/ohbarye/pbt/blob/master/CODE_OF_CONDUCT.md).
 
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
+## Credits
+
+This project draws a lot of inspiration from other testing tools, namely
+
+- [fast-check](https://fast-check.dev/)
+- [Loupe](https://github.com/vinistock/loupe)
+- [RSpec](https://github.com/rspec/rspec)
+- [Minitest](https://github.com/seattlerb/minitest)
+- [Parallel](https://github.com/grosser/parallel)
+- [PropCheck for Ruby](https://github.com/Qqwy/ruby-prop_check)
+- [PropCheck for Elixir](https://github.com/alfert/propcheck)
+
 ## Code of Conduct
 
-Everyone interacting in the Pbt project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/pbt/blob/master/CODE_OF_CONDUCT.md).
+Everyone interacting in the Pbt project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/ohbarye/pbt/blob/master/CODE_OF_CONDUCT.md).

data/lib/pbt/arbitrary/arbitrary.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+module Pbt
+  module Arbitrary
+    # Abstract class for generating random values on type `T`.
+    #
+    # @abstract
+    class Arbitrary
+      # Generate a value of type `T`, based on the provided random number generator.
+      #
+      # @abstract
+      # @param rng [Random] Random number generator.
+      # @return [Object] Random value of type `T`.
+      def generate(rng)
+        raise NotImplementedError
+      end
+
+      # Shrink a value of type `T`.
+      # Must never be called with possibly invalid values.
+      #
+      # @abstract
+      # @param current [Object]
+      # @return [Enumerator<Object>]
+      def shrink(current)
+        raise NotImplementedError
+      end
+
+      # Create another arbitrary by applying `mapper` value by value.
+      #
+      # @example
+      #   integer_generator = Pbt.integer
+      #   num_str_generator = integer_arb.map(->(n){ n.to_s }, ->(s) {s.to_i})
+      #
+      # @param mapper [Proc] Proc to map generated values. Mainly used for generation.
+      # @param unmapper [Proc] Proc to unmap generated values. Used for shrinking.
+      # @return [MapArbitrary] New arbitrary with mapped elements
+      def map(mapper, unmapper)
+        MapArbitrary.new(self, mapper, unmapper)
+      end
+
+      # Create another arbitrary by filtering values against `refinement`.
+      # All the values produced by the resulting arbitrary satisfy `!!refinement(value) == true`.
+      #
+      # Be aware that using `filter` may reduce possible valid values and may impact the time required to generate a valid value.
+      #
+      # @example
+      #   integer_generator = Pbt.integer
+      #   even_integer_generator = integer_arb.filter { |x| x.even? }
+      #   # or `integer_arb.filter(&:even?)`
+      #
+      # @param refinement [Proc] Predicate proc to test each produced element. Return true to keep the element, false otherwise.
+      # @return [FilterArbitrary] New arbitrary filtered using `refinement`.
+      def filter(&refinement)
+        FilterArbitrary.new(self, &refinement)
+      end
+    end
+  end
+end