webspicy 0.16.1 → 0.19.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 445a2cc26015bc9a25efcbef22a2611aedc4d6fa5ff57fdbeac0aeed8447cdb6
-  data.tar.gz: 94e713312855382b55b55d6a1709635900426bc8d9929fefa22d79207cbb5ae0
+  metadata.gz: 614e5904931375bd6f4df435d33736cde757e752d0d759fbc3af4e185b9cd9e4
+  data.tar.gz: 8d63e07e93809bd5139300e5206f0a3b3204e86d77faea60a29302e5787413d1
 SHA512:
-  metadata.gz: 33afca9646dec9268ed8fc5581f9a10cd73fce7cd170b8f999bb43d6c66e94f1decc9161803134d7885d5ee27a1f4d262511faaed7ffb3b9da4ba92e93b95ee2
-  data.tar.gz: e11ab7abb09d9a8796e9946b2424b9bd11c46e82f9ab43219fdfec2ba969580fa5a6d5e89d202d17494f9f7d659990efdd8428f0a730770d5d1ccccbc79996f1
+  metadata.gz: 33d6a69e097e14a90f8b69edb4833a4a1325cec38513bf8f88d017b31847e08b310acce50dc7fb3d3cbdeefdff76b3497b489adbc2e965c7e8b12f0228a63524
+  data.tar.gz: 0d6cf55d54a6bb94b48511f9c178a1e5e673772749784b3c7c720d9413670dc1e93029cfa0687efd8fced6f91eb699ea5cae4ab15e40a674dea7843712dec114

data/README.md

@@ -1,11 +1,13 @@
+[![Build Status](https://travis-ci.com/enspirit/webspicy.svg?branch=master)](https://travis-ci.com/enspirit/webspicy)
 # Webspicy
 
 A specification and test framework for web services seen as black-box software
-operations. Webspicy yields a better test coverage for a smaller testing effort,
-because software quality matters.
+operations. Webspicy yields a better test coverage for a smaller testing effort.
 
 See webspicy in action and make the tutorial on https://yourbackendisbroken.dev
 
+Have a look at `doc/*.md` for vocabulary and vision as well as `ROADMAP.md`.
+
 ## Features
 
 * Declarative specification of HTTP web services + their tests
@@ -19,7 +21,7 @@ See webspicy in action and make the tutorial on https://yourbackendisbroken.dev
   your API design better.
 
 * Formal and human-friendly data schema with strong data matching semantics,
-  thanks to finitio.io
+  thanks to [http://finitio.io](http://finitio.io)
 
 * Test instrumentation and generation, based on PRE & POST contracts.
 
@@ -30,6 +32,10 @@ See webspicy in action and make the tutorial on https://yourbackendisbroken.dev
 * Extra goodies: when a specification is written, it can also be used for
   mocking the API, generating an openapi file, etc.
 
+## Is this used on real-world cases?
+
+Yes, `webspicy` is currently used on a dozen production components. Our biggest specification has 324 specification files for thousands of tests, 35% of them being generated.
+
 ## Getting started with the commandline
 
 To install webspicy on your developer computer, install ruby then:
@@ -60,7 +66,7 @@ to use our `:tester` docker image. Just mount your test suite as a volume
 in `/home/app` and you are good to go:
 
 ```
-docker run -v path/to/tests:/formalspec enspirit/webspicy-tester
+docker run -v path/to/tests:/formalspec enspirit/webspicy:tester
 ```
 
 If your plan is to test a backend that runs on your own machine (vs.
@@ -69,7 +75,7 @@ you will need to add some networking option, as shown below. Please
 refer to Docker documentation.
 
 ```
-docker run -v path/to/tests:/formalspec --network=host enspirit/webspicy-tester
+docker run -v path/to/tests:/formalspec --network=host enspirit/webspicy:tester
 ```
 
 ## Contributing

data/bin/webspicy

@@ -9,6 +9,7 @@
 #/
 #/ Options:
 #/ -h, --help       Show this help message
+#/ -k, --insecure   Allow insecure server connections when using SSL
 #/ -v, --version    Show webspicy version
 #/     --debug      Same as LOG_LEVEL=DEBUG, takes precedence
 #/
@@ -19,6 +20,7 @@
 #/ - METHOD=GET|POST|DELETE...    execute only tests matching the HTTP verb
 #/ - TAG=...                      execute only tests matching the given tag
 #/ - FAILFAST=yes|no              stop executing tests on first failure
+#/ - INSECURE=yes|no              allow insecure server connections when using SSL
 #/
 require 'webspicy'
 require 'webspicy/tester'
@@ -37,6 +39,9 @@ ARGV.options do |opts|
     puts "webspicy v#{Webspicy::VERSION}, (c) Enspirit SRL"
     exit(0)
   }
+  opts.on('-k', '--insecure') {
+    ENV['INSECURE'] = 'yes'
+  }
   opts.on("--debug") {
     ENV['LOG_LEVEL'] = 'DEBUG'
     Webspicy::LOGGER.level = Logger.const_get(ENV['LOG_LEVEL'])
@@ -51,4 +56,4 @@ end
 
 config = Webspicy::Configuration.dress(ARGV[0])
 res = Webspicy::Tester.new(config).call
-abort("Some tests failed") unless res == 0
+abort("#{res} errors occured") unless res == 0

data/doc/1-black-box-scene.md

@@ -0,0 +1,109 @@
+# Setting the black-box scene
+
+This document sets the black-box testing & specification scene. We start with the testing vocabulary (that is probably well known) and gradually introduce specification concepts (that are a bit less).
+
+## Test object
+
+The `Test object` is the software under test, generally a backend component exposing web services. As `webspicy` is also used for documentation & specification, we sometimes call it the `Specification object` or `Documentation object`.
+
+The test object is seen as a black-box: its specification and testing are in terms of observable behavior, not driven by the internal structure or implementation.
+
+## Test item
+
+A `Test item` (resp. `Specification item`) is an individual element to be tested (resp. specified). `webspicy` is generally used to test web services invoked thanks to HTTP. If we abstract from HTTP details - and see it as the request/response invocation protocol only - test items are high-level software operations called with input/output data.
+
+## Test case
+
+A `Test case` is a complete setting for invoking a test item, together with assertions on the result. The setting includes a `Current system state` and a `Request` (that contains `Input` data). The result covers the `Resulting system state`, and a `Response` (that contains `Output` data).
+
+## Test suite
+
+A `Test suite` of an object is the set of all its test cases.
+
+## System state
+
+Most of the time we are interested in (deterministic) *stateful* testing. That is, we consider that the behavior of a test item is not only influenced by the `Input` data but also by the current system state.
+
+The `System state` covers the content of all stateful components the test object interacts with in its environment: files, databases, caches, buses, etc.
+
+For effective testing, one needs to be able to control and check the system state easily.
+
+## Current system state
+
+A `Current system state` is the state of the system just before invoking a test item.
+
+## Resulting system state
+
+A `Resulting system state` is the state of the system just after invoking a test item from an current state.
+
+## Request
+
+A `Request` is what is sent to a test object to invoke a test item. When test items are web services, it is a HTTP request specifically.
+
+## Response
+
+A `Response` is what is received from the test object in return from a test item invocation request. When test items are web services, it is a HTTP response specifically.
+
+## Input
+
+The `Input` is the data on which a high-level operation is executed. In a similar way that a high-level operation abstracts from web services details, the input abstracts from HTTP request details.
+
+## Output
+
+The `Output` is the data returned by a high-level operation after execution. In a similar way that a high-level operation abstracts from web services details, the output abstracts from HTTP response details.
+
+## Precondition
+
+A `Precondition` is a necessary condition to be met for the test item to execute successfully when invoked. A precondition can refer to the current state, the request, and/or the input.
+
+The set of preconditions of an item is sometimes written `PRE`.
+
+## Postcondition
+
+A `Postcondition` is a condition that is guaranteed by the test item (if implemented correctly) when invoked while its preconditions holds. A postcondition can refer to the current state (before execution), the request,  the input, the resulting state, the response, and/or the output.
+
+## Input schema
+
+The `Input schema` is a formal definition of the set of valid input data of a test item (hence the underlying high-level operation). The input schema is a `webspicy` formal shortcut for a precondition stating that "input data must be valid, that is ...".
+
+(The input schema can also be seen as the signature of the high-level operation.)
+
+## Output schema
+
+The `Output schema` is a formal definition of the set of possible output data of a test item (idem). The output schema is a `webspicy` formal shortcut for a postcondition stating that "output data is valid, that is ...".
+
+(The output schema can also be seen as the return type of the high-level operation.)
+
+## Ideal specification
+
+An `Ideal specification` is a formal identification of an item, together with the set of its pre and post conditions. By nature the ideal specification covers the input and output schema.
+
+## Example
+
+An `Example` is a positive test case. That is, it is the description of an invocation setting that meets all preconditions of a test item together with assertions on the successful result.
+
+## Counterexample
+
+A `Counterexample` is a negative test case. That is, it is the description of an invocation setting that violates at least one precondition, together with assertions on the error result.
+
+## Errcondition
+
+An `Errcondition` is a condition that is guaranteed by the test item (if implemented correctly) when invoked while at least one precondition is violated. An errcondition can refer to the current state (before execution), the request, the input, the resulting state, the response, and/or the output.
+
+An item is said to be robust when it has strong errconditions: it guarantees some behavior and results even in case of execution failures.
+
+## Error schema
+
+The `Error schema` is a formal definition of the set of possible output data of a test item when it fails to execute successfully, typically because a precondition is violated. The error schema is a `webspicy` formal shortcut for an errcondition stating that "error data must be valid, that is ...".
+
+## Exceptional specification
+
+An `Exceptional specification` is a formal identification of an item, together with the set of its pre and err conditions. By nature the exceptional specification covers the input and err schema.
+
+## Item specification
+
+An `Item specification` is the set of its ideal and exception specications. Therefore it includes all schema, pre, post and err conditions.
+
+## Specification
+
+A `Specification` of an object is the set of all its item specifications.

data/doc/2-black-box-testing.md

@@ -0,0 +1,27 @@
+# Black-box testing
+
+We define here what we call `black-box testing`, using the vocabulary installed previously.
+
+## Objective
+
+Black-box testing aims at checking whether a test object (actually its implementation) meets its specification. It does so by looking at the object's observable behavior only.
+
+## Method
+
+Testing is never exhaustive and gives no correctness guarantees. As one says, testing can only show the presence of defects, not their absence.
+
+In practice, black-box testing meets its objective by checking whether:
+
+* all positive test cases (i.e. examples) yield successful executions meeting the postconditions.
+
+* all negative test cases (i.e. counterexamples) yield unsuccessful executions meeting the errconditions.
+
+## Prerequisites
+
+Applying black-box testing to web services requires at least:
+
+* A test item: a web service endpoint
+* A specification: pre, post and err conditions of the web service
+* Test cases: examples and counterexamples of web service usage
+
+The second element, i.e. the *specification*, is both important and useful. Modern test frameworks leave the specification implicit (not written at all) or informal (written as code documentation). We think this is both a mistake and a lost opportunity. We will explain why in next section.

data/doc/3-specification-importance.md

@@ -0,0 +1,41 @@
+# On the importance of the specification
+
+Modern software development methods and frameworks, especially Test-Driven (TDD) and Behavior-Driven Development (BDD), use tests extensively. As their name suggests, they use tests to *drive* the development. In fact, it is a major contribution of Agile methods to software development: driving the development by tests invites the developer to improve his code through testing and thereby also improves test coverage as a side effect of the development itself.
+
+However this is not the traditional objective of software testing, which is to test the implementation's conformance to a specification. This yields strange situations where a lot of tests are written while the specification is not:
+
+* Most of the time the specification is implicit: it is simply not written at all,
+* Sometimes the specification is written as informal comments on the tested method or class,
+* Some test frameworks even suggest that the set of tests *is* the specification, which is notably wrong.
+
+## What is a specification?
+
+As a reminder, a `Specification` is the set of `PRE-`, `POST-` and `ERR-` `conditions` defining the contract between the test item (e.g. a web service) and its consumer. In short,
+
+- if the consumer calls the service while the `PRE` are met, then the service guarantees that the `POST` are met.
+- if the consumer calls the service while a `PRE` is not met, and the service is robust, then it guarantees that the `ERR` are met.
+
+## Advantages of an explicit specification
+
+Having a explicit specification has traditional advantages:
+
+- It documents the software, generally in a better and more consise way than a full test suite
+- It enables reasoning about correctness: if you don't know the contractual conditions you can't actually know whether the test suite passes by chance of because the software implementation is correct
+
+Beyond those, specifications can also be used somewhat "against the grain", in the test item / specification / test-case triangle. This is the reason why `webspicy` exists, as we want to explore the following advantages in the specific context of API testing:
+
+- A specification can be used to make tests easier to write
+  * by instrumenting positive test cases so that preconditions are met with almost no effort from the tester
+  * by generating current system states automatically, instead of having to script them
+
+- A specification can be used to generate test cases automatically
+  * generating examples aims at testing correctness
+  * generating counterexamples aims at testing robustness & security
+
+- A specification can be used as a roadmap
+  * documenting unchecked preconditions highlights robustness weakenesses and calls for counterexamples
+  * documenting unmet postconditions highlights bugs and call for examples
+
+- A specification may capture best practices and conventions
+  * Pre and post-conditions can sometimes be reused across different softwares, when they apply to conventions such as RESTful APIs
+

data/doc/4-sequence-diagram.md

@@ -0,0 +1,82 @@
+```
+@startuml
+participant Tester
+participant Client
+participant Result
+participant PrePost
+participant Config
+Tester -> Config : before_all(Scope, Client)
+group * test case
+  Tester -> Config   : before_each(TestCase, Client)
+  Tester -> Client   : instrument(TestCase)
+  group * pre/post/err
+    Client -> PrePost  : instrument(TestCase, Client)
+  end
+  Client -> Config   : instrument(TestCase, Client)
+  Tester -> Client   : call(TestCase)
+  Tester -> Result   : check_response!
+  Tester -> Result   : check_output!
+  Tester -> Result   : check_assertions!
+  Tester -> Result   : check_postconditions!
+  group * post/err
+    Result -> PrePost  : check(Invocation)
+  end
+  Tester -> Config   : after_each(TestCase, Client)
+end
+Tester -> Config : after_all(Scope, Client)
+@enduml
+```
+
+                    ┌──────┐              ┌──────┐          ┌──────┐          ┌───────┐          ┌──────┐          
+                    │Tester│              │Client│          │Result│          │PrePost│          │Config│          
+                    └──┬───┘              └──┬───┘          └──┬───┘          └───┬───┘          └──┬───┘          
+                       │                     │                 │                  │                 │              
+                       │                     │    before_all(Scope, Client)       │                 │              
+                       │───────────────────────────────────────────────────────────────────────────>│              
+                       │                     │                 │                  │                 │              
+          ╔══════════════╤═══════════════════╪═════════════════╪══════════════════╪═════════════════╪═════════════╗
+          ║ * TEST CASE  │                   │                 │                  │                 │             ║
+          ╟──────────────┘                   │  before_each(TestCase, Client)     │                 │             ║
+          ║            │───────────────────────────────────────────────────────────────────────────>│             ║
+          ║            │                     │                 │                  │                 │             ║
+          ║            │ instrument(TestCase)│                 │                  │                 │             ║
+          ║            │────────────────────>│                 │                  │                 │             ║
+          ║            │                     │                 │                  │                 │             ║
+          ║            │        ╔════════════╪════╤════════════╪══════════════════╪═════════════╗   │             ║
+          ║            │        ║ * PRE/POST/ERR  │            │                  │             ║   │             ║
+          ║            │        ╟─────────────────┘            │                  │             ║   │             ║
+          ║            │        ║            │    instrument(TestCase, Client)    │             ║   │             ║
+          ║            │        ║            │───────────────────────────────────>│             ║   │             ║
+          ║            │        ╚════════════╪═════════════════╪══════════════════╪═════════════╝   │             ║
+          ║            │                     │                 │                  │                 │             ║
+          ║            │                     │             instrument(TestCase, Client)             │             ║
+          ║            │                     │─────────────────────────────────────────────────────>│             ║
+          ║            │                     │                 │                  │                 │             ║
+          ║            │    call(TestCase)   │                 │                  │                 │             ║
+          ║            │────────────────────>│                 │                  │                 │             ║
+          ║            │                     │                 │                  │                 │             ║
+          ║            │            check_response!            │                  │                 │             ║
+          ║            │──────────────────────────────────────>│                  │                 │             ║
+          ║            │             check_output!             │                  │                 │             ║
+          ║            │──────────────────────────────────────>│                  │                 │             ║
+          ║            │           check_assertions!           │                  │                 │             ║
+          ║            │──────────────────────────────────────>│                  │                 │             ║
+          ║            │         check_postconditions!         │                  │                 │             ║
+          ║            │──────────────────────────────────────>│                  │                 │             ║
+          ║            │                     │                 │                  │                 │             ║
+          ║            │                     │    ╔═════════════╤═════════════════╪═════════════╗   │             ║
+          ║            │                     │    ║ * POST/ERR  │                 │             ║   │             ║
+          ║            │                     │    ╟─────────────┘                 │             ║   │             ║
+          ║            │                     │    ║            │ check(Invocation)│             ║   │             ║
+          ║            │                     │    ║            │─────────────────>│             ║   │             ║
+          ║            │                     │    ╚════════════╪══════════════════╪═════════════╝   │             ║
+          ║            │                     │                 │                  │                 │             ║
+          ║            │                     │  after_each(TestCase, Client)      │                 │             ║
+          ║            │───────────────────────────────────────────────────────────────────────────>│             ║
+          ╚════════════╪═════════════════════╪═════════════════╪══════════════════╪═════════════════╪═════════════╝
+                       │                     │                 │                  │                 │              
+                       │                     │    after_all(Scope, Client)        │                 │              
+                       │───────────────────────────────────────────────────────────────────────────>│              
+                    ┌──┴───┐              ┌──┴───┐          ┌──┴───┐          ┌───┴───┐          ┌──┴───┐          
+                    │Tester│              │Client│          │Result│          │PrePost│          │Config│          
+                    └──────┘              └──────┘          └──────┘          └───────┘          └──────┘           

data/lib/webspicy.rb

@@ -5,11 +5,11 @@ require 'finitio'
 require 'logger'
 require 'ostruct'
 require 'yaml'
-require 'rspec'
 require 'rack/test'
 require 'mustermann'
 require 'colorized_string'
 require 'securerandom'
+require 'forwardable'
 module Webspicy
 
   ###
@@ -20,20 +20,24 @@ module Webspicy
   require 'webspicy/support'
   require 'webspicy/specification'
   require 'webspicy/configuration'
-  require 'webspicy/checker'
   require 'webspicy/tester'
+  require 'webspicy/web'
+
+  class Error < StandardError; end
+  class TimeoutError < Error; end
 
   ###
   ### Backward compatibility
   ###
   Client = Tester::Client
-  HttpClient = Tester::HttpClient
-  RackTestClient = Tester::RackTestClient
+  HttpClient = Web::HttpClient
+  RackTestClient = Web::RackTestClient
   Resource = Specification
   Precondition = Specification::Precondition
   Postcondition = Specification::Postcondition
   FileUpload = Specification::FileUpload
   Scope = Configuration::Scope
+  Checker = Tester::FileChecker
 
   ###
   ### About folders
@@ -52,6 +56,12 @@ module Webspicy
   FIO
   FORMALDOC = Finitio.system(Path.dir/("webspicy/formaldoc.fio"))
 
+  ###
+  ### Exceptions that we let pass during testing
+  ###
+
+  PASSTHROUGH_EXCEPTIONS = [NoMemoryError, SignalException, SystemExit]
+
   # Returns a default scope instance.
   def default_scope
     Configuration::Scope.new(Configuration.new)
@@ -62,6 +72,7 @@ module Webspicy
     raw = YAML.load(raw) if raw.is_a?(String)
     with_scope(scope) do
       r = FORMALDOC["Specification"].dress(raw)
+      r.config = scope.config
       r.located_at!(file) if file
       r
     end

data/lib/webspicy/configuration.rb

@@ -3,14 +3,29 @@ module Webspicy
 
     LISTENER_KINDS = [ :before_all, :before_each, :after_all, :after_each, :around_each ]
 
-    def initialize(folder = Path.pwd, parent = nil)
-      @folder = folder
+    class << self
+      attr_accessor :default_folder
+
+      def with_default_folder(f)
+        old = default_folder
+        @default_folder = f
+        yield.tap{
+          @default_folder = old
+        }
+      end
+    end
+
+    def initialize(folder = Configuration.default_folder || Path.pwd, parent = nil)
+      @folder = folder.expand_path
       @parent = parent
       @children = []
       @preconditions = []
       @postconditions = []
+      @errconditions = []
+      @insecure = default_insecure
       @listeners = Hash.new{|h,k| h[k] = [] }
       @rspec_options = default_rspec_options
+      @failfast = default_failfast
       @run_examples = default_run_examples
       @run_counterexamples = default_run_counterexamples
       @run_generated_counterexamples = default_run_generated_counterexamples
@@ -23,14 +38,16 @@ module Webspicy
         :success => :green
       }
       @scope_factory = ->(config){ Scope.new(config) }
-      @client = Tester::HttpClient
-      Path.require_tree(folder/'support') if (folder/'support').exists?
+      @client = Web::HttpClient
+      Path.require_tree(@folder/'support') if (@folder/'support').exists?
+      @world = Support::World.new(folder/'world', self)
       yield(self) if block_given?
     end
     attr_accessor :folder
     protected :folder=
 
     attr_accessor :colors
+    attr_reader :world
 
     def self.dress(arg, &bl)
       case arg
@@ -43,9 +60,11 @@ module Webspicy
       when ->(f){ Path(f).exists? }
         arg = Path(arg)
         if arg.file? && arg.ext == ".rb"
-          c = Kernel.instance_eval arg.read, arg.to_s
-          yield(c) if block_given?
-          c
+          Configuration.with_default_folder(arg.parent) do
+            c = Kernel.instance_eval arg.read, arg.to_s
+            yield(c) if block_given?
+            c
+          end
         elsif arg.file? && arg.ext == '.yml'
           folder = arg.backfind("[config.rb]")
           if folder && folder.exists?
@@ -81,7 +100,7 @@ module Webspicy
           config.each_scope(&bl)
         end
       else
-        yield factor_scope
+        yield(factor_scope)
       end
     end
 
@@ -126,6 +145,13 @@ module Webspicy
     attr_accessor :postconditions
     protected :postconditions=
 
+    # Registers an errcondition matcher
+    def errcondition(clazz)
+      errconditions << clazz
+    end
+    attr_accessor :errconditions
+    protected :errconditions=
+
     # Returns whether this configuration has children configurations or not
     def has_children?
       !children.empty?
@@ -369,6 +395,28 @@ module Webspicy
     attr_accessor :rspec_options
     protected :rspec_options=
 
+    # Returns the default value for failfast
+    #
+    # The following environment variables <-> option are supported:
+    #
+    # - FAILFAST=yes <-> true
+    #
+    def default_failfast
+      ENV['FAILFAST'] == 'yes' || ENV['FAILFAST'] == "1"
+    end
+    attr_accessor :failfast
+
+    # Returns the default value to use for insecure.
+    #
+    # The following environment variables <-> option are supported:
+    #
+    # - INSECURE=yes <-> true
+    #
+    def default_insecure
+      ENV['INSECURE'] == 'yes' || ENV['INSECURE'] == "1"
+    end
+    attr_accessor :insecure
+
     # Returns the default rspec options.
     #
     # By default rspec colors are enabled and the format set to 'documentation'.
@@ -422,6 +470,7 @@ module Webspicy
         d.children = []
         d.preconditions = self.preconditions.dup
         d.postconditions = self.postconditions.dup
+        d.errconditions = self.errconditions.dup
         d.rspec_options = self.rspec_options.dup
         d.listeners = LISTENER_KINDS.inject({}){|ls,kind|
           ls.merge(kind => self.listeners(kind).dup)