brine-dsl 0.8.1 → 0.9.0

data/docs/src/cookbook.adoc

@@ -1,160 +0,0 @@
-= icon:cutlery[] Brine Cookbook
-Matt Whipple <http://github.com/mwhipple[@mwhipple]>
-:description: Cookbook for the Brine REST Testing DSL
-:keywords: Brine, Cucumber, REST, DSL
-:gh_repo: https://github.com/brightcove/brine
-
-== Overview
-
-The following are some recipes to address issues which may arise while using
-Brine but are not considered part of Brine's (current) responsibility.
-Many of these are focused on simplicity and ease rather than robustness or
-elegance, and modification to address specific cases should be expected.
-
-Many of these will also be platform specific and be subject to further
-organization once platforms beyond ruby are supported.
-
-== Intercepting of HTTP Calls
-
-=== Context
-
-There may be cases where the request or reponse may need to be modified
-in some way before or after transmission, for example to add a custom header
-which is somehow specific to the testing configuration and therefore outside
-of anything that belongs in the specification.
-
-=== Solution
-
-This can be done through the registration of custom Faraday middleware on
-the Brine client. All of the registered middleware attaches
-to the generated connection through an array of functions in
-{gh_repo}/blob/master/lib/brine/requester.rb[requester.rb].
-Each function accepts the connection object as its single parameter.
-The array of functions is exposed as `connection_handlers` and can be modified
-through getting that property from either the default `World`
-(for the default client) or from an appropriate `ClientBuilder`
-if creating custom clients and mutating it accordingly (setting the reference
-is not suported).
-You could just build your own client without the facilities provided by Brine.
-
-=== Recipe
-
-An example to add a custom header could be implemented as follows:
-
-[source,ruby]
-----
-require 'faraday'
-class CustomHeaderAdder < Faraday::Middleware
-  def initialize(app, key, val)
-    super(app)
-    @key = key
-    @val = val
-  end
-
-  def call(env)
-    env[:request_headers].merge!({@key => @val})
-    @app.call(env)
-  end
-end
-
-...
-
-connection_handlers.unshift(proc do |conn|
-  conn.use CustomHeaderAdder, 'x-header', 'I am a test'
-end)
-----
-
-== "Assurances"
-
-=== Context
-
-Ideally all tests should be as self-contained and isolated as possible;
-when writing functional tests, however, there are cases where this isn't
-feasible or possible. In some cases a system depends on another external
-system which is not a system that is under test and which (for whatever reason)
-cannot be easily worked with. In white box testing such a system would likely be
-represented by some form of test double, but this may be unfeasible and/or
-undesirable when testing a deployed system.
-
-An example of such a system is user/account management which often incurs
-additional overhead to provision a new account. When testing a secured
-system valid accounts are needed for representative testing, but provisioning
-a new account may be difficult or outside the scope of the system that is being
-actively tested. If tested functionality involves enacting account-wide changes
-and the number of accounts is limited, then that is likely to unfortunately
-prevent complete test isolation.
-
-=== Solution
-
-In such cases a standard solution is to designate certain resources to be
-reused for certain tests. These are analogous to the concept of "fixtures" in
-some test suites though there may be slight differences in implementation and
-reliance on them. Here the term "assurances" is used...primarily because it
-starts with `a` which lends itself to relevant files being listed towards the
-beginning alphabetically in a given directory.
-
-The goal of assurances is to specify conditions which are expected before other
-tests are to be run. Preferably the dependent tests should also explicitly
-declare the dependency but a significant solution for that is not established.
-Assurances therefore validate that preconditions are met; ideally if such
-preconditions can be established idempotently then the assurances can do so
-before the validation.
-
-==== Assurances are NOT Tests
-
-**Assurances validate a state which is desired to be consistently retained
-within the system rather than being changed**. This means that they should _not_
-be used for tests as that would require state changes, nor should they clean up
-after themselves (as that would also imply a state change). If assurances are
-configured for a system which should also be tested, then appropriate tests
-should exist (including those that may validate any behavior relied upon by
-the assurance).
-
-==== Consequences
-
-As mentioned previously assertions help in cases where tests cannot be fully
-isolated, and therefore some known state must be established and reused across
-tests (and such state should *not* change). A practical reason for this is to
-allow for overlapping test executions.
-If tests are not fully isolated, state is being changed, and test runs overlap,
-then tests may fail non-deterministically due to one test run pulling the state
-out from another. This in the simplest form can be a nuisance but it also
-effectively precludes the ability to speed up test runs through the use of
-parallelism/asynchronicity.
-
-_TODO: Enumerate drawbacks_
-
-=== Recipe
-
-This can be done using standard cucumber tags. Assurances can be defined in
-designated `assure_*.feature` files where each Feature is appropriately tagged:
-
-[source,gherkin]
-----
-@assure
-Feature: Some preconditions are verified...
-----
-
-And then a Rake task is added to run those tagged features:
-
-[source,ruby]
-----
-Cucumber::Rake::Task.new(:assure) do |t|
-   t.cucumber_opts = "#{ENV['CUCUMBER_OPTS']} --tags @assure"
-end
-----
-
-The Rake task that runs the other tests then depends on that task:
-
-[source,ruby]
-----
-task :invoke_cuke => [:assure] do
-  #Run cucumber, potentially in parallel and likely with --tags `@assure`
-end
-----
-
-This approach tests preconditions and will avoid running the rest of the tests
-if they are not (relying on standard Rake behavior). The assurances can also be
-run with different Cucumber behavior so that the full test suite can be more
-stochastic (randomized/non-serialized) while the assurances can be more
-controlled.

data/docs/src/guide.adoc

@@ -1,524 +0,0 @@
-= icon:book[] Brine User Guide
-Matt Whipple <http://github.com/mwhipple[@mwhipple]>
-:description: The User Guide for using the Brine REST Testing DSL
-:keywords: Brine, Cucumber, REST, DSL
-:grave: `
-:response_attribute: (body|status|headers)
-
-Cucumber DSL for testing REST APIs
-
-== Introduction
-
-=== Motivation
-
-REpresentational State Transfer APIs expose their functionality
-through combinations of fairly coarse primitives that generally
-revolve around the use of transferring data in a standard exchange
-format (such as JSON) using HTTP methods and other aspects of the very
-simple HTTP protocol. Tests for such an API can therefore be defined
-using a domain specific language (DSL) built around those higher level
-ideas rather than requiring a general purpose language (the equivalent
-of scripted `curl` commands with some glue code and assertions).
-This project provides such a DSL by using select libraries
-integrated into Cucumber, where Cucumber provides a test-oriented
-framework for DSL creation.
-
-=== Sample Usage
-
-The general usage pattern revolves around construction of a request
-and performing assertions against the received response.
-
-[source,gherkin]
-----
-When the request body is assigned:
-  """
-  {"first_name": "John",
-   "last_name": "Smith"}
-  """
-And a POST is sent to `/users`
-Then the value of the response status is equal to `200`
-And the value of the response body is including:
-  """
-  {"first_name": "John",
-   "last_name": "Smith"}
-  """
-----
-
-=== Key Features
-
-Variable Binding/Expansion::
-  In cases where dynamic data is in the response or is desired for the
-  request, then values can be bound to identifiers which can then be
-  expanded using http://mustache.github.io[Mustache] templates in your
-  feature files.
-
-Type Transforms::
-  Different types of data can be expressed directly in the feature files
-  or expanded into variables by using the appropriate syntax for that
-  type.
-
-Type Coercion::
-  Related to transforms, a facility to coerce types is also provided. This
-  allows more intelligent comparison of inputs which have been transformed to a
-  richer data type with those that have not been transformed (normally strings).
-  As an example comparing a date/time value with a string will attempt to parse
-  the string to a date/time so that the values can be compared using the proper
-  semantics.
-
-<<_resource_cleanup>>::
-  Tests are likely to create resources which should then be cleaned up,
-  restoring the pre-test state of the system: steps to facilitate this
-  are provided.
-
-Authentication::
-  Presently OAuth2 is supported to issue authenticated requests during a
-  test (likely using a feature `Background`).
-
-Request Construction and Response Assertion Step Definitions::
-  The previous features combined with the library of provide steps should
-  cover all of the functionality needed to exercise and validate all of
-  the functionality exposed by your REST API.
-
-== Installation
-
-Brine is published as `brine-dsl` on rubygems, the page for which is
-at https://rubygems.org/gems/brine-dsl. The latest version and other
-gem metadata can be viewed on that page. Brine can be used by
-declaring that gem in your project Gemfile such as:
-
-[source,ruby]
-----
-gem 'brine-dsl', '~> 1.0'
-----
-
-(after version 1.0 is released).
-
-Brine can then be "mixed in" to your project (which adds assorted
-modules to the `World` and loads all the step definitions and other
-Cucumber magic) by adding the following to your `support/env.rb` or
-other ruby file:
-
-[source,ruby]
-----
-require 'brine'
-
-World(brine_mix)
-----
-
-Select pieces can also be loaded (to be documented). With the above,
-feature files should be able to be written and executed without
-requiring any additional ruby code.
-
-== Tutorial
-
-We'll write some tests against http://myjson.com/api
-(selected fairly arbitrary from the list at
-https://github.com/toddmotto/public-apis).
-The API is being explored for the sake of this tutorial,
-which also serves to bolster this library to support the effort.
-
-=== Selecting a ROOT_URL
-
-Brine expects steps to use relative URLs. The feature files specify
-the behavior of an API (or multiple APIs), while the root of the
-URLs define where that API is, so this is a natural mapping.
-More practically, when developing an API it's likely to
-be promoted across various environments such as
-local, qa, stage, and production so having a parameterized root for
-the URLs eases this while encouraging inter-environment consistency.
-
-For simple cases where all tests are to be run against the same root,
-the root url can be specified with the environment variable `ROOT_URL`,
-such as `ROOT_URL=https://api.myjson.com/ cucumber`, or letting `rake`
-take care of this for you such as:
-
-[source,ruby]
-----
-Cucumber::Rake::Task.new do
-  ENV['ROOT_URL'] = 'https://api.myjson.com/'
-end
-----
-
-which could then be called with `rake cucumber`. The rake approach
-can be extended for different tasks for each environment, each
-of which sets the appropriate environment variables allowing the
-test code itself to follow
-https://12factor.net/config[Twelve-Factor App guidelines]
-where Rake provides sugary convenience.
-
-=== A Basic GET
-
-Most tests will involve some form of issuing requests and performing assertions
-on the responses. Let's start with a simple version of that pattern,
-testing the response status from a GET request.
-
-[source,gherkin]
-----
-include::../../tutorial/missing.feature[]
-----
-
-=== A Write Request
-
-For POST, PATCH and PUT requests you'll normally want to include a request body.
-To support this, additional data can be added to the requests before they are
-sent (see <<_request_construction>>).
-
-[source,gherkin]
-----
-include::../../tutorial/post_status.feature[]
-----
-
-=== Test Response Properties
-
-The API that was chosen for testing returns the link to the created resource
-which is based off of a generated id. That means that the exact response cannot
-be verified, but instead property based testing can be done to verify that the
-data is sane and therefore likely trustworthy. In this case we
-can check that the `uri` response child matches the expected pattern.
-
-[source,gherkin]
-----
-include::../../tutorial/post_matching.feature[]
-----
-
-////
-=== Known Response Data
-
-One of the simplest and most obvious things to test for is that the response
-contains data for which exact values are expected. Continuing from above we
-can check that the response body returns the fields that we provided.
-
-[source,gherkin]
-----
-include::../../tutorial/post_including.feature[]
-----
-////
-
-== Environment Variables
-
-Some Brine behavior can be tuned by passing it appropriate environment
-variables, listed here.
-
-`BRINE_LOG_HTTP`::
-  Output HTTP traffic to stdout. Any truthy value will result in request and
-  response metadata being logged, a value of `DEBUG` (case insensitive) will
-  also log the bodies.
-
-`BRINE_LOG_BINDING`::
-  Log values as they are assigned to variables in Brine steps.
-
-== Language Conventions
-
-=== The use of ``{grave}``s
-
-Backticks/grave accents are used as _parameter delimiters_. It is perhaps
-most helpful to think of them in those explicit terms rather than thinking of
-them as an alternate _quote_ construct. In particular quoting implies that the
-parameter value is a string value, while the step transforms allow for
-alternative data types.
-
-``{grave}``s were chosen as they are less common than many other syntactical
-elements and also allow for the use of logically significant
-quoting within paremeter values while hopefully avoiding the need for escape
-artistry (as used for argument transforms).
-
-== Selection and Assertion
-
-As tests are generally concerned with performing assertions, a testing DSL
-should be able to express the variety of assertions that may be needed. Because
-these are likely to be numerous, it could easily lead to duplicated logic or
-geometric growth of code due to the combinations of types of assertions and the
-means to select the inputs for the assertion.
-
-To avoid this issue the concepts of selection and assertion are considered
-separate operations in Brine. Internally this corresponds to two steps: the
-first assigns a selector;
-the second passes the assertion to that selector which is responsible for
-applying the assertion against the selected value(s). In standard step use this
-will still be expressed as a single step,
-and dynamic step definitions are used to split the work appropriately.
-
-For example the step:
-
-[source,gherkin]
-----
-Then the value of the response body is equal to `foo`
-----
-
-Will be split where the subject of the step (`the value of the response body`)
-defines the selector and the predicate of the step
-`is equal to {grave}foo{grave}` defines the assertion (which is translated to a
-step such as `Then it is equal to {grave}foo{grave}`).
-
-The result of this is that the assertion steps will always follow a pattern
-where the subject resembles `the value of ...` and the predicate always
-resembles `is ...`. Learning the selection phrases and the assertion phrases
-and combining them should be a more efficient and flexible way to become
-familiar with the language instead of focusing on the resulting combined steps.
-
-The chosen approach sacrifices eloquence for the sake of consistency.
-The predicate will always start with `is` which can lead to awkward language
-such as `is including` rather than simply `includes`.
-The consistency provides additional benefits such as consistent modification:
-for instance `is not` can always be use for negation rather than working out the
-appropriate phrasing for a more natural sounding step (let alone the logic).
-
-One of the secondary goals of this is that assertion step definitions should
-be very simple to write and modifiers (such as negation) should be provided for
-free to those definitions.
-As assertion definitions are likely to be numerous and potentially customized,
-this should help optimize code economy.
-
-=== Selection Modifiers
-
-To pursue economical flexibility Brine steps attempt to balance step definitions
-which accommodate variations while keeping the step logic and patterns fairly
-simple. Selection steps in particular generally accept some parameters that
-affect their behavior. This allows the relatively small number of selection
-steps to provide the flexibility to empower the more numerous assertion steps.
-
-==== Traversal
-
-Selection steps can generally target the root of the object specified (such as
-the response body) or some nodes within the object if it is a non-scalar value
-(for instance a child of the response body). This is indicated in the
-<<_selection,step reference selection steps>> by the `[$TRAVERSAL]` placeholder.
-`child {grave}$EXPRESSION{grave}` or `children {grave}$EXPRESSION{grave}` can
-optionally be inserted at the placeholder to select nested nodes as described
-in <<_traversal_2>>.
-
-==== Negation
-
-The selectors also currently handle negation of the associated assertions.
-This is potentially counter-intuitive but as previously mentioned the intent is
-that this should ease the creation of assertions. If negation is added to a
-selector that it is expected that the assertion will _fail_.
-
-Negation will be normally indicated in the
-<<_selection,step reference selection steps>> by the presence
-of the `[not]` placeholder. A similar placeholder may be used that is more
-readable but leads to an equivalent inversion of the semantics of the statement.
-To negate the step, the text within the ``[]``s should be inserted
-in the indicated position.
-
-[NOTE, caption='Future Versions']
-
-Handling this in the selectors is (as mentioned) counter-intuitive and
-unnecessarily couples the selector to the assertion. It is currently done for
-practical reasons but is likely to be replaced in a future version after (or as
-part of) the initial port of the library to another platform. When it is
-replaced, all existing steps will remain supported through at least one more
-major revision and most should (most should remain unchanged).
-
-=== Chained Assertions
-
-[WARNING, caption='Unsupported Feature']
-
-Use at your own risk, this feature is *not presently supported*.
-
-For anyone that likes to live on the (relative) edge or if this gathers notable
-interest...the above also provides an implicit feature: after a value is
-selected multiple assertions could be performed against it.
-
-For instance:
-
-[source,gherkin]
-----
-Then the value of the response body is equal to `foo`
-And it is of the type `String`
-----
-
-Though this may work in simple cases the present design is likely to produce
-surprising results since some aspects (such as negation) are handled by the
-selector so it would be inherited by the conjunctions even though it wouldn't
-read that way.
-
-== Traversal
-
-The language exposed by Brine is flat but the data returned by the server is
-likely to include deeper data structures such as objects and collections. To
-allow selection within such structures a `traversal` language is embedded within
-some steps which will be indicated by the use of the `TRAVERSAL` placeholder.
-
-The traversal language consists of a selected subset of
-http://goessner.net/articles/JsonPath/[JsonPath].
-
-[NOTE, caption='The Selected Subset']
-
-The subset of JsonPath functionality has been chosen that is believed to support
-all needed test cases without requiring deep familiarity with JsonPath. This may
-lead to more numerous simple steps in place of fewer steps that use unsupported
-expressions. Additionally Brine is intended to be ported to a range of platforms
-and so only those steps outlined here will be supported across those platforms.
-JsonPath expressions _not_ listed below will not be explicitly disallowed but
-are not officially supported (will not be tested and will not be ported to
-another platform if needed).
-
-=== Cardinality
-
-Each traversal expression will select _all_ matching nodes which is therefore
-represented as a collection. Often, however, only a single node is expected or
-desired. Therefore the traversal expression will also be accompanied by a phrase
-which defines the expected cardinality, normally `child` vs. `children`.
-`children` will _always_ return an array while `child` will return what would be
-the first element in that array. `child` should be used when accessing a
-specific node within the tree, while `children` should be used for what amounts
-to a query across multiple nodes (such as testing the value of a field for every
-element in a collection).
-
-=== Expressions
-
-`.$KEY`::
-  Access the `KEY` named child of the starting node. The leading `.` can be
-  omitted if at the start of an expression.
-
-`.[$INDEX]`::
-  Access the element of the array at index `INDEX`
-
-`.[$FROM:$TO]`::
-  Access a slice of the array containing the elements at index `FROM` through
-  `TO` (including both limits).
-
-== Resource Cleanup
-
-All test suites should clean up after themselves as a matter of hygiene and to
-help enforce test independence and reproducibility. This is particularly
-important for this library given that it is likely the systems under test
-are likely to remain running; accumulated uncleaned resources are at best a
-nuisance to have to weed through and at worst raise some costs or other due to
-heightened consumption of assorted resources (as opposed to more ephemeral test
-environments).
-
-Brine therefore provides mechanisms to assist in cleaning up those resources
-which are created as part of a test run. A conceptual hurdle for this type of
-functionality is that it is very unlikely to be part of the feature that is
-being specified, and therefore should ideally not be part of the specification.
-Depending on the functionality (and arguably the
-https://www.martinfowler.com/articles/richardsonMaturityModel.html[maturity])
-of the API, most or all of the cleanup can be automagically done based on
-convention. There are tentative plans to support multiple techniques for
-cleaning up resources based on how much can be implicitly
-ascertained...though presently there exists only one.
-
-=== Step indicating resource to DELETE
-
-If the API supports DELETE requests to remove created resources but it is either
-desirable or necessary to specify what those resource PATHS are, a step can be
-used to indicate which resources should be DELETEd upon test completion.
-
-_see <<_cleanup,Cleanup Step Definitions>>_
-
-== Step Reference
-
-=== Request Construction
-
-link:specs.html#_request_construction[icon:cogs[] Specification]
-
-The requests which are sent as part of a test are constructed using
-a https://en.wikipedia.org/wiki/Builder_pattern[Builder].
-
-`When a $METHOD is sent to {grave}$PATH{grave}`::
-  As every request to a REST API is likely to have a significant
-  HTTP `METHOD` and `PATH`, this step is considered required and is therefore
-  used to send the built request. This should therefore be the *last* step for
-  any given request that is being built.
-
-`When the request body is assigned:`::
-  The multiline content provided will be assigned to the body of the request.
-  This will normally likely be the JSON representation of data.
-
-`When the request query parameter {grave}$PARAMETER{grave} is assigned
-{grave}$VALUE{grave}`::
-  Assign `VALUE` to the request query `PARAMETER`.
-  The value will be URL encoded and the key/value pair appended to the URL using
-  the appropriate `?` or `&` delimiter.
-  The order of the parameters in the resulting URL should be considered
-  undefined.
-
-`When the request header {grave}$HEADER{grave} is assigned {grave}$VALUE{grave}`::
-  Assign `VALUE` to the request header `HEADER`.
-  Will overwrite any earlier value for the specified header, including earlier
-  steps or defaults.
-
-=== Cleanup
-
-`When a resouce is created at {grave}$PATH{grave}`::
-  Mark `PATH` as a resource to DELETE after the test is run.
-  See <<_resource_cleanup>>
-
-=== Assignment
-
-link:specs.html#_assignment[icon:cogs[] Specification]
-
-`When {grave}$IDENTIFIER{grave} is assigned {grave}$VALUE{grave}`::
-  Assigns `VALUE` to `IDENTIFIER`.
-
-`When {grave}$IDENTIFIER{grave} is assigned a random string`::
-  Assigns a random string (UUID) to `IDENTIFIER`.
-  This is particularly useful to assist with test isolation.
-
-`When {grave}$IDENTIFIER{grave} is assigned a timestamp`::
-  Assigns to `IDENTIFIER` a timestamp value representing the instant at
-  which the step was evaluated.
-
-`When {grave}$IDENTIFIER{grave} is assigned the response {response_attribute} [$TRAVERSAL]`::
-  Assigns to `IDENTIFIER` the value extracted from the specified response
-  attribtute (at the optional traversal path).
-
-=== Selection
-
-link:specs.html#_selection[icon:cogs[] Specification]
-
-_see <<_selection_and_assertion>>_
-
-_TODO: Replace all the explicit attributes._
-
-`Then the value of the response {response_attribute} is`::
-  Select the sepecified attribute of the current HTTP response.
-
-`Then the value of the response body [$TRAVERSAL] is [not]`::
-  Select the value from the body of the response.
-
-`Then the value of the response body [$TRAVERSAL] does have any element that is [not]`::
-  Select any (at least one) element from the structure within the response body.
-
-`Then the value of the response body [$TRAVERSAL] has elements which are all`::
-  Select all elements from the structure within the response body.
-
-=== Assertion
-
-link:specs.html#_assertion[icon:cogs[] Specification]
-
-_see <<_selection_and_assertion>>_
-
-`Then it is equal to {grave}$VALUE{grave}`::
-  Assert that the current selected value is equivalent to `VALUE`
-
-`Then it is matching {grave}$VALUE{grave}`::
-  Assert that the current select value matches the regular expression `VALUE`
-
-`Then it is including {grave}$VALUE{grave}`::
-  Assert that the current select value includes/is a superset of `VALUE`.
-
-`Then it is empty`::
-  Assert that value is empty or null. Any type which is not testable for
-  emptiness (such as booleans or numbers) will always return false. Null is
-  treated as an empty value so that it can be treated as such for endpoints that
-  return null in place of empty collections, and non-null empty values can
-  easily be tested for using conjunction.
-
-`Then it is of length {grave}$VALUE{grave}`::
-  Assert that the value exposes a length attribute and the value of that
-  attribute is `VALUE`.
-
-`Then it is a valid {grave}$TYPE{grave}`::
-  Assert that the selected value is a valid instance of a `TYPE`. Presently this
-is focused on standard data types (intially based on those specified by JSON),
-but it is designed to handle user specified domain types pending some minor
-wiring and documentation. The current supported types are:
- * `Object`: A JSON style object/associative array
- * `String`
- * `Number`
- * `Integer`
- * `Array`
- * `Boolean`