apirunner 0.4.5 → 0.4.6

data/README.rdoc

@@ -1,8 +1,8 @@
 = apirunner
 
-*apirunner* let's you test your _JSON_ _API_ from the outside. Sometimes model-, controlle-r and routing-test's are not enough, you want to send requests to your application and validate the response in ganular detail. Then apirunner will be your best friend.
+*apirunner* let's you test your _JSON_ _API_ from the outside. Sometimes model-, controller and routing-test's are not enough, you want to send requests to your application and validate the response in ganular detail. Then apirunner will be your best friend.
 
-apirunner is no replacement to rspec or cucumber tests, nor does it replace webrat or capable tools like that. It's an addition that lets you query you API, specify your queries in detail, parse the expected response code, message, header and body and compare all (or any) of 'em to your expectation.
+apirunner is no replacement to rspec or cucumber tests, nor does it replace webrat or capable tools like that. It's an addition that lets you query you API, specify your queries in detail, parse the expected response code, message, header and body and compare all (or any) of 'em to your expectation, as well as check and document every testcases performance.
 
 The request and expectation can (and have to be) written down in easily createable YAML Files. The provided expectation matchers can match strings, integers and regular expressions. So apirunner provides you with a simple but powerful tool to examine your api's bugs.
 
@@ -14,16 +14,23 @@ apirunner was initially developed for testing of the mighty (m8ty) i18n recommen
 
 * be configured for as many environemnts as you wish (your local machine, you staging environment, your production boxes, your wifes handbag)
 * send GET, POST, PUT and DELETE requests via HTTP
+* wait arbitrary but well specifief amount of time before sending a request
 * read as many testcases as you wish from YAML files and execute them in the order of file appearance
+* generate iterational testcases at runtime (for mass/performance test')
 * read more then one testcase from a file
 * match the response code's of your applications responses
 * match the syntactical correctnes of the response format (as long as it is JSON)
-* proof the occurance and match the content of your app's HTTP header
-* proof the occurence and match the content of your app's Body (as long as it responds JSON)
+* proof the occurance and match the content of your app's HTTP headers
+* proof the occurence and match the content of your app's body (as long as it responds JSON)
 * optionally match only parts of header / body (you dont have to specify them in more detail than you are interested in)
 * exclude certain value test's from certain environments (by reading excludes from excludes.yml)
+* build several priority layers, so that you can run only parts of your testspec
 * provide you with some nice feedback at the console .... yeah sexy dots (".") and fancy F's ("F") ....
 * print out a nice error report (that you as a awesome ruby coder will never see)
+* print out a nice success report if you wish
+* measure the performance of your api from the outsite (no concurrency provided today, sry)
+* print out a nice performance report
+* substitute defined resource names of you testcases (resource namespacing) so that several testruns on the same box dont interfere (Hudson vs. developer)
 * be invoked from within rake to generate some example configuration and testcase files
 * be invoked also from within rake to run your test's
 * not travel to Ibiza
@@ -32,6 +39,28 @@ apirunner was initially developed for testing of the mighty (m8ty) i18n recommen
 
   gem install apirunner
 
+== Prerequisites
+
+Until today apirunner runs only in connection with a rails application itself. In the future it (hopefully) will be able to run even isolated without a Rails environment. Releases of Rails prior to 3.0.0.rc are untested and will likely fail. Please don't blame the author put submit you patches.
+
+== Invocation
+
+Assuming you defined your environments as seen in the following section "Configuration", apirunner provides you with the following rake tasks:
+
+  rake -T api
+
+should result in:
+
+  rake api:performance:local       # runs a series of nessecary api calls for performance measuring and parses their response in environment local
+  rake api:performance:production  # runs a series of nessecary api calls for performance measuring and parses their response in environment production
+  rake api:performance:staging     # runs a series of nessecary api calls for performance measuring and parses their response in environment staging
+  rake api:run:local               # runs a series of nessecary api calls and parses their response in environment local
+  rake api:run:production          # runs a series of nessecary api calls and parses their response in environment production
+  rake api:run:staging             # runs a series of nessecary api calls and parses their response in environment staging
+  rake api:scaffold                # generates configuration and a skeleton for apirunner tests as well as excludes
+
+Tasks are speaking so far ... 
+
 == Configuration
 
   rake api::scaffold
@@ -42,9 +71,25 @@ The latter one generates a starter configuration file in your config directory:
 
 Additionally there will be some example testcases which can be found in:
 
-  test/apirunner/001_create_ressource.yml
-  test/apirunner/002_delete_ressource.yml
-  test/apirunner/excludes.yml
+  /test/apirunner/001_create_user.yml
+  /test/apirunner/002_update_resources.yml
+  /test/apirunner/003_update_ratings.yml
+  /test/apirunner/004_update_series_ratings.yml
+  /test/apirunner/005_rateables_and_pagination.yml
+  /test/apirunner/006_recommendations.yml
+  /test/apirunner/007_item_predictions.yml
+  /test/apirunner/008_discovery.yml
+  /test/apirunner/009_cacheable_operations.yml
+  /test/apirunner/010_fsk.yml
+  /test/apirunner/011_misc.yml
+  /test/apirunner/012_telekom_error_reports.yml
+  /test/apirunner/013-extended-unpersonalized-discovery.yml
+  /test/apirunner/014-extended-personalized-discovery.yml
+  /test/apirunner/015_create_10000_users.yml
+  /test/apirunner/999_delete_user.yml
+  /test/apirunner/excludes.yml
+
+These testcases are specific to recent requirements regarding the moviepilot API but can be helpful to understand, how the YAML expectation files have to be created.
 
 At first take some time and change config/api_runner.yml to your needs. You might for example want to test your app locally on localhost:3000, on staging machine and on production environment too. So your api_runner.yml could look like that:
 
@@ -52,31 +97,58 @@ At first take some time and change config/api_runner.yml to your needs. You migh
     protocol: http
     host: localhost
     port: 3000
-    namespace: api
+    namespace: api1v0
   staging:
     protocol: http
-    host: staging.yourstagingdomain.dom
+    host: staging.moviepilot.com
     port: 80
-    namespace: api
+    namespace: api1v0
   production:
-    protocol: https
-    host: www.yourproductiondomain.dom
+    protocol: http
+    host: production.moviepilot.com
     port: 80
-    namespace: prod_api
-
-Take a look at "namespace" here. It makes the expectation matcher build ressource URI's like so:
-
-  http://localhost:3000/api
-  http://staging.yourstagingdomain.dom/api
-  http://www.yourproductiondomain.dom/prod_api
+    namespace: api1v0
+  general:
+    verbosity:
+      - verbose_on_error
+      - verbose_on_success
+      - rspec
+      - performance
+      - verbose_with_curl
+    priority: 0
+    substitution:
+      substitutes:
+        - duffybasic
+        - daisyduck
+        - duffyduck
+        - duffyduck2
+        - duffyduck3
+        - duffydad
+        - duffykid
+        - roadrunner
+        - teletubby
+        - luckyluke
+        - wileecoyote
+      prefix: abc_
+    csv_mode:
+      - append
+      - create
+      - none
+
+The configuration options above need some explanation (uuuuugh) but have to follow the YAML standard, so BE CAREFUL(!) about proper indentation (two spaces).
+So far you can define as many environments as you would like to query. The example above specifies 3 of them [:local, :staging, :production].
+
+You can specify a :protocol, :host, :port as well as a (URL) :namespace per environment. The namespace option is mandatory, so you can omit it. We introduced it, so we can support different versions of our api at the same time and question different versions on different boxes with one setup.
+
+The option makes the expectation matcher build ressource URI's like so:
+
+  http://localhost:3000/api1v0
+  http://staging.moviepilot.com/api1v0
+  http://production.moviepilot.com/api1v0
 
 The ressource pathes are simply appended before the request is sent.
 
-The file also generates your rake tasks dynamically. The above config will generate 3 new tasks:
-
-  api:run:local
-  api:run:staging
-  api:run:production
+TODO
 
 == Excludes
 
@@ -84,15 +156,6 @@ You may also want to define some excludes for some of your environment. Imageine
 
 Simply define your story to check "Last-Modified" generally and exclude it for 
 
-== Invocation
-
-Assuming you defined an environment "local" and "staging" you can invoke your masterpiece with:
-
-  rake api:run:local
-
-and
-
-  rake api:run:staging
 
 == Testing
 

data/VERSION

@@ -1 +1 @@
-0.4.5
+0.4.6

data/apirunner.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{apirunner}
-  s.version = "0.4.5"
+  s.version = "0.4.6"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["jan@moviepilot.com"]
-  s.date = %q{2010-10-25}
+  s.date = %q{2010-10-27}
   s.description = %q{apirunner is a testsuite to query your RESTful JSON API and match response with your defined expectations}
   s.email = %q{developers@moviepilot.com}
   s.extra_rdoc_files = [
@@ -31,10 +31,18 @@ Gem::Specification.new do |s|
     "examples/test/api_runner/001_create_user.yml",
     "examples/test/api_runner/002_update_resources.yml",
     "examples/test/api_runner/003_update_ratings.yml",
-    "examples/test/api_runner/004_rateables_and_pagination.yml",
-    "examples/test/api_runner/005_recommendations.yml",
-    "examples/test/api_runner/006_discovery.yml",
-    "examples/test/api_runner/007_cacheable_operations.yml",
+    "examples/test/api_runner/004_update_series_ratings.yml",
+    "examples/test/api_runner/005_rateables_and_pagination.yml",
+    "examples/test/api_runner/006_recommendations.yml",
+    "examples/test/api_runner/007_item_predictions.yml",
+    "examples/test/api_runner/008_discovery.yml",
+    "examples/test/api_runner/009_cacheable_operations.yml",
+    "examples/test/api_runner/010_fsk.yml",
+    "examples/test/api_runner/011_misc.yml",
+    "examples/test/api_runner/012_telekom_error_reports.yml",
+    "examples/test/api_runner/013-extended-unpersonalized-discovery.yml",
+    "examples/test/api_runner/014-extended-personalized-discovery.yml",
+    "examples/test/api_runner/015_create_10000_users.yml",
     "examples/test/api_runner/999_delete_user.yml",
     "examples/test/api_runner/excludes.yml",
     "features/apirunner.feature",

data/examples/config/api_runner.yml

@@ -5,8 +5,8 @@ local:
   namespace: api1v0
 staging:
   protocol: http
-  host: staging.moviepilot.dom
-  port: 8976
+  host: staging.premium.moviepilot.com
+  port: 80
   namespace: api1v0
 production:
   protocol: http
@@ -18,5 +18,24 @@ general:
     - verbose_on_error
     - verbose_on_success
     - rspec
+    - performance
     - verbose_with_curl
   priority: 0
+  substitution:
+    substitutes:
+      - duffybasic
+      - daisyduck
+      - duffyduck
+      - duffyduck2
+      - duffyduck3
+      - duffydad
+      - duffykid
+      - roadrunner
+      - teletubby
+      - luckyluke
+      - wileecoyote
+    prefix: abc_
+  csv_mode:
+    - append
+    - create
+    - none

data/examples/test/api_runner/001_create_user.yml

@@ -1,4 +1,19 @@
 ---
+- name: 'Create new Basic User'
+  request:
+    headers:
+      Content-Type: 'application/json'
+    path:        '/users/duffybasic'
+    method:      'PUT'
+    body:
+      username:  'duffybasic'
+      expires_at: 2011-09-09T22:41:50+00:00
+  response_expectation:
+    status_code: 201
+    headers:
+      Last-Modified:    /.*/
+    body:
+      username:       'duffybasic'
 - name: 'Create new User'
   request:
     headers:
@@ -8,20 +23,20 @@
     body:
       username:  'duffyduck'
       watchlist:
-      - m333
-      - m79
+      - m1035
+      - m2087
       blacklist:
-      - m334
-      - m77
+      - m1554
+      - m2981
       skiplist:
-      - m335
-      - m78
+      - m1590
+      - m1056
       ratings:
-        m336: 4
-        m79: 2.5
-        m777: 3.0
-        m567: 4.0
-        m354: 5.0
+        m12493: 4.0
+        m1875: 2.5
+        m7258: 3.0
+        m7339: 4.0
+        m3642: 5.0
       expires_at: 2011-09-09T22:41:50+00:00
   response_expectation:
     status_code: 201
@@ -30,17 +45,20 @@
     body:
       username:       'duffyduck'
       watchlist:
-      - m333
-      - m79
+      - m1035
+      - m2087
       blacklist:
-      - m334
-      - m77
+      - m1554
+      - m2981
       skiplist:
-      - m335
-      - m78
+      - m1590
+      - m1056
       ratings:
-        m336: 4.0
-        m79: 2.5
+        m12493: 4.0
+        m1875: 2.5
+        m7258: 3.0
+        m7339: 4.0
+        m3642: 5.0
       fsk:            "18"
 - name: 'Update existing User - Update watchlist'
   request:
@@ -71,14 +89,17 @@
       - m367
       - m73
       blacklist:
-      - m334
-      - m77
+      - m1554
+      - m2981
       skiplist:
-      - m335
-      - m78
+      - m1590
+      - m1056
       ratings:
-        m336: 4.0
-        m79: 2.5
+        m12493: 4.0
+        m1875: 2.5
+        m7258: 3.0
+        m7339: 4.0
+        m3642: 5.0
 - name: 'Set 10 Ratings'
   request:
     headers:
@@ -110,15 +131,6 @@
     body:
       username:       'duffyduck'
       fsk:            "18"
-      watchlist:
-      - m367
-      - m73
-      blacklist:
-      - m334
-      - m77
-      skiplist:
-      - m335
-      - m78
       ratings:
         "m1035": "1.0"
         "m1554": "2.0"
@@ -127,19 +139,17 @@
         "m12493": "4.0"
         "m1875": "5.0"
         "m7258": "2.5"
-- name: 'User creation should fail on wrong username'
-  priority: 1
+- name: 'User creation should fail on bogus characters in username'
+  priority: 3
   request:
     headers:
       Content-Type: 'application/json'
-    path:        '/users/duffyduck'
+    path:        '/users/duffyduck_fix_me'
     method:      'PUT'
     body:
-     username: '!"§$%&/()=?QWERTZUI"§$%&      /('
   response_expectation:
     status_code: 409
 - name: 'User creation should fail on wrong fsk'
-  priority: 1
   request:
     headers:
       Content-Type: 'application/json'
@@ -151,7 +161,6 @@
   response_expectation:
     status_code: 409
 - name: 'User creation should fail on wrong expiration_date'
-  priority: 1
   request:
     headers:
       Content-Type: 'application/json'

data/examples/test/api_runner/002_update_resources.yml

@@ -10,6 +10,11 @@
     - m73
     - m42
     - m23
+    - s70
+    - s71
+    - s86
+    - s87
+    - s88
   response_expectation:
     status_code: 204
     body:
@@ -30,6 +35,11 @@
       - m73
       - m42
       - m23
+      - s70
+      - s71
+      - s86
+      - s87
+      - s88
 - name: 'Update existing user - clean watchlist'
   request:
     headers:
@@ -63,6 +73,11 @@
     - m72
     - m41
     - m22
+    - s101
+    - s100
+    - s99
+    - s69
+    - s70
   response_expectation:
     status_code: 204
     body:
@@ -83,6 +98,11 @@
       - m72
       - m41
       - m22
+      - s101
+      - s100
+      - s99
+      - s69
+      - s70
 - name: 'Update existing user - clean skiplist'
   request:
     headers:
@@ -117,6 +137,11 @@
     - m71
     - m40
     - m21
+    - s87
+    - s88
+    - s51
+    - s52
+    - s99
   response_expectation:
     status_code: 204
     body:
@@ -137,6 +162,11 @@
       - m71
       - m40
       - m21
+      - s87
+      - s88
+      - s51
+      - s52
+      - s99
 - name: 'Update existing user - clean blacklist'
   request:
     headers:
@@ -170,6 +200,8 @@
       watchlist:
       - m365
       - m71
+      - s70
+      - s88
   response_expectation:
     status_code: 404
 - name : 'Skiplist update of a non existing profile should return 404'
@@ -182,6 +214,8 @@
       skiplist:
       - m365
       - m71
+      - s99
+      - s100
   response_expectation:
     status_code: 404
 - name : 'Blacklist update of a non existing profile should return 404'
@@ -194,6 +228,8 @@
       blacklist:
       - m365
       - m71
+      - s71
+      - s70
   response_expectation:
     status_code: 404
 - name : 'Watchlist update with non existing movies should return 409'
@@ -209,6 +245,35 @@
       - m7777777777777777
   response_expectation:
     status_code: 409
+- name : 'Watchlist update with non existing series should return 409'
+  request:
+    headers:
+      Content-Type: 'application/json'
+    path: '/users/duffyduck/watchlist'
+    method: 'PUT'
+    body:
+      watchlist:
+      - s9999999999999999
+      - s8888888888888888
+      - s7777777777777777
+  response_expectation:
+    status_code: 409
+- name : 'Watchlist update with non existing movies AND series should return 409'
+  request:
+    headers:
+      Content-Type: 'application/json'
+    path: '/users/duffyduck/watchlist'
+    method: 'PUT'
+    body:
+      watchlist:
+      - m9999999999999999
+      - m8888888888888888
+      - m7777777777777777
+      - s9999999999999999
+      - s8888888888888888
+      - s7777777777777777
+  response_expectation:
+    status_code: 409
 - name : 'Blacklist update with non existing movies should return 409'
   request:
     headers:
@@ -222,6 +287,35 @@
       - m7777777777777777
   response_expectation:
     status_code: 409
+- name : 'Blacklist update with non existing series should return 409'
+  request:
+    headers:
+      Content-Type: 'application/json'
+    path: '/users/duffyduck/blacklist'
+    method: 'PUT'
+    body:
+      blacklist:
+      - s9999999999999999
+      - s8888888888888888
+      - s7777777777777777
+  response_expectation:
+    status_code: 409
+- name : 'Blacklist update with non existing movies AND series should return 409'
+  request:
+    headers:
+      Content-Type: 'application/json'
+    path: '/users/duffyduck/blacklist'
+    method: 'PUT'
+    body:
+      blacklist:
+      - m9999999999999999
+      - m8888888888888888
+      - m7777777777777777
+      - s9999999999999999
+      - s8888888888888888
+      - s7777777777777777
+  response_expectation:
+    status_code: 409
 - name : 'Skiplist update with non existing movies should return 409'
   request:
     headers:
@@ -235,3 +329,32 @@
       - m7777777777777777
   response_expectation:
     status_code: 409
+- name : 'Skiplist update with non existing series should return 409'
+  request:
+    headers:
+      Content-Type: 'application/json'
+    path: '/users/duffyduck/skiplist'
+    method: 'PUT'
+    body:
+      skiplist:
+      - s9999999999999999
+      - s8888888888888888
+      - s7777777777777777
+  response_expectation:
+    status_code: 409
+- name : 'Skiplist update with non existing movies AND series should return 409'
+  request:
+    headers:
+      Content-Type: 'application/json'
+    path: '/users/duffyduck/skiplist'
+    method: 'PUT'
+    body:
+      skiplist:
+      - m9999999999999999
+      - m8888888888888888
+      - m7777777777777777
+      - s9999999999999999
+      - s8888888888888888
+      - s7777777777777777
+  response_expectation:
+    status_code: 409

data/examples/test/api_runner/003_update_ratings.yml

@@ -1,5 +1,5 @@
 ---
-- name: "One putted rating should be createable with 204 response"
+- name: "One putted movie rating should be createable with 204 response"
   request:
     headers:
       Content-Type:         'application/json'
@@ -10,7 +10,7 @@
   response_expectation:
     status_code: 204
     body:
-- name: "There should be exactly one rating in that profile"
+- name: "There should be exactly one movie rating in that profile"
   request:
     headers:
       Content-Type:         'application/json'
@@ -21,7 +21,7 @@
     body:
       ratings:
         m765: 1.0
-- name: "5 putted ratings should be createable with 204 response"
+- name: "5 putted movie ratings should be createable with 204 response"
   request:
     headers:
       Content-Type:         'application/json'
@@ -36,7 +36,7 @@
   response_expectation:
     status_code: 204
     body:
-- name: "There should be exactly 5 ratings in that profile"
+- name: "There should be exactly 5 movie ratings in that profile"
   request:
     headers:
       Content-Type:         'application/json'
@@ -51,7 +51,7 @@
         m89: 3.0
         m91: 4.0
         m98: 5.0
-- name: "It should return a 404 in case ratings for a non existant user are put"
+- name: "It should return a 404 in case movie ratings for a non existant user are put"
   request:
     headers:
       Content-Type:         'application/json'
@@ -62,7 +62,7 @@
       m42: 2.0
   response_expectation:
     status_code: 404
-- name: "It should return a 409 in case ratings are put with ambigous moviemaster_id"
+- name: "It should return a 409 in case movie ratings are put with ambigous moviemaster_id"
   request:
     headers:
       Content-Type:         'application/json'
@@ -74,8 +74,7 @@
   response_expectation:
     status_code: 409
     body:
-- name: "It should return a 409 in case ratings are put with ambigous ratings values"
-  priority: 1
+- name: "It should return a 409 in case movie ratings are put with ambigous ratings values"
   request:
     headers:
       Content-Type:         'application/json'