rainforest-cli 1.6.5 → 1.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 70b20280040fce6f6790e5e18149596eccb2824e
-  data.tar.gz: 1f83c0cc0e433bfc36baf775023c0b3f7f49780d
+  metadata.gz: e73f6818b72f61f2b152943bc46f066b95c43630
+  data.tar.gz: a5a82d24cf2b5d5d4be4fafe0c3a0ebab1cca626
 SHA512:
-  metadata.gz: d9a8325fe43ba3a84017701f0ab3be81bad42c7e6bd04853909835a7afa2fa3867e8e4e91320fa24c6549df9691015003b6a25c5c141a8c12754523be4ae3051
-  data.tar.gz: 9be91de080050e86660f0962d534d97632f22298f5240747e52aa05855e7cfa961daa0d21002794641409dd9731909425351ecba3d047c9677c0b4daa57996c4
+  metadata.gz: add946c11dd743c5fb9796d00a12526c87a7453bfe52341a28272d22ae0fb05458b95bf2cb52080623a67d17bc1318a09db75a2c96c9d94ec4041b0724e48da5
+  data.tar.gz: 712af1181c2892308dd5396c63653f0e61e5944ad74044788589326258e74b3075cf40a93a672e5c19dabe7d38fba7328d43fe85b7630683c2ee9542dad183e2

data/.rubocop.yml

@@ -3,3 +3,6 @@ inherit_gem:
 
 Style/FileName:
   Exclude: [rainforest-cli.gemspec]
+
+Metrics/LineLength:
+  Exclude: [spec/**]

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Rainforest CLI Changelog
 
+## 1.7.0 - 8th August 2016
+- New run flag: `--wait` for hooking into existing run instead of starting a new one.
+(77df41bf79b8635fb8c2d8a93968f975db092c69, @shosti)
+- New run flag: `--junit-file` for exporting run results into JUnit format.
+(349f2b1f5c8b423766875751c7cafed692fc2bed, @briancolby)
+- New feature: Embed screenshots and file download links in your RFML tests.
+(e081072c591c810b8bc3edce6d2e507d12a1a18e, @epaulet)
+
 ## 1.6.5 - 13th June 2016
 - Exit with non-zero status code when `validate` fails. (8db1d38be39aa50d2afcdef817f78c654b3108b6,
 @DyerSituation)

data/README.md

@@ -32,9 +32,9 @@ You can customize the amount of threads to use when making HTTP requests by sett
 API errors when fetching or updating multiple tests, lowering this value can help.
 The default value is `16`.
 
-#### Commands
+## Options
 
-###### Running Tests
+#### Running Tests
 
 Run all tests.
 
@@ -54,7 +54,13 @@ Run all tests with tag 'run-me' and abort previous in-progress runs.
 rainforest run --tag run-me --fg --conflict abort
 ```
 
-###### Creating and Managing Tests
+Run all in the foreground and generate a junit xml report.
+
+```bash
+rainforest run all --fg --junit-file rainforest.xml
+```
+
+#### Creating and Managing Tests
 
 Create new Rainforest test in RFML format (Rainforest Markup Language).
 
@@ -82,12 +88,19 @@ Upload tests to Rainforest
 rainforest upload
 ```
 
+Remove RFML file and remove test from Rainforest test suite.
+
+```bash
+rainforest rm /path/to/test/file.rfml
+```
+
 Export all tests from Rainforest
+
 ```bash
 rainforest export
 ```
 
-###### Viewing Account Specific Information
+#### Viewing Account Specific Information
 
 See a list of all of your sites and their IDs
 ```bash
@@ -104,6 +117,11 @@ See a list of all of your browsers and their IDs
 rainforest browsers
 ```
 
+To generate a junit xml report for a test run which has already completed
+```bash
+rainforest report --run-id <run-id> --junit-file rainforest.xml
+```
+
 ## Options
 
 ### General
@@ -128,8 +146,11 @@ Rainforest Tests written using RFML have the following format
 # redirect: [REDIRECT FLAG]
 - [EMBEDDED TEST RFML ID]
 
-[ACTION 2]
-[QUESTION 2]
+Action with an embedded screenshot: {{ file.screenshot(./relative/path/to/screenshot.jpg) }}
+Response with an embedded file download: {{ file.download(./relative/path/to/file.txt) }}
+
+[ACTION 3]
+[QUESTION 3]
 
 ... etc.
 ```
@@ -161,6 +182,13 @@ tests and the first step of a test.
 - `EMBEDDED TEST RFML ID` - Embed the steps of another test within the current test
 using the embedded test's RFML ID.
 
+Embedding Screenshots and Downloadable Files:
+- Your embedded files must be checked into version control with along your RFML tests.
+- `{{ file.screenshot(RELATIVE_PATH) }}` will embed screenshot found at RELATIVE_PATH
+which is a path to the screenshot relative to the RFML file.
+- `{{ file.download(RELATIVE_PATH) }}` will embed downloadable file link found at RELATIVE_PATH
+which is a path to the downloadable file relative to the RFML file.
+
 For more information on test writing, please visit our [documentation](http://support.rainforestqa.com/hc/en-us/sections/200585603-Writing-Tests).
 
 ### Command Line Options
@@ -173,6 +201,7 @@ Popular command line options are:
 - `--environment-id` - run your tests using this environment. Otherwise it will use your default environment
 - `--conflict OPTION` - use the `abort` option to abort any runs in progress in the same environment as your new run. use the `abort-all` option to abort all runs in progress.
 - `--fg` - results in the foreground - rainforest-cli will not return until the run is complete. This is what you want to make the build pass / fail dependent on rainforest results
+- `--wait RUN_ID` - wait for an existing run to finish instead of starting a new one, and exit with a non-0 code if the run fails. rainforest-cli will exit immediately if the run is already complete.
 - `--fail-fast` - fail the build as soon as the first failed result comes in. If you don't pass this it will wait until 100% of the run is done. Use with `--fg`.
 - `--custom-url` - use a custom url for this run. Example use case: an ad-hoc QA environment with [Fourchette](https://github.com/rainforestapp/fourchette). You will need to specify a `site_id` too for this to work. Note that we will be creating a new environment for this particular run.
 - `--git-trigger` - only trigger a run when the last commit (for a git repo in the current working directory) has contains `@rainforest` and a list of one or more tags. E.g. "Fix checkout process. @rainforest #checkout" would trigger a run for everything tagged `checkout`. This over-rides `--tag` and any tests specified. If no `@rainforest` is detected it will exit 0.
@@ -180,6 +209,12 @@ Popular command line options are:
 - `--embed-tests` - Use with `rainforest export` to export your tests without extracting the
 steps of an embedded test.
 - `--test-folder /path/to/directory` - Use with `rainforest [new, upload, export]`. If this option is not provided, rainforest-cli will, in the case of 'new' create a directory, or in the case of 'upload' and 'export' use the directory, at the default path `./spec/rainforest/`.
+- `--junit-file` - Create a junit xml report file with the specified name.  Must be run in foreground mode, or with the report command. Uses the rainforest
+api to construct a junit report.  This is useful to track tests in CI such as Jenkins or Bamboo.
+- `--run-id` - Only used with the report command.  Specify a past rainforest run by ID number to generate a report for.
+- `--import-variable-csv-file /path/to/csv/file.csv` - Use with `run` and `--import-variable-name` to upload new tabular variable values before your run to specify the path to your CSV file.
+- `--import-variable-name NAME` - Use with `run` and `import-variable-csv-file` to upload
+new tabular variable values before your run to specify the name of your tabular variable.
 
 
 #### Specifying Test IDs
@@ -192,7 +227,7 @@ test IDs taken from the Rainforest dashboard. ie:
 All other argument types should be specified as seen above.
 
 
-More detailed info on options can be [found here](https://github.com/rainforestapp/rainforest-cli/blob/master/lib/rainforest/cli/options.rb#L23-L74).
+More detailed info on options can be [found here](https://github.com/rainforestapp/rainforest-cli/blob/master/lib/rainforest_cli/options.rb#L23-L74).
 
 ## Support
 

data/bin/rainforest

@@ -2,6 +2,6 @@
 
 $LOAD_PATH.unshift File.join(File.dirname(__FILE__), "../lib")
 
-require "rainforest/cli"
+require "rainforest_cli"
 
 RainforestCli.start(ARGV)

data/lib/{rainforest/cli.rb → rainforest_cli.rb}

@@ -2,25 +2,28 @@
 require 'erb'
 require 'json'
 require 'logger'
-require 'rainforest/cli/version'
-require 'rainforest/cli/constants'
-require 'rainforest/cli/options'
-require 'rainforest/cli/runner'
-require 'rainforest/cli/http_client'
-require 'rainforest/cli/git_trigger'
-require 'rainforest/cli/csv_importer'
-require 'rainforest/cli/test_parser'
-require 'rainforest/cli/test_files'
-require 'rainforest/cli/remote_tests'
-require 'rainforest/cli/validator'
-require 'rainforest/cli/exporter'
-require 'rainforest/cli/deleter'
-require 'rainforest/cli/uploader'
-require 'rainforest/cli/resources'
+require 'rainforest_cli/version'
+require 'rainforest_cli/constants'
+require 'rainforest_cli/options'
+require 'rainforest_cli/runner'
+require 'rainforest_cli/http_client'
+require 'rainforest_cli/git_trigger'
+require 'rainforest_cli/csv_importer'
+require 'rainforest_cli/test_parser'
+require 'rainforest_cli/test_files'
+require 'rainforest_cli/remote_tests'
+require 'rainforest_cli/validator'
+require 'rainforest_cli/exporter'
+require 'rainforest_cli/deleter'
+require 'rainforest_cli/uploader'
+require 'rainforest_cli/resources'
+require 'rainforest_cli/junit_outputter'
+require 'rainforest_cli/reporter'
 
 module RainforestCli
   def self.start(args)
     options = OptionParser.new(args)
+    @http_client = HttpClient.new(token: options.token)
     OptionParser.new(['--help']) if args.size == 0
 
     begin
@@ -37,6 +40,7 @@ module RainforestCli
     when 'upload' then Uploader.new(options).upload
     when 'rm' then Deleter.new(options).delete
     when 'export' then Exporter.new(options).export
+    when 'report' then Reporter.new(options).report
     when 'sites', 'folders', 'browsers'
       Resources.new(options).public_send(options.command)
     else
@@ -54,4 +58,8 @@ module RainforestCli
   def self.logger=(logger)
     @logger = logger
   end
+
+  def self.http_client
+    @http_client
+  end
 end

data/lib/{rainforest/cli → rainforest_cli}/http_client.rb

@@ -17,7 +17,7 @@ module RainforestCli
       response = HTTParty.delete make_url(url), {
         body: body,
         headers: headers,
-        verify: false
+        verify: false,
       }
 
       JSON.parse(response.body)
@@ -28,7 +28,7 @@ module RainforestCli
         response = HTTParty.post make_url(url), {
           body: body,
           headers: headers,
-          verify: false
+          verify: false,
         }
 
         return JSON.parse(response.body)
@@ -40,12 +40,13 @@ module RainforestCli
         response = HTTParty.get make_url(url), {
           body: body,
           headers: headers,
-          verify: false
+          verify: false,
         }
 
         if response.code == 200
           return JSON.parse(response.body)
         else
+          RainforestCli.logger.warn("Status Code: #{response.code}, #{response.body}")
           return nil
         end
       end
@@ -89,7 +90,7 @@ module RainforestCli
     def headers
       {
         'CLIENT_TOKEN' => @token,
-        'User-Agent' => "Rainforest-cli-#{RainforestCli::VERSION}"
+        'User-Agent' => "Rainforest-cli-#{RainforestCli::VERSION}",
       }
     end
   end

data/lib/rainforest_cli/junit_outputter.rb

@@ -0,0 +1,69 @@
+require 'builder'
+require 'time'
+require 'json'
+
+# frozen_string_literal: true
+module RainforestCli
+  class JunitOutputter
+    attr_reader :builder, :client
+
+    def initialize(token, run, tests)
+      @client = HttpClient.new token: token
+      @json_run = run # JSON containing the results of /1/runs/{run_id}.json
+      @json_tests = tests # JSON containing the results of /1/runs/{run_id}/tests.json
+      @builder = Builder::XmlMarkup.new( :indent => 2)
+    end # end initialize
+
+    def build_test_suite
+      @json_tests.each do | test |
+        build_test test
+      end # end do
+    end # end process_run_results
+
+    def build_test(test)
+      test_name = test['title']
+      execution_time = Time.parse(test['updated_at']) - Time.parse(test['created_at'])
+      test_status = test['result']
+      @builder.testcase(:name => test_name, :time => execution_time) do
+        case test_status
+        when "failed"
+          build_failed_test test
+        end # end case
+      end # end do
+    end # end build_test
+
+    def build_failed_test(test)
+      response = client.get("/runs/#{@json_run['id']}/tests/#{test['id']}.json")
+      response['steps'].each do | step |
+        step['browsers'].each do | browser |
+          browser_name = browser['name']
+          browser['feedback'].each do | opinion |
+            if opinion['answer_given'] == 'no' and opinion['job_state'] == 'approved'
+              if opinion['note'] != ""
+                @builder.failure(:type => browser_name, :message => opinion['note'])
+              end
+            end
+          end
+        end
+      end
+    end # end build_failed_test
+
+    def parse
+      @builder.instruct! :xml, :version => "1.0", :encoding => "UTF-8"
+      @builder.testsuite(
+        :name => @json_run['description'],
+        :errors => @json_run['total_no_result_tests'],
+        :failures => @json_run['total_failed_tests'],
+        :tests => @json_run['total_tests'],
+        :time => Time.parse(@json_run['timestamps']['complete']) - Time.parse(@json_run['timestamps']['created_at']),
+        :timestamp => @json_run['created_at']) do
+          build_test_suite
+        end
+    end # end parse
+
+    def output(stream)
+      stream.write(@builder.target!)
+    end #end output
+
+  end # end class
+end # end module

data/lib/{rainforest/cli → rainforest_cli}/options.rb

@@ -6,7 +6,8 @@ module RainforestCli
     attr_writer :file_name, :tags
     attr_reader :command, :token, :tags, :conflict, :browsers, :site_id, :environment_id,
                 :import_file_name, :import_name, :custom_url, :description, :folder,
-                :debug, :file_name, :test_folder, :embed_tests, :app_source_url, :crowd
+                :debug, :file_name, :test_folder, :embed_tests, :app_source_url, :crowd, :run_id,
+                :junit_file
 
     TOKEN_NOT_REQUIRED = %w{new validate}.freeze
 
@@ -15,6 +16,8 @@ module RainforestCli
       @tags = []
       @browsers = nil
       @debug = false
+      @junit_file = nil
+      @run_id = nil
       @token = ENV['RAINFOREST_API_TOKEN']
 
       # NOTE: Disabling line length cop to allow for consistency of syntax
@@ -55,6 +58,11 @@ module RainforestCli
           @foreground = value
         end
 
+        opts.on('--wait ID', Integer, 'Wait for run #ID to complete') do |value|
+          @wait = true
+          @run_id = value
+        end
+
         opts.on('--fail-fast', String, "Fail as soon as there is a failure (don't wait for completion)") do |_value|
           @failfast = true
         end
@@ -103,6 +111,14 @@ module RainforestCli
           @app_source_url = value
         end
 
+        opts.on('--junit-file FILE', 'Gather the results of a run and create junit output in FILE.xml, must be run with --fg') do |value|
+          @junit_file = value
+        end
+
+        opts.on('--run-id ID', 'Gather the results of a completed run, must be run with export and --junit-file') do |value|
+          @run_id = value
+        end
+
         opts.on_tail('--help', 'Display help message and exit') do |_value|
           puts opts
           exit 0
@@ -142,10 +158,18 @@ module RainforestCli
       @failfast
     end
 
+    def wait?
+      !!@wait
+    end
+
     def foreground?
       @foreground
     end
 
+    def junit_file?
+      @junit_file
+    end
+
     def validate!
       if !TOKEN_NOT_REQUIRED.include?(command)
         unless token
@@ -169,6 +193,21 @@ module RainforestCli
         raise ValidationError, 'You must include a file name'
       end
 
+      if command == 'run' && junit_file?
+        unless foreground?
+          raise ValidationError, 'You can only generate junit test output in foreground mode'
+        end
+      end
+
+      if command == 'report'
+        if junit_file.nil?
+          raise ValidationError, 'You must specify a junit ouptut filename'
+        end
+        if run_id.nil?
+          raise ValidationError, 'You must specify a run-id to generate a report for'
+        end
+      end
+
       true
     end
 

data/lib/rainforest_cli/reporter.rb

@@ -0,0 +1,64 @@
+require 'fileutils'
+
+#frozen_string_literal: true
+module RainforestCli
+  class Reporter
+    attr_reader :client
+    attr_writer :run_id
+
+    def initialize(options)
+      @options = options
+      @client = HttpClient.new token: options.token
+      @run_id = options.run_id
+      @output_filename = options.junit_file
+    end
+
+    def report
+      if @run_id == nil
+        logger.fatal "Reporter needs a valid run_id to report on"
+      else
+        logger.info "Generating JUNIT report for #{@run_id} : #{@output_filename}"
+      end
+
+      run = client.get("/runs/#{@run_id}.json")
+
+      if run == nil
+        logger.fatal "Non 200 code recieved"
+        exit 1
+      end
+
+      if run['error']
+        logger.fatal "Error retrieving results for your run: #{run['error']}"
+        exit 1
+      end
+
+      if run.has_key?('total_tests') and run['total_tests'] != 0
+        tests = client.get("/runs/#{@run_id}/tests.json?page_size=#{run['total_tests']}")
+
+        if tests == nil
+          logger.fatal "Non 200 code recieved"
+          exit 1
+        end
+
+        if tests.kind_of?(Hash) and tests['error'] # if this had worked tests would be an array
+          logger.fatal "Error retrieving test details for your run: #{tests['error']}"
+          exit 1
+        end
+
+        outputter = JunitOutputter.new(@options.token, run, tests)
+        outputter.parse
+      end
+
+      unless File.directory?(File.dirname(@output_filename))
+        FileUtils.mkdir_p(File.dirname(@output_filename))
+      end
+
+      File.open(@output_filename, 'w') { |file| outputter.output(file) }
+    end
+
+    def logger
+      RainforestCli.logger
+    end
+
+  end
+end