test_spec 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e5f6724300184d3600f08828c91a0d61a21208aa
-  data.tar.gz: 896de6158af5a26a3a288d9297867e03db771a19
+  metadata.gz: e2a3e183e6720aca0feee09a37aa7c46523b4108
+  data.tar.gz: 6d71391ab4677b7ece78333417a8a12f6aa857c4
 SHA512:
-  metadata.gz: fec3edcf80e74b76bb0d262df4aadd56a5470790fa32653bb176022675fa01736553dd34d7f5079f9ab188ecf1c603588259c091aa23340168912c1aa41906bf
-  data.tar.gz: cdc157ed2065d4b9f58ccfb186128c88f1bdb28c819801f0a149d1d277b16f375f45a87d022327d71169b5621a29e01f58581268b03489f63d3a22dc78778d31
+  metadata.gz: b4602a79d42b6b133d49d3d5a1751760299b87dcdba3f9ddbf604cfab6a90580d6c170dd7daf8b2513d54ed410050365408366452743435dc01ca810280cb391
+  data.tar.gz: 8d576bdf2e726b3f2cd6f8b502d9502867955bb8fca1ab1235be912864fa49551472e04a6419d9565edab526d56d759d1df1fcf902ab26fc2e259b587170de4f

data/.gitignore

@@ -11,6 +11,9 @@
 /spec/coverage/
 /tmp/
 
+# Generated Reports
+reports/
+
 # Rspec Failure Tracking
 
 .rspec_status

data/.hound.yml

@@ -67,3 +67,13 @@ Metrics/ParameterLists:
 # Remove execute permissions check.
 Lint/ScriptPermission:
   Enabled: false
+
+# Allow methods with has_ for predicates.
+Naming/PredicateName:
+  NameWhitelist:
+    - has_failed_screenshot?
+    - has_exception?
+    - has_screenshots?
+    - has_screenrecord?
+    - has_spec?
+    - has_comment?

data/README.md

@@ -60,7 +60,7 @@ Because TestSpec uses a custom formatter, you should have an `.rspec` file with
 --format RSpec::TestSpec::Formatter
 ```
 
-You can use RSpec constructs within Specify constructs although there are some things to be aware of. Here is an example:
+You can use RSpec constructs within TestSpec constructs although there are some things to be aware of. Here is an example:
 
 ```ruby
 Feature 'Bank Accounts' do
@@ -68,7 +68,7 @@ Feature 'Bank Accounts' do
   subject { Account.new(valid_account_number) }
 
   Scenario 'starting a new account' do
-    Test 'will have a starting balance of 0' do
+    test 'will have a starting balance of 0' do
       expect(subject.balance).to eq(0)
     end
 
@@ -79,7 +79,7 @@ Feature 'Bank Accounts' do
 end
 ```
 
-You can see that within the Feature construct I have let and subject elements. Within the Scenario you can see I use a Specify method (Test) and an RSpec method (it).
+You can see that within the Feature construct I have let and subject elements. Within the Scenario you can see I use a TestSpec method (Test) and an RSpec method (it).
 
 ## Documentation
 
@@ -183,6 +183,171 @@ You can also see here that multiple **Scenario** blocks can be included within a
 
 This should give a rough idea of how TestSpec provides an internal DSL.
 
+### TestSpec API
+
+The [unit tests](https://github.com/jeffnyman/test_spec/tree/master/spec) will give you some idea of how TestSpec works.
+
+To use TestSpec as an RSpec overlay, you have **descriptive containers** with the following keywords:
+
+* Feature, Ability, Story, Component, Workflow
+
+These are defined in the [spec.rb](https://github.com/jeffnyman/test_spec/blob/master/lib/test_spec/spec.rb) file.
+
+You have **example group sequences** with the following keywords:
+
+* Scenario, Condition, Behavior
+* Step, Test, Rule, Fact
+* steps, tests, rules, facts
+
+These are defined in the [test_spec.rb](https://github.com/jeffnyman/test_spec/blob/master/lib/test_spec.rb) file.
+
+You have **example steps** with the following keywords:
+
+* (Gherkin steps) Given, When, Then, And, But
+* (RSpec steps) it, specify, example
+* (Specify steps) step, test, rule, fact
+
+These are defined in the [example_group.rb](https://github.com/jeffnyman/test_spec/blob/master/lib/test_spec/rspec/example_group.rb) file.
+
+The API is basically this: **Containers contain example groups that contain example steps.** Everything must essentially be nested in that order. This allows you to follow just about every xSpec or xBehave pattern out there.
+
+Some representative examples:
+
+```ruby
+Feature 'some feature description' do
+  Scenario 'some test condition' do
+    Given 'some context' do
+    end
+
+    When 'some action' do
+    end
+
+    Then 'some observable' do
+    end
+  end
+end
+```
+
+```ruby
+Workflow 'some workflow description' do
+  Behavior 'some behavior description' do
+    example 'some test condition' do
+    end
+
+    example 'some other test condition' do
+    end
+  end
+end
+```
+
+```ruby
+Component 'some component name' do
+  rules 'some high-level rules condition' do
+    rule 'some specific rule test condition' do
+    end
+
+    rule 'some other specific rule test condition' do
+    end
+  end
+end
+```
+
+A few notes on this.
+
+Example group sequences and example steps cannot be top level. Only descriptive containers can be the top level artifact in a test spec. Also, example steps cannot be directly under descriptive containers. For example, you can't do this:
+
+```ruby
+Ability 'ability keyword' do
+  When 'when keyword' do
+  end
+end
+```
+
+Here the `When` (an example step) is under a descriptive container (`Ability`). But it needs to be within an example group (such as `Scenario`).
+
+It's also worth noting that descriptive containers cannot be used inside example groups. For example, you can't do this:
+
+```ruby
+Ability 'ability keyword' do
+  Scenario 'scenario keyword' do
+    Story 'story keyword' do
+    end
+  end
+end
+```
+
+Here the `Story` (a descriptive container) is used under a example group (`Scenario`). Descriptive containers, however, are top-level constructs. They provide a grouping method for one or more example groups.
+
+You can nest descriptive containers if you feel that provides better understanding of your test spec. For example:
+
+```ruby
+Feature 'feature keyword' do
+  Workflow 'workflow keyword' do
+    Component 'component #1' do
+      Scenario 'scenario' do
+      end
+    end
+
+    Component 'component #2' do
+      Scenario 'scenario' do
+      end
+    end
+  end
+end
+```
+
+Here notice that the containers `Feature`, `Workflow` and `Component` are used together to indicate a relationship about what is being tested. Each of the innermost containers then has some example groups (`Scenario` in this case) which can contain specific tests.
+
+## Design Rationale
+
+TestSpec is a micro-framework.
+
+A micro-framework provides a focused solution, which means it does one thing and one thing only, instead of trying to solve each and every problem. While doing that one thing it does well, the micro-framework should do it while being expressive and concise. Further, it should be able to serve as one component of your own custom modularized framework, allowing you to compose solutions.
+
+### Frameworks are Effective and Efficient
+
+I believe that test solution frameworks require effective functionality, an efficient user experience, and demonstrable business impact. This all has to be delivered, deployed, and supported quickly and cost-effectively. You want the equivalent of a lower total cost of ownership and a faster time-to-market for whatever you produce (such as test artifacts).
+
+### Frameworks are Comfortable
+
+Like a programming language, a framework needs to be something you're comfortable with -- something that reflects your personal style and mode of working. That's what TestSpec is for me.
+
+TestSpec allows me to continue working with a Gherkin-like structure but putting my test specs closer to the code. Just as in RSpec, everything is in one place: the Ruby file. The benefit here is that you don't have to change the text in two places -- feature files and step definition files -- every time you change something. Further, there are no more matchers to sync up with natural language. TestSpec uses plain Ruby helper methods coupled with various patterns.
+
+### Frameworks Provide
+
+A framework provides a few key aspects.
+
+A place for everything: Structure and convention drive a good framework. Everything should have a proper place within the system; this eliminates guesswork and increases productivity.
+
+A culture and aesthetic to help inform programming decisions: Rather than seeing the structure imposed by a framework as constraining, see it as liberating. A good framework encodes its opinions, gently guiding you. Often, difficult decisions are made for you by virtue of convention. The culture of the framework helps you make fewer menial decisions and helps you focus on what matters most.
+
+### Frameworks Have Qualities
+
+I believe a test framework with good qualities will ...
+
+* Encode opinions.
+* Have an elegant, concise syntax.
+* Have powerful metaprogramming features.
+* Be well suited as a host language for creating DSLs.
+* Allow for an open ecosystem.
+
+### TestSpec (Conceptually) Compared
+
+Even though their domains are different, consider TestSpec in light of Rails. Rails is more than a programming framework for creating web applications. It's also a framework for thinking about web applications.
+
+Rails ships not as a blank slate equally tolerant of every kind of expression. On the contrary, Rails trades that flexibility for the convenience of "what most people need most of the time to do most things."
+
+You could argue that Rails is a designer straightjacket. Yet this straightjacket sets you free from focusing on the things that just don't matter and focuses your attention on the stuff that does. To be able to accept that trade, you need to understand not just how to do something in Rails, but also why it's done like that. Only by understanding the why will you be able to consistently work with the framework instead of against it.
+
+It doesn't mean that you'll always have to agree with a certain choice, but you will need to agree to the overachieving principle of conventions. You have to learn to relax and let go of your attachment to personal idiosyncrasies when the productivity rewards are right.
+
+So it is with TestSpec.
+
+A framework goal is to solve 80% of the problems that occur in your testing domain, assuming that the remaining 20% are problems that are unique to the application's domain. This implies that 80% of the code in an application is infrastructure. So here instead of focusing on the details of knitting an application together, you get to focus on the 20% that really matters.
+
+The framework lets you start right away by encompassing a set of intelligent decisions about how your logic should work and alleviating the amount of low-level decision making you need to do up front. As a result, you can focus on the problems you're trying to solve and get the job done more quickly.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rake spec:all` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/lib/test_spec/rspec/example.rb

@@ -0,0 +1,97 @@
+require "test_spec/rspec/problem"
+
+class Example
+  attr_reader :run_time, :status, :duration, :exception, :description,
+    :full_description, :example_group, :failed_screenshot,
+    :screenshots, :screenrecord, :file_path, :metadata, :spec
+
+  # rubocop:disable Metrics/AbcSize
+  def initialize(example)
+    @execution_result = example.execution_result
+    @run_time = @execution_result.run_time.round(5)
+    @status = @execution_result.status.to_s
+    @example_group = example.example_group.to_s
+    @description = example.description
+    @full_description = example.full_description
+    @metadata = example.metadata
+    @duration = @execution_result.run_time.to_s(:rounded, precision: 5)
+    @screenrecord = @metadata[:screenrecord]
+    @screenshots = @metadata[:screenshots]
+    @spec = nil
+    @file_path = @metadata[:file_path]
+    @exception = Problem.new(example, @file_path)
+    @failed_screenshot = @metadata[:failed_screenshot]
+    @comment = @metadata[:comment]
+  end
+  # rubocop:enable Metrics/AbcSize
+
+  def example_title
+    title_arr = @example_group.to_s.split('::') - %w[RSpec ExampleGroups]
+    title_arr.push @description
+    title_arr.join(' → ')
+  end
+
+  def comment
+    ERB::Util.html_escape(@comment)
+  end
+
+  def has_spec?
+    !@spec.nil?
+  end
+
+  def has_comment?
+    !@comment.nil?
+  end
+
+  def has_screenrecord?
+    !@screenrecord.nil?
+  end
+
+  def has_screenshots?
+    !@screenshots.nil? && !@screenshots.empty?
+  end
+
+  def has_failed_screenshot?
+    !@failed_screenshot.nil?
+  end
+
+  def has_exception?
+    !@exception.klass.nil?
+  end
+
+  def klass(prefix = 'label-')
+    class_map = {
+      passed: "#{prefix}success",
+      failed: "#{prefix}danger",
+      pending: "#{prefix}warning"
+    }
+
+    class_map[@status.to_sym]
+  end
+
+  def create_spec_line(spec_text)
+    formatter = Rouge::Formatters::HTML.new(css_class: 'highlight')
+    lexer = Rouge::Lexers::Gherkin.new
+    @spec = formatter.format(lexer.lex(spec_text.gsub('#->', '')))
+  end
+
+  # rubocop:disable Metrics/LineLength
+  # rubocop:disable Metrics/AbcSize
+  def self.load_spec_comments!(examples)
+    examples.group_by(&:file_path).each do |file_path, file_examples|
+      lines = File.readlines(file_path)
+
+      file_examples.zip(file_examples.rotate).each do |ex, next_ex|
+        lexically_next = next_ex &&
+                         next_ex.file_path == ex.file_path &&
+                         next_ex.metadata[:line_number] > ex.metadata[:line_number]
+        start_line_idx = ex.metadata[:line_number] - 1
+        next_start_idx = (lexically_next ? next_ex.metadata[:line_number] : lines.size) - 1
+        spec_lines = lines[start_line_idx...next_start_idx].select { |l| l.match(/#->/) }
+        ex.create_spec_line(spec_lines.join) unless spec_lines.empty?
+      end
+    end
+  end
+  # rubocop:enable Metrics/LineLength
+  # rubocop:enable Metrics/AbcSize
+end

data/lib/test_spec/rspec/formatter.rb

@@ -1,24 +1,75 @@
+require "test_spec/rspec/example"
+require "erb"
+require "rouge"
+require "fileutils"
+require "active_support"
+require "active_support/inflector"
+require "active_support/core_ext/numeric"
 require "rspec/core/formatters/documentation_formatter"
 
 module RSpec
   module TestSpec
+    # rubocop:disable Metrics/ClassLength
     class Formatter < ::RSpec::Core::Formatters::DocumentationFormatter
       ::RSpec::Core::Formatters.register(
         self,
         :example_started, :example_passed, :example_step_passed,
-        :example_step_pending, :example_step_failed
+        :example_step_pending, :example_pending, :example_step_failed,
+        :example_group_finished, :example_group_started
       )
 
+      DEFAULT_REPORT_PATH = File.join(
+        '.', 'reports', Time.now.strftime('%Y%m%d-%H%M%S')
+      )
+
+      REPORT_PATH = ENV['REPORT_PATH'] || DEFAULT_REPORT_PATH
+      SCREENRECORD_DIR = File.join(REPORT_PATH, 'screenrecords')
+      SCREENSHOT_DIR   = File.join(REPORT_PATH, 'screenshots')
+
+      def initialize(_output)
+        super
+        create_report_directory
+        create_screenshots_directory
+        create_screenrecords_directory
+        provide_resources
+
+        @all_groups = {}
+
+        # REPEATED FROM THE SUPER
+        # @group_level = 0
+      end
+
       # rubocop:disable Metrics/LineLength
       def example_started(notification)
         return unless notification.example.metadata[:with_steps]
 
         full_message = "#{current_indentation}#{notification.example.description}"
         output.puts Core::Formatters::ConsoleCodes.wrap(full_message, :default)
+
+        # For reporter
+        @group_example_count += 1
       end
 
+      # This comes from the DocumentationFormatter.
       def example_passed(notification)
         super unless notification.example.metadata[:with_steps]
+
+        # For reporter
+        @group_example_success_count += 1
+        @examples << Example.new(notification.example)
+      end
+
+      # This comes from the DocumentationFormatter.
+      def example_failed(notification)
+        @group_example_failure_count += 1
+        @examples << Example.new(notification.example)
+      end
+
+      # This comes from the DocumentationFormatter.
+      # Needed for Reporter.
+      def example_pending(notification)
+        @group_example_pending_count += 1
+        @examples << Example.new(notification.example)
       end
 
       def example_step_passed(notification)
@@ -48,6 +99,151 @@ module RSpec
         output.puts Core::Formatters::ConsoleCodes.wrap(full_message, :failure)
       end
       # rubocop:enable Metrics/LineLength
+
+      # ADDED FOR REPORTING
+
+      def example_group_started(_notification)
+        if @group_level.zero?
+          @examples = []
+          @group_example_count = 0
+          @group_example_success_count = 0
+          @group_example_failure_count = 0
+          @group_example_pending_count = 0
+        end
+
+        super
+
+        # REPEATED FROM THE SUPER
+        # @group_level += 1
+      end
+
+      # rubocop:disable Metrics/LineLength
+      # rubocop:disable Metrics/MethodLength
+      # rubocop:disable Metrics/AbcSize
+      def example_group_finished(notification)
+        super
+
+        return unless @group_level.zero?
+
+        # rubocop:disable Metrics/BlockLength
+        File.open("#{REPORT_PATH}/#{notification.group.description.parameterize}.html", "w") do |f|
+          @passed = @group_example_success_count
+          @failed = @group_example_failure_count
+          @pending = @group_example_pending_count
+
+          duration_values = @examples.map(&:run_time)
+          duration_keys = duration_values.size.times.to_a
+
+          if duration_values.size < 2 && !duration_values.empty?
+            duration_values.unshift(duration_values.first)
+            duration_keys = duration_keys << 1
+          end
+
+          @title = notification.group.description
+          @durations = duration_keys.zip(duration_values)
+          @summary_duration = duration_values.inject(0) { |sum, i| sum + i }.to_s(:rounded, precision: 5)
+
+          Example.load_spec_comments!(@examples)
+
+          class_map = {
+            passed: 'success',
+            failed: 'danger',
+            pending: 'warning'
+          }
+
+          statuses = @examples.map(&:status)
+
+          status =
+            if statuses.include?('failed')
+              'failed'
+            elsif statuses.include?('passed')
+              'passed'
+            else
+              'pending'
+            end
+
+          @all_groups[notification.group.description.parameterize] = {
+            group: notification.group.description,
+            examples: @examples.size,
+            status: status,
+            klass: class_map[status.to_sym],
+            passed: statuses.select { |s| s == 'passed' },
+            failed: statuses.select { |s| s == 'failed' },
+            pending: statuses.select { |s| s == 'pending' },
+            duration: @summary_duration
+          }
+
+          template_file = File.read(
+            File.dirname(__FILE__) + "/../../../templates/report.erb"
+          )
+
+          f.puts ERB.new(template_file).result(binding)
+        end
+        # rubocop:enable Metrics/BlockLength
+
+        # THIS ONE IS FROM THE SUPER
+        # @group_level -= 1 if @group_level > 0
+      end
+      # rubocop:enable Metrics/MethodLength
+      # rubocop:enable Metrics/AbcSize
+
+      # This is from BaseTextFormatter.
+      # rubocop:disable Metrics/MethodLength
+      # rubocop:disable Metrics/AbcSize
+      def close(notification)
+        File.open("#{REPORT_PATH}/overview.html", "w") do |f|
+          @overview = @all_groups
+
+          @passed = @overview.values.map { |g| g[:passed].size }.inject(0) { |sum, i| sum + i }
+          @failed = @overview.values.map { |g| g[:failed].size }.inject(0) { |sum, i| sum + i }
+          @pending = @overview.values.map { |g| g[:pending].size }.inject(0) { |sum, i| sum + i }
+
+          duration_values = @overview.values.map { |e| e[:duration] }
+          duration_keys = duration_values.size.times.to_a
+
+          if duration_values.size < 2
+            duration_values.unshift(duration_values.first)
+            duration_keys = duration_keys << 1
+          end
+
+          @durations = duration_keys.zip(duration_values.map { |d| d.to_f.round(5) })
+          @summary_duration = duration_values.map { |d| d.to_f.round(5) }.inject(0) { |sum, i| sum + i }.to_s(:rounded, precision: 5)
+          @total_examples = @passed + @failed + @pending
+
+          template_file = File.read(
+            File.dirname(__FILE__) + "/../../../templates/overview.erb"
+          )
+
+          f.puts ERB.new(template_file).result(binding)
+        end
+
+        super
+      end
+      # rubocop:enable Metrics/AbcSize
+      # rubocop:enable Metrics/MethodLength
+      # rubocop:enable Metrics/LineLength
+
+      private
+
+      def create_report_directory
+        FileUtils.rm_rf(REPORT_PATH) if File.exist?(REPORT_PATH)
+        FileUtils.mkpath(REPORT_PATH)
+      end
+
+      def create_screenshots_directory
+        FileUtils.mkdir_p SCREENSHOT_DIR unless File.exist?(SCREENSHOT_DIR)
+      end
+
+      def create_screenrecords_directory
+        FileUtils.mkdir_p SCREENRECORD_DIR unless File.exist?(SCREENRECORD_DIR)
+      end
+
+      def provide_resources
+        FileUtils.cp_r(
+          File.dirname(__FILE__) + "/../../../resources", REPORT_PATH
+        )
+      end
     end
+    # rubocop:enable Metrics/ClassLength
   end
 end