busser-behave 0.1.3

data/vendor/behave/features/formatter.progress3.feature

@@ -0,0 +1,235 @@
+@sequential
+Feature: Progress3 Formatter
+
+    In order to effectively analyze output of many runs
+    As a tester
+    I want that behave generates to present results for each scenario
+
+    @setup
+    Scenario: Feature Setup
+        Given a new working directory
+        And a file named "features/steps/steps.py" with:
+            """
+            from behave import step
+
+            @step('{word:w} step passes')
+            def step_passes(context, word):
+                pass
+
+            @step('{word:w} step fails')
+            def step_fails(context, word):
+                assert False, "XFAIL-STEP"
+
+            @step(u'a step raises an error "{message}"')
+            def step_raises_exception(context, message):
+                raise RuntimeError(message)
+            """
+
+    Scenario: Use Progress3 formatter on simple feature
+        Given a file named "features/simple_feature_with_name.feature" with:
+            """
+            Feature: Simple, empty Feature
+            """
+        When I run "behave -f progress3 features/simple_feature_with_name.feature"
+        Then it should pass with:
+            """
+            0 features passed, 0 failed, 1 skipped
+            0 scenarios passed, 0 failed, 0 skipped
+            """
+        And the command output should contain:
+            """
+            Simple, empty Feature    # features/simple_feature_with_name.feature
+            """
+
+    Scenario: Use Progress3 formatter with feature and one scenario without steps
+        Given a file named "features/simple_scenario.feature" with:
+            """
+            Feature: Simple feature with one scenario
+              Scenario: Simple scenario without steps
+            """
+        When I run "behave -f progress3 features/simple_scenario.feature"
+        Then it should pass with:
+            """
+            1 feature passed, 0 failed, 0 skipped
+            1 scenario passed, 0 failed, 0 skipped
+            """
+        And the command output should contain:
+            """
+            Simple feature with one scenario    # features/simple_scenario.feature
+              Simple scenario without steps
+            """
+
+    Scenario: Use Progress3 formatter with feature and one scenario with all passing steps
+        Given a file named "features/scenario_with_steps.feature" with:
+            """
+            Feature: Feature with scenario
+              Scenario: Simple scenario with passing steps
+                  Given a step passes
+                  When another step passes
+                  Then third step passes
+                  And fourth step passes
+                  But fifth step passes
+            """
+        When I run "behave -f progress3 features/scenario_with_steps.feature"
+        Then it should pass with:
+            """
+            1 feature passed, 0 failed, 0 skipped
+            1 scenario passed, 0 failed, 0 skipped
+            """
+        And the command output should contain:
+            """
+            Feature with scenario    # features/scenario_with_steps.feature
+              Simple scenario with passing steps  .....
+            """
+
+    Scenario: Use Progress3 formatter with feature and one scenario with a failing step
+        Given a file named "features/scenario_with_steps.feature" with:
+            """
+            Feature: Feature with scenario
+              Scenario: Simple scenario with last failing step
+                  Given a step passes
+                  When second step passes
+                  Then third step passes
+                  And another step passes
+                  But last step fails
+            """
+        When I run "behave -f progress3 features/scenario_with_steps.feature"
+        Then it should fail with:
+            """
+            0 features passed, 1 failed, 0 skipped
+            0 scenarios passed, 1 failed, 0 skipped
+            """
+        And the command output should contain:
+            """
+            Feature with scenario    # features/scenario_with_steps.feature
+                Simple scenario with last failing step  ....F
+            --------------------------------------------------------------------------------
+            FAILURE in step 'last step fails' (features/scenario_with_steps.feature:7):
+            Assertion Failed: XFAIL-STEP
+            --------------------------------------------------------------------------------
+            """
+
+    Scenario: Use Progress3 formatter with feature and one scenario with an exception in the step
+        Given a file named "features/scenario_with_steps.feature" with:
+            """
+            Feature: Feature with scenario
+              Scenario: Simple scenario with error in the step
+                  Given a step passes
+                  When second step passes
+                  Then third step passes
+                  And another step passes
+                  But a step raises an error "Error message here"
+            """
+        When I run "behave -f progress3 features/scenario_with_steps.feature"
+        Then it should fail with:
+            """
+            0 features passed, 1 failed, 0 skipped
+            0 scenarios passed, 1 failed, 0 skipped
+            """
+        And the command output should contain:
+            """
+            Feature with scenario    # features/scenario_with_steps.feature
+                Simple scenario with error in the step  ....E
+            --------------------------------------------------------------------------------
+            FAILURE in step 'a step raises an error "Error message here"' (features/scenario_with_steps.feature:7):
+            """
+        And the command output should contain:
+            """
+            RuntimeError: Error message here
+            
+            --------------------------------------------------------------------------------
+            """
+
+    Scenario: Use Progress3 formatter with feature and three scenarios with all passing steps
+        Given a file named "features/scenario_with_steps.feature" with:
+            """
+            Feature: Feature with three scenarios
+              Scenario: First scenario with passing steps
+                  Given a step passes
+                  When second step passes
+                  Then third step passes
+                  And another step passes
+                  But last step passes
+
+              Scenario: Second scenario with passing steps
+                  Given a step passes
+                  When second step passes
+                  Then third step passes
+                  And another step passes
+                  But last step passes
+
+              Scenario: Third scenario with passing steps
+                  Given a step passes
+                  When second step passes
+                  Then third step passes
+                  And another step passes
+                  But last step passes
+            """
+        When I run "behave -f progress3 features/scenario_with_steps.feature"
+        Then it should pass with:
+            """
+            1 feature passed, 0 failed, 0 skipped
+            3 scenarios passed, 0 failed, 0 skipped
+            15 steps passed, 0 failed, 0 skipped, 0 undefined
+            """
+        And the command output should contain:
+            """
+            Feature with three scenarios    # features/scenario_with_steps.feature
+                First scenario with passing steps  .....
+                Second scenario with passing steps  .....
+                Third scenario with passing steps  .....
+            """
+
+    Scenario: Use Progress3 formatter with feature and three scenarios with a failing step
+        Given a file named "features/scenario_with_steps.feature" with:
+            """
+            Feature: Feature with various results in scenarios
+              Scenario: First scenario with passing steps
+                  Given a step passes
+                  When second step passes
+                  Then third step passes
+                  And another step passes
+                  But last step passes
+
+              Scenario: Second scenario with second failing step
+                  Given a step passes
+                  When second step fails
+                  Then third step passes
+                  And another step passes
+                  But last step passes
+
+              Scenario: Third scenario with fourth failing step
+                  Given a step passes
+                  When second step passes
+                  Then third step passes
+                  And fourth step fails
+                  But last step passes
+            """
+        When I run "behave -f progress3 features/scenario_with_steps.feature"
+        Then it should fail with:
+            """
+            0 features passed, 1 failed, 0 skipped
+            1 scenario passed, 2 failed, 0 skipped
+            9 steps passed, 2 failed, 4 skipped, 0 undefined
+            """
+        And the command output should contain:
+            """
+            Feature with various results in scenarios    # features/scenario_with_steps.feature
+                First scenario with passing steps  .....
+                Second scenario with second failing step  .F
+            --------------------------------------------------------------------------------
+            FAILURE in step 'second step fails' (features/scenario_with_steps.feature:11):
+            Assertion Failed: XFAIL-STEP
+            --------------------------------------------------------------------------------
+            
+                Third scenario with fourth failing step  ...F
+            --------------------------------------------------------------------------------
+            FAILURE in step 'fourth step fails' (features/scenario_with_steps.feature:20):
+            Assertion Failed: XFAIL-STEP
+            --------------------------------------------------------------------------------
+            
+            
+            Failing scenarios:
+              features/scenario_with_steps.feature:9  Second scenario with second failing step
+              features/scenario_with_steps.feature:16  Third scenario with fourth failing step
+            """

data/vendor/behave/features/formatter.rerun.feature

@@ -0,0 +1,296 @@
+@sequential
+Feature: Rerun Formatter
+
+    To simplify to run the scenarios that have failed during the last test run
+    As a tester
+    I want that behave generates the necessary information for me.
+
+    . SOLUTION:
+    .
+    .   Put RerunFormatter into "behave.ini" configuration file:
+    .
+    .       # -- file:behave.ini
+    .       [behave]
+    .       format   = rerun
+    .       outfiles = rerun.txt
+    .
+    .   Then a "rerun.txt" file is generated during each test run
+    .   that contains the file locations of the failing scenarios, like:
+    .
+    .       # -- file:rerun.txt
+    .       # RERUN: Failing scenarios during last test run.
+    .       features/alice.feature:10
+    .       features/alice.feature:42
+    .
+    .   To rerun the failing scenarios of the last test run, use:
+    .
+    .       behave @rerun.txt
+
+
+    @setup
+    Scenario: Feature Setup
+        Given a new working directory
+        And a file named "features/steps/steps.py" with:
+            """
+            from behave import step
+
+            @step('a step passes')
+            def step_passes(context):
+                pass
+
+            @step('a step fails')
+            def step_fails(context):
+                assert False, "XFAIL-STEP"
+            """
+        And a file named "features/alice.feature" with:
+            """
+            Feature: Alice
+
+              Scenario: Passing
+                Given a step passes
+                When a step passes
+                Then a step passes
+
+              @xfail
+              Scenario: When-step fails
+                Given a step passes
+                When a step fails
+                Then a step passes
+
+              @xfail
+              Scenario: Then-step fails
+                Given a step passes
+                When a step passes
+                Then a step fails
+            """
+        And a file named "features/bob.feature" with:
+            """
+            Feature: Bob
+
+              @xfail
+              Scenario: Failing early
+                Given a step fails
+                When a step passes
+                Then a step fails
+
+              Scenario: Passing
+                Given a step passes
+
+              @xfail
+              Scenario: Failing
+                When a step passes
+                Then a step fails
+            """
+
+    @usecase.step1
+    Scenario: Rerun file is generated when failures occur
+        When I run "behave -f rerun -o rerun.txt -f plain -T features/"
+        Then it should fail with:
+            """
+            0 features passed, 2 failed, 0 skipped
+            2 scenarios passed, 4 failed, 0 skipped
+            """
+        And the command output should contain:
+            """
+            Failing scenarios:
+              features/alice.feature:9  When-step fails
+              features/alice.feature:15  Then-step fails
+              features/bob.feature:4  Failing early
+              features/bob.feature:13  Failing
+            """
+        And the file "rerun.txt" should contain:
+            """
+            # -- RERUN: 4 failing scenarios during last test run.
+            features/alice.feature:9
+            features/alice.feature:15
+            features/bob.feature:4
+            features/bob.feature:13
+            """
+
+    @usecase.step2
+    Scenario: Use rerun file during next test run to test only failing scenarios
+        Given a file named "rerun.txt" with:
+            """
+            # -- RERUN: 4 failing scenarios during last test run.
+            features/alice.feature:9
+            features/alice.feature:15
+            features/bob.feature:4
+            features/bob.feature:13
+            """
+        When I run "behave -f plain -T --no-skipped @rerun.txt"
+        Then it should fail with:
+            """
+            0 features passed, 2 failed, 0 skipped
+            0 scenarios passed, 4 failed, 2 skipped
+            """
+        And the command output should contain:
+            """
+            Feature: Alice
+
+              Scenario: When-step fails
+                Given a step passes ... passed
+                When a step fails ... failed
+            Assertion Failed: XFAIL-STEP
+
+              Scenario: Then-step fails
+                Given a step passes ... passed
+                When a step passes ... passed
+                Then a step fails ... failed
+            Assertion Failed: XFAIL-STEP
+
+            Feature: Bob
+
+              Scenario: Failing early
+                Given a step fails ... failed
+            Assertion Failed: XFAIL-STEP
+
+              Scenario: Failing
+                When a step passes ... passed
+                Then a step fails ... failed
+            Assertion Failed: XFAIL-STEP
+            """
+        And the command output should contain:
+            """
+            Failing scenarios:
+              features/alice.feature:9  When-step fails
+              features/alice.feature:15  Then-step fails
+              features/bob.feature:4  Failing early
+              features/bob.feature:13  Failing
+            """
+
+
+    Scenario: Rerun file is deleted when no failures occur
+
+      The rerun output file should be deleted
+      if the test run was successful and no failures occurred.
+
+        Given an empty file named "rerun.txt"
+        When I run "behave -f rerun -o rerun.txt -f plain -T --no-skipped --tags=~@xfail features/"
+        Then it should pass with:
+            """
+            2 features passed, 0 failed, 0 skipped
+            2 scenarios passed, 0 failed, 4 skipped
+            """
+        And a file named "rerun.txt" should not exist
+
+
+    Scenario: Use RerunFormatter with output=stdout
+        When I run "behave -f rerun -f plain -T --no-skipped features/"
+        Then it should fail with:
+            """
+            0 features passed, 2 failed, 0 skipped
+            2 scenarios passed, 4 failed, 0 skipped
+                 """
+        And the command output should contain:
+            """
+            # -- RERUN: 4 failing scenarios during last test run.
+            features/alice.feature:9
+            features/alice.feature:15
+            features/bob.feature:4
+            features/bob.feature:13
+            """
+
+
+    @with.behave_configfile
+    Scenario: Use rerun file with RerunFormatter in behave configuration file
+        Given a file named "behave.ini" with:
+            """
+            [behave]
+            format   = rerun
+            outfiles = rerun.txt
+            show_timings = false
+            show_skipped = false
+            """
+        When I run "behave -f plain features/"
+        Then it should fail with:
+            """
+            0 features passed, 2 failed, 0 skipped
+            2 scenarios passed, 4 failed, 0 skipped
+            """
+        And the file "rerun.txt" should contain:
+            """
+            # -- RERUN: 4 failing scenarios during last test run.
+            features/alice.feature:9
+            features/alice.feature:15
+            features/bob.feature:4
+            features/bob.feature:13
+            """
+        When I run "behave -f plain @rerun.txt"
+        Then it should fail with:
+            """
+            0 features passed, 2 failed, 0 skipped
+            0 scenarios passed, 4 failed, 2 skipped
+            """
+        And the command output should contain:
+            """
+            Feature: Alice
+
+              Scenario: When-step fails
+                Given a step passes ... passed
+                When a step fails ... failed
+            Assertion Failed: XFAIL-STEP
+
+              Scenario: Then-step fails
+                Given a step passes ... passed
+                When a step passes ... passed
+                Then a step fails ... failed
+            Assertion Failed: XFAIL-STEP
+
+            Feature: Bob
+
+              Scenario: Failing early
+                Given a step fails ... failed
+            Assertion Failed: XFAIL-STEP
+
+              Scenario: Failing
+                When a step passes ... passed
+                Then a step fails ... failed
+            Assertion Failed: XFAIL-STEP
+            """
+        And the file "rerun.txt" should contain:
+            """
+            # -- RERUN: 4 failing scenarios during last test run.
+            features/alice.feature:9
+            features/alice.feature:15
+            features/bob.feature:4
+            features/bob.feature:13
+            """
+
+    @sad.case
+    @with.behave_configfile
+    Scenario: Two RerunFormatter use same output file
+        Given a file named "behave.ini" with:
+            """
+            [behave]
+            format   = rerun
+            outfiles = rerun.txt
+            """
+        When I run "behave -f rerun -o rerun.txt -f plain features/"
+        Then it should fail with:
+            """
+            0 features passed, 2 failed, 0 skipped
+            2 scenarios passed, 4 failed, 0 skipped
+            """
+        And the file "rerun.txt" should contain:
+            """
+            # -- RERUN: 4 failing scenarios during last test run.
+            features/alice.feature:9
+            features/alice.feature:15
+            features/bob.feature:4
+            features/bob.feature:13
+            """
+        But the file "rerun.txt" should not contain:
+            """
+            # -- RERUN: 4 failing scenarios during last test run.
+            features/alice.feature:9
+            features/alice.feature:15
+            features/bob.feature:4
+            features/bob.feature:13
+
+            # -- RERUN: 4 failing scenarios during last test run.
+            features/alice.feature:9
+            features/alice.feature:15
+            features/bob.feature:4
+            features/bob.feature:13
+            """
+        And note that "the second RerunFormatter overwrites the output of the first one"