polytrix 0.1.0.pre → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1c91cf7e97c088e23aa0134c2e20acfab0640195
-  data.tar.gz: 9bd8c63ae3cfbbc3bc75fc54c3b094ad848dd8c7
+  metadata.gz: 57453fe422a3e27d1aaa87f79689d05c8aa9bcc0
+  data.tar.gz: b3fd502c953d29fba9948eb12053b1341d1ce15b
 SHA512:
-  metadata.gz: 5361e579064d58b4168e5929f995d1a42506c2faa49b246ddccd83eec3f2adef3d28b2761ab0bfd9b173e6ef690702b7eba56caef2e1a75746045c48c29e97b3
-  data.tar.gz: 61bcc9b099b67a95695cc59ac0d23baf3791cb8eec7c1bed382964c25e03ebc18fac01e6bce5bed6be1666e45a58ada4ef611323121dcf1bdc5c43c3c4f6823c
+  metadata.gz: 3ad963f3478969e5d44cfce5f10e652d94acaa488a9fe5969c377f0ca1133325c022201e211338be5d1b9917664ccdef702b307564cd9258e621d62ca26050c7
+  data.tar.gz: 80e47b9677116fbc2652dc38a54ab0979bc11b10a00cffee865a1026a3bb1609763464a1a67d455714ec3832df1cac33f9676dbae827f309a54d1e4164900d4a

data/.gitignore

@@ -11,3 +11,4 @@ node_modules/
 pkg/
 coverage/
 .yardoc/
+.polytrix/

data/.rubocop.yml

@@ -1,10 +1,15 @@
 inherit_from: .rubocop-todo.yml
 
+Encoding:
+  Enabled: false
+  # EnforcedStyle: when_needed
+
 AllCops:
   Include:
     - Guardfile
     - Rakefile
     - pacto.gemspec
   Exclude:
-    - bin/**
-    - tmp/**
+    - bin/**/*
+    - tmp/**/*
+    - tmp/**/*.rb

data/.travis.yml

@@ -1,11 +1,12 @@
 before_install:
- - sudo add-apt-repository ppa:cwchien/gradle
+ - sudo add-apt-repository -y ppa:cwchien/gradle
  - sudo apt-get update -qq
  - sudo apt-get install -qq gradle
 rvm:
   - 1.9.3
   - 2.0.0
   - 2.1.0
+  - jruby
 allow_failures:
   rvm:
     - 1.9.3 # YAML formatting differs

data/Gemfile

@@ -5,3 +5,4 @@ gemspec
 gem "pry"
 gem "rake-notes"
 gem "simplecov"
+gem 'hashie', :git => 'https://github.com/intridea/hashie'

data/README.md

@@ -1,139 +1,231 @@
 # Polytrix - the Polyglot Testing Matrix
 
-Polytrix is a polyglot test runner and documentation generator. It aims to let you run sample code written in any language. It's especially useful if you want to run similar code samples in multiple languages. Simply put, if a project like [Slate](https://github.com/tripit/slate) looks like an interesting documentation option, then you might be interested in Polytrix for testing.
+Polytrix is a polyglot test runner and documentation generator. It aims to let you run sample code written in any language. It's especially useful if you want to run similar code samples in multiple languages, a project that has been ported to several languages, or API clients for the same service that are provided in several languages.
 
-# Features
+Polytrix was influenced by a number of polyglot projects, including [Travis-CI](travis-ci.org), [Docco](https://github.com/jashkenas/docco), [Slate](https://github.com/tripit/slate), and polyglot test-suites like the [JSON Schema Test Suite](https://github.com/json-schema/JSON-Schema-Test-Suite) and the [JSON-LD Test Suite](http://json-ld.org/test-suite/).
 
-Polytrix samples defined in a "test manifest" written in YAML. The test manifest is meant to be portable so you can use you can create a "lightweight sample runner" in your preferred build/test tool of choice for your language, and then integrate the samples with Polytrix later to get the extra features. Polytrix can:
-- Run sample code in any language and several platforms
-- Perform compatibility testing checking multiple implementations (in different langauges) against the same set of assertions
-- Generate documentation from sample code and test results
-- Generate compatibility or feature matrix reports
+The user-experience was heavily influenced by [test-kitchen](http://kitchen.ci/). Polytrix aims to do for multi-language development of command-line applications what test-kitchen has done for multi-platform development of infrastructure.
 
-Polytrix provides a few built-in assertions, but also has a plugin system that you can use to do more advanced validation, like using [Pacto](https://github.com/thoughtworks/pacto) to intercept and validate the usage of RESTful services.
+## Installing Polytrix
 
-# Usage preview
+Polytrix is distributed as a Ruby Gem. It should be installed using Bundler by adding this line to your Gemfile:
 
-Polytrix is currently run via rspec. You can create a script that looks like this and run it with rspec:
-
-```ruby
-require 'polytrix/rspec'
-
-Polytrix.implementors = Dir['sdks/*'].map{ |sdk|
-  name = File.basename(sdk)
-  Polytrix::Implementor.new :name => name
-}
-
-Polytrix.load_manifest 'polytrix_tests.yml'
-Polytrix.bootstrap
-Polytrix.run_tests
+```shell
+gem 'polytrix', '~> 0.1'
 ```
 
-Polytrix will use the information in the Manifest and Implementors (see the sections below) to build an rspec test suite. It will setup tags for each Implementor, and names corresponding with the tests in the manifest.
+And then running `bundle install`.
 
-So in our Polytrix examples project you can use commands like:
+It can also be installed without Bundler by running `gem install polytrix`.
 
-```sh
-$ # Only run tests for the Java implementor
-$ bundle exec rspec -t java
-$ # Run the "hello world" tests in each language
-$ bundle exec rspec -e "hello world"
-```
+## Getting Help
 
-## Usage Breakdown
+Polytrix is primarily intented to be run as a standalone utility, though it does have an API for integrating with RSpec or other Ruby-based test frameworks.
 
-### Defining Implementors (SDKs)
+Note: You may need to prefix commands with `bundle exec` if you installed Polytrix using Bundler.
 
-Polytrix can run the tests against multiple implementors. This usually means an SDK, but we used the generic term implementor because Polytrix works equally well for testing code katas, coursework, or other items. Perhaps even things like multi-platform plugins for [Calatrava](https://github.com/calatrava/calatrava/wiki/Plugins) or [PhoneGap](http://docs.phonegap.com/en/3.4.0/guide_hybrid_plugins_index.md.html#Plugin%20Development%20Guide).
+If you need a quick reminder of what the `polytrix` command gives you, then use the **help** subcommand:
 
-This snippet defines the implementors:
-```ruby
-Polytrix.implementors = Dir['sdks/*'].map{ |sdk|
-  name = File.basename(sdk)
-  Polytrix::Implementor.new :name => name
-}
+```
+$ bundle exec polytrix help
+Commands:s
+  polytrix bootstrap [(all|<REGEX>)] [opts] # Bootstraps the code samples for one or more tests
+  polytrix code2doc [(all|<REGEX>)] [opts]  # Converts annotated code samples to Markdown or reStructuredText
+  polytrix clean [(all|<REGEX>)] [opts]     # Removes test results for one or more tests
+  polytrix clone [(all|<REGEX>)] [opts]     # Clones the code samples from git for one or more tests
+  polytrix exec [(all|<REGEX>)] [opts]      # Executes code samples for one or more tests
+  polytrix help [COMMAND]                   # Describe available commands or one specific command
+  polytrix list [(all|<REGEX>)] [opts]      # List all tests
+  polytrix test [all|<REGEX>)] [opts]       # Test one or more tests
+  polytrix verify [(all|<REGEX>)] [opts]    # Verify the execution results of one or more tests
+  polytrix version                          # Print Polytrix's version information
 ```
 
-See the full Implementor documentation for details on other attributes you can set, like `:language`. Polytrix will try to infer any information you don't pass.
-
-#### Bootstrapping, compiling, wrapper scripts
+## The polytrix.yml file
 
-Polytrix currently uses the [script/bootstrap](http://wynnnetherland.com/linked/2013012801/bootstrapping-consistency) pattern to allow each implementor to hook into dependency management, build tools, or other systems. Polytrix will look for three scripts (on Windows it will look for a *.ps1 version written in PowerShell):
+The Polytrix test suites and the implementors that should be tested are defined in a polytrix.yml file.
+A basic file looks like this:
 
-| File             | Purpose                                                        |
-| ---------------- | -------------------------------------------------------------- |
-| script/bootstrap | Prepare the SDK, usually by running depenency management tool. |
-| script/wrapper   | Wrapper script instead of executing code samples as scripts    |
+```yaml
+polytrix:
+  implementors:
+    ruby_samples:
+      language: ruby
+      bootstrap_cmd: bundle install
+      exec_cmd: bundle exec ruby
+      git:
+        repo: https://github.com/polytrix/ruby_samples
+    python_samples:
+      language: python
+      bootstrap_script: scripts/pip_install
+      exec_script: scripts/pip_install
+      git:
+        repo: https://github.com/polytrix/python_samples
+    java_samples:
+      language: java
+      # The default is to look for scripts/bootstrap and scripts/exec
+      # or try to use the default bootstrap/exec behavior for the language
+      git:
+        repo: 'https://github.com/polytrix/java_samples'
+
+  suites:
+    katas:
+      code_samples:
+        hello_world:
+          - validate:
+              stdout: Hello, World!
+        quine:
+          - validate:
+              quine_validator: true
+    utilities:
+      code_samples:
+        word_count:
+          - name: short text
+            input: |
+              I am a word count utility
+            validate:
+              stdout: 6 words
+          - name: small file
+            input_file: fixtures/small_file.txt
+            validate:
+              stdout: 10 words
+          - name: large file
+            input_file: fixtures/large_file.txt
+            validate:
+              stdout: 1000 words
+```
 
-The bootstrap script is called by `Polytrix.bootstrap`. The wrapper script, if it exists, wraps the executino of the code sample. If there is no wrapper script, Polytrix will try to execute the sample code as a script. That works for many non-compiled scripting languages, like Ruby or Python, but won't work for something like Java.
+### Defining Implementors
+
+Polytrix tests scenarios across one or more implementors. The implementors may each be in a different
+language, but you can also have multiple implementors for the same langauge. The structure used to define implementors within polytrix.yml is described below.
+
+#### Implementor Object
+
+Field Name | Type | Description
+---|:---:|---
+<a name="implementor_name"/>name | `string` | The name of the implementor. (*If omitted, this attribute’s value defaults to the key name associated with this object.*)
+<a name="implementor_directory"/>directory | `string` | The location of the implementor project containing code samples.
+<a name="implementor_language"/>language | `string` | The primary programming language of the code samples in this implementor.
+<a name="implementor_git"/>git | [Git Object](#git_object) | Defines how to clone the project via Git.
+<a name="implementor_scripts"/>scripts | [Scripts Object](#scripts_object) | Defines scripts to bootstrap, compile, and execute the samples.
+
+#### Git Object
+<a name="git_object"/>Field Name | Type | Description
+---|:---:|---
+<a name="git_repo"/>repo | `string` | **Required.** The git repo to clone.
+<a name="git_branch"/>branch | `string` | The git branch to clone.
+<a name="git_to"/>to | `string` | The local directory to clone into.
+
+#### Scripts Object
+<a name="scripts_object"/>Field Name | Type | Description
+---|:---:|---
+<a name="directory"/>directory | `directory` | The directory that contains the script files. **Default**: `scripts/`.
+<a name="bootstrap_script"/>bootstrap_script | `file` | The file to execute to bootstrap the samples. **Default**: `bootstrap`
+<a name="compile_script"/>compile_script | `file` | The file to execute to compile the samples. **Default**: `compile`
+<a name="exec_script"/>exec_script | `file` | The file to use as an execute wrapper script for running samples. **Default**: `exec`
+<a name="bootstrap_cmd"/>bootstrap_cmd | `string` | An inline script to execute instead of executing *bootstrap_file*.
+<a name="compile_cmd"/>compile_cmd | `string` | An inline script to execute instead of executing *bootstrap_file*.
+<a name="exec_cmd"/>exec_cmd | `string` | An inline script to use as an execution wrapper, instead of searching for a `exec` script.
+
+#### Example Implementor Definition
 
-If there is a wrapper script, Polytrix will call it with teh sample source file as the first argument, e.g.:
-```sh
-$ cd my_java_sdk
-$ ./script/wrapper src/samples/HelloWorld.java
+```yaml
+implementors:
+  ruby_samples:
+    directory: samples/ruby
+    language: ruby
+    git:
+      repo: https://github.com/polytrix/ruby_samples
+      branch: master
+      to: samples/ruby
+    scripts:
+      bootstrap_cmd: bundle install
+      exec_cmd: bundle exec ruby "$@"
 ```
 
-### Defining tests - the test manifest
+### Attaching Code Samples
 
-Tests are defined in a YAML "test manifest" which defines what sample code should be executed, and what input it should receive. Standardizing the input is important for compliance testing, becaues it is difficult to maintain tests where one example expects "FOO=bar" and another expects "--foo bar".
+Polytrix tests are divided into suites, code samples, and scenarios. Each tuple (suite, code sample, scenario) is a test.
 
-A simple test manifest looks like this:
-```yaml
----
-  global_env:                          # global_env defines input available for all scenarios
-    LOCALE: <%= ENV['LANG'] %>         # templating is allowed
-    FAVORITE_NUMBER: 5
-  suites:                              # suites defines the test suites that can be executed
-    Katas:                             # "Katas" is the name of the first suite
-      env:                             # These "env" values are only available within the "Katas" suite
-        NAME: 'Max'
-      samples:                         # samples defines the individual tests in a suite
-        - hello world
-        - quine
-    Tutorials:                         # "Tutorials" is the name of the second suite
-          env:
-          samples:
-            - deploying
-            - documenting
-```
+The structure used to define tests is:
+
+#### Suite Object
 
-### Test setup
+Field Name | Type | Description
+---|:---:|---
+<a name="suite_name"/>name | `string` | The name of the suite. (*If omitted, this attribute’s value defaults to hash key associated with this object.*)
+<a name="suite_input"/>input | `string` | The content to send to the code sample as standard input.
+<a name="suite_input_file"/>input_file | `file` | Reads the specified file and sets *input*.
+<a name="code_samples"/>code_samples | Hash of [Code Samples](#code_samples) | Defines the code samples to test
 
-`Polytrix.run_tests` runs the tests. Actually, right now it really just defines them in rspec, you still need to run the whole script via the RSpec command for the tests to run.
+#### Code Sample Object
 
-### Finding samples
+The code sample object determines which scenario to runwhich determines which code sample will be run. It can also
+override the default values set in the Suite Object.
 
-Polytrix finds samples based on a loose naming convention. This makes it easier to use file names that are idiomatic for each implementor, while still allowing Polytrix to find the right file.
+If the Scenario Object is a list, it will be automatically split into distinct but related scenarios. This is useful
+for testing the same code sample with a variety of different input values.
 
-Polytrix basically does a case-insensitive search for a file whose name matches the scenario name, ignoring subfolders, spaces, prefixes, puctuation and file extension.  So these files all match a scenario named "hello world":
-- hello_world.rb
-- src/com/world/HelloWorld.java
-- samples/01_hello_world.go
+Field Name | Type | Description
+---|:---:|---
+<a name="code_sample_name"/>name | `string` | The name of the suite. (*If omitted, this attribute’s value defaults to hash key associated with this object.*)
+<a name="scenarios"/>scenarios | List of [Scenario Objects](#scenario_object) | Defines the scenarios to test for the code sample. **Default**: An scenario named default with no input.
 
-### Reports and documentation
 
-You can use any RSpec formatter with Polytrix, since Polytrix is based on RSpec.
+#### Scenario Object
 
-There are also some Polytrix specific documentation and report generators. You can generate:
-- Scenario-level reports: Documentation or reports for a single scenario
-- Global reports: Documentation or reports summarizing all tested scenarios
+Field Name | Type | Description
+---|:---:|---
+<a name="scenario_name"/>name | `string` | **Required.** The scenario name.
+<a name="scenario_input"/>input | `string` | The content to send to the code sample as standard input.
+<a name="scenario_input_file"/>input_file | `file` | Reads the specified file and sets *input*.
+<a name="scenarios"/>language | Hash of [Scenario Objects](#scenario_object) | Defines the scenarios within the suite
 
-The documentatio/reports are generated via a template processing system. Polytrix searches the template directory ('doc-src/' by default) for scenario-level samples using the same logic as in the "Finding samples" section above. It looks for a template matching "index" (e.g. index.md, index.rst, index.html) for the global report.
+#### Example Suite Definition
 
-The templates are processed as ERB. In addition to being able to access the top-level Polytrix API, the following variables are bound:
-| Variable   | Description                              |
-| ---------- | ---------------------------------------- |
-| scenario   | The name of the scenario being processed |
-| challenges | One or more Challenge objects containing the scenario configuration and results |
+```yaml
+  suites:
+    katas:
+      scenarios:
+        hello_world:
+          validate:
+            stdout: Hello, World!
+        quine:
+          validate:
+            quine_validator: true
+    utilities:
+      scenarios:
+        word_count:
+          - name: short text
+            input: |
+              I am a word count utility
+            validate:
+              stdout: 6 words
+          - name: small file
+            input_file: fixtures/small_file.txt
+            validate:
+              stdout: 10 words
+          - name: large file
+            input_file: fixtures/large_file.txt
+            validate:
+              stdout: 1000 words
+```
 
-### Common compliance tests
+## Plugins
 
-Refactoring... documentation coming soon.
+TBD
 
-### Plugins
+## Custom Validators
 
-Refactoring... documentation coming soon.
+TBD
 
-# Influences
+## Usage
 
-Several projects have influenced ideas in Polytrix. If you find Polytrix interesting or want to contribute, you may want to look at those projects. See influence.md.
+### Cloning implementors
+### Bootstrapping implementors
+### Executing scenarios
+### Verifying scenario results
+### Full test
+
+```

data/Rakefile

@@ -13,7 +13,7 @@ RSpec::Core::RakeTask.new('spec') do |t|
 end
 
 Cucumber::Rake::Task.new(:features) do |t|
-  t.cucumber_opts = "features --require features/support --require features/step_definitions"
+  t.cucumber_opts = "features --require features/support --require features/step_definitions -t ~@wip"
 end
 
 desc 'Remove reports and other generated artifacts'
@@ -25,11 +25,13 @@ end
 desc 'Self-test and self-document'
 task :self do
   sh 'bundle exec polytrix bootstrap'
-  sh 'bundle exec polytrix test'
-  sh 'bundle exec polytrix code2doc samples/sdks/java/challenges/*.java --target-dir=docs/samples/code2doc/java'
-  sh 'bundle exec polytrix code2doc samples/sdks/python/challenges/*.py --target-dir=docs/samples/code2doc/python'
-  sh 'bundle exec polytrix code2doc samples/sdks/ruby/challenges/*.rb --target-dir=docs/samples/code2doc/ruby'
-  # sh 'bundle exec polytrix code2doc samples/*.rb'
+  Dir.chdir 'samples' do
+    # sh 'bundle exec polytrix test'
+    sh 'bundle exec polytrix code2doc java --target-dir=docs/samples/code2doc/java'
+    sh 'bundle exec polytrix code2doc python --target-dir=docs/samples/code2doc/python'
+    sh 'bundle exec polytrix code2doc ruby --target-dir=docs/samples/code2doc/ruby'
+    # sh 'bundle exec polytrix code2doc samples/*.rb'
+  end
 end
 
 Rubocop::RakeTask.new(:rubocop) do |task|

data/bin/polytrix

@@ -1,5 +1,6 @@
 #!/usr/bin/env ruby
+require 'polytrix/error'
 require 'polytrix/cli'
 
 $LOAD_PATH << 'spec'
-Polytrix::CLI::Main.start
+Polytrix.with_friendly_errors { Polytrix::CLI.start }

data/docs/samples/code2doc/java/HelloWorld.md

@@ -1,13 +1,17 @@
 This is an example of the standard [Hello World](http://en.wikipedia.org/wiki/Hello_world_program)
 program written in [Java][java].
+
 ```java
 public class HelloWorld {
 ```
+
 A Java CLI application is defined by creating a main method with this signature:
+
 ```java
   public static void main(String[] args) {
     System.out.println("Hello, world!");
   }
 }
 ```
+
 [java]: http://en.wikipedia.org/wiki/Java_(programming_language)

data/docs/samples/code2doc/java/Quine.md

@@ -1,3 +1,4 @@
+
 ```java
 public class Quine
 {
@@ -31,3 +32,4 @@ public class Quine
   }
 }
 ```
+

data/docs/samples/code2doc/python/hello_world.md

@@ -1,3 +1,5 @@
+
 ```py
 print 'Hello, world!'
 ```
+

data/docs/samples/code2doc/python/quine.md

@@ -1,4 +1,6 @@
+
 ```py
 s = 's = %r\nprint(s%%s)'
 print(s%s)
 ```
+

data/docs/samples/code2doc/ruby/hello_world.md

@@ -1,7 +1,11 @@
+
 ```rb
 name = 'world'
 ```
+
 This will print Hello, world!
+
 ```rb
 puts "Hello, #{name}!"
 ```
+

data/features/bootstrapping.feature

@@ -0,0 +1,36 @@
+Feature: Boostrapping
+
+  Polytrix uses the [script/bootstrap](http://wynnnetherland.com/linked/2013012801/bootstrapping-consistency) pattern to prepare SDKs for testing. You can hook into any package manager, compiler, build tool, or any other toolchain to prepare to build and run samples.
+
+  Scenario: Bootstrapping all SDKs
+    Given the ruby SDK
+    And the java SDK
+    And the python SDK
+    And the hello_world polytrix config
+    And the standard rspec setup
+    When I run `bundle exec polytrix bootstrap`
+    Then the output should contain "-----> Bootstrapping java"
+    Then the output should contain "-----> Bootstrapping python"
+    Then the output should contain "-----> Bootstrapping ruby"
+
+  Scenario: Bootstrapping selected SDKs
+    Given the ruby SDK
+    And the java SDK
+    And the python SDK
+    And the hello_world polytrix config
+    And the standard rspec setup
+    When I run `bundle exec polytrix bootstrap "(java|ruby)"`
+    Then the output should contain "-----> Bootstrapping java"
+    Then the output should not contain "-----> Bootstrapping python"
+    Then the output should contain "-----> Bootstrapping ruby"
+
+  Scenario: Bootstrapping by scenario
+    Given the ruby SDK
+    And the java SDK
+    And the python SDK
+    And the hello_world polytrix config
+    And the standard rspec setup
+    When I run `bundle exec polytrix bootstrap hello`
+    Then the output should contain "-----> Bootstrapping java"
+    Then the output should contain "-----> Bootstrapping python"
+    Then the output should contain "-----> Bootstrapping ruby"

data/features/cloning.feature

@@ -0,0 +1,34 @@
+@wip
+Feature: Cloning
+
+  Polytrix can clone projects from git.
+
+  Scenario: Cloning all SDKs
+    Given the hello_world polytrix config
+    And the standard rspec setup
+    When I run `bundle exec polytrix clone`
+    Then the output should contain "-----> Cloning java"
+    Then the output should contain "-----> Cloning python"
+    Then the output should contain "-----> Cloning ruby"
+
+  Scenario: Cloning selected SDKs
+    Given the ruby SDK
+    And the java SDK
+    And the python SDK
+    And the hello_world polytrix config
+    And the standard rspec setup
+    When I run `bundle exec polytrix clone "(java|ruby)"`
+    Then the output should contain "-----> Cloning java"
+    Then the output should not contain "-----> Cloning python"
+    Then the output should contain "-----> Cloning ruby"
+
+  Scenario: Cloning by scenario
+    Given the ruby SDK
+    And the java SDK
+    And the python SDK
+    And the hello_world polytrix config
+    And the standard rspec setup
+    When I run `bundle exec polytrix clone hello`
+    Then the output should contain "-----> Cloning java"
+    Then the output should contain "-----> Cloning python"
+    Then the output should contain "-----> Cloning ruby"

data/features/execution.feature

@@ -4,15 +4,7 @@ Feature: Running SDKs
 
   Polytrix also sets up tags for the SDKs so you can use the normal rspec `-t` option to select which SDK to run.
 
-  Scenario: Bootstrap an SDK
-    Given the java SDK
-    And the empty polytrix config
-    And the standard rspec setup
-    When I run `bundle exec rspec`
-    Then the output should contain ":compileJava"
-    And the output should contain "BUILD SUCCESSFUL"
-
-Scenario: Running all SDKs
+  Scenario: Running all SDKs
     Given the ruby SDK
     And the java SDK
     And the python SDK
@@ -41,13 +33,6 @@ Scenario: Running all SDKs
     """
     require 'polytrix/rspec'
 
-    Polytrix.configure do |polytrix|
-      Dir['sdks/*'].each do |sdk|
-        polytrix.implementor sdk
-      end
-      polytrix.test_manifest = 'polytrix_tests.yml'
-    end
-
     # This is more of an integration test, but makes sure the rspec API is working.
     # Expect results to all be pending, because there's no implementors in this proj.
 
@@ -62,6 +47,7 @@ Scenario: Running all SDKs
       expect(challenge[:result].execution_result.stdout).to eq File.read(challenge[:result].source)
     end
     Polytrix.load_tests
+
     """
     When I run `bundle exec rspec spec/custom_spec.rb`
     And the output should match /expected: 1\s+got: 0/

data/features/fixtures/configs/empty.yml

@@ -1,4 +1,15 @@
 ---
+  implementors:
+    ruby:
+      language: 'ruby'
+      basedir: 'sdks/ruby'
+    java:
+      language: 'java'
+      basedir: 'sdks/java'
+    python:
+      language: 'python'
+      basedir: 'sdks/python'
+
   global_env:
     LOCALE: <%= ENV['LANG'] %>
     FAVORITE_NUMBER: 5
@@ -9,4 +20,4 @@
       samples:
     Tutorials:
       env:
-      samples:
+      samples:

data/features/fixtures/configs/hello_world.yml

@@ -1,4 +1,14 @@
 ---
+  implementors:
+    ruby:
+      language: 'ruby'
+      basedir: 'sdks/ruby'
+    java:
+      language: 'java'
+      basedir: 'sdks/java'
+    python:
+      language: 'python'
+      basedir: 'sdks/python'
   global_env:
     LOCALE: <%= ENV['LANG'] %>
     FAVORITE_NUMBER: 5
@@ -7,4 +17,4 @@
       env:
         NAME: 'Max'
       samples:
-        - hello world
+        - hello world