xcknife 0.1.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ef0dca6ea2940fdd267c0d6a1343c6cca06cd061
-  data.tar.gz: 743037f45ece209c60620bda130cd52554370a0c
+  metadata.gz: 0b6205c8d36ae8e54613311d15ca1819c311e091
+  data.tar.gz: fee83a2de8da43399901a1b1dd90185b6dcbf8e1
 SHA512:
-  metadata.gz: 5739e9f4a633136067a27dca08c1c383e73d166cc20cf0d55e36e9d2a63dc23160c58aca97eb5f0dc5a506fffa446b850f371166966d181e6efa38ce8bc137eb
-  data.tar.gz: e73093a4a3ddaedc77868f545e0aecf66496940aac1782d4bc0280f6148171c39d2848a104dd3668003f6aa1360bdb6f4a565cda9ef89ff85f054576e863c5de
+  metadata.gz: 938970059eaeb4463ee976b0b22a05b0912098817f007dc22e2ac1401fb7dac8fc148ed8c4c1da8bbb99dfd82951fcf51d19873d8d14c116f38b695d3f554667
+  data.tar.gz: 9bf514e7e67fdf21ddbf358bdf9fc58569a0eef1e25a0c4715aa6524b0cfdae0fa14214db30eee8c50e880b366905686316d53f6acf3ae9a85d1eb27648d6de9

data/CONTRIBUTING.md

@@ -0,0 +1,15 @@
+### Sign the CLA
+
+Any contributors to the master *xcknife* repository must sign the
+[Individual Contributor License Agreement (CLA)]. It's a short form that covers
+our bases and makes sure you're eligible to contribute.
+
+### Submitting a Pull Request
+
+When you have a change you'd like to see in the master repository, send a
+[pull request]. Before we merge your request, we'll make sure you're in the list
+of people who have signed a CLA.
+
+[Individual Contributor License Agreement (CLA)]: https://spreadsheets.google.com/spreadsheet/viewform?formkey=dDViT2xzUHAwRkI3X3k5Z0lQM091OGc6MQ&ndplr=1
+[pull request]: https://github.com/square/xcknife/pulls
+

data/README.md

@@ -1 +1,218 @@
-WIP
+# XCKnife
+[![Apache 2 licensed](https://img.shields.io/badge/license-Apache2-blue.svg)](https://github.com/square/xcknife/blob/master/LICENSE)
+
+XCKnife is a tool that partitions XCTestCase tests in a way that minimizes total execution time (particularly useful for distributed test executions).
+ 
+It works by leveraging [xctool's](https://github.com/facebook/xctool) [json-stream](https://github.com/facebook/xctool#included-reporters) reporter output.
+
+XCKnife generates a list of only arguments meant to be pass to Xctool's [*-only* test arguments](https://github.com/facebook/xctool#testing), but alternatively could used to generate multiple xcschemes with the proper test partitions.
+ 
+## Install
+
+`$ gem install xcknife`
+
+## Using as command line tool
+ 
+```
+$ xcknife -h
+Usage: xcknife [options] worker-count historical-timings-json-stream-file [current-tests-json-stream-file]
+    -p, --partition TARGETS          Comma separated list of targets. Can be used multiple times
+    -o, --output FILENAME            Output file. Defaults to STDOUT
+    -a, --abbrev                     Results are abbreviated
+    -h, --help                       Show this message
+```
+
+## Example 
+
+The data provided on the [example](https://github.com/square/xcknife/tree/master/example) folder:
+
+`$ xcknife  -p iPhoneTestTarget 3 example/xcknife-exemplar-historical-data.json-stream example/xcknife-exemplar.json-stream`
+
+This will balance the tests onthe `iPhoneTestTarget` into 3 machines. The output is:
+
+```json
+{
+  "metadata": {
+    "worker_count": 3,
+    "partition_set_count": 1,
+    "total_time_in_ms": 910,
+    "historical_total_tests": 5,
+    "current_total_tests": 5,
+    "class_extrapolations": 0,
+    "target_extrapolations": 0
+  },
+  "partition_set_data": [
+    {
+      "partition_set": "iPhoneTestTarget",
+      "size": 3,
+      "imbalance_ratio": 1.0,
+      "partitions": [
+        {
+          "shard_number": 1,
+          "cli_arguments": [ "-only", "iPhoneTestTarget:iPhoneTestClassGama" ],
+          "partition_imbalance_ratio": 0.9923076923076923
+        },
+        {
+          "shard_number": 2,
+          "cli_arguments": [ "-only", "iPhoneTestTarget:iPhoneTestClassAlpha,iPhoneTestClassDelta" ],
+          "partition_imbalance_ratio": 1.0054945054945055
+        },
+        {
+          "shard_number": 3,
+          "cli_arguments": [ "-only", "iPhoneTestTarget:iPhoneTestClassBeta,iPhoneTestClassOmega" ],
+          "partition_imbalance_ratio": 1.0021978021978022
+        }]}]}
+```
+
+This provides a lot of data about the partitions and their imabalances (both internal to the partition sets, and amongst them).
+
+If you only want the *-only* arguments, run with the `-a` flag:
+
+`$ xcknife  -p iPhoneTestTarget 3 example/xcknife-exemplar-historical-data.json-stream example/xcknife-exemplar.json-stream -a`
+
+outputing:
+
+```json
+[
+  [
+    [
+      "-only",
+      "iPhoneTestTarget:iPhoneTestClassGama"
+    ],
+    [
+      "-only",
+      "iPhoneTestTarget:iPhoneTestClassAlpha,iPhoneTestClassDelta"
+    ],
+    [
+      "-only",
+      "iPhoneTestTarget:iPhoneTestClassBeta,iPhoneTestClassOmega"
+    ]
+  ]
+]
+```
+
+## Example: Multiple partition Sets
+
+You can pass the partition flag mutliple times, so that XCKnife will do two level partiotioning: inside each partition, and then for all partitions.
+  
+This is useful if each partition is tested against multiple devices, simulator versions or configurations. On the following example picture `CommonTestTarget` being tested against iPhones only, while `CommonTestTarget,iPadTestTarget` is tested against iPads.
+
+`$ xcknife  -p CommonTestTarget -p CommonTestTarget,iPadTestTarget 6 example/xcknife-exemplar-historical-data.json-stream example/xcknife-exemplar.json-stream`
+
+This will balance two partition sets: `CommonTestTarget` and `CommonTestTarget,iPadTestTarget` into 6 machines. The output is:
+
+```json
+{
+  "metadata": {
+    "worker_count": 6,
+    "partition_set_count": 2,
+    "total_time_in_ms": 8733,
+    "historical_total_tests": 6,
+    "current_total_tests": 7,
+    "class_extrapolations": 1,
+    "target_extrapolations": 0
+  },
+  "partition_set_data": [
+    {
+      "partition_set": "CommonTestTarget",
+      "size": 1,
+      "imbalance_ratio": 0.5480554313813143,
+      "partitions": [
+        {
+          "shard_number": 1,
+          "cli_arguments": [
+            "-only",
+            "CommonTestTarget:CommonTestClass"
+          ],
+          "partition_imbalance_ratio": 1.0
+        }
+      ]
+    },
+    {
+      "partition_set": "CommonTestTarget,iPadTestTarget",
+      "size": 5,
+      "imbalance_ratio": 1.4519445686186858,
+      "partitions": [
+        {
+          "shard_number": 2,
+          "cli_arguments": [
+            "-only",
+            "iPadTestTarget:iPadTestClassTwo"
+          ],
+          "partition_imbalance_ratio": 3.0800492610837438
+        },
+        {
+          "shard_number": 3,
+          "cli_arguments": [
+            "-only",
+            "iPadTestTarget:iPadTestClassOne"
+          ],
+          "partition_imbalance_ratio": 0.6169950738916257
+        },
+        {
+          "shard_number": 4,
+          "cli_arguments": [
+            "-only",
+            "iPadTestTarget:iPadTestClassFour"
+          ],
+          "partition_imbalance_ratio": 0.6169950738916257
+        },
+        {
+          "shard_number": 5,
+          "cli_arguments": [
+            "-only",
+            "CommonTestTarget:CommonTestClass"
+          ],
+          "partition_imbalance_ratio": 0.3774630541871921
+        },
+        {
+          "shard_number": 6,
+          "cli_arguments": [
+            "-only",
+            "iPadTestTarget:iPadTestClassThree"
+          ],
+          "partition_imbalance_ratio": 0.30849753694581283
+        }
+      ]
+    }
+  ]
+}
+```
+
+## Using as Ruby gem
+
+Described [here](https://github.com/square/xcknife/tree/master/example).
+
+## Minimizing json-stream files
+
+XCKnife uses only a few attributes of a json-stream file. If you are storing the files in repository, you may want to remove unecessary data with `xcknife-min`. For example:
+
+`$ xcknife-min example/xcknife-exemplar-historical-data.json-stream minified.json-stream` 
+
+## Contributing
+
+Any contributors to the master *xcknife* repository must sign the
+[Individual Contributor License Agreement (CLA)]. It's a short form that covers
+our bases and makes sure you're eligible to contribute.
+
+When you have a change you'd like to see in the master repository, send a
+[pull request]. Before we merge your request, we'll make sure you're in the list
+of people who have signed a CLA.
+
+[Individual Contributor License Agreement (CLA)]: https://spreadsheets.google.com/spreadsheet/viewform?formkey=dDViT2xzUHAwRkI3X3k5Z0lQM091OGc6MQ&ndplr=1
+[pull request]: https://github.com/square/xcknife/pulls
+
+
+## License
+
+Copyright 2016 Square Inc.
+ 
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ 
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/Rakefile

@@ -0,0 +1,6 @@
+require 'rspec/core/rake_task'
+require "bundler/gem_tasks"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/bin/xcknife

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+require 'xcknife'
+
+XCKnife::Runner.invoke

data/bin/xcknife-min

@@ -0,0 +1,38 @@
+#!/usr/bin/env ruby
+# encoding: utf-8
+
+# Script for cleaning up historical events, so that we only store the ones we need
+require 'set'
+require 'json' unless defined?(::JSON)
+
+INTERESTING_EVENTS = %w[begin-ocunit end-test].to_set
+def cleanup(input_file_name, output_file_name)
+  if input_file_name.nil? or output_file_name.nil?
+    return puts "Usage: xcknife_clean [input] [output]"
+  end
+  lines = IO.readlines(input_file_name)
+  lines_written = 0
+  total = lines.size
+  File.open(output_file_name, "w") do |f|
+    lines.each do |line|
+      data = JSON.load(line)
+      if INTERESTING_EVENTS.include?(data["event"])
+        lines_written += 1
+        %w[output sdk timestamp exceptions result succeeded methodName].each do |k|
+          data.delete(k)
+        end
+        if data["test"]
+          data["test"] = "1"
+        end
+        f.puts(data.to_json)
+      end
+    end
+  end
+  lines_removed = total - lines_written
+  percent = (100.0 * lines_removed / total).round(2)
+  puts "Done. Removed #{lines_removed} lines (#{percent}%) out of #{total}"
+  puts "Written new json-stream file to: #{output_file_name}"
+end
+
+input_file_name, output_file_name = ARGV
+cleanup input_file_name, output_file_name

data/example/README.md

@@ -0,0 +1,15 @@
+# Example
+
+Data gathered from the [xcknife-exemplar sample project](https://github.com/danielribeiro/xcknife-exemplar).
+
+To run the example run:
+
+```
+$ ruby run_example.rb
+```
+
+This is equivalent to the example of: 
+
+```
+$ xcknife -p CommonTestTarget -p CommonTestTarget,iPadTestTarget 6 example/xcknife-exemplar-historical-data.json-stream example/xcknife-exemplar.json-stream
+```

data/example/run_example.rb

@@ -0,0 +1,34 @@
+require_relative '../lib/xcknife'
+require 'pp'
+
+# Gem usage of xcknife. Functionaly equivalent to
+# $ xcknife -p CommonTestTarget -p CommonTestTarget,iPadTestTarget 6 example/xcknife-exemplar-historical-data.json-stream example/xcknife-exemplar.json-stream
+include XCKnife::XCToolCmdHelper
+TARGET_PARTITIONS = {
+  "AllTests" => ["CommonTestTarget", "iPadTestTarget"],
+  "OnlyCommon" => ["CommonTestTarget"]
+}
+
+def run(historical_file, current_file)
+  test_target_names = TARGET_PARTITIONS.keys
+  knife = XCKnife::StreamParser.new(6, TARGET_PARTITIONS.values)
+  result = knife.compute_shards_for_file(historical_file, current_file)
+  partition_sets = result.test_maps
+  puts "total = #{result.total_test_time}"
+  puts "test times = #{result.test_times.inspect}"
+  puts "stats = #{result.stats.to_h.pretty_inspect}"
+  puts "imbalances = #{result.test_time_imbalances.to_h.inspect}"
+  shard_number = 0
+  puts "size = #{partition_sets.size}"
+  puts "sizes = #{partition_sets.map(&:size).join(", ")}"
+  partition_sets.each_with_index do |partition_set, i|
+    target_name = test_target_names[i]
+    partition_set.each do |partition|
+      puts "target name for worker #{shard_number} = #{target_name}"
+      puts "only is: #{xctool_only_arguments(partition).inspect}"
+      shard_number += 1
+    end
+  end
+end
+
+run("xcknife-exemplar-historical-data.json-stream", "xcknife-exemplar.json-stream")

data/example/xcknife-exemplar-historical-data.json-stream

@@ -0,0 +1,14 @@
+{"bundleName":"CommonTestTarget.xctest","deviceName":"iPad Air","event":"begin-ocunit","targetName":"CommonTestTarget","testType":"logic-test","sdkName":"iphonesimulator9.2"}
+{"test":"1","className":"CommonTestClass","event":"end-test","totalDuration":0.1075069904327393}
+{"test":"1","className":"CommonTestClass","event":"end-test","totalDuration":0.303464949131012}
+{"test":"1","className":"CommonTestClass","event":"end-test","totalDuration":0.2003870010375977}
+{"bundleName":"iPadTestTarget.xctest","deviceName":"iPad Air","event":"begin-ocunit","targetName":"iPadTestTarget","testType":"application-test","sdkName":"iphonesimulator9.2"}
+{"test":"1","className":"iPadTestClassOne","event":"end-test","totalDuration":1.001249969005585}
+{"test":"1","className":"iPadTestClassThree","event":"end-test","totalDuration":0.5002140402793884}
+{"test":"1","className":"iPadTestClassTwo","event":"end-test","totalDuration":5.001157999038696}
+{"bundleName":"iPhoneTestTarget.xctest","deviceName":"iPad Air","event":"begin-ocunit","targetName":"iPhoneTestTarget","testType":"application-test","sdkName":"iphonesimulator9.2"}
+{"test":"1","className":"iPhoneTestClassAlpha","event":"end-test","totalDuration":0.2037490010261536}
+{"test":"1","className":"iPhoneTestClassBeta","event":"end-test","totalDuration":0.2012439966201782}
+{"test":"1","className":"iPhoneTestClassDelta","event":"end-test","totalDuration":0.1004489660263062}
+{"test":"1","className":"iPhoneTestClassGama","event":"end-test","totalDuration":0.3001730442047119}
+{"test":"1","className":"iPhoneTestClassOmega","event":"end-test","totalDuration":0.101157009601593}

data/example/xcknife-exemplar.json-stream

@@ -0,0 +1,15 @@
+{"bundleName":"iPadTestTarget.xctest","deviceName":"iPad Air","event":"begin-ocunit","targetName":"iPadTestTarget","testType":"application-test","sdkName":"iphonesimulator9.2"}
+{"test":"1","className":"iPadTestClassFour","event":"end-test","totalDuration":"0"}
+{"test":"1","className":"iPadTestClassOne","event":"end-test","totalDuration":"0"}
+{"test":"1","className":"iPadTestClassThree","event":"end-test","totalDuration":"0"}
+{"test":"1","className":"iPadTestClassTwo","event":"end-test","totalDuration":"0"}
+{"bundleName":"iPhoneTestTarget.xctest","deviceName":"iPad Air","event":"begin-ocunit","targetName":"iPhoneTestTarget","testType":"application-test","sdkName":"iphonesimulator9.2"}
+{"test":"1","className":"iPhoneTestClassAlpha","event":"end-test","totalDuration":"0"}
+{"test":"1","className":"iPhoneTestClassBeta","event":"end-test","totalDuration":"0"}
+{"test":"1","className":"iPhoneTestClassDelta","event":"end-test","totalDuration":"0"}
+{"test":"1","className":"iPhoneTestClassGama","event":"end-test","totalDuration":"0"}
+{"test":"1","className":"iPhoneTestClassOmega","event":"end-test","totalDuration":"0"}
+{"bundleName":"CommonTestTarget.xctest","deviceName":"iPad Air","event":"begin-ocunit","targetName":"CommonTestTarget","testType":"logic-test","sdkName":"iphonesimulator9.2"}
+{"test":"1","className":"CommonTestClass","event":"end-test","totalDuration":"0"}
+{"test":"1","className":"CommonTestClass","event":"end-test","totalDuration":"0"}
+{"test":"1","className":"CommonTestClass","event":"end-test","totalDuration":"0"}

data/lib/xcknife.rb

@@ -0,0 +1,9 @@
+require 'xcknife/events_analyzer'
+require 'xcknife/stream_parser'
+require 'xcknife/xctool_cmd_helper'
+require 'xcknife/runner'
+require 'xcknife/exceptions'
+
+module XCKnife
+  VERSION = '0.5.0'
+end

data/lib/xcknife/events_analyzer.rb

@@ -0,0 +1,55 @@
+require 'xcknife/json_stream_parser_helper'
+require 'set'
+
+module XCKnife
+  class EventsAnalyzer
+    include JsonStreamParserHelper
+    attr_reader :target_class_map, :total_tests
+
+    def self.for(events, relevant_partitions)
+      return NullEventsAnalyzer.new if events.nil?
+      new(events, relevant_partitions)
+    end
+
+    def initialize(events, relevant_partitions)
+      @total_tests = 0
+      @relevant_partitions = relevant_partitions
+      @target_class_map = analyze_events(events)
+    end
+
+    def has_test_target?(target)
+      target_class_map.has_key?(target)
+    end
+
+    def has_test_class?(target, clazz)
+      has_test_target?(target) and target_class_map[target].include?(clazz)
+    end
+
+    private
+    def analyze_events(events)
+      ret = Hash.new { |h, key| h[key] = Set.new }
+      each_test_event(events) do |target_name, result|
+        next unless @relevant_partitions.include?(target_name)
+        @total_tests += 1
+        ret[target_name] << result.className
+      end
+      ret
+    end
+  end
+
+  # Null object for EventsAnalyzer
+  # @ref https://en.wikipedia.org/wiki/Null_Object_pattern
+  class NullEventsAnalyzer
+    def has_test_target?(target)
+      true
+    end
+
+    def has_test_class?(target, clazz)
+      true
+    end
+
+    def total_tests
+      0
+    end
+  end
+end