startup-time 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a9ba80fb5d1e63389c3ca3b8b3dc3b0a8cc338504f6a23c810d14335e0e044f5
-  data.tar.gz: 8e199bbea99112e8dc993d6c3500bad99048cb72fd8f6bd96c498da40ee9b20d
+  metadata.gz: b4a81d32c87d33dc6e8dd9099982be606ac59256d5968350e109f97979ee96eb
+  data.tar.gz: 0200de23b714da4f6d45f4267e563e54217573fb58fbf66bf1d0ca1016eebd63
 SHA512:
-  metadata.gz: 9de0b24fbd653e0d52e95b39838d2af9258de9fbd0e2e16f75827413c37d4ed86522da5f74b531fd71404faf9cc1f73d3bfe5afd19ded0be2eb0960b220d297c
-  data.tar.gz: a2432428c758ccc48fa0e7616c88fa61e0acd73abab6ec4751d4f3cb2cc93f4fe7efed31c58f611c4e6173d2aa993ff13e6053b1559945a34e036d8e92d12de1
+  metadata.gz: 49c7f9e3e518859dcd7735bb101f6f4e5f48bf6d5a7da890efaf1951de58c0e89cef93a2f447fa4bc677d4cda2fa2aaf24ae9d58fe09683507a65dd9f6b5ca1c
+  data.tar.gz: 9b7dfff84fb1b333aefdd5ff19c176f9e9238754fd52aeac4c9cdf2c18f3fdc5930af7d9ed61ce4fdd2ec44520e15ce8a268a453ed367afd5b4e4d7c4bd60a44

data/CHANGELOG.md

@@ -1,6 +1,12 @@
+## 1.1.0 - 2019-02-27
+
+- dump the version of the compiler/interpreter in verbose mode
+- fix gcc 4 (hello.c): add missing return value
+- update obsolete dmd options
+
 ## 1.0.0 - 2019-02-21
 
-- gemify
+- convert the script to a gem and release it
 - add tests for Deno, GraalVM JavaScript, Nim, and TruffleRuby
 - add --json option to render the results as a JSON array
 - add changelog

data/README.md

@@ -7,17 +7,18 @@
 <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
 
 - [NAME](#name)
-- [USAGE](#usage)
 - [INSTALLATION](#installation)
+- [SYNOPSIS](#synopsis)
 - [DESCRIPTION](#description)
   - [Why?](#why)
-  - [Example Output](#example-output)
+- [OPTIONS](#options)
+- [SAMPLE OUTPUT](#sample-output)
 - [PREREQUISITES](#prerequisites)
-- [Further Reading](#further-reading)
-- [See Also](#see-also)
-- [Author](#author)
-- [Version](#version)
-- [Copyright and License](#copyright-and-license)
+- [REFERENCES](#references)
+- [SEE ALSO](#see-also)
+- [AUTHOR](#author)
+- [VERSION](#version)
+- [COPYRIGHT AND LICENSE](#copyright-and-license)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
@@ -25,7 +26,13 @@
 
 startup-time - a command-line benchmarking tool which measures the startup times of programs in various languages
 
-## USAGE
+## INSTALLATION
+
+```sh
+$ gem install startup-time
+```
+
+## SYNOPSIS
 
 ```sh
 # run all available tests
@@ -44,48 +51,73 @@ $ startup-time --only jvm
 $ startup-time --only fast --omit slow-compile
 
 # increase the number of times each test is run (default: 10)
-$ startup-time --rounds 100
-```
-
-## INSTALLATION
-
-```sh
-$ gem install startup-time
+$ startup-time --count 100
 ```
 
 ## DESCRIPTION
 
-A command-line tool which measures how long it takes to execute ["Hello, world!"](https://en.wikipedia.org/wiki/%22Hello,_World!%22_program) programs written in various languages. It records the fastest time for each program and prints a sorted table of the times after each run. Apart from the [prerequisites](#prerequisites) listed below, the tool doesn't require any of the tested languages to be installed: if a compiler/interpreter is not available, the test is skipped.
+A command-line tool which measures how long it takes to execute ["Hello, world!"](https://en.wikipedia.org/wiki/%22Hello,_World!%22_program)
+programs written in various languages. It records the fastest time for each program and prints a sorted table of the times after each run.
+Apart from the [prerequisites](#prerequisites) listed below, the tool doesn't require any of the tested languages to be installed: if a
+compiler/interpreter is not available, the test is skipped.
 
 ### Why?
 
-To determine which languages are practical (or impractical) to use for command-line interface (CLI) programs. Anything under
-[100 milliseconds](https://www.nngroup.com/articles/response-times-3-important-limits/) is perceived as instantaneous. Anything over that is perceived as delayed, which can tangibly impair productivity and flow, and even risk breaking the user's train of thought.
+To determine which languages are practical (or impractical) to use for command-line interface (CLI) tools. Anything under
+[100 milliseconds](https://www.nngroup.com/articles/response-times-3-important-limits/) is perceived as instantaneous.
+Anything over that is perceptibly delayed, which can impair interactivity and productivity on the command line, and can
+mean the difference between staying in the zone and losing your train of thought.
+
+## OPTIONS
+
+```
+USAGE:
+
+    startup-time [options]
+
+OPTIONS:
+
+    -c, --count, --rounds INTEGER    The number of times to execute each test (default: 10)
+        --clean                      Remove the build directory and exit
+                                     (targets will be recompiled on the next run)
+    -d, --dir PATH                   Specify the build directory
+                                     (default: ${XDG_CONFIG_HOME:-~/.cache/startup-time})
+    -h, --help                       Show this help message and exit
+    -H, --help-only, --help-omit     Show the IDs and groups that can be passed to --only and --omit
+    -j, --json                       Output the results in JSON format (implies --quiet)
+    -o, --only LIST                  Only execute the specified tests (comma-separated list of IDs/groups)
+    -O, --omit LIST                  Don't execute the specified tests (comma-separated list of IDs/groups)
+    -q, --quiet                      Suppress all inessential output
+    -v, --verbose                    Enable verbose logging
+    -V, --version                    Display the version and exit
+```
 
-### Example Output
+## SAMPLE OUTPUT
 
     Test                  Time (ms)
     C (gcc)                    0.33
-    Nim                        0.45
-    LuaJIT                     0.62
-    Rust                       0.63
-    Kotlin Native (konan)      0.66
-    Go                         0.75
-    Lua                        1.00
-    Haskell (GHC)              1.17
+    Nim                        0.44
+    Kotlin Native              0.61
+    LuaJIT                     0.64
+    Go                         0.66
+    Rust                       0.67
+    D (DMD)                    0.88
+    Lua                        0.98
+    D (GDC)                    1.10
+    Haskell (GHC)              1.14
     C++ (g++)                  1.24
-    Perl                       1.75
+    Perl                       1.69
     Java Native (GraalVM)      1.95
-    Crystal                    2.20
-    Bash                       3.37
-    JavaScript (Deno)         11.18
-    Python 3                  26.41
-    JavaScript (GraalVM)      35.58
-    Java                      56.40
-    Python 2                  57.00
-    JavaScript (Node.js)      71.61
-    Ruby                      77.30
-    Ruby (TruffleRuby)        91.28
+    Crystal                    2.10
+    Bash                       3.36
+    JavaScript (Deno)         11.15
+    Python 3                  25.68
+    JavaScript (GraalVM)      32.38
+    Java                      54.59
+    Python 2                  56.30
+    JavaScript (Node.js)      70.39
+    Ruby                      77.04
+    Ruby (TruffleRuby)        81.77
     Kotlin                   103.02
     Scala                    801.21
 
@@ -93,27 +125,26 @@ To determine which languages are practical (or impractical) to use for command-l
 
 - Ruby >= 2.4
 
-## Further Reading
+## REFERENCES
 
 - [Response Times: The 3 Important Limits](https://www.nngroup.com/articles/response-times-3-important-limits/)
 - [100 milliseconds](http://cogsci.stackexchange.com/questions/1664/what-is-the-threshold-where-actions-are-perceived-as-instant)
-- [The Great Startup Problem](http://mail.openjdk.java.net/pipermail/mlvm-dev/2014-August/005866.html)
 
-## See Also
+## SEE ALSO
 
 - [gnustavo/startup-times](https://github.com/gnustavo/startup-times) - a script to investigate the startup times of several programming languages
 - [jwiegley/helloworld](https://github.com/jwiegley/helloworld) - a comparison of "Hello, world" startup times in various languages
 - [sharkdp/hyperfine](https://github.com/sharkdp/hyperfine) - a command-line benchmarking tool
 
-## Author
+## AUTHOR
 
 [chocolateboy](mailto:chocolate@cpan.org)
 
-## Version
+## VERSION
 
-1.0.0
+1.1.0
 
-## Copyright and License
+## COPYRIGHT AND LICENSE
 
 Copyright © 2015-2019 by chocolateboy
 

data/lib/startup_time/app.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'benchmark'
+require 'bundler/setup'
 require 'komenda'
 require 'shellwords' # for Array#shelljoin
 require 'tty/table'
@@ -11,8 +12,9 @@ module StartupTime
   class App
     EXPECTED_OUTPUT = /\AHello, world!\r?\n\z/
 
+    include FileUtils # for `sh`
     include Util # for `which`
-    include Services.mixin %i[builder ids_to_groups options selected_tests]
+    include Services.mixin %i[builder ids_to_groups selected_tests]
 
     def initialize(args = ARGV)
       @options = Options.new(args)
@@ -65,7 +67,7 @@ module StartupTime
 
       sorted = @times.sort_by { |result| result[:time] }
 
-      if options.format == :json
+      if @options.format == :json
         require 'json'
         puts sorted.to_json
       elsif !sorted.empty?
@@ -94,16 +96,47 @@ module StartupTime
       args = Array(test[:command])
 
       if args.size == 1 # native executable
+        compiler = test[:compiler] || id
         cmd = File.absolute_path(args.first)
-        return unless File.exist? cmd
+        return unless File.exist?(cmd)
       else # interpreter + source
-        cmd = which(args.first)
+        compiler = args.first
+        cmd = which(compiler)
         return unless cmd
       end
 
+      # dump the compiler/interpreter's version if running in verbose mode
+      if @verbosity == :verbose
+        puts
+        puts "test: #{id}"
+
+        # false (don't print the program's version); otherwise, a command
+        # (template string) to execute to dump the version
+        version = test[:version]
+
+        unless version == false
+          version ||= '%{compiler} --version'
+
+          # the compiler may have been uninstalled since the target was
+          # built, so make sure it still exists
+          if (compiler_path = which(compiler))
+            version_command = version % { compiler: compiler_path }
+            sh version_command
+          end
+        end
+      end
+
       argv0 = args.shift
       command = [cmd, *args]
 
+      unless @verbosity == :quiet
+        if @verbosity == :verbose
+          puts "command: #{command.shelljoin}"
+        else
+          print '.'
+        end
+      end
+
       # make sure the command produces the expected output
       result = Komenda.run(command)
       output = result.output
@@ -118,17 +151,13 @@ module StartupTime
 
       times = []
 
-      unless @verbosity == :quiet
-        if @verbosity == :verbose
-          puts "#{id}: #{command.shelljoin}"
-        else
-          print '.'
-        end
-      end
-
-      @options.rounds.times do
-        times << Benchmark.realtime do
-          system([cmd, argv0], *args, out: File::NULL)
+      # the bundler environment slows down ruby and breaks truffle-ruby,
+      # so make sure it's disabled for the benchmark
+      Bundler.with_clean_env do
+        @options.rounds.times do
+          times << Benchmark.realtime do
+            system([cmd, argv0], *args, out: File::NULL)
+          end
         end
       end
 

data/lib/startup_time/options.rb

@@ -48,8 +48,8 @@ module StartupTime
 
         opts.on(
           '--clean',
-          'Remove the build directory and exit (targets will be ',
-          'recompiled on the next run)'
+          'Remove the build directory and exit',
+          '(targets will be recompiled on the next run)'
         ) do
           @action = :clean
         end
@@ -58,7 +58,8 @@ module StartupTime
           '-d',
           '--dir PATH',
           String,
-          "Specify the build directory (default: #{BUILD_DIR})"
+          'Specify the build directory',
+          "(default: #{BUILD_DIR})"
         ) do |value|
           @build_dir = value
         end
@@ -93,8 +94,7 @@ module StartupTime
           '-o',
           '--only LIST',
           Array, # comma-separated strings
-          'Only execute the specified tests (comma-separated list of ',
-          'IDs/groups)'
+          'Only execute the specified tests (comma-separated list of IDs/groups)'
         ) do |values|
           values.each { |value| registry.only(value.strip) }
         end
@@ -103,8 +103,7 @@ module StartupTime
           '-O',
           '--omit LIST',
           Array, # comma-separated strings
-          "Don't execute the specified tests (comma-separated list ",
-          'of IDs/groups)'
+          "Don't execute the specified tests (comma-separated list of IDs/groups)"
         ) do |values|
           values.each { |value| registry.omit(value.strip) }
         end

data/lib/startup_time/registry.rb

@@ -46,7 +46,7 @@ module StartupTime
       @only = Set.new
     end
 
-    # remove the specified test(s) from the set of enabled tests
+    # add the specified test(s) to the set of disabled tests
     def omit(id)
       @omit.merge(ids_for(id))
     end

data/lib/startup_time/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module StartupTime
-  VERSION = '1.0.0'
+  VERSION = '1.1.0'
 end

data/resources/src/hello.c

@@ -2,4 +2,5 @@
 
 int main(int argc, char * argv[]) {
     printf("Hello, world!\n");
+    return 0;
 }

data/resources/tests.yaml

@@ -1,12 +1,13 @@
 ---
-# anything < 10ms on my system is arbitrarily tagged :fast
-# anything > 50ms on my system is arbitrarily tagged :slow
+# anything < 10 ms on my system is arbitrarily tagged :fast
+# anything > ~100 ms on my system is tagged :slow
 
 bash:
   name: Bash
   groups:
     - fast
     - script
+  version: '%{compiler} --version | head -n1'
   command:
     - bash
     - hello.bash
@@ -17,6 +18,7 @@ crystal:
     - compiled
     - fast
     - native
+    - slow-compile
   source: hello.cr
   compile: '%{compiler} build --release -o %{target} %{source}'
   command: hello.cr.out
@@ -40,7 +42,8 @@ dmd:
     - fast
     - native
   source: hello.d
-  compile: '%{compiler} -O3 -o %{target} %{source}'
+  compile: '%{compiler} -O -of=%{target} %{source}'
+  version: '%{compiler} --version | head -n1'
   command: hello.dmd.d.out
 
 g++:
@@ -53,6 +56,7 @@ g++:
     - native
   source: hello.cpp
   compile: '%{compiler} -O3 -o %{target} %{source}'
+  version: '%{compiler} --version | head -n1'
   command: hello.gpp.out
 
 gcc:
@@ -62,9 +66,9 @@ gcc:
     - compiled
     - fast
     - native
-    - no-ci # XXX flaky on Travis
   source: hello.c
   compile: '%{compiler} -O3 -o %{target} %{source}'
+  version: '%{compiler} --version | head -n1'
   command: hello.gcc.out
 
 gdc:
@@ -76,6 +80,7 @@ gdc:
     - native
   source: hello.d
   compile: '%{compiler} -O3 -o %{target} %{source}'
+  version: '%{compiler} --version | head -n1'
   command: hello.gdc.d.out
 
 ghc:
@@ -97,6 +102,7 @@ go:
     - native
   source: hello.go
   compile: '%{compiler} build -o %{target} %{source}'
+  version: '%{compiler} version'
   command: hello.go.out
 
 graalvm-js:
@@ -104,6 +110,7 @@ graalvm-js:
   groups:
     - js
     - script
+  version: '%{compiler} --version'
   command:
     - js
     - hello.js
@@ -115,6 +122,7 @@ javac:
     - java
     - jvm
   source: HelloJava.java
+  version: '%{compiler} -version'
   command:
     - java
     - HelloJava
@@ -140,6 +148,8 @@ kotlinc-native:
     - native
     - slow-compile
   source: HelloKotlin.kt
+  compiler: konanc
+  version: '%{compiler} -version'
   command: hello.kt.out
 
 kotlinc:
@@ -152,6 +162,7 @@ kotlinc:
   source: HelloKotlin.kt
   target: HelloKotlinKt.class
   compile: '%{compiler} %{source}'
+  version: '%{compiler} -version'
   command:
     - kotlin
     - HelloKotlinKt
@@ -173,6 +184,7 @@ lua:
     - fast
     - luas
     - script
+  version: '%{compiler} -v'
   command:
     - lua
     - hello.lua
@@ -183,6 +195,7 @@ luajit:
     - fast
     - luas
     - script
+  version: '%{compiler} -v'
   command:
     - luajit
     - hello.lua
@@ -195,6 +208,7 @@ nim:
     - native
   source: hello.nim
   compile: '%{compiler} compile --out:%{target} --opt:speed --verbosity:0 --hints:off %{source}'
+  version: '%{compiler} --version | head -n1'
   command: hello.nim.out
 
 node:
@@ -212,6 +226,8 @@ perl5:
     - fast
     - perl
     - script
+  # trim perl's lengthy version output
+  version: "%{compiler} -MConfig -le 'print qq($^V $Config{archname})'"
   command:
     - perl
     - hello.pl
@@ -271,6 +287,7 @@ scalac:
     - slowest
   source: HelloScala.scala
   compile: '%{compiler} %{source}'
+  version: '%{compiler} -version'
   command:
     - scala
     - HelloScala
@@ -278,9 +295,9 @@ scalac:
 truffle-ruby:
   name: Ruby (TruffleRuby)
   groups:
-    - no-test # doesn't work inside bundler
     - rubies
     - script
+  version: '%{compiler} --version'
   command:
     - truffleruby
     - hello.rb

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: startup-time
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - chocolateboy
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-02-21 00:00:00.000000000 Z
+date: 2019-02-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -24,6 +24,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '5.2'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: cli-pasta
   requirement: !ruby/object:Gem::Requirement
@@ -122,20 +136,6 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 0.0.2
-- !ruby/object:Gem::Dependency
-  name: bundler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '2.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: rubocop
   requirement: !ruby/object:Gem::Requirement
@@ -150,7 +150,7 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 0.65.0
-description: A benchmarking tool which tests how long it takes to execute "Hello,
+description: A benchmarking tool which measures how long it takes to execute "Hello,
   world!" programs in various languages.
 email: chocolate@cpan.org
 executables: