tumugi 0.5.3 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ee5ae43e408b981ed963063ab589495702a573ba
-  data.tar.gz: 0bded13c3f3eb229613b724b04c96963741b0d09
+  metadata.gz: 8245cbb305b7100cc7b3c522ee232d89ac6704c6
+  data.tar.gz: 3a7b0605e4f5b7e7fbc5b8fa0b77c1c5c9c932c2
 SHA512:
-  metadata.gz: 50c955ce2e7f277bcda63d977aae3eaafe1c4ed0e989df7a490592b61d43cc1478fa844221dc2f3c6f6187f50dbc15882f4beceefdd0ca1a64ad5acaff51ffe0
-  data.tar.gz: a4e031251a9b2a4077bde42d735f7aa06f29c55ec1c9b6452bd5d3b0dbd672a933fbb9a89cb384bda9ac53bc0bf0e52863c3cb3ca2b540c029d98842a20f8f03
+  metadata.gz: 081770a32be9360b5c3dec706f5277cfe22c42b1d2ad21771bfa3567b83a75451d7f2ab71dfcf69c5e0391e11b8224172d9ec71a9e7c1b1c40ddf2eaed0c05cf
+  data.tar.gz: 010e6fcb0d19a20c8e7e33fddb06f63cbac19b5f15a45f40cc381cc0c36522fee4ef80b7939358318d2e7b20566ffb65c0f0b4174e2ccb74fdc0aaa0f35dfd38

data/.gitignore

@@ -7,3 +7,5 @@
 /pkg/
 /spec/reports/
 /tmp/
+/.ruby-version
+/tumugi-plugin-*

data/.travis.yml

@@ -6,11 +6,14 @@ rvm:
   - 2.3.1
   - ruby-head
   - jruby-9.0.5.0
+  - jruby-9.1.2.0
+  - jruby-head
 before_install:
   - gem install bundler
 matrix:
   allow_failures:
     - rvm: ruby-head
+    - rvm: jruby-head
 addons:
   apt:
     packages:

data/CHANGELOG.md

@@ -1,12 +1,54 @@
 # Change Log
 
-## [0.5.3](https://github.com/tumugi/tumugi/tree/0.5.3) (2016-06-14)
-[Full Changelog](https://github.com/tumugi/tumugi/compare/v0.5.2...0.5.3)
+## [0.6.0](https://github.com/tumugi/tumugi/tree/0.6.0) (2016-07-09)
+[Full Changelog](https://github.com/tumugi/tumugi/compare/v0.5.3...0.6.0)
+
+**Implemented enhancements:**
+
+- Refactoring executor [\#95](https://github.com/tumugi/tumugi/issues/95)
+- Improve log [\#87](https://github.com/tumugi/tumugi/issues/87)
+- DSL improvement [\#81](https://github.com/tumugi/tumugi/issues/81)
+- Support dryrun option [\#55](https://github.com/tumugi/tumugi/issues/55)
+- param\_set accept block [\#53](https://github.com/tumugi/tumugi/issues/53)
+- Plugin test helper [\#29](https://github.com/tumugi/tumugi/issues/29)
+- Plugin template generator [\#28](https://github.com/tumugi/tumugi/issues/28)
+- Add Tumugi.app which is alias of Tumugi.application [\#92](https://github.com/tumugi/tumugi/pull/92) ([hakobera](https://github.com/hakobera))
+- Add short cut method to set parmeter in DSL [\#84](https://github.com/tumugi/tumugi/pull/84) ([hakobera](https://github.com/hakobera))
+- Improve result table [\#83](https://github.com/tumugi/tumugi/pull/83) ([hakobera](https://github.com/hakobera))
+- Introduce new DSL syntax: set [\#82](https://github.com/tumugi/tumugi/pull/82) ([hakobera](https://github.com/hakobera))
+
+**Fixed bugs:**
+
+- Fix result reporting failed if parameter type is not :string [\#96](https://github.com/tumugi/tumugi/pull/96) ([hakobera](https://github.com/hakobera))
+- Fix task timeout logic [\#93](https://github.com/tumugi/tumugi/pull/93) ([hakobera](https://github.com/hakobera))
+
+**Closed issues:**
+
+- \[Breaking Change\] Remove auto\_bind feature and make accessible CLI param from task [\#88](https://github.com/tumugi/tumugi/issues/88)
+
+**Merged pull requests:**
+
+- Remove File.exists? [\#100](https://github.com/tumugi/tumugi/pull/100) ([hakobera](https://github.com/hakobera))
+- Remove development dependency: github\_changelog\_generator [\#98](https://github.com/tumugi/tumugi/pull/98) ([hakobera](https://github.com/hakobera))
+- Refactoring executor [\#97](https://github.com/tumugi/tumugi/pull/97) ([hakobera](https://github.com/hakobera))
+- Update README [\#94](https://github.com/tumugi/tumugi/pull/94) ([hakobera](https://github.com/hakobera))
+- Test with jruby 9.1.2.0 on travis [\#91](https://github.com/tumugi/tumugi/pull/91) ([hakobera](https://github.com/hakobera))
+- Add command to create plugin template project [\#90](https://github.com/tumugi/tumugi/pull/90) ([hakobera](https://github.com/hakobera))
+- Improve log [\#89](https://github.com/tumugi/tumugi/pull/89) ([hakobera](https://github.com/hakobera))
+- Add plugin test helper [\#86](https://github.com/tumugi/tumugi/pull/86) ([hakobera](https://github.com/hakobera))
+- Parameter can be set by block [\#85](https://github.com/tumugi/tumugi/pull/85) ([hakobera](https://github.com/hakobera))
+
+## [v0.5.3](https://github.com/tumugi/tumugi/tree/v0.5.3) (2016-06-14)
+[Full Changelog](https://github.com/tumugi/tumugi/compare/v0.5.2...v0.5.3)
 
 **Fixed bugs:**
 
 - Fix logger not defiend in Tumugi::Config [\#79](https://github.com/tumugi/tumugi/pull/79) ([hakobera](https://github.com/hakobera))
 
+**Merged pull requests:**
+
+- Preapre release for 0.5.3 [\#80](https://github.com/tumugi/tumugi/pull/80) ([hakobera](https://github.com/hakobera))
+
 ## [v0.5.2](https://github.com/tumugi/tumugi/tree/v0.5.2) (2016-06-13)
 [Full Changelog](https://github.com/tumugi/tumugi/compare/v0.5.1...v0.5.2)
 

data/README.md

@@ -1,102 +1,16 @@
 [![Build Status](https://travis-ci.org/tumugi/tumugi.svg)](https://travis-ci.org/tumugi/tumugi) [![Code Climate](https://codeclimate.com/github/tumugi/tumugi/badges/gpa.svg)](https://codeclimate.com/github/tumugi/tumugi) [![Coverage Status](https://coveralls.io/repos/tumugi/tumugi/badge.svg?branch=master&service=github)](https://coveralls.io/github/tumugi/tumugi?branch=master) [![Gem Version](https://badge.fury.io/rb/tumugi.svg)](https://badge.fury.io/rb/tumugi)
 
+![tumugi](http://tumugi.github.io/images/logo.png)
+
 # Tumugi
 
 Tumugi is a ruby library to build, run and manage complex workflows. Tumugi enables you to define workflows as a ruby code.
 
-## Installation
-
-**Tumugi only support Ruby 2.1 and above**
-
-Add this line to your application's Gemfile:
-
-```ruby
-gem 'tumugi'
-```
-
-And then execute:
-
-```bash
-$ bundle
-```
-
-Or install it yourself as:
-
-```bash
-$ gem install tumugi
-```
-
-## Usage
-
-![Alt text](http://g.gravizo.com/g?
-  digraph G {
-    rankdir=LR;
-    Task2 -> Task1
-    Task3 -> Task1;
-    Task4 -> Task2;
-    Task4 -> Task3;
-  }
-)
-
-You can define workflow above as ruby code:
-
-```rb
-task :task1 do
-  requires [:task2, :task3]
-  run { puts 'task1#run' }
-end
-
-task :task2 do
-  requires [:task4]
-  run { puts 'task2#run' }
-end
-
-task :task3 do
-  requires [:task4]
-  run { puts 'task3#run' }
-end
-
-task :task4 do
-  # You can use do ... end style
-  run do
-    puts 'task4#run'
-    sleep 3
-  end
-end
-```
+**Tumugi only support Ruby >= 2.1 and JRuby >= 9.0.0.0**
 
-Save these code into `workflow.rb`,
-then run this script by `tumugi` command like this:
+## [Documentation](https://tumugi.github.io)
 
-```bash
-$ tumugi run -f workflow.rb task1
-I, [2016-05-06T22:58:28.271234 #76156]  INFO -- : start: task4
-I, [2016-05-06T22:58:28.271310 #76156]  INFO -- : run: task4
-I, [2016-05-06T22:58:28.271386 #76156]  INFO -- : task4#run
-I, [2016-05-06T22:58:29.276218 #76156]  INFO -- : completed: task4
-I, [2016-05-06T22:58:29.276373 #76156]  INFO -- : start: task2
-I, [2016-05-06T22:58:29.276437 #76156]  INFO -- : run: task2
-I, [2016-05-06T22:58:29.276546 #76156]  INFO -- : task2#run
-I, [2016-05-06T22:58:29.276606 #76156]  INFO -- : completed: task2
-I, [2016-05-06T22:58:29.276650 #76156]  INFO -- : start: task3
-I, [2016-05-06T22:58:29.276688 #76156]  INFO -- : run: task3
-I, [2016-05-06T22:58:29.276733 #76156]  INFO -- : task3#run
-I, [2016-05-06T22:58:29.276765 #76156]  INFO -- : completed: task3
-I, [2016-05-06T22:58:29.276798 #76156]  INFO -- : start: task1
-I, [2016-05-06T22:58:29.276823 #76156]  INFO -- : run: task1
-I, [2016-05-06T22:58:29.276861 #76156]  INFO -- : task1#run
-I, [2016-05-06T22:58:29.276899 #76156]  INFO -- : completed: task1
-I, [2016-05-06T22:58:29.278919 #76156]  INFO -- : Result report:
-+-------+----------+------------+-----------+
-| Task  | Requires | Parameters | State     |
-+-------+----------+------------+-----------+
-| task1 | task2    |            | completed |
-|       | task3    |            |           |
-| task3 | task4    |            | completed |
-| task2 | task4    |            | completed |
-| task4 |          |            | completed |
-+-------+----------+------------+-----------+
-```
+Please check [tumugi.github.io](https://tumugi.github.io) for installation & usage.
 
 ## Development
 

data/Rakefile

@@ -1,6 +1,5 @@
 require "bundler/gem_tasks"
 require "rake/testtask"
-require "github_changelog_generator/task"
 
 Rake::TestTask.new(:test) do |t|
   t.libs << "test"
@@ -8,7 +7,4 @@ Rake::TestTask.new(:test) do |t|
   t.test_files = FileList['test/**/*_test.rb']
 end
 
-GitHubChangelogGenerator::RakeTask.new :changelog do |config|
-end
-
 task :default => :test

data/examples/task_inheritance.rb

@@ -1,12 +1,15 @@
 class FileTask < Tumugi::Task
   Tumugi::Plugin.register_task(:file, self)
 
+  param :param1, type: :string
+
   def output
     target(:local_file, "tmp/#{id}.txt")
   end
 
   def run
     log "#{id}#run"
+    log param1
     output.open('w') do |f|
       f.puts('done')
     end
@@ -16,16 +19,22 @@ end
 # Task type can specified by task plugin ID
 task :task1, type: :file do
   requires [:task2, :task3]
+  param1 'param1 of task1'
 end
 
 task :task2, type: :file do
   requires [:task4]
+  param1 'param1 of task2'
 end
 
 # You can also specify type by Class object
 task :task3, type: FileTask do
   requires [:task4]
+  param1 'param1 of task3'
 end
 
 task :task4, type: FileTask do
+  param1 do
+    'param1 of task4'
+  end
 end

data/examples/task_parameter.rb

@@ -1,5 +1,5 @@
 task :task1 do
-  param :key1, auto_bind: true, required: true #=> 'value1', get value from CLI parameter
+  param :key1, auto_bind: true, required: true #=> get value from CLI parameter
 
   requires :task2
   run do
@@ -8,10 +8,9 @@ task :task1 do
 end
 
 task :task2 do
-  # If you want do disable auto binding feature, set `auto_bind: false`
-  param :key1, auto_bind: false #=> nil
-  param :key2
-  param_set :key2, 'value2' #=> 'value'
+  param :key1
+  param :key2, type: :time
+  key2 Time.parse('2016-06-27')
 
   requires :task3
   run do

data/lib/tumugi.rb

@@ -1,4 +1,4 @@
-require 'tumugi/application'
+require 'tumugi/workflow'
 require 'tumugi/config'
 require 'tumugi/error'
 require 'tumugi/logger'
@@ -6,9 +6,12 @@ require 'tumugi/version'
 
 module Tumugi
   class << self
-    def application
-      @application ||= Tumugi::Application.new
+    def workflow
+      @workflow ||= Tumugi::Workflow.new
     end
+    # alias for backward compatibility
+    alias_method :app, :workflow
+    alias_method :application, :workflow
 
     def configure(&block)
       raise Tumugi::ConfigError.new 'Tumugi.configure must have block' unless block_given?

data/lib/tumugi/cli.rb

@@ -1,5 +1,6 @@
 require 'thor'
 require 'tumugi'
+require 'tumugi/command/new'
 
 module Tumugi
   class CLI < Thor
@@ -12,6 +13,7 @@ module Tumugi
         option :params, aliases: '-p', type: :hash, desc: 'Task parameters'
         option :quiet, type: :boolean, desc: 'Suppress log', default: false
         option :verbose, type: :boolean, desc: 'Show verbose log', default: false
+        option :log_format, type: :string, desc: 'Log format', enum: ['text', 'json'], default: 'text'
       end
 
       def exit_on_failure?
@@ -43,20 +45,43 @@ module Tumugi
       execute(:show, task, opts.freeze)
     end
 
+    desc "new PLUGIN_NAME", "Create new plugin project"
+    def new(name)
+      generate_plugin(name, options)
+    end
+
     private
 
     def execute(command, task, options)
-      success = Tumugi.application.execute(command, task, options)
+      success = Tumugi.workflow.execute(command, task, options)
       unless success
-        logger.error "#{command} command failed"
-        raise Thor::Error, 'failed'
+        raise Thor::Error, "execute finished, but return it's failed"
       end
-      success
+      logger.info "status: success, command: #{command}, task: #{task}, options: #{options}"
     rescue => e
       logger.error "#{command} command failed"
       logger.error e.message
-      e.backtrace.each { |line| logger.error line }
-      raise Thor::Error, 'failed'
+      if options[:verbose]
+        e.backtrace.each { |line| logger.debug line }
+      else
+        logger.error "If you want to know more detail, run with '--verbose' option"
+      end
+      logger.error "status: failed, command: #{command}, task: #{task}, options: #{options}"
+      raise Thor::Error.new("tumugi #{command} failed, please check log")
+    end
+
+    def generate_plugin(name, options)
+      Tumugi::Command::New.new.execute(name, options)
+      logger.info "status: success, command: new, name: #{name}, options: #{options}"
+    rescue => e
+      logger.error e.message
+      if options[:verbose]
+        e.backtrace.each { |line| logger.debug line }
+      else
+        logger.error "If you want to know more detail, run with '--verbose' option"
+      end
+      logger.error "status: failed, command: new, name: #{name}, options: #{options}"
+      raise Thor::Error.new("tumugi new failed, please check log")
     end
 
     def logger

data/lib/tumugi/command/new.rb

@@ -0,0 +1,64 @@
+require 'fileutils'
+require 'erubis'
+
+module Tumugi
+  module Command
+    class New
+      def execute(name, options={})
+        full_project_name = "tumugi-plugin-#{name}"
+        templates = [
+          [ "Gemfile.erb", "Gemfile" ],
+          [ "gemspec.erb", "#{full_project_name}.gemspec" ],
+          [ "gitignore.erb", ".gitignore" ],
+          [ "Rakefile.erb", "Rakefile" ],
+          [ "README.md.erb", "README.md" ],
+          [ "examples/example.rb.erb", "examples/example.rb" ],
+          [ "lib/tumugi/plugin/target/target.rb.erb", "lib/tumugi/plugin/target/#{name}.rb" ],
+          [ "lib/tumugi/plugin/task/task.rb.erb", "lib/tumugi/plugin/task/#{name}.rb" ],
+          [ "test/test_helper.rb.erb", "test/test_helper.rb" ],
+          [ "test/test.rb.erb", "test/#{name}_test.rb" ],
+          [ "test/plugin/target/target_test.rb.erb", "test/plugin/target/#{name}_test.rb" ],
+          [ "test/plugin/task/task_test.rb.erb", "test/plugin/task/#{name}_test.rb" ],
+        ]
+
+        @data_dir = "#{File.expand_path(File.dirname(__FILE__))}/../data/new"
+        @dest_dir = "#{(options[:path] || '.')}/#{full_project_name}"
+        if File.exist?(@dest_dir)
+          logger.error("#{@dest_dir} is already exists. Please delete it first")
+          return false
+        end
+
+        FileUtils.mkdir_p(@dest_dir)
+
+        templates.each do |value|
+          src_file, dest_file = value
+          eruby = Erubis::Eruby.new(File.read(src_path(src_file)))
+          eruby.filename = src_path(src_file)
+          context = {
+            full_project_name: full_project_name,
+            name: name,
+            tumugi_version: Tumugi::VERSION,
+          }
+          logger.info("Create #{dest_path(dest_file)}")
+          FileUtils.mkdir_p(File.dirname(dest_path(dest_file)))
+          File.write(dest_path(dest_file), eruby.result(context))
+        end
+        return true
+      end
+
+      def logger
+        Tumugi::Logger.instance
+      end
+
+      private
+
+      def src_path(file)
+        "#{@data_dir}/#{file}"
+      end
+
+      def dest_path(file)
+        "#{@dest_dir}/#{file}"
+      end
+    end
+  end
+end

data/lib/tumugi/command/run.rb

@@ -1,120 +1,30 @@
-require 'parallel'
-require 'retriable'
-require 'terminal-table'
-require 'thor'
-require 'timeout'
-
-require 'tumugi/error'
-require 'tumugi/mixin/listable'
+require 'tumugi/dag_result_reporter'
+require 'tumugi/executor/local_executor'
 
 module Tumugi
   module Command
     class Run
-      include Tumugi::Mixin::Listable
-
       def execute(dag, options={})
-        workers = options[:workers] || Tumugi.config.workers
-        settings = { in_threads: workers }
-        start(task_queue(dag), settings)
+        worker_num = options[:workers] || Tumugi.config.workers
+        executor = Tumugi::Executor::LocalExecutor.new(dag, logger, worker_num: worker_num)
+        result = start(executor)
         show_result_report(dag)
-        !dag.tsort.any? { |t| t.state == :failed }
+        result
       end
 
       private
 
-      def task_queue(dag)
-        waiting_task_queue = Queue.new
-        available_task_queue = Queue.new
-        dag.tsort.each do |t|
-          if t.ready?
-            available_task_queue << t
-          else
-            waiting_task_queue << t
-          end
-        end
-        lambda {
-          task = (available_task_queue.empty? ? nil : available_task_queue.pop)
-          if task.nil?
-            task = (waiting_task_queue.empty? ? nil : waiting_task_queue.pop)
-          end
-          task || Parallel::Stop
-        }
-      end
-
-      def start(proc, settings)
-        Parallel.each(proc, settings) do |t|
-          logger.info "start: #{t.id}"
-          timeout = t.timeout || Tumugi.config.timeout
-          Timeout::timeout(timeout, Tumugi::TimeoutError) do
-            until t.ready? || t.requires_failed?
-              sleep 5
-            end
-
-            if t.completed?
-              t.state = :skipped
-              logger.info "#{t.state}: #{t.id} is already completed"
-            elsif t.requires_failed?
-              t.state = :requires_failed
-              logger.info "#{t.state}: #{t.id} has failed requires task"
-            else
-              logger.info "run: #{t.id}"
-              t.state = :running
-
-              begin
-                Retriable.retriable retry_options do
-                  t.run
-                end
-              rescue => e
-                t.state = :failed
-                logger.info "#{t.state}: #{t.id}"
-                logger.error "#{e.message}"
-                logger.error e.backtrace.join("\n")
-              else
-                t.state = :completed
-                logger.info "#{t.state}: #{t.id}"
-              end
-            end
-          end
-        end
-      end
-
-      def retry_options
-        {
-          tries: Tumugi.config.max_retry,
-          base_interval: Tumugi.config.retry_interval,
-          max_interval: Tumugi.config.retry_interval * Tumugi.config.max_retry,
-          max_elapsed_time: Tumugi.config.retry_interval * Tumugi.config.max_retry,
-          multiplier: 1.0,
-          rand_factor: 0.0,
-          on_retry: on_retry
-        }
-      end
-
-      def on_retry
-        Proc.new do |exception, try, elapsed_time, next_interval|
-          if next_interval
-            logger.error "#{exception.class}: '#{exception.message}' - #{try} tries in #{elapsed_time} seconds and #{next_interval} seconds until the next try."
-          else
-            logger.error "#{exception.class}: '#{exception.message}' - #{try} tries in #{elapsed_time} seconds. Task failed."
-          end
-        end
+      def start(executor)
+        logger.info "start workflow: #{Tumugi.workflow.id}"
+        result = executor.execute
+        logger.info "end workflow: #{Tumugi.workflow.id}"
+        result
       end
 
       def show_result_report(dag)
-        headings = ['Task', 'Requires', 'Parameters', 'State']
-        table = Terminal::Table.new headings: headings do |t|
-          dag.tsort.reverse.map do |task|
-            proxy = task.class.merged_parameter_proxy
-            requires = list(task.requires).map do |r|
-              r.id
-            end
-            params = proxy.params.map do |name, _|
-              "#{name}=#{task.send(name.to_sym)}"
-            end
-            t << [ task.id, requires.join("\n"), params.join("\n"), task.state ]
-          end
-        end
-        logger.info "Result report:\n#{table.to_s}"
+        reporter = Tumugi::DAGResultReporter.new
+        report = reporter.show(dag)
+        logger.info "Result report:\n#{report.to_s}"
       end
 
       def logger