rworkflow 0.6.5 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 338ee9efd55c80b7aead2a33138d200a60ab6d9f
-  data.tar.gz: fb8f2820425891429247f155b66006420571038a
+  metadata.gz: 200f47a61942919470a013a36c8cc25b874bf445
+  data.tar.gz: 2764ce1f1c296d2499723be7a995e618b348b543
 SHA512:
-  metadata.gz: 94ff46a524a656091ae7971d0878221cae8b0b239b91e10d8199ab14debdc4f1fb32141c07a57e0a816c0fd8c721c335a9ac55226927f5456a3d6201ce908c4f
-  data.tar.gz: f8a1c8f480275ef5b895cc96fda83a3f9c4bcd82211a178b2764a7aba03defd50b9f2c78e6860f73f1958901ff66f52621c6f6fb097c06c231275a10b1f311c0
+  metadata.gz: 53f8c990a5557b835ae4acdb2a7cf63e1c1dcf8202d626a8d523af6fca4578841e57cdbba6c41c7fa1912543c415e337ebb31a8b99a82ed5c634e4d18e7ab0a7
+  data.tar.gz: d8547f66c05957d1ea952039bbfc8c3ce50e0c0d4bc15ec520cd1bde76d6d52998e6f5ed14336d5d829a0b0acafe12663083e1a2de6668b08383a737f1847e11

data/README.md

@@ -0,0 +1,110 @@
+# Rworkflow
+
+[![GitHub release](https://img.shields.io/badge/release-0.6.5-blue.png)](https://github.com/barcoo/rworkflow/releases/tag/0.6.5)
+[![Build Status](https://travis-ci.org/barcoo/rworkflow.svg?branch=master&cache=busted)](https://travis-ci.org/barcoo/rworkflow)
+[![Coverage Status](https://coveralls.io/repos/github/barcoo/rworkflow/badge.svg?branch=master)](https://coveralls.io/github/barcoo/rworkflow?branch=master)
+
+The rworkflow framework removes many headaches when it comes to asynchronous tasks that need to run after the other, depending on their previous state.
+
+A workflow is basically a set of defined states with transitions between them (a state machine or lifecycle) in which every state can contain a set of jobs (a job can be any serializable object). A set of jobs can then be pushed to the workflow on the initial state and then they can transit between the states following the transitions defined. Moreover, the jobs can be also transformed (e.g. a state receives 100 jobs, groups them into one unique job and then pushes this unique job to the next state with a transition).
+
+A simple flow (Flow class) only implements this model, but there is a subclass of the simple flow (SidekiqFlow) which interprets every state of the lifecycle as a sidekiq job. Thus, whenever some jobs are pushed to the workflow, this dynamically creates the needed sidekiq workers to complete the workflow.
+
+
+## Define a lifecycle
+
+The lifecycle is the definition of the state machine that every job pushed to the workflow will transit:
+
+```ruby
+lifecycle = Workflow::Lifecycle.new do |lc|
+  lc.state("Floating", {cardinality: 10}) do |state|
+    state.transition :rescued, 'Lifeboat'
+    state.transition :drowned, Flow::STATE_FAILED
+  end
+
+  lc.state("Lifeboat", {cardinality: 2}) do |state|
+    state.transition :landed, 'Land'
+    state.transition :starved, Flow::STATE_FAILED
+  end
+
+  lc.state("Land") do |state|
+    state.transition :rescued, Flow::STATE_SUCCESSFUL
+    state.transition :died, Flow::STATE_FAILED
+  end
+
+  lc.initial = "Floating"
+end
+```
+
+Notes:
+
+- For SidekiqFlow worflows the cycle state names need to be the same as an existing class that derives from Workflow::Worker (which implements a SidekiqWork)
+- The transition state names (e.g. :rejected, :generated) are arbitrary, the Worker needs to call those later. There can be more than two.
+- There are some predefined final states (Flow::STATE_FAILED, Flow::STATE_SUCCESSFUL). When all jobs are pushed via transitions to one of these states, the workflow is then finished.
+- The state cardinality indicates how many jobs will be served to a state (by default one)
+
+## Create Workers
+
+For SidekiqFlow create a subclass of Workflow::Worker for each state defined on the lifecycle (except for predefined final states)
+
+```ruby
+class Floating < Workflow::Worker
+  def process(objects)
+    # The size of objects will be at the most the cardinality defined on the lifecycle
+    rescued, drowned = objects.partition { |object| object.even? }
+
+    transition(:rescued, rescued)
+    transition(:drowned, drowned)
+  end
+end
+
+class Lifeboat < Workflow::Worker
+  def process(objects)
+    landed, starved = objects.partition { |object| object < 4 }
+
+    transition(:landed, landed)
+    transition(:starved, starved)
+  end
+end
+
+class Land < Workflow::Worker
+  def process(objects)
+    rescued, died = objects.partition { |object| object == 0 }
+
+    transition(:rescued, rescued)
+    transition(:died, died)
+  end
+end
+```
+
+Notes:
+
+- Create a class with the exact name that you defined above in the lifecycle definition
+- You will be given an array of objects of a size to a maximum of the defined cardinality in the state. By default is 1.
+- The worker is responsible for the jobs that receives: it has to define a transition for them or otherwise they will be out of the workflow.
+
+## Create and execute the Workflow
+
+```ruby
+options = {}
+workflow = Workflow::SidekiqFlow.create(lifecycle, 'SafeBoatWorkflow', options)
+initial_jobs = [1,2,3,45,6,7,8,9,10]
+workflow.start(initial_jobs)
+```
+
+Notes:
+
+- Create a new Sidekiq flow using the lifecycle object defined in the first step above
+- Run flow.start passing in an array of objects
+- The objects need to be serializable
+- _options_ can contain several properties for the workflow (TODO: complete/expand)
+
+# Roadmap
+
+1. Decouple persistence layer (for now rworkflow depends on redis_rds which, in turn, depends on redis)
+2. See [State and Transition Policies](doc/states_and_transitions_policies.rdoc).
+3. Test Helper (simplify tests)
+4. Improve logging
+5. Use a separated Redis instance/db instead of a namespace?
+6. sidekiq and rails dependencies should be optional
+7. Move Web UI from CimRails to here

data/Rakefile

@@ -4,16 +4,6 @@ rescue LoadError
   puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
 end
 
-require 'rdoc/task'
-
-RDoc::Task.new(:rdoc) do |rdoc|
-  rdoc.rdoc_dir = 'rdoc'
-  rdoc.title    = 'Rworkflow'
-  rdoc.options << '--line-numbers'
-  rdoc.rdoc_files.include('README.rdoc')
-  rdoc.rdoc_files.include('lib/**/*.rb')
-end
-
 Bundler::GemHelper.install_tasks
 
 require 'rake/testtask'
@@ -26,116 +16,3 @@ Rake::TestTask.new(:test) do |t|
 end
 
 task default: :test
-
-namespace :cim do
-  desc 'Tags, updates README, and CHANGELOG and pushes to Github. Requires ruby-git'
-  task :release do
-    tasks = ['cim:assert_clean_repo', 'cim:git_fetch', 'cim:set_new_version', 'cim:update_readme', 'cim:update_changelog', 'cim:commit_changes', 'cim:tag']
-    begin
-      tasks.each { |task| Rake::Task[task].invoke }
-      `git push && git push origin '#{Rworkflow::VERSION}'`
-    rescue => error
-      puts ">>> ERROR: #{error}; might want to reset your repository"
-    end
-  end
-
-  desc 'Fails if the current repository is not clean'
-  task :assert_clean_repo do
-    status = `git status -s`.chomp.strip
-    if status.strip.empty?
-      status = `git log origin/master..HEAD`.chomp.strip # check if we have unpushed commits
-      if status.strip.empty?
-        puts '>>> Repository is clean!'
-      else
-        puts '>>> Please push your committed changes before releasing!'
-        exit(-1)
-      end
-    else
-      puts '>>> Please stash or commit your changes before releasing!'
-      exit(-1)
-    end
-  end
-
-  desc 'Fetches latest tags/commits'
-  task :git_fetch do
-    puts '>>> Fetching latest git refs'
-    `git fetch --tags`
-  end
-
-  desc 'Requests the new version number'
-  task :set_new_version do
-    STDOUT.print(">>> New version number (current: #{Rworkflow::VERSION}; leave blank if already updated): ")
-    input = STDIN.gets.strip.tr("'", "\'")
-
-    current = if input.empty?
-      Rworkflow::VERSION
-    else
-      unless input =~ /[0-9]+\.[0-9]+\.[0-9]+/
-        puts '>>> Please use semantic versioning!'
-        exit(-1)
-      end
-
-      input
-    end
-
-    latest = `git describe --abbrev=0`.chomp.strip
-    unless Gem::Version.new(current) > Gem::Version.new(latest)
-      puts ">>> Latest tagged version is #{latest}; make sure gem version (#{current}) is greater!"
-      exit(-1)
-    end
-
-    if !input.empty?
-      `sed -i -u "s@VERSION = '#{Rworkflow::VERSION}'@VERSION = '#{input}'@" #{File.expand_path('../lib/rworkflow/version.rb', __FILE__)}`
-      $VERBOSE = nil
-      Rworkflow.const_set('VERSION', input)
-      $VERBOSE = false
-
-      `bundle check` # force updating version
-    end
-  end
-
-  desc 'Updates README with latest version'
-  task :update_readme do
-    puts '>>> Updating README.md'
-    replace = %([![GitHub release](https://img.shields.io/badge/release-#{Rworkflow::VERSION}-blue.png)](https://github.com/barcoo/rworkflow/releases/tag/#{Rworkflow::VERSION}))
-
-    `sed -i -u 's@^\\[\\!\\[GitHub release\\].*$@#{replace}@' README.md`
-  end
-
-  desc 'Updates CHANGELOG with commit log from last tag to this one'
-  task :update_changelog do
-    puts '>>> Updating CHANGELOG.md'
-    latest = `git describe --abbrev=0`.chomp.strip
-    log = `git log --pretty=format:'- [%h](https://github.com/barcoo/rworkflow/commit/%h) *%ad* __%s__ (%an)' --date=short '#{latest}'..HEAD`.chomp
-
-    changelog = File.open('.CHANGELOG.md', 'w')
-    changelog.write("# Changelog\n\n###{Rworkflow::VERSION}\n\n#{log}\n\n")
-    File.open('CHANGELOG.md', 'r') do |file|
-      begin
-        file.readline # skip first two lines
-        file.readline
-        while buffer = file.read(2048)
-          changelog.write(buffer)
-        end
-      rescue => error
-      end
-    end
-
-    changelog.close
-    `mv '.CHANGELOG.md' 'CHANGELOG.md'`
-  end
-
-  desc 'Commits the README/CHANGELOG changes'
-  task :commit_changes do
-    puts '>>> Committing updates to README/CHANGELOG'
-    `git commit -am'Updated README.md and CHANGELOG.md on new release'`
-  end
-
-  desc 'Creates and pushes the tag to git'
-  task :tag do
-    puts '>>> Tagging'
-    STDOUT.print('>>> Please enter a tag message: ')
-    input = STDIN.gets.strip.tr("'", "\'")
-    `git tag -a '#{Rworkflow::VERSION}' -m '#{input}'`
-  end
-end

data/lib/rworkflow/flow.rb

@@ -40,7 +40,7 @@ module Rworkflow
 
     def finished?
       return false unless started?
-      total = get_counters.reduce(0) do |sum, pair|
+      total = self.counters.reduce(0) do |sum, pair|
         self.class.terminal?(pair[0]) ? sum : (sum + pair[1].to_i)
       end
 
@@ -49,13 +49,13 @@ module Rworkflow
 
     def status
       status = 'Running'
-      status = (successful?) ? 'Finished' : 'Failed' if finished?
+      status = successful? ? 'Finished' : 'Failed' if finished?
 
       return status
     end
 
     def created_at
-      return @created_at ||= begin Time.at(get(:created_at, 0)) end
+      return @created_at ||= begin Time.zone.at(get(:created_at, 0)) end
     end
 
     def started?
@@ -71,11 +71,11 @@ module Rworkflow
     end
 
     def start_time
-      return Time.at(get(:start_time, 0))
+      return Time.zone.at(get(:start_time, 0))
     end
 
     def finish_time
-      return Time.at(get(:finish_time, 0))
+      return Time.zone.at(get(:finish_time, 0))
     end
 
     def expected_duration
@@ -90,22 +90,22 @@ module Rworkflow
       return get_list(state).size
     end
 
-    def get_counters
-      counters = @storage.get(:counters)
-      if !counters.nil?
-        counters = begin
-          self.class.serializer.load(counters)
+    def counters
+      the_counters = @storage.get(:counters)
+      if !the_counters.nil?
+        the_counters = begin
+          self.class.serializer.load(the_counters)
         rescue => e
           Rails.logger.error("Error loading stored flow counters: #{e.message}")
           nil
         end
       end
-      return counters || get_counters!
+      return the_counters || counters!
     end
 
     # fetches counters atomically
-    def get_counters!
-      counters = { processing: 0 }
+    def counters!
+      the_counters = { processing: 0 }
 
       names = @lifecycle.states.keys
       results = RedisRds::Object.connection.multi do
@@ -115,14 +115,14 @@ module Rworkflow
       end
 
       (self.class::STATES_TERMINAL + names).each do |name|
-        counters[name] = results.shift.to_i
+        the_counters[name] = results.shift.to_i
       end
 
-      counters[:processing] = results.shift.reduce(0) { |sum, pair| sum + pair.last.to_i }
+      the_counters[:processing] = results.shift.reduce(0) { |sum, pair| sum + pair.last.to_i }
 
-      return counters
+      return the_counters
     end
-    private :get_counters!
+    private :counters!
 
     def fetch(fetcher_id, state_name)
       @processing.set(fetcher_id, 1)
@@ -159,7 +159,7 @@ module Rworkflow
 
     def list_objects(state_name, limit = -1)
       list = get_list(state_name)
-      return list.get(0, limit).map {|object| self.class.serializer.load(object)}
+      return list.get(0, limit).map { |object| self.class.serializer.load(object) }
     end
 
     def get_state_list(state_name)
@@ -183,9 +183,9 @@ module Rworkflow
           post_process
 
           if self.public?
-            counters = get_counters!
-            counters[:processing] = 0 # Some worker might have increased the processing flag at that time even if there is no more jobs to be done
-            @storage.setnx(:counters, self.class.serializer.dump(counters))
+            the_counters = self.counters!
+            the_counters[:processing] = 0 # Some worker might have increased the processing flag at that time even if there is no more jobs to be done
+            @storage.setnx(:counters, self.class.serializer.dump(the_counters))
             states_cleanup
           else
             self.cleanup
@@ -194,8 +194,7 @@ module Rworkflow
       end
     end
 
-    def post_process
-    end
+    def post_process; end
     protected :post_process
 
     def metadata_string
@@ -268,7 +267,7 @@ module Rworkflow
 
     def get(key, default = nil)
       value = @flow_data.get(key)
-      value = if value.nil? then default else self.class.serializer.load(value) end
+      value = value.nil? ? default : self.class.serializer.load(value)
 
       return value
     end
@@ -319,7 +318,7 @@ module Rworkflow
     end
 
     def total_objects_processed(counters = nil)
-      return (counters || get_counters).reduce(0) do |sum, pair|
+      return (counters || self.counters).reduce(0) do |sum, pair|
         if self.class.terminal?(pair[0])
           sum + pair[1]
         else
@@ -329,11 +328,11 @@ module Rworkflow
     end
 
     def total_objects(counters = nil)
-      return (counters || get_counters).reduce(0) { |sum, pair| sum + pair[1] }
+      return (counters || self.counters).reduce(0) { |sum, pair| sum + pair[1] }
     end
 
     def total_objects_failed(counters = nil)
-      return (counters || get_counters).reduce(0) do |sum, pair|
+      return (counters || self.counters).reduce(0) do |sum, pair|
         if self.class.failure?(pair[0])
           sum + pair[1]
         else
@@ -406,12 +405,14 @@ module Rworkflow
       def read_flow_class(id)
         klass = nil
         raw_class = id.split('__').first
-        klass = begin
-          raw_class.constantize
-        rescue NameError => _
-          Rails.logger.warn("Unknown flow class for workflow id #{id}")
-          nil
-        end if !raw_class.nil?
+        if !raw_class.nil?
+          klass = begin
+            raw_class.constantize
+          rescue NameError => _
+            Rails.logger.warn("Unknown flow class for workflow id #{id}")
+            nil
+          end
+        end
 
         return klass
       end

data/lib/rworkflow/lifecycle.rb

@@ -5,11 +5,12 @@ module Rworkflow
 
     CARDINALITY_ALL_STARTED = :all_started # Indicates a cardinality equal to the jobs pushed at the start of the workflow
 
+    DEFAULT_CARDINALITY = State::DEFAULT_CARDINALITY
+    STATE_POLICY_NO_WAIT = State::STATE_POLICY_NO_WAIT
     DEFAULT_STATE_OPTIONS = {
-      cardinality: State::DEFAULT_CARDINALITY,
-      priority: State::DEFAULT_PRIORITY,
-      policy: State::STATE_POLICY_NO_WAIT
-    }
+      cardinality: self::DEFAULT_CARDINALITY,
+      policy: self::STATE_POLICY_NO_WAIT
+    }.freeze
 
     def initialize(state_class: State, state_options: {})
       @state_options = DEFAULT_STATE_OPTIONS.merge(state_options)
@@ -30,7 +31,7 @@ module Rworkflow
 
     def transition(from, name)
       from_state = @states[from]
-      fail(StateError, from) if from_state.nil?
+      raise(StateError, from) if from_state.nil?
 
       return from_state.perform(name, @default)
     end