dynflow 0.8.1 → 0.8.2

checksums.yaml

@@ -1,15 +1,15 @@
 ---
 !binary "U0hBMQ==":
   metadata.gz: !binary |-
-    YzQyNGMyZjRlNGM4NDQ3ODg0MjAwN2QzZDVmZDM3ZDhkYTk3YTg2OA==
+    MGY3ZTU0Nzk1ZTJlMjQ3YzRjMTI4ZjkzYTNkODZhYWIzYzZhYzc4ZQ==
   data.tar.gz: !binary |-
-    ZjlkYWNiYTkzMTZmZTYwNTAyOGI0MTk1MjQ4ZmY0ODJlZDJkYWNjYw==
+    NTU1MDY2YTBhNjk2Mjc2YjJkNDhkM2JhYzFjYzk1ODU3ZDUzNDc2OQ==
 SHA512:
   metadata.gz: !binary |-
-    OTA5MTVjODI5NzJjODkxNmYzY2ZhMzNjY2RjZTI2ZTY1NmZmYzhjYTZhYzIy
-    MDQ1NmUxN2ViZDE2NmIxZTdmNTIzZmJjOWFhNjM0YzZmNDIxYjNmNzA3YmE2
-    ZjhhMzY4MDdkMzMyZWE4ZTU0MTA1Yjk0NTYzMDAxYWJiZmY0NDg=
+    ZDRiZmQ2ODQ1MjEyYThmMDg4YjkyNDE1Y2Q5MTI1OWNkMDNjODlmODE5ODlk
+    N2Q2NDQyNzYyNDQzOWFmZGRhMjZmYTEwYWM1YTkyZTdlMDExMWY0NmU4N2Zl
+    MzkxYjFlZDgxY2I1MWQ4ZjhhZTgxZDY4NTI3NmRkMTBhNDE5NzE=
   data.tar.gz: !binary |-
-    NzY2NDVmNTBmNDQwZWE3NDIwMTM4ZjA2NTUzNjUwYWE2ODhkYmJiNTcyODFl
-    NzhmNTU1NDFjNTJiOWY0YzYwNDJhMDkyNThjYTVhZGM5NDgzYTUwOTk3NDU4
-    Nzk4YmFmZDJiNTYyZDQ5YTdhNmFjMGQzZjI1ZDFiMTBiOGE5ZGM=
+    MGQyNzY3Y2QwZTZhYTQwNGIwYzY4NzgyNzg3NjVhZTliMDBmZTNjMzBkNDc0
+    ZTRhMGZhNGJiODEwZjIzODNlMDczMmU0ZDNlNDIwYzQyZmJkMmUzNzhjNDQz
+    MDRmMzMxNzdmMWZhNWRlOWE4Yjk1Nzc5Njc2MjAyMmUwNzBhMzU=

data/doc/pages/source/documentation/index.md

@@ -123,7 +123,7 @@ end
 ```
 
 Note that it does not have to be only other actions that are planned to run.
-In fact it's very common that the action plan itself, which means it will
+In fact it's very common that the action plans itself, which means it will
 put its own `run` method call in the execution plan. In order to do that
 you can use `plan_self`. This could be used in MyActions::File::Destroy
 used in previous example
@@ -142,7 +142,7 @@ end
 
 In example above, it seems that `plan_self` is just shortcut to
 `plan_action MyActions::File::Destroy, filename` but it's not entirely true.
-Note that `plan_action` always trigger `plan` of a given action while `plan_self`
+Note that `plan_action` always triggers `plan` of a given action while `plan_self`
 plans only the `run` of Action, so by using `plan_action` we'd end up in
 endless loop.
 
@@ -235,6 +235,8 @@ TriggerResult = Algebrick.type do
   PlaningFailed   = type { fields! execution_plan_id: String, error: Exception }
   # Returned by #trigger when planning is successful but execution fails to start.
   ExecutionFailed = type { fields! execution_plan_id: String, error: Exception }
+  # Returned by #schedule when scheduling succeeded.
+  Scheduled       = type { fields! execution_plan_id: String }
   # Returned by #trigger when planning is successful, #future will resolve after
   # ExecutionPlan is executed.
   Triggered       = type { fields! execution_plan_id: String, future: Future }
@@ -268,6 +270,37 @@ def self.trigger_task(async, action, *args, &block)
 end
 ```
 
+#### Scheduling
+
+Scheduling an action means setting it up to be triggered at set time in future.
+Any action can be scheduled by calling:
+
+```ruby
+world_instance.schedule(AnAction,
+                        { start_at: Time.now + 360, start_before: Time.now + 400 },
+                        *args)
+```
+
+This snippet of code would schedule `AnAction` with arguments `args` to be executed
+in the time interval between `start_at` and `start_before`. Setting `start_before` to `nil`
+would schedule this action without the timeout limit.
+
+When an action is scheduled, an execution plan object is created with state set
+to `scheduled`, but it doesn't run the the plan phase yet, the planning happens
+when the `start_at` time comes. If the planning doesn't happen in time
+(e.g. after `start_before`), the execution plan is marked as failed
+(its state is set to `stopped` and result to `error`).
+
+Since the `args` have to be saved, there must be a mechanism to safely serialize and deserialize them
+in order to make them survive being saved in a database. This is handled by a serializer.
+Different serializers can be set per action by overriding its `schedule` method.
+
+Planning of the scheduled plans is handled by `Scheduler`, an object which
+periodically checks for scheduled execution plans and plans them. Scheduled execution
+plans don't do anything by themselves, they just wait to be picked up and planned by a Scheduler.
+It means that if no scheduler is present, their planning will be delayed until a scheduler
+is spawned.
+
 #### Plan phase
 
 Planning always uses the thread triggering the action. Plan phase
@@ -319,7 +352,7 @@ end
 
 {% info_block %}
 
-It's considered a good practice to use the just enough data for the
+It's considered a good practice to use just enough data for the
 input for the action to perform the job. That means not too much
 (such as using ActiveRecord's attributes), as it might have
 performance impact as well as causes issues when changing the
@@ -488,8 +521,8 @@ def plan
   # so it's added to the above sequence to be executed as 4th.
   action1 = plan_action AnAction, actions_executed_sequentially.last.output
 
-  # It's planned in default plan's concurrency scope it's executed concurrently
-  # to about four actions.
+  # It's planned in default plan's concurrency scope so it's executed concurrently
+  # with the other four actions.
   action2 = plan_action AnAction
 end
 ```
@@ -557,7 +590,7 @@ The usual execution looks as follows, we use an ActiveRecord User as example of
     used. Potentially, saving some data that were retrieved in the `run`
     phase back to the local database.
 
-For that reason there are transactions around whole `plan` and `finale` phase
+For that reason there are transactions around whole `plan` and `finalize` phase
 (all action's plan methods are in one transaction).
 If anything goes wrong in the `plan` phase any change made during planning to local DB is
 reverted. Same holds for finalizing, if anything goes wrong, all changes are reverted. Therefore
@@ -833,6 +866,7 @@ Each **Action phase** can be in one of the following states:
 **Execution plan** has following states:
 
 -   **Pending** - Planning did not start yet.
+-   **Scheduled** - Scheduled for later execution, not yet planned.
 -   **Planning** - It's being planned.
 -   **Planned** - It've been planned, running phase did not start yet.
 -   **Running** - It's running, `run` and `finalize` phases of actions are executed.
@@ -851,12 +885,12 @@ Each **Action phase** can be in one of the following states:
 
 ### Error handling
 
-If there is an error risen in **`plan` phase**, the error is persisted in the Action object 
-for later inspection and it bubbles up in `World#trigger` method which was used to trigger 
-the action leading to this error. 
+If an error is raised in **`plan` phase**, it is persisted in the Action object
+for later inspection and it bubbles up in `World#trigger` method which was used to trigger
+the action leading to this error.
 If you compare it to errors raised during `run` and `finalize` phase,
-there's the major difference: Those never bubble up in `trigger` because they are running
-in executor not in triggering Thread, they are just persisted in Action object.
+there's one major difference: Those never bubble up in `trigger` because they are running
+in executor not in triggering Thread, they are persisted just in the Action object.
 
 If there is an error in **`run` phase**, the execution pauses. You can inspect the error in
 [console](#console). The error may be intermittent or you may fix the problem manually. After
@@ -981,6 +1015,7 @@ client worlds: useful in production, see [develpment vs. production](#developmen
 client requests and other worlds
 1. **executor dispatcher** - responsible for getting requests from
 other worlds and sending the responses
+1. **scheduler** - responsible for planning and exectuion of scheduled tasks
 
 {% plantuml %}
 
@@ -1234,7 +1269,7 @@ The persistence making sure that the serialized states of the
 execution plans are persisted for recovery and status tracking. The
 execution plan data are stored in it, with the actual state.
 
-Unlike coordinator, the all the persisted data don't have to be
+Unlike coordinator, all the persisted data don't have to be
 available for all the worlds at the same time: every world needs just
 the data that it is actively working on. Also, all the data don't have to
 be fully synchronized between worlds (as long as the up-to-date data

data/dynflow.gemspec

@@ -22,8 +22,8 @@ Gem::Specification.new do |s|
   s.add_dependency "multi_json"
   s.add_dependency "apipie-params"
   s.add_dependency "algebrick", '~> 0.7.0'
-  s.add_dependency "concurrent-ruby", '~> 0.9.0.pre3'
-  s.add_dependency "concurrent-ruby-edge", '~> 0.1.0.pre3'
+  s.add_dependency "concurrent-ruby", '~> 0.9.0'
+  s.add_dependency "concurrent-ruby-edge", '~> 0.1.0'
 
   s.add_development_dependency "rack-test"
   s.add_development_dependency "minitest"

data/examples/future_execution.rb

@@ -0,0 +1,73 @@
+#!/usr/bin/env ruby
+
+require_relative 'example_helper'
+
+class CustomPassedObject
+  attr_reader :id, :name
+
+  def initialize(id, name)
+    @id = id
+    @name = name
+  end
+end
+
+class CustomPassedObjectSerializer < ::Dynflow::Serializers::Abstract
+  def serialize(*args)
+    object = args.first
+    # Serialized output can be anything that is representable as JSON: Array, Hash...
+    { :id => object.id, :name => object.name }
+  end
+
+  def deserialize(serialized_args)
+    # Deserialized output must be an Array
+    [CustomPassedObject.new(serialized_args[:id], serialized_args[:name])]
+  end
+end
+
+class DelayedAction < Dynflow::Action
+
+  def schedule(schedule_options, *args)
+    CustomPassedObjectSerializer.new
+  end
+
+  def plan(passed_object)
+    plan_self :object_id => passed_object.id, :object_name => passed_object.name
+  end
+
+  def run
+  end
+
+end
+
+if $0 == __FILE__
+  ExampleHelper.world.action_logger.level = 1
+  ExampleHelper.world.logger.level = 0
+
+  past = Time.now - 200
+  near_future = Time.now + 29
+  future = Time.now + 180
+
+  object = CustomPassedObject.new(1, 'CPS')
+
+  past_plan = ExampleHelper.world.schedule(DelayedAction, { :start_at => past, :start_before => past }, object)
+  near_future_plan = ExampleHelper.world.schedule(DelayedAction, { :start_at => near_future, :start_before => future }, object)
+  future_plan = ExampleHelper.world.schedule(DelayedAction, { :start_at => future }, object)
+
+  puts <<-MSG.gsub(/^.*\|/, '')
+    |
+    |  Future Execution Example
+    |  ========================
+    |
+    |  This example shows the future execution functionality of Dynflow, which allows to plan actions to be executed at set time.
+    |
+    |  Execution plans:
+    |    #{past_plan.id} is scheduled to execute before #{past} and should timeout on the first run of the scheduler.
+    |    #{near_future_plan.id} is scheduled to execute at #{near_future} and should run successfully.
+    |    #{future_plan.id} is scheduled to execute at #{future} and should run successfully.
+    |
+    |  Visit http://localhost:4567 to see their status.
+    |
+  MSG
+
+  ExampleHelper.run_web_console
+end

data/lib/dynflow.rb

@@ -9,7 +9,7 @@ require 'concurrent-edge'
 
 logger                          = Logger.new($stderr)
 logger.level                    = Logger::INFO
-Concurrent.configuration.logger = lambda do |level, progname, message = nil, &block|
+Concurrent.global_logger = lambda do |level, progname, message = nil, &block|
   logger.add level, message, progname, &block
 end
 
@@ -36,11 +36,14 @@ module Dynflow
   require 'dynflow/flows'
   require 'dynflow/execution_history'
   require 'dynflow/execution_plan'
+  require 'dynflow/scheduled_plan'
   require 'dynflow/action'
   require 'dynflow/executors'
   require 'dynflow/logger_adapters'
   require 'dynflow/world'
   require 'dynflow/connectors'
   require 'dynflow/dispatcher'
+  require 'dynflow/serializers'
+  require 'dynflow/schedulers'
   require 'dynflow/config'
 end

data/lib/dynflow/action.rb

@@ -282,6 +282,17 @@ module Dynflow
       recursion.(input)
     end
 
+    def execute_schedule(schedule_options, *args)
+      world.middleware.execute(:schedule, self, schedule_options, *args) do
+        @serializer = schedule(schedule_options, *args)
+      end
+    end
+
+    def serializer
+      raise "The action must be scheduled in order to access the serializer" if @serializer.nil?
+      @serializer
+    end
+
     protected
 
     def state=(state)
@@ -298,6 +309,10 @@ module Dynflow
       @step.save
     end
 
+    def schedule(schedule_options, *args)
+      Serializers::Noop.new
+    end
+
     # @override to implement the action's *Plan phase* behaviour.
     # By default it plans itself and expects input-hash.
     # Use #plan_self and #plan_action methods to plan actions.

data/lib/dynflow/config.rb

@@ -70,7 +70,15 @@ module Dynflow
     end
 
     config_attr :auto_rescue, Algebrick::Types::Boolean do
-      false
+      true
+    end
+
+    config_attr :auto_validity_check, Algebrick::Types::Boolean do |world, config|
+      !!config.executor
+    end
+
+    config_attr :validity_check_timeout, Fixnum do
+      5
     end
 
     config_attr :exit_on_terminate, Algebrick::Types::Boolean do
@@ -85,6 +93,12 @@ module Dynflow
       true
     end
 
+    config_attr :scheduler, Schedulers::Abstract, NilClass do |world|
+      options = { :poll_interval => 15,
+                  :time_source => -> { Time.now.utc } }
+      Schedulers::Polling.new(world, options)
+    end
+
     config_attr :action_classes do
       Action.all_children
     end

data/lib/dynflow/coordinator.rb

@@ -157,6 +157,13 @@ module Dynflow
 
     end
 
+    class SchedulerLock < LockByWorld
+      def initialize(world)
+        super
+        @data[:id] = "scheduler"
+      end
+    end
+
     class WorldInvalidationLock < LockByWorld
       def initialize(world, invalidated_world)
         super(world)

data/lib/dynflow/execution_plan.rb

@@ -16,11 +16,12 @@ module Dynflow
                 :started_at, :ended_at, :execution_time, :real_time, :execution_history
 
     def self.states
-      @states ||= [:pending, :planning, :planned, :running, :paused, :stopped]
+      @states ||= [:pending, :scheduled, :planning, :planned, :running, :paused, :stopped]
     end
 
     def self.state_transitions
-      @state_transitions ||= { pending:  [:planning],
+      @state_transitions ||= { pending:  [:scheduled, :planning],
+                               scheduled: [:planning, :stopped],
                                planning: [:planned, :stopped],
                                planned:  [:running],
                                running:  [:paused, :stopped],
@@ -72,7 +73,7 @@ module Dynflow
         @started_at = Time.now
       when :stopped
         @ended_at       = Time.now
-        @real_time      = @ended_at - @started_at
+        @real_time      = @ended_at - @started_at unless @started_at.nil?
         @execution_time = compute_execution_time
       else
         # ignore
@@ -151,6 +152,17 @@ module Dynflow
       @last_step_id += 1
     end
 
+    def schedule(action_class, options, schedule_options, *args)
+      prepare(action_class, options)
+      execution_history.add("schedule", @world.id)
+      update_state :scheduled
+      entry_action.execute_schedule(schedule_options, args)
+    end
+
+    def schedule_record
+      @schedule_record ||= persistence.load_scheduled_plan(id)
+    end
+
     def prepare(action_class, options = {})
       options = options.dup
       caller_action = Type! options.delete(:caller_action), Dynflow::Action, NilClass

data/lib/dynflow/execution_plan/steps/plan_step.rb

@@ -51,7 +51,7 @@ module Dynflow
       end
 
       def self.state_transitions
-        @state_transitions ||= { pending:   [:running],
+        @state_transitions ||= { pending:   [:running, :error],
                                  running:   [:success, :error],
                                  success:   [],
                                  suspended: [],
@@ -76,6 +76,10 @@ module Dynflow
             hash[:children]
       end
 
+      def load_action
+        @action = @world.persistence.load_action(self)
+      end
+
       def initialize_action(caller_action)
         attributes = { execution_plan_id: execution_plan_id,
                        id:                action_id,

data/lib/dynflow/middleware.rb

@@ -21,6 +21,10 @@ module Dynflow
       @stack.action or raise "the action is not available"
     end
 
+    def schedule(*args)
+      pass(*args)
+    end
+
     def run(*args)
       pass(*args)
     end

data/lib/dynflow/middleware/stack.rb

@@ -14,7 +14,7 @@ module Dynflow
       @middleware_class = Child! middleware_class, Middleware
       @middleware       = middleware_class.new self
       @action           = Type! action, Dynflow::Action, NilClass
-      @method           = Match! method, :plan, :run, :finalize, :plan_phase, :finalize_phase
+      @method           = Match! method, :schedule, :plan, :run, :finalize, :plan_phase, :finalize_phase
       @next_stack       = Type! next_stack, Middleware::Stack, Proc
     end
 

data/lib/dynflow/middleware/world.rb

@@ -14,7 +14,7 @@ module Dynflow
     end
 
     def execute(method, action_or_class, *args, &block)
-      Match! method, :plan, :run, :finalize, :plan_phase, :finalize_phase
+      Match! method, :schedule, :plan, :run, :finalize, :plan_phase, :finalize_phase
       if Child? action_or_class, Dynflow::Action
         action = nil
         action_class = action_or_class

data/lib/dynflow/persistence.rb

@@ -49,6 +49,25 @@ module Dynflow
       adapter.save_execution_plan(execution_plan.id, execution_plan.to_hash)
     end
 
+    def find_past_scheduled_plans(time)
+      adapter.find_past_scheduled_plans(time).map do |plan|
+        ScheduledPlan.new_from_hash(@world, plan)
+      end
+    end
+
+    def delete_scheduled_plans(filters, batch_size = 1000)
+      adapter.delete_scheduled_plans(filters, batch_size)
+    end
+
+    def save_scheduled_plan(schedule)
+      adapter.save_scheduled_plan(schedule.execution_plan_uuid, schedule.to_hash)
+    end
+
+    def load_scheduled_plan(execution_plan_id)
+      hash = adapter.load_scheduled_plan(execution_plan_id)
+      ScheduledPlan.new_from_hash(@world, hash)
+    end
+
     def load_step(execution_plan_id, step_id, world)
       step_hash = adapter.load_step(execution_plan_id, step_id)
       ExecutionPlan::Steps::Abstract.from_hash(step_hash, execution_plan_id, world)