conflow 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b85fd52b7c1a2161135fc389c45b3f1e6754462a17ef30463435f1c919685950
-  data.tar.gz: a5f05aa2c69b8426a72dc1070834ec74120a5053a74f93250098765c461105ef
+  metadata.gz: 572a2e10d542d2914de25fc748c1c153938e59b081aaef7829208395a86e462b
+  data.tar.gz: 99a32aa33c314eeabfa4feed3b386072ba2146139b552b257c59cd4b693fcca0
 SHA512:
-  metadata.gz: 3291b765c557576651568d735b3bad21e7a191919a858e9365bbf7da97d632d09f99f3c7a2f35d3133b03d3e82a9802489dd1b63542bd1343325f89c74c22845
-  data.tar.gz: a6797479cf68b310791680b2709ed54ebb9e372e26e87a1ffb2e685418a33eefa1b621bf5f68aa6ec05024c54d58f427123335ba204049a3fd7bfba4aa6d026e
+  metadata.gz: 8538390ead7bcc20bcea935aee1ec4d4ec7359eba95a89a8a5ddd14a3fbec25b69f9900d4548a4fd6370ae94eff13471ac794fe39a593d64a16ae0f2bf0abdf3
+  data.tar.gz: 53f5221e6bb9c0bd7dffdfdae3f0cb955f170e3aaa22a281a0e7f682952d9ad946de7daa93b0899935a5e77864f8178e5219e766182f74cbddfc16ea0d12881a

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    conflow (0.2.0)
+    conflow (0.3.0)
       redis (~> 4.0)
 
 GEM
@@ -11,11 +11,11 @@ GEM
     coderay (1.1.2)
     connection_pool (2.2.1)
     diff-lcs (1.3)
-    docile (1.1.5)
+    docile (1.3.0)
     json (2.1.0)
     method_source (0.9.0)
     parallel (1.12.1)
-    parser (2.5.0.3)
+    parser (2.5.0.4)
       ast (~> 2.4.0)
     powerpack (0.1.1)
     pry (0.11.3)
@@ -45,8 +45,8 @@ GEM
       ruby-progressbar (~> 1.7)
       unicode-display_width (~> 1.0, >= 1.0.1)
     ruby-progressbar (1.9.0)
-    simplecov (0.15.1)
-      docile (~> 1.1.0)
+    simplecov (0.16.0)
+      docile (~> 1.1)
       json (>= 1.8, < 3)
       simplecov-html (~> 0.10.0)
     simplecov-html (0.10.2)

data/README.md

@@ -1,8 +1,8 @@
 # Conflow
 
-[![Build Status](https://travis-ci.org/fanfilmu/conflow.svg?branch=master)](https://travis-ci.org/fanfilmu/conflow) [![Maintainability](https://api.codeclimate.com/v1/badges/80b66a285ca1803f391a/maintainability)](https://codeclimate.com/github/fanfilmu/conflow/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/80b66a285ca1803f391a/test_coverage)](https://codeclimate.com/github/fanfilmu/conflow/test_coverage)
+[![Gem Version](https://badge.fury.io/rb/conflow.svg)](https://badge.fury.io/rb/conflow) [![Build Status](https://travis-ci.org/fanfilmu/conflow.svg?branch=master)](https://travis-ci.org/fanfilmu/conflow) [![Maintainability](https://api.codeclimate.com/v1/badges/80b66a285ca1803f391a/maintainability)](https://codeclimate.com/github/fanfilmu/conflow/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/80b66a285ca1803f391a/test_coverage)](https://codeclimate.com/github/fanfilmu/conflow/test_coverage)
 
-Conflow allows defining comlicated workflows with dependencies. Inspired by [Gush](https://github.com/chaps-io/gush) (the idea) and [Redis::Objects](https://github.com/nateware/redis-objects) (the implementation) it focuses solely on dependency logic, while leaving queueing jobs and executing them entirely in hands of the programmer.
+Conflow allows defining complicated workflows with dependencies. Inspired by [Gush](https://github.com/chaps-io/gush) (the idea) and [Redis::Objects](https://github.com/nateware/redis-objects) (the implementation) it focuses solely on dependency logic, while leaving queueing jobs and executing them entirely in hands of the programmer.
 
 Please have a look at `Gush` if you already use Rails and ActiveJob - it might suit your needs better.
 
@@ -97,6 +97,24 @@ end
 
 ![Created graph](https://camo.githubusercontent.com/0b1ee59994323900906264ea50fbc9169e4d21dd/68747470733a2f2f63686172742e676f6f676c65617069732e636f6d2f63686172743f63686c3d646967726170682b472b2537422530442530412b2b72616e6b6469722533444c522533422530442530412b2b25323253544152542532322b2d2533452b25323246697273744a6f622532322530442530412b2b25323253544152542532322b2d2533452b253232496e646570656e64656e744a6f622532322530442530412b2b25323246697273744a6f622532322b2d2533452b2532325365636f6e644a6f622532322530442530412b2b253232496e646570656e64656e744a6f622532322b2d2533452b2532325365636f6e644a6f622532322530442530412b2b2532325365636f6e644a6f622532322b2d2533452b25323246696e69736855702532322530442530412b2b25323246696e69736855702532322b2d2533452b253232454e442532322530442530412537442530442530412b266368743d6776)
 
+### Hooks
+
+If you want to modify your flow dynamically depending on results of your jobs, you can use hooks.
+
+Hooks are methods defined on the `Flow` objects which will be called with result of the job (value returned by `Worker`, see [Performing jobs](https://github.com/fanfilmu/conflow#performing-jobs)).
+
+```ruby
+class MyFlow < ApplicationFlow
+  def configure
+    run CollectEmails, hook: :send_notifications
+  end
+
+  def send_notifications(emails)
+    emails.each { |email| run SendNotification, params: { email: email } }
+  end
+end
+```
+
 ### Performing jobs
 
 To perform job, use `Conflow::Worker` mixin. It adds `#perform` method, which accepts two arguments: IDs of the flow and the job.
@@ -127,6 +145,24 @@ SecondJob.new({}).call
 FinishUp.new({}).call
 ```
 
+## Theory
+
+The main idea of the gem is, obviously, a directed graph containing information about dependencies. It is stored in Redis in following fields:
+
+* `conflow:job:<id>:successors` - ([List](https://redis.io/topics/data-types#lists)) containing IDs of jobs which depend on `<id>`
+* `conflow:flow:<id>:indegee` - ([Sorted Set](https://redis.io/topics/data-types#sorted-sets)) set of all unqueued jobs with score representing how many dependencies are not yet fulfilled
+
+There are three main actions that can be performed on this graph (Redis-wise):
+
+1. Queue jobs
+   Removes all jobs with score 0 from `:indegree` set
+2. Complete job
+   Decrement scores of all of the job's successors by one
+3. Add job
+   Add job ID to `:successors` list for all jobs on which it depends and add job itself to `:indegree` set
+
+All of these actions are performed via `eval`/`evalsha` - it lifts problems with synchronization (as scripts are executed as if in transaction) and significantly reduces amount of requests made to Redis.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/lib/conflow.rb

@@ -6,6 +6,7 @@ require "digest"
 require "json"
 require "redis"
 
+require "conflow/redis"
 require "conflow/redis/connection_wrapper"
 require "conflow/redis/field"
 require "conflow/redis/value_field"
@@ -34,6 +35,12 @@ require "conflow/worker"
 # while leaving queueing jobs and executing them entirely in hands of the programmer.
 module Conflow
   class << self
+    # Assigns Redis connection to be used by {Conflow}. It will be wrapped in ConnectionWrapper
+    # for Redis instances, in order to have single API containing #with method.
+    # You can also assign ConnectionPool instance.
+    # @param conn [Redis, ConnectionPool] Redis connection
+    # @example
+    #   Conflow.redis = Redis.new
     def redis=(conn)
       @redis =
         if defined?(ConnectionPool) && conn.is_a?(ConnectionPool)
@@ -43,6 +50,7 @@ module Conflow
         end
     end
 
+    # @return [Conflow::Redis::ConnectionWrapper, ConnectionPool] Wrapped Redis connection
     def redis
       self.redis = ::Redis.current unless defined?(@redis)
       @redis

data/lib/conflow/flow/job_handler.rb

@@ -15,13 +15,16 @@ module Conflow
       def run(job_class, params: {}, after: [], hook: nil)
         build_job(job_class, params, hook).tap do |job|
           job_classes[job_class] = job
-          after = prepare_dependencies(after)
+          after = prepare_dependencies(after).compact
 
           call_script(Conflow::Redis::AddJobScript, job, after: after)
           queue_available_jobs
         end
       end
 
+      # Finishes job, changes its status, runs hook if it's present and queues new available jobs
+      # @param job [Conflow::Job] job to be marked as finished
+      # @param result [Object] result of the job to be passed to hook
       def finish(job, result = nil)
         send(job.hook.to_s, result) unless job.hook.nil?
         call_script(Conflow::Redis::CompleteJobScript, job)

data/lib/conflow/redis.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module Conflow
+  # Redis module contains all logic related to handling flows in Redis
+  # @api private
+  module Redis
+  end
+end

data/lib/conflow/redis/add_job_script.rb

@@ -3,6 +3,7 @@
 module Conflow
   module Redis
     # Adds new job to flow
+    # @api private
     class AddJobScript < Script
       # script accepts keys: flow.job_ids, flow.indegree, and keys successors of the jobs on which new job depends.
       # It also accepts one argument: id of the new job

data/lib/conflow/redis/array_field.rb

@@ -3,55 +3,28 @@
 module Conflow
   module Redis
     # Represents Redis list. It's methods mirror most used Array methods.
+    # @api private
     class ArrayField < Field
       include Enumerable
 
-      def [](index)
-        command :lindex, [key, index]
-      end
-
-      def []=(index, value)
-        command :lset, [key, index, value]
-      end
-
-      def insert(value, after: nil, before: nil)
-        if after
-          command :linsert, [key, :after, after, value]
-        elsif before
-          command :linsert, [key, :before, before, value]
-        else
-          raise ArgumentError, "You need to pass one of [:after, :before] keywords"
-        end
-      end
-
+      # @return [Integer] number of elements in list
       def size
         command :llen, [key]
       end
 
+      # @return [Array] Ruby Array representation of Redis list
       def to_a
         command :lrange, [key, 0, -1]
       end; alias to_ary to_a
 
-      def pop
-        command :rpop, [key]
-      end
-
+      # @param values [String...] one or more values to be added to the list
+      # @return [String] Redis response
       def push(*values)
         command :rpush, [key, values]
       end; alias << push
 
-      def concat(ary)
-        push(*ary)
-      end
-
-      def shift
-        command :lpop, [key]
-      end
-
-      def unshift(value)
-        command :lpush, [key, value]
-      end
-
+      # Replace contents of Redis list
+      # @param new_array [Array] array of new values
       def overwrite(new_array)
         redis.with do |conn|
           conn.pipelined do
@@ -61,10 +34,14 @@ module Conflow
         end
       end
 
+      # Iterates over list
+      # @see to_a
       def each(&block)
         to_a.each(&block)
       end
 
+      # @param other [Object] Object to compare value with. Handles Arrays and other {ArrayField} objects
+      # @return [Boolean] true if equal
       def ==(other)
         case other
         when Array      then to_a == other
@@ -73,6 +50,8 @@ module Conflow
         end
       end
 
+      # String representation of the list
+      # @see to_a
       def to_s
         to_a.to_s
       end

data/lib/conflow/redis/complete_job_script.rb

@@ -3,6 +3,7 @@
 module Conflow
   module Redis
     # Adds new job to flow
+    # @api private
     class CompleteJobScript < Script
       self.script = <<~LUA
         local indegree_set = KEYS[1]

data/lib/conflow/redis/connection_wrapper.rb

@@ -3,6 +3,7 @@
 module Conflow
   module Redis
     # Wraps Redis connection to behave like connection pool
+    # @api private
     class ConnectionWrapper
       # @param redis [Redis] Redis connection to be wrapped
       def initialize(redis)

data/lib/conflow/redis/field.rb

@@ -3,10 +3,13 @@
 module Conflow
   module Redis
     # Base class for fields. All fields are assigned a Redis key.
+    # @api private
     class Field
+      # Redis key
       attr_reader :key
       alias id key
 
+      # @param key [String] Redis key to store the field in
       def initialize(key)
         @key = key
       end

data/lib/conflow/redis/field_builder.rb

@@ -3,9 +3,11 @@
 module Conflow
   module Redis
     # Helper class for defining getter and setter methods for Fields
+    # @api private
     class FieldBuilder
       # This dynamic module contains accessor methods for a field
       class FieldAccessor < Module
+        # Creates new dynamic module with accessor methods for a field
         def initialize(field_name, klass, methods)
           super() do
             define_getter(field_name, klass) if methods.include?(:getter)
@@ -13,6 +15,7 @@ module Conflow
           end
         end
 
+        # Defines getter method, which will return proper {Conflow::Redis::Field}
         def define_getter(field_name, klass)
           instance_var = "@#{field_name}"
 
@@ -22,6 +25,7 @@ module Conflow
           end
         end
 
+        # Setter uses Field#overwrite method to replace contents of the field
         def define_setter(field_name)
           define_method("#{field_name}=") do |value|
             send(field_name).tap { |field| field.overwrite(value) }
@@ -29,13 +33,19 @@ module Conflow
         end
       end
 
-      attr_reader :field_name, :klass
+      # Name of the accessor methods
+      attr_reader :field_name
+      # Class of the field. Should inherit from {Conflow::Redis::Field}
+      attr_reader :klass
 
+      # @see field_name
+      # @see klass
       def initialize(field_name, klass)
         @field_name = field_name
         @klass = klass
       end
 
+      # Creates dynamic {FieldAccessor} module and includes it in base class
       def call(base, methods: %i[getter setter])
         base.include(FieldAccessor.new(field_name, klass, methods))
       end

data/lib/conflow/redis/findable.rb

@@ -3,12 +3,15 @@
 module Conflow
   module Redis
     # Findable module allows to use .find method on models with identifiers. It requires additional field: :type
+    # @api private
     module Findable
+      # Adds :type field to store class name of the Findable object
       def self.included(base)
         base.extend ClassMethods
         base.field :type, :value
       end
 
+      # Adds logic for assigning class name as type
       def initialize(*args)
         super
         self.type = self.class.name
@@ -16,6 +19,10 @@ module Conflow
 
       # Adds .find method which accepts ID and returns model of proper (sub)type
       module ClassMethods
+        # Creates new {Findable} object from ID.
+        # @example
+        #   MyFlow.new.id #=> 13
+        #   Conflow::Flow.find(13) #=> Myflow object
         def find(id)
           class_name = ValueField.new(format(key_template + ":type", id: id)).value
           raise ::Redis::CommandError, "#{name} with ID #{id} doesn't exist" unless class_name

data/lib/conflow/redis/hash_field.rb

@@ -3,26 +3,26 @@
 module Conflow
   module Redis
     # Represents Redis hash. It's methods mirror most used Hash methods.
+    # @api private
     class HashField < Field
       include Enumerable
 
-      def [](field)
-        value = command(:hget, [key, field])
-        value ? JSON.parse(value) : value
-      end
-
-      def []=(field, value)
-        command :hset, [key, field, JSON.dump(value)]
-      end
-
+      # Merges hashes, similar to Hash#merge
+      # @param hash [Hash] hash of keys and values to be merged
+      # @return [String] Redis response
+      #
+      # @example
+      #   field.merge(smart: true, degree: :none)
       def merge(hash)
         command :hmset, [key, hash.flatten]
       end
 
-      def delete(*fields)
-        command :hdel, [key, fields]
-      end
-
+      # Replaces currently stored hash with one given as param
+      # @param new_hash [Hash] hash of keys and values to be stored
+      # @return [String] Redis response
+      #
+      # @example
+      #   field.overwrite(smart: true, degree: :none)
       def overwrite(new_hash)
         redis.with do |conn|
           conn.pipelined do
@@ -32,24 +32,26 @@ module Conflow
         end
       end
 
-      def keys
-        command(:hkeys, [key]).map(&:to_sym)
-      end
-
-      def size
-        command :hlen, [key]
-      end
-
+      # Creates Ruby Hash based on soted values. Keys will be symbolized and values JSON-parsed
+      # @return [String] Redis response
+      #
+      # @example
+      #   field.to_h #=> { smart: true, degree: "none" }
       def to_h
         command(:hgetall, [key]).each_with_object({}) do |(key, value), hash|
           hash[key.to_sym] = JSON.parse(value)
         end
       end; alias to_hash to_h
 
+      # Iterates over hash.
+      # @see to_h
+      # @return [Enumerator] if no block given
       def each(&block)
         to_h.each(&block)
       end
 
+      # @param other [Object] Object to compare value with. Handles Hashes and other {HashField} objects
+      # @return [Boolean] true if equal
       def ==(other)
         case other
         when Hash      then to_h == other
@@ -58,6 +60,7 @@ module Conflow
         end
       end
 
+      # @return [String] string representation of the hash
       def to_s
         to_h.to_s
       end

data/lib/conflow/redis/identifier.rb

@@ -5,6 +5,7 @@ module Conflow
     # Identifier changes logic of fields so that they can be found by an id.
     # ID is a counter stored in redis under .counter_key
     # Key is build with template stored in .key_template
+    # @api private
     module Identifier
       # Extends base class with {ClassMethods}
       def self.included(base)
@@ -13,7 +14,10 @@ module Conflow
 
       # class methods for classes with identifier
       module ClassMethods
-        attr_writer :counter_key, :key_template
+        # @see counter_key
+        attr_writer :counter_key
+        # @see key_template
+        attr_writer :key_template
 
         # Copies *counter_key* and *key_template* to child classes
         def inherited(base)

data/lib/conflow/redis/model.rb

@@ -3,6 +3,7 @@
 module Conflow
   module Redis
     # Models adds .field method which allows to define fields easily.
+    # @api private
     module Model
       # Extends base class with .field and .has_many methods
       def self.included(base)

data/lib/conflow/redis/queue_jobs_script.rb

@@ -3,6 +3,7 @@
 module Conflow
   module Redis
     # Removes jobs from indegree set, adds them to queued list, and returns IDs of queued jobs.
+    # @api private
     class QueueJobsScript < Script
       self.script = <<~LUA
         local indegree_set = KEYS[1]

data/lib/conflow/redis/script.rb

@@ -3,6 +3,7 @@
 module Conflow
   module Redis
     # Main class for scripts, handling logic of executing and caching scripts.
+    # @api private
     class Script
       class << self
         # @return [Boolean] whether scripts are cached or not

data/lib/conflow/redis/set_field.rb

@@ -3,6 +3,7 @@
 module Conflow
   module Redis
     # Represents Redis list. It's methods mirror most used Set methods.
+    # @api private
     class SetField < Field
       include Enumerable
 

data/lib/conflow/redis/sorted_set_field.rb

@@ -4,6 +4,7 @@ module Conflow
   module Redis
     # Represents Redis sorted set. Closest Ruby representation is a Hash
     # where keys are elements of the set and values represent score.
+    # @api private
     class SortedSetField < Field
       # Adds one or more keys to the set.
       # @param hash [Hash] hash of values and scores to be added
@@ -25,17 +26,6 @@ module Conflow
         command :zscore, [key, value]
       end
 
-      # Set score of given element.
-      # @param value [String, Symbol] element of the set
-      # @param rank [Numeric] score to be assigned
-      # @return [Integer] Number of added elements (1 if key didn't exist, 0 otherwise)
-      #
-      # @example
-      #   field[:last] = 24 #=> 0
-      def []=(value, rank)
-        command :zadd, [key, rank, value]
-      end
-
       # Number of elements in the set
       # @return [Integer] Size of the set
       #
@@ -55,22 +45,6 @@ module Conflow
         command :zrem, [key, value]
       end
 
-      # Returns first *n* elements of the sorted set
-      # @param num [Integer] amount of elements to be returned. Defaults to 1.
-      # @return [String, Array<String>] first *num* elements from the set
-      def first(num = 1)
-        result = command :zrange, [key, 0, num - 1]
-        num == 1 ? result[0] : result
-      end
-
-      # Returns last *n* elements of the sorted set
-      # @param num [Integer] amount of elements to be returned. Defaults to 1.
-      # @return [String, Array<String>] last *num* elements from the set
-      def last(num = 1)
-        result = command :zrevrange, [key, 0, num - 1]
-        num == 1 ? result[0] : result
-      end
-
       # Creates regular Ruby Hash based on Redis values.
       # @return [Hash] Hash representing this Sorted set
       def to_h

data/lib/conflow/redis/value_field.rb

@@ -3,6 +3,7 @@
 module Conflow
   module Redis
     # Represents single value (Redis String). Values are serialized as JSON in order to preserve type.
+    # @api private
     class ValueField < Field
       # @note *value* must be serializable through JSON.dump
       # @param value [Object] new value to be saved
@@ -20,7 +21,7 @@ module Conflow
 
       # @param other [Object] Object to compare value with. Handles Strings, Numerics,
       #   Symbols and other {ValueField} objects
-      # @return [String] Redis response
+      # @return [Boolean] true if equal
       def ==(other)
         case other
         when String, Numeric then value == other

data/lib/conflow/version.rb

@@ -2,5 +2,5 @@
 
 module Conflow
   # Current version of the gem
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: conflow
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Michał Begejowicz
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-03-11 00:00:00.000000000 Z
+date: 2018-03-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
@@ -148,6 +148,7 @@ files:
 - lib/conflow/flow.rb
 - lib/conflow/flow/job_handler.rb
 - lib/conflow/job.rb
+- lib/conflow/redis.rb
 - lib/conflow/redis/add_job_script.rb
 - lib/conflow/redis/array_field.rb
 - lib/conflow/redis/complete_job_script.rb