gush 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a7d04a1298b3a3171ff5a24b5d3057e040f07ec7
-  data.tar.gz: f63ebc5de7186f2d042e316e519807ebf1819724
+  metadata.gz: 5f2be71d1a76f021091ae90270ea5f01a2413ae0
+  data.tar.gz: c4160594dd6ad163239548623a7e67b29aa1727c
 SHA512:
-  metadata.gz: 7492e091cce7f5c40acda1a29981db317e701a239a1e06bb9567d341fd0904ed31982a0484b1746968cdec999ce1f41aea5e2a4078756669ebc316d8352d060e
-  data.tar.gz: 8264f120a6a876f2f265ca7270c2d2ce0177265645ed595da778433a4ff53415107cc51cb19b9f3af638e7ca4b67ec31fb6ac0969b37c214a95718a3cdd831c4
+  metadata.gz: 271ac326d01d2dafd4c0092e3011a5d57206b988bfc7e5e62e375b913bcb2e8750440058e07a58d60663446fc7b7254f2b91a39ed6d038976995dccc3af0952b
+  data.tar.gz: a28e4ef3338b305523e7d7d275aa6d3e357ddfe5161aaef95e323b12268a301c8d59352f12c68c445392df613c29ae11f38079a6838006c3ca2920c32ce418ad

data/README.md

@@ -1,7 +1,12 @@
-# Gush [![Build Status](https://travis-ci.org/pokonski/gush.svg?branch=master)](https://travis-ci.org/pokonski/gush)
+# Gush [![Build Status](https://travis-ci.org/chaps-io/gush.svg?branch=master)](https://travis-ci.org/chaps-io/gush)
+
+## [![](http://i.imgur.com/ya8Wnyl.png)](https://chaps.io) proudly made by [Chaps](https://chaps.io)
 
 Gush is a parallel workflow runner using only Redis as its message broker and Sidekiq for workers.
 
+## Theory
+
+Gush relies on directed acyclic graphs to store dependencies, see [Parallelizing Operations With Dependencies](https://msdn.microsoft.com/en-us/magazine/dd569760.aspx) by Stephen Toub.
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -26,9 +31,9 @@ Here is a complete example of a workflow you can create:
 ```ruby
 # workflows/sample_workflow.rb
 class SampleWorkflow < Gush::Workflow
-  def configure
-    run FetchJob1
-    run FetchJob2
+  def configure(url_to_fetch_from)
+    run FetchJob1, params: { url: url_to_fetch_from }
+    run FetchJob2, params: {some_flag: true, url: 'http://url.com'}
 
     run PersistJob1, after: FetchJob1
     run PersistJob2, after: FetchJob2
@@ -52,19 +57,62 @@ For the Workflow above, the graph will look like this:
 
 ![SampleWorkflow](http://i.imgur.com/SmeRRVT.png)
 
-### Defining jobs
+
+#### Passing parameters to jobs
+
+You can pass any primitive arguments into jobs while defining your workflow:
+
+```ruby
+# workflows/sample_workflow.rb
+class SampleWorkflow < Gush::Workflow
+  def configure
+    run FetchJob1, params: { url: "http://some.com/url" }
+  end
+end
+```
+
+See below to learn how to access those params inside your job.
+
+#### Defining jobs
 
 Jobs are classes inheriting from `Gush::Job`:
 
 ```ruby
-#workflows/sample/fetch_job1.rb
 class FetchJob1 < Gush::Job
   def work
     # do some fetching from remote APIs
+
+    params #=> {url: "http://some.com/url"}
   end
 end
 ```
 
+`params` method is a hash containing your (optional) parameters passed to `run` method in the workflow.
+
+#### Passing arguments to workflows
+
+Workflows can accept any primitive arguments in their constructor, which then will be availabe in your
+`configure` method.
+
+Here's an example of a workflow responsible for publishing a book:
+
+```ruby
+# workflows/sample_workflow.rb
+class PublishBookWorkflow < Gush::Workflow
+  def configure(url, isbn)
+    run FetchBook, params: { url: url }
+    run PublishBook, params: { book_isbn: isbn }
+  end
+end
+```
+
+and then create your workflow with those arguments:
+
+```ruby
+PublishBookWorkflow.new("http://url.com/book.pdf", "978-0470081204")
+```
+
+
 ### Running workflows
 
 Now that we have defined our workflow we can use it:
@@ -72,14 +120,14 @@ Now that we have defined our workflow we can use it:
 #### 1. Initialize and save it
 
 ```ruby
-flow = SampleWorkflow.new
+flow = SampleWorkflow.new(optional, arguments)
 flow.save # saves workflow and its jobs to Redis
 ```
 
 **or:** you can also use a shortcut:
 
 ```ruby
-flow = SampleWorkflow.create
+flow = SampleWorkflow.create(optional, arguments)
 ```
 
 #### 2. Start workflow
@@ -99,6 +147,50 @@ flow.start!
 Now Gush will start processing jobs in background using Sidekiq
 in the order defined in `configure` method inside Workflow.
 
+### Pipelining
+
+Gush offers a useful feature which lets you pass results of a job to its dependencies, so they can act accordingly.
+
+**Example:**
+
+Let's assume you have two jobs, `DownloadVideo`, `EncodeVideo`.
+The latter needs to know where the first one downloaded the file to be able to open it.
+
+
+```ruby
+class DownloadVideo < Gush::Job
+  def work
+    downloader = VideoDownloader.fetch("http://youtube.com/?v=someytvideo")
+
+    output(downloader.file_path)
+  end
+end
+```
+
+`output` method is Gush's way of saying: "I want to pass this down to my descendants".
+
+Now, since `DownloadVideo` finished and its dependant job `EncodeVideo` started, we can access that payload down the (pipe)line:
+
+```ruby
+class EncodeVideo < Gush::Job
+  def work
+    video_path = payloads["DownloadVideo"]
+  end
+end
+```
+
+`payloads` is a hash containing outputs from all parent jobs, where job class names are the keys.
+
+**Note:** `payloads` will only contain outputs of the job's ancestors. So if job `A` depends on `B` and `C`,
+the `paylods` hash will look like this:
+
+```ruby
+{
+  "B" => (...),
+  "C" => (...)
+}
+```
+
 
 ### Checking status:
 
@@ -126,9 +218,8 @@ flow.status
   bundle gush list
   ```
 
-### Requiring workflows inside your projects
 
-**Skip this step if using Gush inside Rails application, workflows will already be loaded**
+### Requiring workflows inside your projects
 
 When using Gush and its CLI commands you need a Gushfile.rb in root directory.
 Gushfile should require all your Workflows and jobs, for example:
@@ -141,6 +232,11 @@ Dir[Rails.root.join("app/workflows/**/*.rb")].each do |file|
 end
 ```
 
+## Contributors
+
+- [Mateusz Lenik](https://github.com/mlen)
+- [Michał Krzyżanowski](https://github.com/krzyzak)
+
 ## Contributing
 
 1. Fork it ( http://github.com/pokonski/gush/fork )

data/gush.gemspec

@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
   spec.name          = "gush"
-  spec.version       = "0.1.1"
+  spec.version       = "0.1.2"
   spec.authors       = ["Piotrek Okoński"]
   spec.email         = ["piotrek@okonski.org"]
   spec.summary       = "Fast and distributed workflow runner using only Sidekiq and Redis"

data/lib/gush/cli.rb

@@ -131,7 +131,7 @@ module Gush
 
     def load_gushfile
       file = client.configuration.gushfile
-      if !@gushfile.exist?
+      if !gushfile.exist?
         raise Thor::Error, "#{file} not found, please add it to your project".colorize(:red)
       end
 

data/lib/gush/client.rb

@@ -117,7 +117,7 @@ module Gush
       sidekiq.push(
         'class' => Gush::Worker,
         'queue' => configuration.namespace,
-        'args'  => [workflow_id, job.class.to_s, configuration.to_json]
+        'args'  => [workflow_id, job.class.to_s]
       )
     end
 
@@ -126,7 +126,7 @@ module Gush
     attr_reader :sidekiq, :redis
 
     def workflow_from_hash(hash, nodes = nil)
-      flow = hash[:klass].constantize.new(false)
+      flow = hash[:klass].constantize.new
       flow.stopped = hash.fetch(:stopped, false)
       flow.id = hash[:id]
 

data/lib/gush/job.rb

@@ -1,10 +1,8 @@
 module Gush
   class Job
-
-    attr_accessor :workflow_id, :incoming, :outgoing,
-      :finished_at, :failed_at, :started_at, :enqueued_at
-
-    attr_reader :name
+    attr_accessor :workflow_id, :incoming, :outgoing, :params,
+      :finished_at, :failed_at, :started_at, :enqueued_at, :payloads
+    attr_reader :name, :output_payload, :params
 
     def initialize(workflow, opts = {})
       @workflow = workflow
@@ -14,18 +12,16 @@ module Gush
 
     def as_json
       {
-        name: @name,
+        name: name,
         klass: self.class.to_s,
-        finished: finished?,
-        enqueued: enqueued?,
-        failed: failed?,
-        incoming: @incoming,
-        outgoing: @outgoing,
+        incoming: incoming,
+        outgoing: outgoing,
         finished_at: finished_at,
         enqueued_at: enqueued_at,
         started_at: started_at,
         failed_at: failed_at,
-        running: running?
+        params: params,
+        output_payload: output_payload
       }
     end
 
@@ -37,6 +33,10 @@ module Gush
       hash[:klass].constantize.new(flow, hash)
     end
 
+    def output(data)
+      @output_payload = data
+    end
+
     def work
     end
 
@@ -56,8 +56,7 @@ module Gush
     end
 
     def fail!
-      @finished_at = current_timestamp
-      @failed_at = current_timestamp
+      @finished_at = @failed_at = current_timestamp
     end
 
     def enqueued?
@@ -89,19 +88,20 @@ module Gush
     end
 
     private
-
     def current_timestamp
       Time.now.to_i
     end
 
-    def assign_variables(options)
-      @name        = options[:name]
-      @incoming    = options[:incoming] || []
-      @outgoing    = options[:outgoing] || []
-      @failed_at   = options[:failed_at]
-      @finished_at = options[:finished_at]
-      @started_at  = options[:started_at]
-      @enqueued_at = options[:enqueued_at]
+    def assign_variables(opts)
+      @name           = opts[:name]
+      @incoming       = opts[:incoming] || []
+      @outgoing       = opts[:outgoing] || []
+      @failed_at      = opts[:failed_at]
+      @finished_at    = opts[:finished_at]
+      @started_at     = opts[:started_at]
+      @enqueued_at    = opts[:enqueued_at]
+      @params         = opts[:params] || {}
+      @output_payload = opts[:output_payload]
     end
   end
 end

data/lib/gush/worker.rb

@@ -6,19 +6,18 @@ module Gush
     include ::Sidekiq::Worker
     sidekiq_options retry: false
 
-    def perform(workflow_id, job_id, configuration_json)
-      configure_client(configuration_json)
+    def perform(workflow_id, job_id)
+      setup_job(workflow_id, job_id)
 
-      workflow = client.find_workflow(workflow_id)
-      job = workflow.find_job(job_id)
+      job.payloads = incoming_payloads
 
       start = Time.now
-      report(workflow, job, :started, start)
+      report(:started, start)
 
       failed = false
       error = nil
 
-      mark_as_started(workflow, job)
+      mark_as_started
       begin
         job.work
       rescue Exception => e
@@ -27,46 +26,68 @@ module Gush
       end
 
       unless failed
-        report(workflow, job, :finished, start)
-        mark_as_finished(workflow, job)
+        report(:finished, start)
+        mark_as_finished
 
-        enqueue_outgoing_jobs(workflow.id, job)
+        enqueue_outgoing_jobs
       else
-        mark_as_failed(workflow, job)
-        report(workflow, job, :failed, start, error.message)
+        mark_as_failed
+        report(:failed, start, error.message)
       end
     end
 
     private
+    attr_reader :client, :workflow, :job
 
-    attr_reader :client
+    def client
+      @client ||= Gush::Client.new(Gush.configuration)
+    end
+
+    def setup_job(workflow_id, job_id)
+      @workflow ||= client.find_workflow(workflow_id)
+      @job ||= workflow.find_job(job_id)
+    end
+
+    def incoming_payloads
+      payloads = {}
+      job.incoming.each do |job_name|
+        payloads[job_name] = client.load_job(workflow.id, job_name).output_payload
+      end
 
-    def configure_client(config_json)
-      @client = Client.new(Configuration.from_json(config_json))
+      payloads
     end
 
-    def mark_as_finished(workflow, job)
+    def mark_as_finished
       job.finish!
       client.persist_job(workflow.id, job)
     end
 
-    def mark_as_failed(workflow, job)
+    def mark_as_failed
       job.fail!
       client.persist_job(workflow.id, job)
     end
 
-    def mark_as_started(workflow, job)
+    def mark_as_started
       job.start!
       client.persist_job(workflow.id, job)
     end
 
-    def report_workflow_status(workflow, job)
-      message = {workflow_id: workflow.id, status: workflow.status, started_at: workflow.started_at, finished_at: workflow.finished_at }
-      client.workflow_report(message)
+    def report_workflow_status
+      client.workflow_report({
+        workflow_id:  workflow.id,
+        status:       workflow.status,
+        started_at:   workflow.started_at,
+        finished_at:  workflow.finished_at
+      })
     end
 
-    def report(workflow, job, status, start, error = nil)
-      message = {status: status, workflow_id: workflow.id, job: job.name, duration: elapsed(start)}
+    def report(status, start, error = nil)
+      message = {
+        status: status,
+        workflow_id: workflow.id,
+        job: job.name,
+        duration: elapsed(start)
+      }
       message[:error] = error if error
       client.worker_report(message)
     end
@@ -75,11 +96,11 @@ module Gush
       (Time.now - start).to_f.round(3)
     end
 
-    def enqueue_outgoing_jobs(workflow_id, job)
+    def enqueue_outgoing_jobs
       job.outgoing.each do |job_name|
-        out = client.load_job(workflow_id, job_name)
+        out = client.load_job(workflow.id, job_name)
         if out.ready_to_start?
-          client.enqueue_job(workflow_id, out)
+          client.enqueue_job(workflow.id, out)
         end
       end
     end

data/lib/gush/workflow.rb

@@ -2,19 +2,15 @@ require 'securerandom'
 
 module Gush
   class Workflow
-    attr_accessor :id, :jobs, :stopped, :persisted
+    attr_accessor :id, :jobs, :stopped, :persisted, :arguments
 
-    def initialize(should_run_configure = true)
+    def initialize(*args)
       @id = id
       @jobs = []
       @dependencies = []
       @persisted = false
       @stopped = false
-
-      if should_run_configure
-        configure
-        create_dependencies
-      end
+      @arguments = args
     end
 
     def self.find(id)
@@ -28,14 +24,12 @@ module Gush
     end
 
     def save
-      if @id.nil?
-        assign_id
-      end
-
+      configure(*@arguments)
+      resolve_dependencies
       client.persist_workflow(self)
     end
 
-    def configure
+    def configure(*args)
     end
 
     def mark_as_stopped
@@ -58,7 +52,7 @@ module Gush
       @stopped = false
     end
 
-    def create_dependencies
+    def resolve_dependencies
       @dependencies.each do |dependency|
         from = find_job(dependency[:from])
         to   = find_job(dependency[:to])
@@ -69,7 +63,7 @@ module Gush
     end
 
     def find_job(name)
-      @jobs.find { |node| node.name == name.to_s || node.class.to_s == name.to_s }
+      jobs.find { |node| node.name == name.to_s || node.class.to_s == name.to_s }
     end
 
     def finished?
@@ -88,23 +82,29 @@ module Gush
       stopped
     end
 
-    def run(klass, deps = {})
-      node = klass.new(self, name: klass.to_s)
-      @jobs << node
+    def run(klass, opts = {})
+      options =
+
+      node = klass.new(self, {
+        name: klass.to_s,
+        params: opts.fetch(:params, {})
+      })
 
-      deps_after = [*deps[:after]]
+      jobs << node
+
+      deps_after = [*opts[:after]]
       deps_after.each do |dep|
         @dependencies << {from: dep.to_s, to: klass.to_s }
       end
 
-      deps_before = [*deps[:before]]
+      deps_before = [*opts[:before]]
       deps_before.each do |dep|
         @dependencies << {from: klass.to_s, to: dep.to_s }
       end
     end
 
     def reload
-      self.class.find(@id)
+      self.class.find(id)
     end
 
     def initial_jobs
@@ -138,11 +138,12 @@ module Gush
       name = self.class.to_s
       {
         name: name,
-        id: @id,
-        total: @jobs.count,
-        finished: @jobs.count(&:finished?),
+        id: id,
+        arguments: @arguments,
+        total: jobs.count,
+        finished: jobs.count(&:finished?),
         klass: name,
-        jobs: @jobs.map(&:as_json),
+        jobs: jobs.map(&:as_json),
         status: status,
         stopped: stopped,
         started_at: started_at,
@@ -158,12 +159,12 @@ module Gush
       ObjectSpace.each_object(Class).select { |klass| klass < self }
     end
 
-    private
-
-    def assign_id
-      @id = client.next_free_id
+    def id
+      @id ||= client.next_free_id
     end
 
+    private
+
     def client
       @client ||= Client.new
     end

data/spec/features/integration_spec.rb

@@ -0,0 +1,72 @@
+require 'spec_helper'
+
+
+describe "Workflows" do
+  it "runs the whole workflow in proper order" do
+    flow = TestWorkflow.create
+    flow.start!
+
+    expect(Gush::Worker).to have_jobs(flow.id, ["Prepare"])
+
+    Gush::Worker.perform_one
+    expect(Gush::Worker).to have_jobs(flow.id, ["FetchFirstJob", "FetchSecondJob"])
+
+    Gush::Worker.perform_one
+    expect(Gush::Worker).to have_jobs(flow.id, ["FetchSecondJob", "PersistFirstJob"])
+
+    Gush::Worker.perform_one
+    expect(Gush::Worker).to have_jobs(flow.id, ["PersistFirstJob", "NormalizeJob"])
+
+    Gush::Worker.perform_one
+    expect(Gush::Worker).to have_jobs(flow.id, ["NormalizeJob"])
+
+    Gush::Worker.perform_one
+
+    expect(Gush::Worker.jobs).to be_empty
+
+    flow = flow.reload
+    expect(flow).to be_finished
+    expect(flow).to_not be_failed
+  end
+
+  it "passes payloads down the workflow" do
+    class UpcaseJob < Gush::Job
+      def work
+        output params[:input].upcase
+      end
+    end
+
+    class PrefixJob < Gush::Job
+      def work
+        output params[:prefix].capitalize
+      end
+    end
+
+    class PrependJob < Gush::Job
+      def work
+        string = "#{payloads["PrefixJob"]}: #{payloads["UpcaseJob"]}"
+        output string
+      end
+    end
+
+    class PayloadWorkflow < Gush::Workflow
+      def configure
+        run UpcaseJob, params: {input: "some text"}
+        run PrefixJob, params: {prefix: "a prefix"}
+        run PrependJob, after: [UpcaseJob, PrefixJob]
+      end
+    end
+
+    flow = PayloadWorkflow.create
+    flow.start!
+
+    Gush::Worker.perform_one
+    expect(flow.reload.find_job("UpcaseJob").output_payload).to eq("SOME TEXT")
+
+    Gush::Worker.perform_one
+    expect(flow.reload.find_job("PrefixJob").output_payload).to eq("A prefix")
+
+    Gush::Worker.perform_one
+    expect(flow.reload.find_job("PrependJob").output_payload).to eq("A prefix: SOME TEXT")
+  end
+end

data/spec/lib/gush/client_spec.rb

@@ -16,8 +16,7 @@ describe Gush::Client do
 
     context "when given workflow exists" do
       it "returns Workflow object" do
-        expected_workflow = TestWorkflow.new(SecureRandom.uuid)
-        client.persist_workflow(expected_workflow)
+        expected_workflow = TestWorkflow.create
         workflow = client.find_workflow(expected_workflow.id)
 
         expect(workflow.id).to eq(expected_workflow.id)
@@ -28,8 +27,7 @@ describe Gush::Client do
 
   describe "#start_workflow" do
     it "enqueues next jobs from the workflow" do
-      workflow = TestWorkflow.new
-      client.persist_workflow(workflow)
+      workflow = TestWorkflow.create
       expect {
         client.start_workflow(workflow)
       }.to change{Gush::Worker.jobs.count}.from(0).to(1)
@@ -38,15 +36,14 @@ describe Gush::Client do
     it "removes stopped flag when the workflow is started" do
       workflow = TestWorkflow.new
       workflow.mark_as_stopped
-      client.persist_workflow(workflow)
+      workflow.save
       expect {
         client.start_workflow(workflow)
       }.to change{client.find_workflow(workflow.id).stopped?}.from(true).to(false)
     end
 
     it "marks the enqueued jobs as enqueued" do
-      workflow = TestWorkflow.new
-      client.persist_workflow(workflow)
+      workflow = TestWorkflow.create
       client.start_workflow(workflow)
       job = workflow.reload.find_job("Prepare")
       expect(job.enqueued?).to eq(true)
@@ -55,8 +52,7 @@ describe Gush::Client do
 
   describe "#stop_workflow" do
     it "marks the workflow as stopped" do
-      workflow = TestWorkflow.new
-      client.persist_workflow(workflow)
+      workflow = TestWorkflow.create
       expect {
         client.stop_workflow(workflow.id)
       }.to change{client.find_workflow(workflow.id).stopped?}.from(false).to(true)
@@ -76,8 +72,7 @@ describe Gush::Client do
 
   describe "#destroy_workflow" do
     it "removes all Redis keys related to the workflow" do
-      workflow = TestWorkflow.new
-      client.persist_workflow(workflow)
+      workflow = TestWorkflow.create
       expect(redis.keys("gush.workflows.#{workflow.id}").length).to eq(1)
       expect(redis.keys("gush.jobs.#{workflow.id}.*").length).to eq(5)
 
@@ -98,16 +93,14 @@ describe Gush::Client do
 
   describe "#all_workflows" do
     it "returns all registered workflows" do
-      workflow = TestWorkflow.new(SecureRandom.uuid)
-      client.persist_workflow(workflow)
+      workflow = TestWorkflow.create
       workflows = client.all_workflows
       expect(workflows.map(&:id)).to eq([workflow.id])
     end
   end
 
   it "should be able to handle outdated data format" do
-    workflow = TestWorkflow.new
-    client.persist_workflow(workflow)
+    workflow = TestWorkflow.create
 
     # malform the data
     hash = Gush::JSON.decode(redis.get("gush.workflows.#{workflow.id}"), symbolize_keys: true)

data/spec/lib/gush/job_spec.rb

@@ -2,6 +2,13 @@ require 'spec_helper'
 
 describe Gush::Job do
 
+  describe "#output" do
+    it "saves output to output_payload" do
+      job = described_class.new(name: "a-job")
+      job.output "something"
+      expect(job.output_payload).to eq("something")
+    end
+  end
   describe "#fail!" do
     it "sets finished and failed to true and records time" do
       job = described_class.new(name: "a-job")
@@ -59,16 +66,14 @@ describe Gush::Job do
         expected = {
           name: "a-job",
           klass: "Gush::Job",
-          finished: true,
-          enqueued: true,
-          failed: false,
           incoming: [],
           outgoing: [],
           failed_at: nil,
           started_at: nil,
           finished_at: 123,
           enqueued_at: 120,
-          running: false
+          params: {},
+          output_payload: nil
         }
         expect(job.as_json).to eq(expected)
       end
@@ -82,9 +87,6 @@ describe Gush::Job do
         {
           klass: 'Gush::Job',
           name: 'gob',
-          finished: true,
-          failed: true,
-          enqueued: true,
           incoming: ['a', 'b'],
           outgoing: ['c'],
           failed_at: 123,

data/spec/lib/gush/worker_spec.rb

@@ -23,7 +23,7 @@ describe Gush::Worker do
         allow(job).to receive(:work).and_raise(StandardError)
         expect(client).to receive(:worker_report).with(hash_including(status: :failed)).ordered
 
-        subject.perform(workflow.id, "Prepare", config)
+        subject.perform(workflow.id, "Prepare")
         expect(workflow.find_job("Prepare")).to be_failed
       end
 
@@ -31,7 +31,7 @@ describe Gush::Worker do
         allow(job).to receive(:work).and_raise(StandardError)
         expect(client).to receive(:worker_report).with(hash_including(status: :failed)).ordered
 
-        subject.perform(workflow.id, "Prepare", config)
+        subject.perform(workflow.id, "Prepare")
       end
     end
 
@@ -40,13 +40,13 @@ describe Gush::Worker do
         expect(subject).to receive(:mark_as_finished)
         expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
 
-        subject.perform(workflow.id, "Prepare", config)
+        subject.perform(workflow.id, "Prepare")
       end
 
       it "reports that job succedeed" do
         expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
 
-        subject.perform(workflow.id, "Prepare", config)
+        subject.perform(workflow.id, "Prepare")
       end
     end
 
@@ -54,14 +54,14 @@ describe Gush::Worker do
       expect(job).to receive(:work)
       expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
 
-      subject.perform(workflow.id, "Prepare", config)
+      subject.perform(workflow.id, "Prepare")
     end
 
     it "reports when the job is started" do
       allow(client).to receive(:worker_report)
       expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
 
-      subject.perform(workflow.id, "Prepare", config)
+      subject.perform(workflow.id, "Prepare")
     end
   end
 end

data/spec/lib/gush/workflow_spec.rb

@@ -1,25 +1,26 @@
 require 'spec_helper'
 
 describe Gush::Workflow do
-  subject { TestWorkflow.new("test-workflow") }
+  subject { TestWorkflow.create }
 
   describe "#initialize" do
-    context "when configure option is true" do
-      it "runs #configure method " do
-        expect_any_instance_of(TestWorkflow).to receive(:configure)
-        TestWorkflow.new(true)
-      end
-    end
+  end
 
-    context "when configure option is false" do
-      it "it doesn't run #configure method " do
-        expect_any_instance_of(TestWorkflow).to_not receive(:configure)
-        TestWorkflow.new(false)
+  describe "#save" do
+    it "passes constructor arguments to the method" do
+      klass = Class.new(Gush::Workflow) do
+        def configure(*args)
+          run FetchFirstJob
+          run PersistFirstJob, after: FetchFirstJob
+        end
       end
+
+      flow = klass.new("arg1", "arg2")
+
+      expect(flow).to receive(:configure).with("arg1", "arg2")
+      flow.save
     end
-  end
 
-  describe "#save" do
     context "workflow not persisted" do
       it "sets persisted to true" do
         flow = TestWorkflow.new
@@ -29,7 +30,6 @@ describe Gush::Workflow do
 
       it "assigns new unique id" do
         flow = TestWorkflow.new
-        expect(flow.id).to eq(nil)
         flow.save
         expect(flow.id).to_not be_nil
       end
@@ -61,17 +61,16 @@ describe Gush::Workflow do
 
   describe "#to_json" do
     it "returns correct hash" do
-
       klass = Class.new(Gush::Workflow) do
-        def configure
+        def configure(*args)
           run FetchFirstJob
           run PersistFirstJob, after: FetchFirstJob
         end
       end
 
-      result = JSON.parse(klass.new("workflow").to_json)
+      result = JSON.parse(klass.create("arg1", "arg2").to_json)
       expected = {
-        "id"=>nil,
+        "id" => an_instance_of(String),
         "name" => klass.to_s,
         "klass" => klass.to_s,
         "status" => "pending",
@@ -80,67 +79,109 @@ describe Gush::Workflow do
         "started_at" => nil,
         "finished_at" => nil,
         "stopped" => false,
+        "arguments" => ["arg1", "arg2"],
         "jobs" => [
           {
             "name"=>"FetchFirstJob",
             "klass"=>"FetchFirstJob",
-            "finished"=>false,
-            "enqueued"=>false,
-            "failed"=>false,
             "incoming"=>[],
             "outgoing"=>["PersistFirstJob"],
             "finished_at"=>nil,
             "started_at"=>nil,
             "enqueued_at"=>nil,
             "failed_at"=>nil,
-            "running" => false
+            "params" => {},
+            "output_payload" => nil
           },
           {
             "name"=>"PersistFirstJob",
             "klass"=>"PersistFirstJob",
-            "finished"=>false,
-            "enqueued"=>false,
-            "failed"=>false,
             "incoming"=>["FetchFirstJob"],
             "outgoing"=>[],
             "finished_at"=>nil,
             "started_at"=>nil,
             "enqueued_at"=>nil,
             "failed_at"=>nil,
-            "running" => false
+            "params" => {},
+            "output_payload" => nil
           }
         ]
       }
-      expect(result).to eq(expected)
+      expect(result).to match(expected)
     end
   end
 
   describe "#find_job" do
     it "finds job by its name" do
-      expect(TestWorkflow.new("test").find_job("PersistFirstJob")).to be_instance_of(PersistFirstJob)
+      expect(TestWorkflow.create.find_job("PersistFirstJob")).to be_instance_of(PersistFirstJob)
     end
   end
 
   describe "#run" do
+    it "allows passing additional params to the job" do
+      flow = Gush::Workflow.new
+      flow.run(Gush::Job, params: { something: 1 })
+      flow.save
+      expect(flow.jobs.first.params).to eq ({ something: 1 })
+    end
+
     context "when graph is empty" do
       it "adds new job with the given class as a node" do
-        flow = Gush::Workflow.new("workflow")
+        flow = Gush::Workflow.new
         flow.run(Gush::Job)
+        flow.save
         expect(flow.jobs.first).to be_instance_of(Gush::Job)
       end
     end
 
-    context "when last node is a job" do
-      it "attaches job as a child of the last inserted job" do
-        tree = Gush::Workflow.new("workflow")
-        klass1 = Class.new(Gush::Job)
-        klass2 = Class.new(Gush::Job)
-        tree.run(klass1)
-        tree.run(klass2, after: klass1)
-        tree.create_dependencies
-        expect(tree.jobs.first).to be_an_instance_of(klass1)
-        expect(tree.jobs.first.outgoing.first).to eq(klass2.to_s)
-      end
+    it "allows `after` to accept an array of jobs" do
+      tree = Gush::Workflow.new
+      klass1 = Class.new(Gush::Job)
+      klass2 = Class.new(Gush::Job)
+      klass3 = Class.new(Gush::Job)
+      tree.run(klass1)
+      tree.run(klass2, after: [klass1, klass3])
+      tree.run(klass3)
+
+      tree.resolve_dependencies
+
+      expect(tree.jobs.first.outgoing).to match_array([klass2.to_s])
+    end
+
+    it "allows `before` to accept an array of jobs" do
+      tree = Gush::Workflow.new
+      klass1 = Class.new(Gush::Job)
+      klass2 = Class.new(Gush::Job)
+      klass3 = Class.new(Gush::Job)
+      tree.run(klass1)
+      tree.run(klass2, before: [klass1, klass3])
+      tree.run(klass3)
+
+      tree.resolve_dependencies
+
+      expect(tree.jobs.first.incoming).to match_array([klass2.to_s])
+    end
+
+    it "attaches job as a child of the job in `after` key" do
+      tree = Gush::Workflow.new
+      klass1 = Class.new(Gush::Job)
+      klass2 = Class.new(Gush::Job)
+      tree.run(klass1)
+      tree.run(klass2, after: klass1)
+      tree.resolve_dependencies
+      job = tree.jobs.first
+      expect(job.outgoing).to match_array([klass2.to_s])
+    end
+
+    it "attaches job as a parent of the job in `before` key" do
+      tree = Gush::Workflow.new
+      klass1 = Class.new(Gush::Job)
+      klass2 = Class.new(Gush::Job)
+      tree.run(klass1)
+      tree.run(klass2, before: klass1)
+      tree.resolve_dependencies
+      job = tree.jobs.first
+      expect(job.incoming).to match_array([klass2.to_s])
     end
   end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: gush
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
 platform: ruby
 authors:
 - Piotrek Okoński
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-05-16 00:00:00.000000000 Z
+date: 2015-07-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sidekiq
@@ -207,7 +207,7 @@ files:
 - lib/gush/worker.rb
 - lib/gush/workflow.rb
 - spec/Gushfile.rb
-- spec/features/workflows_spec.rb
+- spec/features/integration_spec.rb
 - spec/lib/gush/client_spec.rb
 - spec/lib/gush/configuration_spec.rb
 - spec/lib/gush/job_spec.rb
@@ -241,7 +241,7 @@ specification_version: 4
 summary: Fast and distributed workflow runner using only Sidekiq and Redis
 test_files:
 - spec/Gushfile.rb
-- spec/features/workflows_spec.rb
+- spec/features/integration_spec.rb
 - spec/lib/gush/client_spec.rb
 - spec/lib/gush/configuration_spec.rb
 - spec/lib/gush/job_spec.rb

data/spec/features/workflows_spec.rb

@@ -1,31 +0,0 @@
-require 'spec_helper'
-
-
-describe "Workflows" do
-  it "runs the whole workflow in proper order" do
-    flow = TestWorkflow.create
-    flow.start!
-
-    expect(Gush::Worker).to have_jobs(flow.id, ["Prepare"])
-
-    Gush::Worker.perform_one
-    expect(Gush::Worker).to have_jobs(flow.id, ["FetchFirstJob", "FetchSecondJob"])
-
-    Gush::Worker.perform_one
-    expect(Gush::Worker).to have_jobs(flow.id, ["FetchSecondJob", "PersistFirstJob"])
-
-    Gush::Worker.perform_one
-    expect(Gush::Worker).to have_jobs(flow.id, ["PersistFirstJob", "NormalizeJob"])
-
-    Gush::Worker.perform_one
-    expect(Gush::Worker).to have_jobs(flow.id, ["NormalizeJob"])
-
-    Gush::Worker.perform_one
-
-    expect(Gush::Worker.jobs).to be_empty
-
-    flow = flow.reload
-    expect(flow).to be_finished
-    expect(flow).to_not be_failed
-  end
-end