ductwork 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 252ad8643950e6ea9e5024d0e63885aea8c3c7020bef1586faf0b414b41206f0
+  data.tar.gz: 1133562c0ff6c84c6a91c87e3d73f734e01626842b42c31e49d87127aa25cd7d
+SHA512:
+  metadata.gz: 36460be6d4d35656be5ce2cc0006eaa539f00992c6b32f0c07610d42bc889ea0e198e17d9c405bdfca73469895f9d3076fda4335aea6ad7e6dc5011de4e41612
+  data.tar.gz: ddb569815f9ac837881fa08b13541084a5048a34fceafe718026fc228966b35377d16af56edddd5dd74a8cf210a76a3b0986e40644a0d3d3d4efb2db91b475f6

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Ductwork Changelog
+
+## [0.1.0]
+
+- Initial release - see [documentation](https://docs.getductwork.io/) for details

data/LICENSE.txt

@@ -0,0 +1,168 @@
+Ductwork is an Open Source project licensed under the terms of
+the LGPLv3.0 license. Please see below for license text:
+
+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (c) 2024-2025 Tyler Ewing
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

data/README.md

@@ -0,0 +1,154 @@
+# Ductwork
+
+A Ruby pipeline framework.
+
+Ductwork lets you build complex pipelines quickly and easily using intuitive Ruby tooling and a natural DSL. No need to learn complicated unified object models or stand up separate runner instances—just write Ruby code and let Ductwork handle the orchestration.
+
+**[Full Documentation](https://docs.getductwork.io/)**
+
+## Installation
+
+Add Ductwork to your application's Gemfile:
+
+```bash
+bundle add ductwork
+```
+
+Run the Rails generator to create the binstub, configuration file, and migrations:
+
+```bash
+bin/rails generate ductwork:install
+```
+
+Run migrations and you're ready to start building pipelines!
+
+## Configuration
+
+
+The only required configuration is specifying which pipelines to run. Edit the default configuration file `config/ductwork.yml`:
+
+```yaml
+default: &default
+  pipelines:
+    - EnrichUserDataPipeline
+    - SendMonthlyStatusReportsPipeline
+```
+
+Or use the wildcard to run all pipelines (use cautiously—this can consume significant resources):
+
+```yaml
+default: &default
+  pipelines: "*"
+```
+
+See the [Configuration Guide](https://docs.getductwork.io/advanced/configuration.html) for all available options including thread counts, timeouts, and database settings.
+
+## Usage
+
+### 1. Create a Pipeline Class
+
+Pipeline classes live in `app/pipelines` and inherit from `Ductwork::Pipeline`. While the "Pipeline" suffix is optional, it can help avoid naming collisions:
+
+```ruby
+# app/pipelines/enrich_user_data_pipeline.rb
+class EnrichUserDataPipeline < Ductwork::Pipeline
+end
+```
+
+### 2. Define Steps
+
+Steps are Plain Old Ruby Objects (POROs) that implement two methods:
+- `initialize` - accepts parameters from the trigger call or previous step's return value
+- `execute` - performs the work and returns data for the next step
+
+Steps live in `app/steps`:
+
+```ruby
+# app/steps/users_requiring_enrichment.rb
+class UsersRequiringEnrichment
+  def initialize(days_outdated)
+    @days_outdated = days_outdated
+  end
+
+  def execute
+    ids = User.where("data_last_refreshed_at < ?", @days_outdated.days.ago).ids
+    Rails.logger.info("Enriching #{ids.length} users' data")
+
+    # Return value becomes input to the next step
+    ids
+  end
+end
+```
+
+### 3. Define Transitions
+
+Connect steps together using Ductwork's fluent interface DSL. The key principle: **each step's return value becomes the next step's input**.
+
+```ruby
+class EnrichUserDataPipeline < Ductwork::Pipeline
+  define do |pipeline|
+    pipeline.start(UsersRequiringEnrichment)       # Start with a single step
+            .expand(to: LoadUserData)              # Fan out to multiple steps
+            .divide(to: [FetchDataFromSourceA,     # Split into parallel branches
+                         FetchDataFromSourceB])
+            .combine(into: CollateUserData)        # Merge branches back together
+            .chain(UpdateUserData)                 # Sequential processing
+            .collapse(into: ReportSuccess)         # Gather expanded steps
+  end
+end
+```
+
+**Important:** Return values must be JSON-serializable.
+
+See [Defining Pipelines](https://docs.getductwork.io/getting_started/defining-pipelines.html) for detailed documentation.
+
+### 4. Run Ductwork
+
+Start the Ductwork supervisor, which manages pipeline advancers and job workers for each configured pipeline:
+
+```bash
+bin/ductwork
+```
+
+Use a custom configuration file if needed:
+
+```bash
+bin/ductwork -c config/ductwork.0.yml
+```
+
+### 5. Trigger Your Pipeline
+
+Trigger pipelines from anywhere in your Rails application. The `trigger` method returns a `Ductwork::Pipeline` instance for monitoring:
+
+```ruby
+# In a Rake task
+task enrich_user_data: :environment do
+  pipeline = EnrichUserDataPipeline.trigger(7)
+  puts "Pipeline #{pipeline.id} started"
+end
+
+# In a controller
+def create
+  pipeline = EnrichUserDataPipeline.trigger(params[:days_outdated])
+
+  render json: { pipeline_id: pipeline.id, status: pipeline.status }
+end
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/ductwork/ductwork. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/ductwork/ductwork/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [LGPLv3.0 License](https://github.com/ductwork/ductwork/blob/main/LICENSE.txt).
+
+## Code of Conduct
+
+Everyone interacting in the Ductwork project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/ductwork/ductwork/blob/main/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "rubocop/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/app/models/ductwork/availability.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module Ductwork
+  class Availability < Ductwork::Record
+    belongs_to :execution, class_name: "Ductwork::Execution"
+
+    validates :started_at, presence: true
+  end
+end

data/app/models/ductwork/execution.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module Ductwork
+  class Execution < Ductwork::Record
+    belongs_to :job, class_name: "Ductwork::Job"
+    has_one :availability, class_name: "Ductwork::Availability", foreign_key: "execution_id"
+    has_one :run, class_name: "Ductwork::Run", foreign_key: "execution_id"
+    has_one :result, class_name: "Ductwork::Result", foreign_key: "execution_id"
+
+    validates :retry_count, presence: true
+    validates :started_at, presence: true
+  end
+end

data/app/models/ductwork/job.rb

@@ -0,0 +1,181 @@
+# frozen_string_literal: true
+
+module Ductwork
+  class Job < Ductwork::Record
+    belongs_to :step, class_name: "Ductwork::Step"
+    has_many :executions, class_name: "Ductwork::Execution", foreign_key: "job_id", dependent: :destroy
+
+    validates :klass, presence: true
+    validates :started_at, presence: true
+    validates :input_args, presence: true
+
+    FAILED_EXECUTION_TIMEOUT = 10.seconds
+
+    def self.claim_latest(klass) # rubocop:todo Metrics/MethodLength
+      process_id = ::Process.pid
+      id = Ductwork::Availability
+           .joins(execution: { job: { step: :pipeline } })
+           .where("ductwork_availabilities.started_at <= ?", Time.current)
+           .where(completed_at: nil)
+           .where(ductwork_pipelines: { klass: })
+           .order(:created_at)
+           .limit(1)
+           .pluck(:id)
+           .first
+
+      if id.present?
+        # TODO: probably makes sense to use SQL here instead of relying
+        # on ActiveRecord to construct the correct `UPDATE` query
+        rows_updated = nil
+        Ductwork::Record.transaction do
+          rows_updated = Ductwork::Availability
+                         .where(id: id, completed_at: nil)
+                         .update_all(completed_at: Time.current, process_id: process_id)
+          Ductwork::Execution
+            .joins(:availability)
+            .where(completed_at: nil)
+            .where(ductwork_availabilities: { id: })
+            .update_all(process_id:)
+        end
+
+        if rows_updated == 1
+          Ductwork.configuration.logger.debug(
+            msg: "Job claimed",
+            role: :job_worker,
+            process_id: process_id,
+            availability_id: id
+          )
+          Ductwork::Job
+            .joins(executions: :availability)
+            .find_by(ductwork_availabilities: { id:, process_id: })
+        else
+          Ductwork.configuration.logger.debug(
+            msg: "Did not claim job, avoided race condition",
+            role: :job_worker,
+            process_id: process_id,
+            availability_id: id
+          )
+          nil
+        end
+      end
+    end
+
+    def self.enqueue(step, *args)
+      Ductwork::Record.transaction do
+        job = step.create_job!(
+          klass: step.klass,
+          started_at: Time.current,
+          input_args: JSON.dump({ args: })
+        )
+        execution = job.executions.create!(
+          started_at: Time.current,
+          retry_count: 0
+        )
+        execution.create_availability!(
+          started_at: Time.current
+        )
+
+        job
+      end
+    end
+
+    def execute(pipeline)
+      # i don't _really_ like this, but it should be fine for now...
+      execution = executions.order(:created_at).last
+      logger.debug(
+        msg: "Executing job",
+        role: :job_worker,
+        pipeline: pipeline,
+        job_klass: klass
+      )
+      args = JSON.parse(input_args)["args"]
+      instance = Object.const_get(klass).new(*args)
+      run = execution.create_run!(
+        started_at: Time.current
+      )
+      result = nil
+
+      begin
+        output_payload = instance.execute
+        execution_succeeded!(execution, run, output_payload)
+        result = "success"
+      rescue StandardError => e
+        execution_failed!(execution, run, e)
+        result = "failure"
+      ensure
+        logger.debug(
+          msg: "Executed job",
+          role: :job_worker,
+          pipeline: pipeline,
+          job_klass: klass,
+          result: result
+        )
+      end
+    end
+
+    def return_value
+      if output_payload.present?
+        JSON.parse(output_payload).fetch("payload", nil)
+      end
+    end
+
+    private
+
+    def logger
+      Ductwork.configuration.logger
+    end
+
+    def execution_succeeded!(execution, run, output_payload)
+      payload = JSON.dump({ payload: output_payload })
+
+      Ductwork::Record.transaction do
+        update!(output_payload: payload, completed_at: Time.current)
+        execution.update!(completed_at: Time.current)
+        run.update!(completed_at: Time.current)
+        execution.create_result!(result_type: "success")
+        step.update!(status: :advancing)
+      end
+    end
+
+    def execution_failed!(execution, run, error) # rubocop:disable Metrics/AbcSize
+      halted = false
+      pipeline = step.pipeline
+
+      Ductwork::Record.transaction do
+        execution.update!(completed_at: Time.current)
+        run.update!(completed_at: Time.current)
+        execution.create_result!(
+          result_type: "failure",
+          error_klass: error.class.to_s,
+          error_message: error.message,
+          error_backtrace: error.backtrace
+        )
+
+        if execution.retry_count < Ductwork.configuration.job_worker_max_retry
+          new_execution = executions.create!(
+            retry_count: execution.retry_count + 1,
+            started_at: FAILED_EXECUTION_TIMEOUT.from_now
+          )
+          new_execution.create_availability!(
+            started_at: FAILED_EXECUTION_TIMEOUT.from_now
+          )
+        elsif execution.retry_count >= Ductwork.configuration.job_worker_max_retry
+          halted = true
+
+          step.update!(status: :failed)
+          pipeline.halted!
+        end
+      end
+
+      # NOTE: perform lifecycle hook execution outside of the transaction as
+      # to not unnecessarily hold it open
+      if halted
+        klass = JSON.parse(pipeline.definition).dig("metadata", "on_halt", "klass")
+
+        if klass.present?
+          Object.const_get(klass).new(error).execute
+        end
+      end
+    end
+  end
+end