gouda 0.1.16 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5ad491a59ec06c6199846bdbbb072f3298ed932533ba87de9b3d5da807b7f6a7
-  data.tar.gz: e703c1552897344e3f26830c7a6c386db5ba961bf97f9bfea27c819696d802f5
+  metadata.gz: 6a5b6d65b9cbfbe699e5bbfe1dd2854eeebc74d711d22243ec9c1c647d6986f7
+  data.tar.gz: d2642c37fe82af94a577d6b0806e9178842b8b0a53b64fe2eefeb1362f748a06
 SHA512:
-  metadata.gz: 730efd4a32941cd408e39e5b43772b5627736f232d7632d10564e6c273d9af45986522b550db1deef32f87abcb89808ac39e2b762174c66a3d0d09537e054f9f
-  data.tar.gz: 9706f6a827bdbc916eb26004e6d267c377b4f51d1f6f568423e3aed1c8eeaf9ec1d784a6bd8623f7d41234618fc88ed31db4a0dddcc4c26ea7ef5a5688e22582
+  metadata.gz: 8d276d47fd5dced75511522900f1328ee1cb7cb076f833e6dc0d8eb597cafdc1933986ebb17a2ec37390d0d3e74b2c183fdabba204cc0e41c2010ccb2adefcdb
+  data.tar.gz: 644d5eba7e6fca4fbf2f21d805d98985be8e72fcfa5e06dc8402a26fefb292c2a204134071e0e31ca345ab74863ea6191aa934479e6eb1b865a31e4bea2ff817

data/Appraisals

@@ -1,7 +1,9 @@
 appraise "rails-7" do
   gem "rails", "< 8.0"
+  gem "async", "~> 2.25"
 end
 
 appraise "rails-8" do
   gem "rails", ">= 8.0"
+  gem "async", "~> 2.25"
 end

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+## [0.2.0]
+
+- Introduce fiber execution for workers
+
 ## [0.1.16]
 
 - Actualy suppress DEBUG level logs in `suppressing_sql_logs` :facepalm:

data/Gemfile

@@ -6,3 +6,4 @@ gemspec
 gem "standard", require: false
 gem "rake", "~> 13.0"
 gem "minitest", "~> 5.0"
+gem "async", "~> 2.25"

data/README.md

@@ -72,7 +72,20 @@ to **prevent concurrent executions** but not to **limit the load on the system**
 
 ## Key concepts in Gouda: `executing_on`
 
-A `Workload` is executing on a particular `executing_on` entity - usually a worker thread. That entity gets a pseudorandom ID . The `executing_on` value can be used to see, for example, whether a particular worker thread has hung. If multiple jobs have a far-behind `updated_at` and are all `executing`, this likely means that the worker has crashed or hung. The value can also be used to build a table of currently running workers.
+A `Workload` is executing on a particular `executing_on` entity - usually a worker thread or fiber. That entity gets a pseudorandom ID. The `executing_on` value can be used to see, for example, whether a particular worker thread has hung. If multiple jobs have a far-behind `updated_at` and are all `executing`, this likely means that the worker executing them has died or has hung.
+
+The `executing_on` field now clearly indicates the execution context:
+- Thread-based execution: `hostname-pid-uuid-thread-abc123` 
+- Hybrid execution (threads + fibers): `hostname-pid-uuid-thread-abc123-fiber-def456`
+
+You can programmatically check the execution context:
+
+```ruby
+workload = Gouda::Workload.last
+workload.executed_on_thread?  # => true/false
+workload.uses_async_execution?  # => true/false  
+workload.execution_context    # => :thread, :fiber, or :unknown
+```
 
 ## Usage tips: bulkify your enqueues
 
@@ -103,7 +116,108 @@ User.transaction do
 end
 ```
 
-## Web UI
+## Hybrid Execution: Threads + Fibers
+
+Gouda supports two execution modes: traditional thread-based execution and hybrid execution that combines multiple threads with multiple fibers per thread. The hybrid mode provides non-blocking IO capabilities while still utilizing multiple CPU cores.
+
+### Execution Modes
+
+**Thread-based execution (default):**
+- Multiple worker threads
+- One job per thread at a time
+- Simple and reliable
+- Good for CPU-bound work
+
+**Hybrid execution (threads + fibers):**
+- Multiple worker threads, each containing multiple fibers
+- Much higher concurrency (threads × fibers per thread)
+- Non-blocking IO within each thread
+- Best of both worlds: CPU parallelism + IO concurrency
+
+### Requirements
+
+- Ruby 3.1+ with Fiber.scheduler support
+- `async` gem dependency (~> 2.25, automatically included)
+- **Rails isolation level set to `:fiber`** (critical for hybrid mode)
+- Async-compatible gems for full benefit
+
+### Critical Configuration for Hybrid Mode
+
+**⚠️ IMPORTANT**: When using hybrid execution **with PostgreSQL**, you **must** configure Rails to use fiber-based isolation:
+
+```ruby
+# config/application.rb
+class Application < Rails::Application
+  # This is REQUIRED for Gouda hybrid mode with PostgreSQL
+  config.active_support.isolation_level = :fiber
+end
+```
+
+**Why this matters for PostgreSQL:**
+- Prevents segmentation faults with Ruby 3.4+ and PostgreSQL adapter
+- Ensures ActiveRecord connection pools work correctly with fibers and PostgreSQL connections
+- Required for proper fiber-based concurrency with PostgreSQL's connection handling
+
+**Note**: This configuration is specifically required when using PostgreSQL. Other database adapters may not have the same requirement, but setting it to `:fiber` is generally safe and recommended when using Gouda's hybrid mode.
 
-At the moment the Gouda UI is proprietary, so this gem only provides a "headless" implementation. We expect this to change in the future.
+Gouda will automatically detect if this setting is missing when using PostgreSQL and warn you during startup.
+
+### Configuration
+
+To enable hybrid execution (threads + fibers):
+
+```ruby
+Gouda.configure do |config|
+  # Enable hybrid scheduler (threads + fibers)
+  config.use_fiber_scheduler = true
+  
+  # Number of worker threads (should match your CPU cores)
+  config.worker_thread_count = 4
+  
+  # Number of fibers per thread (can be much higher)
+  config.fibers_per_thread = 10  # This means 10 fibers PER thread
+  
+  # Total concurrency = worker_thread_count × fibers_per_thread
+  # In this example: 4 threads × 10 fibers = 40 concurrent jobs
+end
+```
+
+**Traditional thread-based execution (default):**
+
+```ruby
+Gouda.configure do |config|
+  # Thread-based execution (default)
+  config.use_fiber_scheduler = false
+  
+  # Number of worker threads
+  config.worker_thread_count = 4  # Total concurrency = 4
+end
+```
+
+### Benefits
+
+- **Non-blocking IO**: Database queries, HTTP requests, and file operations don't block other jobs
+- **CPU parallelism**: Multiple threads utilize multiple CPU cores
+- **Higher concurrency**: Can handle many more concurrent jobs with less memory overhead than pure threading
+- **Better resource utilization**: Cooperative scheduling reduces context switching overhead
+- **Backward compatibility**: Thread-based mode remains the default and continues to work
+
+### When to use hybrid vs thread execution
+
+**Use hybrid execution (threads + fibers) for:**
+- IO-bound jobs (HTTP requests, database queries, file processing)
+- High-concurrency scenarios
+- Jobs that spend time waiting for external resources
+- Mixed workloads with both CPU and IO requirements
+
+**Use thread-based execution for:**
+- CPU-intensive jobs
+- Jobs that use gems without async support
+- Simpler deployment scenarios
+- When you want predictable, simple execution
+
+You can even run both modes simultaneously with different queue constraints to optimize for different job types.
+
+## Web UI
 
+At the moment the Gouda UI is proprietary, so this gem only provides a "headless" implementation. We expect this to change in the future.

data/examples/async_job_example.rb

@@ -0,0 +1,116 @@
+# Example of an async-compatible job for use with Gouda's fiber scheduler
+
+require "async"
+require "async/http"
+
+class AsyncHttpJob < ApplicationJob
+  queue_as :default
+
+  def perform(url)
+    # This job will work with both thread-based and fiber-based workers
+    # When using fiber scheduler, HTTP requests will be non-blocking
+
+    # Use async-compatible HTTP client
+    response = fetch_url_async(url)
+
+    # Process the response
+    process_response(response)
+  end
+
+  private
+
+  def fetch_url_async(url)
+    # When running in Gouda's hybrid mode: concurrent with other async operations in same reactor
+    # When running in thread-only mode: blocks the thread (creates own reactor)
+    Async do |task|
+      internet = Async::HTTP::Internet.new
+      response = internet.get(url)
+      response.read
+    ensure
+      internet&.close
+    end
+  end
+
+  def process_response(data)
+    # Process the fetched data
+    Rails.logger.info "Fetched #{data.size} bytes"
+
+    # Example: Save to database (this will also be async with fiber scheduler)
+    MyModel.create!(data: data, processed_at: Time.current)
+  end
+end
+
+# Example of a job that benefits from fiber concurrency
+class BulkDataProcessorJob < ApplicationJob
+  queue_as :bulk_processing
+
+  def perform(record_ids)
+    # Process multiple records concurrently using fibers
+    results = []
+
+    # When using fiber scheduler, these database operations can be concurrent
+    record_ids.each do |id|
+      # This will be non-blocking with fiber scheduler
+      record = MyModel.find(id)
+      result = process_record(record)
+      results << result
+    end
+
+    # Aggregate results
+    aggregate_results(results)
+  end
+
+  private
+
+  def process_record(record)
+    # Simulate some async work (API call, file processing, etc.)
+    # With fiber scheduler, multiple records can be processed concurrently
+    {
+      id: record.id,
+      processed_at: Time.current,
+      result: expensive_operation(record)
+    }
+  end
+
+  def expensive_operation(record)
+    # This could be an HTTP API call, file processing, etc.
+    # that benefits from non-blocking IO
+    sleep(0.1) # Simulated IO wait
+    "processed_#{record.id}"
+  end
+
+  def aggregate_results(results)
+    # Save aggregated results
+    Rails.logger.info "Processed #{results.size} records"
+  end
+end
+
+# Example of configuring different queue constraints for fiber workers
+class FiberWorkerInitializer
+  def self.start_workers
+    if Rails.env.production?
+      # Start multiple workers with different queue constraints
+
+      # High-throughput fiber worker for IO-bound jobs
+      Thread.new do
+        Gouda.configure do |config|
+          config.use_fiber_scheduler = true
+          config.fibers_per_thread = 20
+        end
+
+        Gouda.start
+      end
+
+      # Traditional thread worker for CPU-bound jobs
+      Thread.new do
+        Gouda.configure do |config|
+          config.use_fiber_scheduler = false
+          config.worker_thread_count = 4
+        end
+
+        ENV["GOUDA_QUEUES"] = "cpu_intensive"
+        Gouda.start
+      end
+    end
+  end
+end

data/examples/fiber_configuration.rb

@@ -0,0 +1,45 @@
+# Example configuration for Gouda fiber-based job processing
+# This shows how to properly configure Rails and Gouda for optimal fiber performance
+
+# CRITICAL: Set Rails isolation level to :fiber (config/application.rb)
+# This MUST be set when using PostgreSQL to prevent segfaults and ensure proper fiber concurrency
+#
+# # config/application.rb
+# class Application < Rails::Application
+#   # Required for fiber mode with PostgreSQL - prevents segfaults and connection issues
+#   config.active_support.isolation_level = :fiber
+# end
+
+# Gouda fiber configuration
+Gouda.configure do |config|
+  # Enable fiber-based worker (instead of threads)
+  config.use_fiber_scheduler = true
+
+  # Number of concurrent worker fibers
+  # Can be higher than CPU cores since fibers are lightweight for IO-bound work
+  config.fibers_per_thread = 10
+
+  # Other standard Gouda configuration options work the same
+  config.preserve_job_records = true
+  config.cleanup_preserved_jobs_before = 3.hours
+end
+
+# Example of starting the fiber-based worker
+# Gouda.start
+
+# The worker will automatically:
+# 1. Check that Rails isolation level is set to :fiber (warn if not)
+# 2. Use fiber-based concurrency for non-blocking IO
+
+# Benefits of fiber-based approach:
+# 1. Non-blocking IO operations (database queries, HTTP requests, etc.)
+# 2. Higher concurrency with less memory overhead than threads
+# 3. Better performance for IO-bound jobs
+# 4. Cooperative scheduling reduces context switching overhead
+
+# Important considerations:
+# 1. Your jobs must use async-compatible gems for full benefit
+# 2. Database connection pool size should be configured appropriately
+# 3. CPU-intensive jobs won't benefit much from fiber approach
+# 4. Requires Ruby 3.1+ with Fiber.scheduler support
+# 5. When using PostgreSQL: Rails isolation level MUST be set to :fiber

data/examples/fiber_demo.rb

@@ -0,0 +1,118 @@
+#!/usr/bin/env ruby
+
+# Demonstration of Gouda's fiber-based non-blocking IO worker
+# Run this script to see fiber workers in action
+
+require_relative "../lib/gouda"
+require "active_record"
+require "active_job"
+
+# Setup database connection (assumes PostgreSQL is available)
+ActiveRecord::Base.establish_connection(
+  adapter: "postgresql",
+  database: "gouda_demo",
+  username: ENV["USER"]
+)
+
+# Create tables if they don't exist
+begin
+  ActiveRecord::Base.connection.execute("SELECT 1 FROM gouda_workloads LIMIT 1")
+rescue ActiveRecord::StatementInvalid
+  ActiveRecord::Schema.define(version: 1) do |schema|
+    Gouda.create_tables(schema)
+  end
+end
+
+# Configure Gouda for fiber-based execution
+Gouda.configure do |config|
+  config.use_fiber_scheduler = true
+  config.fibers_per_thread = 5
+  config.polling_sleep_interval_seconds = 0.1
+  config.preserve_job_records = true
+end
+
+# Demo job that simulates IO-bound work
+class DemoAsyncJob < ActiveJob::Base
+  queue_as :default
+
+  def perform(task_name, duration = 0.1)
+    puts "🚀 Starting #{task_name} (Fiber: #{Fiber.current.object_id})"
+
+    # Simulate async IO work (this will be non-blocking with fiber scheduler)
+    sleep(duration)
+
+    puts "✅ Completed #{task_name} after #{duration}s"
+
+    # Return some result
+    "Task #{task_name} completed at #{Time.now}"
+  end
+end
+
+# Demo job that makes HTTP requests (would be non-blocking with proper async HTTP client)
+class HttpFetchJob < ActiveJob::Base
+  queue_as :http
+
+  def perform(url)
+    puts "🌐 Fetching #{url} (Fiber: #{Fiber.current.object_id})"
+
+    # In a real scenario, you'd use an async HTTP client like:
+    # require 'async/http'
+    # Async do
+    #   internet = Async::HTTP::Internet.new
+    #   response = internet.get(url)
+    #   puts "📄 Fetched #{response.read.size} bytes from #{url}"
+    # end
+
+    # For demo purposes, simulate the request
+    sleep(0.2)
+    puts "📄 Simulated fetch from #{url}"
+  end
+end
+
+puts "🎯 Gouda Fiber Worker Demo"
+puts "========================="
+puts "Configuration:"
+puts "  - Fiber scheduler: #{Gouda.config.use_fiber_scheduler}"
+puts "  - Fiber workers: #{Gouda.config.fibers_per_thread}"
+puts ""
+
+# Clean up old jobs
+Gouda::Workload.delete_all
+
+# Enqueue demo jobs
+puts "📝 Enqueueing jobs..."
+Gouda.in_bulk do
+  # Enqueue several IO-bound jobs
+  5.times do |i|
+    DemoAsyncJob.perform_later("Task-#{i}", 0.1 + (i * 0.05))
+  end
+
+  # Enqueue some HTTP jobs
+  3.times do |i|
+    HttpFetchJob.perform_later("https://example.com/page#{i}")
+  end
+end
+
+queued_count = Gouda::Workload.where(state: "enqueued").count
+puts "📊 Enqueued #{queued_count} jobs"
+puts ""
+
+puts "🔄 Starting fiber workers..."
+puts "Press Ctrl+C to stop"
+puts ""
+
+# Run the fiber worker
+begin
+  Gouda.start
+rescue Interrupt
+  puts "\n🛑 Stopping workers..."
+end
+
+# Show results
+puts "\n📊 Final Results:"
+puts "  - Finished: #{Gouda::Workload.where(state: "finished").count}"
+puts "  - Enqueued: #{Gouda::Workload.where(state: "enqueued").count}"
+puts "  - Executing: #{Gouda::Workload.where(state: "executing").count}"
+
+puts "\n💡 Notice how multiple jobs ran concurrently using fibers!"
+puts "   Each job shows its Fiber object ID to demonstrate concurrency."

data/gemfiles/rails_7.gemfile

@@ -5,6 +5,7 @@ source "https://rubygems.org"
 gem "standard", require: false
 gem "rake", "~> 13.0"
 gem "minitest", "~> 5.0"
+gem "async", "~> 2.25"
 gem "rails", "< 8.0"
 
 gemspec path: "../"

data/gemfiles/rails_7.gemfile.lock

@@ -1,10 +1,11 @@
 PATH
   remote: ..
   specs:
-    gouda (0.1.14)
+    gouda (0.2.0)
       activejob (>= 7.2.0)
       activerecord (>= 7.2.0)
       activesupport (>= 7.2.0)
+      async (~> 2.25)
       fugit (~> 1.10)
       railties (>= 7.2.0)
 
@@ -87,6 +88,12 @@ GEM
       rake
       thor (>= 0.14.0)
     ast (2.4.2)
+    async (2.25.0)
+      console (~> 1.29)
+      fiber-annotation
+      io-event (~> 1.11)
+      metrics (~> 0.12)
+      traces (~> 0.15)
     base64 (0.2.0)
     benchmark (0.4.0)
     bigdecimal (3.1.9)
@@ -94,6 +101,10 @@ GEM
     coderay (1.1.3)
     concurrent-ruby (1.3.5)
     connection_pool (2.5.0)
+    console (1.31.0)
+      fiber-annotation
+      fiber-local (~> 1.1)
+      json
     crass (1.0.6)
     date (3.4.1)
     debug (1.10.0)
@@ -103,6 +114,10 @@ GEM
     erubi (1.13.1)
     et-orbi (1.2.11)
       tzinfo
+    fiber-annotation (0.2.0)
+    fiber-local (1.1.0)
+      fiber-storage
+    fiber-storage (1.0.1)
     fugit (1.11.1)
       et-orbi (~> 1, >= 1.2.11)
       raabro (~> 1.4)
@@ -111,6 +126,7 @@ GEM
     i18n (1.14.7)
       concurrent-ruby (~> 1.0)
     io-console (0.8.0)
+    io-event (1.11.0)
     irb (1.15.1)
       pp (>= 0.6.0)
       rdoc (>= 4.0.0)
@@ -129,6 +145,7 @@ GEM
       net-smtp
     marcel (1.0.4)
     method_source (1.1.0)
+    metrics (0.12.2)
     mini_mime (1.1.5)
     minitest (5.25.4)
     net-imap (0.5.6)
@@ -239,6 +256,7 @@ GEM
     stringio (3.1.5)
     thor (1.3.2)
     timeout (0.4.3)
+    traces (0.15.2)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
     unicode-display_width (3.1.4)
@@ -253,11 +271,13 @@ GEM
 
 PLATFORMS
   arm64-darwin-21
+  arm64-darwin-23
   x86_64-darwin
   x86_64-linux
 
 DEPENDENCIES
   appraisal
+  async (~> 2.25)
   debug
   gouda!
   minitest (~> 5.0)

data/gemfiles/rails_8.gemfile

@@ -5,6 +5,7 @@ source "https://rubygems.org"
 gem "standard", require: false
 gem "rake", "~> 13.0"
 gem "minitest", "~> 5.0"
+gem "async", "~> 2.25"
 gem "rails", ">= 8.0"
 
 gemspec path: "../"

data/gemfiles/rails_8.gemfile.lock

@@ -1,10 +1,11 @@
 PATH
   remote: ..
   specs:
-    gouda (0.1.14)
+    gouda (0.2.0)
       activejob (>= 7.2.0)
       activerecord (>= 7.2.0)
       activesupport (>= 7.2.0)
+      async (~> 2.25)
       fugit (~> 1.10)
       railties (>= 7.2.0)
 
@@ -87,6 +88,12 @@ GEM
       rake
       thor (>= 0.14.0)
     ast (2.4.2)
+    async (2.25.0)
+      console (~> 1.29)
+      fiber-annotation
+      io-event (~> 1.11)
+      metrics (~> 0.12)
+      traces (~> 0.15)
     base64 (0.2.0)
     benchmark (0.4.0)
     bigdecimal (3.1.9)
@@ -94,6 +101,10 @@ GEM
     coderay (1.1.3)
     concurrent-ruby (1.3.5)
     connection_pool (2.5.0)
+    console (1.31.0)
+      fiber-annotation
+      fiber-local (~> 1.1)
+      json
     crass (1.0.6)
     date (3.4.1)
     debug (1.10.0)
@@ -103,6 +114,10 @@ GEM
     erubi (1.13.1)
     et-orbi (1.2.11)
       tzinfo
+    fiber-annotation (0.2.0)
+    fiber-local (1.1.0)
+      fiber-storage
+    fiber-storage (1.0.1)
     fugit (1.11.1)
       et-orbi (~> 1, >= 1.2.11)
       raabro (~> 1.4)
@@ -111,6 +126,7 @@ GEM
     i18n (1.14.7)
       concurrent-ruby (~> 1.0)
     io-console (0.8.0)
+    io-event (1.11.0)
     irb (1.15.1)
       pp (>= 0.6.0)
       rdoc (>= 4.0.0)
@@ -129,6 +145,7 @@ GEM
       net-smtp
     marcel (1.0.4)
     method_source (1.1.0)
+    metrics (0.12.2)
     mini_mime (1.1.5)
     minitest (5.25.4)
     net-imap (0.5.6)
@@ -239,6 +256,7 @@ GEM
     stringio (3.1.5)
     thor (1.3.2)
     timeout (0.4.3)
+    traces (0.15.2)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
     unicode-display_width (3.1.4)
@@ -254,11 +272,13 @@ GEM
 
 PLATFORMS
   arm64-darwin-21
+  arm64-darwin-23
   x86_64-darwin
   x86_64-linux
 
 DEPENDENCIES
   appraisal
+  async (~> 2.25)
   debug
   gouda!
   minitest (~> 5.0)

data/gouda.gemspec

@@ -26,6 +26,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency "railties", rails_min
   spec.add_dependency "activejob", rails_min
   spec.add_dependency "fugit", "~> 1.10"
+  spec.add_dependency "async", "~> 2.25"
 
   spec.add_development_dependency "standard"
   spec.add_development_dependency "pg"

data/lib/gouda/railtie.rb

@@ -43,6 +43,8 @@ module Gouda
           Gouda.config.preserve_job_records = config_from_rails[:preserve_job_records]
           Gouda.config.polling_sleep_interval_seconds = config_from_rails[:polling_sleep_interval_seconds]
           Gouda.config.worker_thread_count = config_from_rails[:worker_thread_count]
+          Gouda.config.use_fiber_scheduler = config_from_rails[:use_fiber_scheduler]
+          Gouda.config.fibers_per_thread = config_from_rails[:fibers_per_thread]
         end
       else
         Gouda.config.preserve_job_records = false

data/lib/gouda/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Gouda
-  VERSION = "0.1.16"
+  VERSION = "0.2.0"
 end