litestack 0.1.7 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2662941f303da99554039370e6e53e4c13956c52070e93ad6862742881ca8063
-  data.tar.gz: ccd5583a9e0e7f5c559f8dafdde913bba256b7b5a679fe927cd31805c1289022
+  metadata.gz: afdf01934662a90b8c67455dd46e1649972ee62eb732cbf08746f8ac5f509dae
+  data.tar.gz: 734300372c194639072fdcfe4b35f630b4fb173bf453e14e9813f6c7569a6877
 SHA512:
-  metadata.gz: 1a139b3c42cd3f9a327fc7d8f1487dab2e95f2c88d07e063eecc2e438c91236f301a8e3928c9a3b41030fd8c08f4c118c449e56ea4be35b73b00fe2aa240faf8
-  data.tar.gz: 409221a087df5707bd6da477bb9091daf2ee0fc5f8114d71a7666324676bc62e5d77b6edfd1e3e7d29d601985f7936b2e3545e618b1c888b6c7a748544bbafde
+  metadata.gz: b0b46de8074b05b6bdab5438e932f53212bd3b98b2d534e02b8b81f1330b41869b2271055e7eeaa3643eeee0df11ef1c3884ce880ff235db063d2e05183f3ad7
+  data.tar.gz: 3bf959fa43c538140724114c56836bcaa5c88adc6d1ca5ec4bf0db9cfdd0c6feb98675f2afba01632352dcc94696a3c85d8b0de5537db0f4aa84d2e00829acd1

data/BENCHMARKS.md

@@ -67,7 +67,7 @@ For testing the cache we attempted to try writing and reading different payload
 
 ### Write
 
-|Payload Size (bytes)|Redis|litecahce|
+|Payload Size (bytes)|Redis|litecache|
 |-:|-:|-:|
 |10|4.2K q/s|11.0K q/s|
 |100|4.7K q/s|11.6K q/s|

data/CHANGELOG.md

@@ -1,19 +1,36 @@
 ## [Unreleased]
 
-## [0.1.7] - 2022-03-05
+## [0.2.0] - 2023-05-08
+
+- Litecable, a SQLite driver for ActionCable
+- Litemetric for metrics collection support (experimental, disabled by default)
+- New schema for Litejob, old jobs are auto-migrated
+- Code refactoring, extraction of SQL statements to external files
+- Graceful shutdown support working properly
+- Fork resilience
+
+## [0.1.8] - 2023-03-08
+
+- More code cleanups, more test coverage
+- Retry support for jobs in Litejob
+- Job storage and garbage collection for failed jobs
+- Initial graceful shutdown support for Litejob (incomplete)
+- More configuration options for Litejob
+
+## [0.1.7] - 2023-03-05
 
 - Code cleanup, removal of references to older name
 - Fix for the litedb rake tasks (thanks: netmute)
 - More fixes for the new concurrency model
 - Introduced a logger for the Litejobqueue (doesn't work with Polyphony, fix should come soon)
 
-## [0.1.6] - 2022-03-03
+## [0.1.6] - 2023-03-03
 
 - Revamped the locking model, more robust, minimal performance hit
 - Introduced a new resource pooling class
 - Litecache and Litejob now use the resource pool
 - Much less memory usage for Litecache and Litejob
 
-## [0.1.0] - 2022-02-26
+## [0.1.0] - 2023-02-26
 
 - Initial release

data/README.md

@@ -16,12 +16,14 @@ litestack provides integration with popular libraries, including:
 - ActiveRecord
 - ActiveSupport::Cache
 - ActiveJob
+- ActionCable
 
 With litestack you only need to add a single gem to your app which would replace a host of other gems and services, for example, a typical Rails app using litestack will no longer need the following services:
 
 - Database Server (e.g. PostgreSQL, MySQL)
 - Cache Server (e.g. Redis, Memcached)
 - Job Processor (e.g. Sidekiq, Goodjob)
+- Pubsub Server (e.g. Redis, PostgreSQL)
 
 To make it even more efficient, litestack will detect the presence of Fiber based IO frameworks like Async (e.g. when you use the Falcon web server) or Polyphony. It will then switch its background workers for caches and queues to fibers (using the semantics of the existing framework). This is done transparently and will generally lead to lower CPU and memory utilization.
 
@@ -50,6 +52,7 @@ litestack currently offers three main components
 - litedb
 - litecache
 - litejob
+- litecable
 
 > ![litedb](https://github.com/oldmoe/litestack/blob/master/assets/litedb_logo_teal.png?raw=true)
 
@@ -113,6 +116,8 @@ litecache spawns a background thread for cleanup purposes. In case it detects th
 
 > ![litejob](https://github.com/oldmoe/litestack/blob/master/assets/litejob_logo_teal.png?raw=true)
 
+More info about Litejob can be found in the [litejob guide](https://github.com/oldmoe/litestack/wiki/Litejob-guide)
+
 litejob is a fast and very efficient job queue processor for Ruby applications. It builds on top of SQLite as well, which provides transactional guarantees, persistence and exceptional performance. 
 
 #### Direct litejob usage
@@ -120,7 +125,7 @@ litejob is a fast and very efficient job queue processor for Ruby applications.
 require 'litestack'
 # define your job class
 class MyJob
-  include ::litejob
+  include ::Litejob
       
   queue = :default
       
@@ -159,6 +164,28 @@ queues:
 
 The queues need to include a name and a priority (a number between 1 and 10) and can also optionally add the token "spawn", which means every job will run it its own concurrency context (thread or fiber)
 
+> ![litecable](https://github.com/oldmoe/litestack/blob/master/assets/litecable_logo_teal.png?raw=true)
+
+#### ActionCable
+
+This is a drop in replacement adapter for actioncable that replaces `async` and other production adapters (e.g. PostgreSQL, Redis). This adapter is currently only tested in local (inline) mode.
+
+Getting up and running with litecable requires configuring your cable.yaml file under the config/ directory
+
+cable.yaml
+```yaml
+development:
+  adapter: litecable
+
+test:
+  adapter: test
+
+staging:
+  adapter: litecable
+
+production:
+  adapter: litecable
+```
 
 ## Contributing
 

data/bench/bench_cache_raw.rb

@@ -8,7 +8,7 @@ require 'async/scheduler'
 Fiber.set_scheduler Async::Scheduler.new
 Fiber.scheduler.run
 
-require_relative '../lib/litestack'
+require_relative '../lib/litestack/litecache'
 #require 'litestack'
 
 cache = Litecache.new({path: '../db/cache.db'}) # default settings
@@ -25,6 +25,9 @@ count.times { keys << random_str(10) }
   end
   
   random_keys = keys.shuffle
+  
+  GC.compact
+  
   puts "Benchmarks for values of size #{size} bytes"
   puts "=========================================================="
   puts "== Writes =="
@@ -32,6 +35,13 @@ count.times { keys << random_str(10) }
     cache.set(keys[i], values[i])
   end
 
+  #bench("file writes", count) do |i|
+  #  f = File.open("../files/#{keys[i]}.data", 'w+')
+  #  f.write(values[i])
+  #  f.close
+  #end
+  
+
   bench("Redis writes", count) do |i|
     redis.set(keys[i], values[i])
   end
@@ -41,12 +51,18 @@ count.times { keys << random_str(10) }
     cache.get(random_keys[i])
   end
 
+  #bench("file reads", count) do |i|
+  #  data = File.read("../files/#{keys[i]}.data")
+  #end
+
   bench("Redis reads", count) do |i|
     redis.get(random_keys[i])
   end
   puts "=========================================================="
 
   values = []
+  
+  
 end
 
 
@@ -64,5 +80,5 @@ end
 cache.clear
 redis.flushdb
 
-sleep
+#sleep
 

data/bench/bench_jobs_rails.rb

@@ -1,22 +1,14 @@
 require './bench'
-require 'async/scheduler'
-
-#ActiveJob::Base.logger = Logger.new(IO::NULL)
 
+count = ARGV[0].to_i rescue 1000
+env = ARGV[1] || "t"
+delay = ARGV[2].to_f rescue 0
 
-Fiber.set_scheduler Async::Scheduler.new
 
-require_relative '../lib/active_job/queue_adapters/litejob_adapter'
-
-ActiveSupport::IsolatedExecutionState.isolation_level = :fiber
+#ActiveJob::Base.logger = Logger.new(IO::NULL)
 
 require './rails_job.rb'
 
-
-puts Litesupport.environment
-
-count = 1000
-
 RailsJob.queue_adapter = :sidekiq
 t = Time.now.to_f
 puts "Make sure sidekiq is started with -c ./rails_job.rb"     
@@ -26,13 +18,27 @@ end
 
 puts "Don't forget to check the sidekiq log for processing time conclusion"
 
+
+# Litejob bench
+###############
+
+if env == "a" # threaded
+  require 'async/scheduler'
+  Fiber.set_scheduler Async::Scheduler.new
+  ActiveSupport::IsolatedExecutionState.isolation_level = :fiber
+end
+
+require_relative '../lib/active_job/queue_adapters/litejob_adapter'
+puts Litesupport.environment
+
 RailsJob.queue_adapter = :litejob
 t = Time.now.to_f
 bench("enqueuing litejobs", count) do 
   RailsJob.perform_later(count, t)
 end
 
-Fiber.scheduler.run
-
+if env == "a" # threaded
+  Fiber.scheduler.run
+end
 
 sleep

data/bench/bench_jobs_raw.rb

@@ -24,8 +24,6 @@ if env == "t" # threaded
 elsif env == "a" # async
   require 'async/scheduler'
   Fiber.set_scheduler Async::Scheduler.new
-elsif env == "p" # polyphony
-  require 'polyphony'
 end
 
 require './uljob.rb'

data/lib/action_cable/subscription_adapter/litecable.rb

@@ -0,0 +1,36 @@
+# frozen_stringe_literal: true
+
+require_relative '../../litestack/litecable'
+
+module ActionCable
+  module SubscriptionAdapter
+    class Litecable < ::Litecable# :nodoc:
+
+      attr_reader :logger, :server
+      
+      prepend ChannelPrefix
+
+      DEFAULT_OPTIONS = {
+        config_path: "./config/litecable.yml",
+        path: "./db/cable.db",
+        sync: 0, # no need to sync at all
+        mmap_size: 16 * 1024 * 1024, # 16MB of memory hold hot messages
+        expire_after: 10, # remove messages older than 10 seconds
+        listen_interval: 0.005, # check new messages every 5 milliseconds
+        metrics: false
+      }
+
+      def initialize(server, logger=nil)
+        @server = server
+        @logger = server.logger
+        super(DEFAULT_OPTIONS.dup)
+      end
+            
+      def shutdown
+        close
+      end
+    
+    end
+  end
+end
+

data/lib/active_job/queue_adapters/litejob_adapter.rb

@@ -7,21 +7,16 @@ require "active_job"
 
 module ActiveJob
   module QueueAdapters
-    # == Ultralite adapter for Active Job
+    # == Litestack adapter for Active Job
     #
     #
     #   Rails.application.config.active_job.queue_adapter = :litejob
     class LitejobAdapter
-      
-      DEFAULT_OPTIONS = {
-        config_path: "./config/litejob.yml",
-        path: "../db/queue.db",
-        queues: [["default", 1, "spawn"]],
-        workers: 1
-      }      
-    
+          
       def initialize(options={})
-        Job.options = DEFAULT_OPTIONS.merge(options)
+        # we currently don't honour individual options per job class
+        # possible in the future?
+        # Job.options = DEFAULT_OPTIONS.merge(options)
       end
     
       def enqueue(job) # :nodoc:
@@ -36,6 +31,15 @@ module ActiveJob
 
       class Job # :nodoc:
         
+        DEFAULT_OPTIONS = {
+          config_path: "./config/litejob.yml",
+          path: "../db/queue.db",
+          queues: [["default", 1]],
+          logger: nil, # Rails performs its logging already
+          retries: 5, # It is recommended to stop retries at the Rails level
+          workers: 5
+        }      
+
         include ::Litejob
   
         def perform(job_data)

data/lib/litestack/litecable.rb

@@ -0,0 +1,138 @@
+# frozen_stringe_literal: true
+
+# all components should require the support module
+require_relative 'litesupport'
+require_relative 'litemetric'
+
+require 'base64'
+require 'oj'
+
+class Litecable
+
+  include Litesupport::Liteconnection
+  include Litemetric::Measurable
+
+
+  DEFAULT_OPTIONS = {
+    config_path: "./litecable.yml",
+    path: "./cable.db",
+    sync: 0,
+    mmap_size: 16 * 1024 * 1024, # 16MB
+    expire_after: 5, # remove messages older than 5 seconds
+    listen_interval: 0.05, # check new messages every 50 milliseconds
+    metrics: false
+  }
+  
+  def initialize(options = {})  
+    init(options)
+    @messages = []
+  end
+  
+  # broadcast a message to a specific channel
+  def broadcast(channel, payload=nil)
+    # group meesages and only do broadcast every 10 ms
+    #run_stmt(:publish, channel.to_s, Oj.dump(payload), @pid)
+    # but broadcast locally normally
+    @mutex.synchronize{ @messages << [channel.to_s, Oj.dump(payload)] }
+    local_broadcast(channel, payload) 
+  end
+  
+  # subscribe to a channel, optionally providing a success callback proc
+  def subscribe(channel, subscriber, success_callback = nil)
+    @mutex.synchronize do
+      @subscribers[channel] = {} unless @subscribers[channel]
+      @subscribers[channel][subscriber] = true
+    end
+  end
+  
+  # unsubscribe from a channel
+  def unsubscribe(channel, subscriber)
+    @mutex.synchronize do
+      @subscribers[channel].delete(subscriber) rescue nil
+    end
+  end
+
+  private 
+    
+  def local_broadcast(channel, payload=nil)
+    return unless @subscribers[channel]
+    subscribers = []
+    @mutex.synchronize do
+      subscribers = @subscribers[channel].keys
+    end
+    subscribers.each do |subscriber|
+      subscriber.call(payload)      
+    end
+  end  
+      
+  def setup
+    super # create connection
+    @pid = Process.pid
+    @subscribers = {}
+    @mutex = Litesupport::Mutex.new
+    @running = true
+    @listener = create_listener
+    @pruner = create_pruner
+    @broadcaster = create_broadcaster
+    @last_fetched_id = nil
+  end
+
+  def create_broadcaster
+    Litesupport.spawn do
+      while @running do
+        @mutex.synchronize do
+          if @messages.length > 0
+            run_sql("BEGIN IMMEDIATE")
+            while msg = @messages.shift
+              run_stmt(:publish, msg[0], msg[1], @pid)
+            end
+            run_sql("END")
+          end
+        end
+        sleep 0.02
+      end      
+    end
+  end
+
+  def create_pruner
+    Litesupport.spawn do
+      while @running do
+        run_stmt(:prune, @options[:expire_after])
+        sleep @options[:expire_after]
+      end      
+    end
+  end
+
+  def create_listener
+    Litesupport.spawn do
+      while @running do
+        @last_fetched_id ||= (run_stmt(:last_id)[0][0] || 0)
+        @logger.info @last_fetched_id
+        run_stmt(:fetch, @last_fetched_id, @pid).to_a.each do |msg|
+          @logger.info "RECEIVED #{msg}"
+          @last_fetched_id = msg[0]
+          local_broadcast(msg[1], Oj.load(msg[2])) 
+        end
+        sleep @options[:listen_interval]
+      end      
+    end
+  end
+
+  def create_connection
+    conn = super
+    conn.wal_autocheckpoint = 10000 
+    sql = YAML.load_file("#{__dir__}/litecable.sql.yml")
+    version = conn.get_first_value("PRAGMA user_version")
+    sql["schema"].each_pair do |v, obj| 
+      if v > version
+        conn.transaction do 
+          obj.each{|k, s| conn.execute(s)}
+          conn.user_version = v
+        end
+      end
+    end 
+    sql["stmts"].each { |k, v| conn.stmts[k.to_sym] = conn.prepare(v) }
+    conn
+  end
+  
+end

data/lib/litestack/litecable.sql.yml

@@ -0,0 +1,24 @@
+schema:
+  1:
+    create_table_messages: >
+      CREATE TABLE IF NOT EXISTS messages(
+        id INTEGER PRIMARY KEY autoincrement, 
+        channel TEXT NOT NULL, 
+        value TEXT NOT NULL, 
+        pid INTEGER, 
+        created_at INTEGER NOT NULL ON CONFLICT REPLACE DEFAULT(unixepoch())
+      );
+    create_index_messages_by_date: >
+      CREATE INDEX IF NOT EXISTS messages_by_date ON messages(created_at);  
+
+stmts:
+
+  publish: INSERT INTO messages(channel, value, pid) VALUES ($1, $2, $3)
+
+  last_id: SELECT max(id) FROM messages
+  
+  fetch: SELECT id, channel, value FROM messages WHERE id > $1 and pid != $2
+
+  prune: DELETE FROM messages WHERE created_at < (unixepoch() - $1)
+
+  check_prune: SELECT count(*) FROM messages WHERE created_at < (unixepoch() - $1)