litestack 0.4.3 → 0.4.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e2ac02213caf3525c40b676550a6daa24380f3012a5d4c8f8042cf39aa477593
-  data.tar.gz: 00d0f60e20bcba2cc22704d4e7159fe9b819920d313e7dc7d4e4e799420e95ca
+  metadata.gz: ddcc1a0b4d5a5547b3e3deec7ce5abec68e31ffce348913867368513276ac69a
+  data.tar.gz: 743a21e710d5351131a2cbe50a94de13653a034b88d08b82e470bb3aac2bdfbb
 SHA512:
-  metadata.gz: 7c9c726aecbfd99ad73d82c50c285178becefb4792ece0672b55dfede10a47f0563700a691398373f372ca829491c13e0332a81b9b075f9a0435ae55ac138a1c
-  data.tar.gz: b4a28527632a3065d0deb8eadb5c806d260edfd89aa9b3f00ebbc20a2aa63ebfb7df907e7bd0a500f9e368ecd91b8967789b1f056632ef5bc73502dbbe1524c9
+  metadata.gz: f54bdc732995e8083791c6b214ec6058bb439548ffdcbb14f33fbbeadbca870f4941cff558cdcbfa26212e53080534c12bfacfdfa2ba8ca689f380679ef0504a
+  data.tar.gz: f0a1fe1c95a592e69175e55ad212e351d5bca907b43c4215a4eca919d8dc2dfda68d80fa0ceb74da5909bf822adf0b2ea45e9108fcf0cd995a0a4b337a908ad0

data/.standard.yml

@@ -1,3 +1,5 @@
 ignore:
   - 'test/**/*':
-    - Style/GlobalVars
+    - Style/GlobalVars
+
+ruby_version: 3.0

data/BENCHMARKS.md

@@ -125,11 +125,11 @@ On average, Litecable is quite faster than the Redis based version and offers be
 
 > ![litesearch](https://github.com/oldmoe/litestack/blob/master/assets/litesearch_logo_teal.png?raw=true)
 
-Litesearch was benchmarked against Mielsearch, both using their respective ActiveRecord integrations. Mielsearch was running on the same machine as the benchmark script and was using the default configuration options. The dataset used for testing was the infamous Enron email corpus. Redisearch was not benchmarked due to the clients being not Rails 7.1 compatible (yet), will probably bench Redisearch when they are.
+Litesearch was benchmarked against Meilisearch, both using their respective ActiveRecord integrations. Meilisearch was running on the same machine as the benchmark script and was using the default configuration options. The dataset used for testing was the infamous Enron email corpus. Redisearch was not benchmarked due to the clients being not Rails 7.1 compatible (yet), will probably bench Redisearch when they are.
 
 ### Building the index
 
-||MieliSearch|Litesearch|
+||Meilisearch|Litesearch|
 |-:|-:|-:|
 |Time to insert 10K docs|265.42 seconds|29.06 seconds|
 |Inserted docs/second|38|344|
@@ -137,4 +137,4 @@ Litesearch was benchmarked against Mielsearch, both using their respective Activ
 |Searches/second|133|19608|
 |Index rebuild|0.822|0.626|
 
-We only limited the test to 10K documents becuause MieliSearch was taking a long time to index, so we decided to stop at a reasonable sample size. The search numbers for litesearch were double checked, event against a 100K document set and they remained virtually the same. It is clear that litesearch is much faster than MieliSearch in both indexing and searching, this could be partially attributed to litesearch being a simpler text search engine, but still, the difference is huge! For rebuilding the index though, Litesearch is not that much faster than Mielisearch.
+We only limited the test to 10K documents becuause Meilisearch was taking a long time to index, so we decided to stop at a reasonable sample size. The search numbers for litesearch were double checked, event against a 100K document set and they remained virtually the same. It is clear that litesearch is much faster than Meilisearch in both indexing and searching, this could be partially attributed to litesearch being a simpler text search engine, but still, the difference is huge! For rebuilding the index though, Litesearch is not that much faster than Meilisearch.

data/CAVEATS.md

@@ -0,0 +1,20 @@
+# CAVEATS
+
+This is a document that lists some caveats that you need to consider when dealing with Litestack and SQLite applications
+
+## Single host (server) only
+
+By design, SQLite can be accessed only (in a performant and safe way) from a single host machine. That means that you cannot have things like auto-scaling of app servers since there is but a single one of them. This app server can be a huge one, with many processes, but only one app server at a time.
+
+## Containerization becomes a bit tricky
+
+Container technology was originally designed for stateless applications. Based on this, one of the core aspects of containers is that they are immutable, deploying an app to container means destroying one container and creating another. With statefull applications, like database servers and similarly SQLite based applications, this will potentially mean a down time will be perceived, which in some approaches is mitigate but not perfectly (yet). Expect some quirks when dealing with SQLite and containers.
+
+## Single writer / multi reader
+
+In its default configuration, Litestack allows one writer at a time to write to the database file, while simultaneously allowing any number of readers to access the database. Generally SQLite is pretty fast in writes, but you should be aware that a long running write operation will prevent any other writers from proceeding, for example creating an index on a very large table will stop other writers until the index creation is finished. There is no concurrent index creation facility in SQLite, yet.
+
+## Litestack does not release the GVL
+
+This also applies to the Ruby SQLite3 gem, when it is performing a query, it will not allow any other threads in the Ruby process to resume until it returns. Care should be taken when writing low latency multi-threaded applications with Litestack, it is generally recommended to use fibers over threads in that case, since everything is serialized anyway it makes sense to avoid the ovreheads of mult-threaded environments
+

data/CHANGELOG.md

@@ -1,15 +1,30 @@
 ## [Unreleased]
 
+- [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.4...master)
+
+## [0.4.4] - 2024-05-21
+
+- [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.3...v0.4.4)
+- Fix a table not defined bug for Litesearch
+- Add conditional mapping of external fields to Litesearch (backed index)
+- As a consequence of the above, support AR polymorphic associations
+- As a consequence of the above, support indexing and searching for ActionText attributes
+- Remove 'hanami-router' as a dependency, rely on vanilla Rack for Liteboard 
+- All configuratin .yml files now support ERB rendering
+- Fix adapter registration with Rails 7.2
+- Ground work for more durable job persistence in case of a process crash
+- Many minor fixes
+
 ## [0.4.3] - 2024-02-15
 
-- [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.1...master)
+- [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.2...v0.4.3)
 - Add "sequel" as a development dependency
 - Diff links in CHANGELOG (thanks Weston Ganger)
-- Fix deamonize type in liteboard (thanks Julian Rubisch)
+- Fix daemonize type in liteboard (thanks Julian Rubisch)
 - Better Litecache schema (streamlined numeric value support)
 - Support for set_multi and get_multi in Litecache (read_multi and write_multi support for Rails Cache store)
 - More tests written for Litecache and Rails Litecache store
-- Experimenting with removing the Rails LocalCache as it doesn't show enough improvement in perfromance to compensate for the memory overhead 
+- Experimenting with removing the Rails LocalCache as it doesn't show enough improvement in performance to compensate for the memory overhead
 - Switch Litecache to a FIFO eviction model vs LRU (thanks Julian Rubisch and Stephen Margheim)
 
 ## [0.4.2] - 2023-11-11
@@ -20,7 +35,7 @@
 - Fix Litesearch tests
 - Suppress chatty Litejob exit detector when there are no jobs in flight
 - Tidy up the test folder
-- [#41](https://github.com/oldmoe/litestack/pull/41) - Fix bug in Litecable where the `connected` event was not getting propogated
+- [#41](https://github.com/oldmoe/litestack/pull/41) - Fix bug in Litecable where the `connected` event was not getting propagated
 - Add Litemetric and Liteboard info to README.doc
 - Fix the testing rake task
 

data/FILESYSTEMS.md

@@ -0,0 +1,55 @@
+# Filesystems & Litestack
+
+In the containerized world we live in, many layers of the hardware/software stacks are far abstracted that we no longer know they exist. For the filesystem enthusiasts out there this is a quick overview of how Litestack (and hence SQLite) can benefit from different filesystem
+
+## XFS
+
+A very stable and trusted filesystem with excellent performance characteristics
+
+- Fast reads / writes
+
+## EXT4
+
+Another stable and performant filesystem
+
+- Fast reads / writes
+
+## F2FS
+
+Specially built for solid state storage, has an atomic write mode that is supported by SQLite
+
+- Fast reads
+- Reasonably fast durable writes in synchronous mode
+- Compression (not very useful)
+
+## ZFS
+
+Copy-on-write filesystem with a nifty set of features, very fast snapshotting but only for the FS as a whole, can send incremental snapshots to another host for backup. Naturally a lot more pages are written on write operations thus it is a bit slower in writes.
+
+- Fast reads
+- Slower writes
+- FS Snapshotting with send/recv
+- fast device cache
+- Compression
+
+## Btrfs
+
+Another CoW filesystem that delivers snapshotting and incremental send/recv at a much granular level.
+
+- Fast reads
+- Slower writes
+- Fast copies (free backups)
+- Sub volume Snapshotting with send/recv
+- Compression
+
+## Bcachefs
+
+A new CoW filesystem built on the foundations of the bcache module. Improving rapidly but still lacks a few of the common CoW fs features
+
+- Fast reads
+- Slower writes
+- fast device cache
+
+
+
+

data/Gemfile

@@ -4,3 +4,5 @@ source "https://rubygems.org"
 
 # Specify your gem's dependencies in litestack.gemspec
 gemspec
+
+# gem "rails"

data/README.md

@@ -59,7 +59,7 @@ litedb is a wrapper around SQLite3, offering a better default configuration that
 
 #### Direct litedb usage
 
-litedb can be used exactly as the SQLite3 gem, since litedb iherits from SQLite3
+litedb can be used exactly as the SQLite3 gem, since litedb inherits from SQLite3
 
 ```ruby
 require 'litestack'
@@ -209,7 +209,7 @@ idx.search('kamal')
 idx.search('subject: awa') 
 ```
 
-Litesearch integrates tightly with ActiveRecord ans Sequel, here are integration examples
+Litesearch integrates tightly with ActiveRecord and Sequel, here are integration examples
 
 #### ActiveRecord
 

data/ROADMAP.md

@@ -21,25 +21,25 @@ This is a list of functional/non-functional features that litestack targets to i
   - [ ] Automated testing matrix
   - [ ] Automated gem builds
 - [ ] Litedb improvements
-  - [ ] ActiveRecord Sqlite3 Adapter compatibility mode    
+  - [ ] ActiveRecord Sqlite3 Adapter compatibility mode
   - [ ] Sequel Sqlite3 Adapter compatibility mode
-  - [ ] Extension bundling/loading    
+  - [ ] Extension bundling/loading
 - [ ] Liteiob improvements
   - [ ] Persist jobs even during execution
   - [ ] Zombie job detection
-  - [ ] Better process exit handling 
+  - [ ] Better process exit handling
   - [ ] Faster job dispatch for the no delay case
 - [ ] Database maintenance scripts
   - [ ] Online backup
   - [ ] Restore
   - [ ] Litestream integration
-  - [ ] CoW filesystems specific backup path
+  - [ ] CoW filesystem-specific backup path
 - [ ] Zero downtime deployment scripts
   - [ ] Rails
-  - [ ] Genereic Rack Applications
+  - [ ] Generic Rack Applications
 - [ ] Litemetric improvements
   - [ ] Rails performance module
   - [ ] Ruby Memory/GC module
   - [ ] Restore generic module
   - [ ] Better HTML/CSS
-- [ ] Kredis replacement implementation?    
+- [ ] Kredis replacement implementation?

data/bench/bench.rb

@@ -1,5 +1,7 @@
 require "sqlite3"
 
+#RubyVM::YJIT.enable
+
 def bench(msg, iterations = 1000)
   GC.start
   # GC.compact

data/bench/bench_cache_rails.rb

@@ -1,6 +1,6 @@
 require "active_support"
 require_relative "../lib/litestack"
-require_relative "./bench"
+require_relative "bench"
 
 cache = ActiveSupport::Cache::Litecache.new
 
@@ -12,10 +12,8 @@ redis = ActiveSupport::Cache.lookup_store(:redis_cache_store, {})
 values = []
 keys = []
 count = 1000
-payload_sizes = [10, 100, 1000, 10000]
-#payload_sizes = [100]
 
-payload_sizes.each do |size|
+[10, 100, 1000, 10000].each do |size|
   count.times do
     keys << random_str(10)
     values << random_str(size)
@@ -34,17 +32,17 @@ payload_sizes.each do |size|
   end
 
   puts "== Multi Writes =="
-  bench("litecache multi-writes", count/5) do |i|
-    idx = i * 5 
-    payload = {} 
-    5.times {|j| payload[keys[idx + j]] = values[idx + j] }
+  bench("litecache multi-writes", count / 5) do |i|
+    idx = i * 5
+    payload = {}
+    5.times { |j| payload[keys[idx + j]] = values[idx + j] }
     cache.write_multi(payload)
   end
 
-  bench("Redis multi-writes", count/5) do |i|
-    idx = i * 5 
-    payload = {} 
-    5.times {|j| payload[keys[idx + j]] = values[idx + j] }
+  bench("Redis multi-writes", count / 5) do |i|
+    idx = i * 5
+    payload = {}
+    5.times { |j| payload[keys[idx + j]] = values[idx + j] }
     redis.write_multi(payload)
   end
 
@@ -58,21 +56,20 @@ payload_sizes.each do |size|
   end
 
   puts "== Multi Reads =="
-  bench("litecache multi-reads", count/5) do |i|
+  bench("litecache multi-reads", count / 5) do |i|
     idx = i * 5
     payload = []
-    5.times {|j| payload << random_keys[idx+j]}
+    5.times { |j| payload << random_keys[idx + j] }
     cache.read_multi(*payload)
   end
 
-  bench("Redis multi-reads", count/5) do |i|
+  bench("Redis multi-reads", count / 5) do |i|
     idx = i * 5
     payload = []
-    5.times {|j| payload << random_keys[idx+j]}
+    5.times { |j| payload << random_keys[idx + j] }
     redis.read_multi(*payload)
   end
 
-
   puts "=========================================================="
 
   keys = []

data/bench/bench_cache_raw.rb

@@ -1,6 +1,6 @@
 require "redis"
 require "sqlite3"
-require_relative "./bench"
+require_relative "bench"
 
 # require 'polyphony'
 require "async/scheduler"
@@ -11,7 +11,7 @@ Fiber.scheduler.run
 require_relative "../lib/litestack/litecache"
 # require 'litestack'
 
-cache = Litecache.new #({path: "../db/cache.db"}) # default settings
+cache = Litecache.new # ({path: "../db/cache.db"}) # default settings
 redis = Redis.new # default settings
 
 values = []
@@ -40,17 +40,20 @@ count.times { keys << random_str(10) }
   end
 
   puts "== Multi Writes =="
-  bench("litecache multi-writes", count/5) do |i|
-    idx = i * 5 
-    payload = {} 
-    5.times {|j| payload[keys[idx + j]] = values[idx + j] }
+  bench("litecache multi-writes", count / 5) do |i|
+    idx = i * 5
+    payload = {}
+    5.times { |j| payload[keys[idx + j]] = values[idx + j] }
     cache.set_multi(payload)
-  end 
+  end
 
-  bench("Redis multi-writes", count/5) do |i|
-    idx = i * 5 
-    payload = [] 
-    5.times {|j| payload << keys[idx + j]; payload <<  values[idx + j]}
+  bench("Redis multi-writes", count / 5) do |i|
+    idx = i * 5
+    payload = []
+    5.times { |j|
+      payload << keys[idx + j]
+      payload << values[idx + j]
+    }
     redis.mset(*payload)
   end
 
@@ -64,17 +67,17 @@ count.times { keys << random_str(10) }
   end
 
   puts "== Multi Reads =="
-  bench("litecache multi-reads", count/5) do |i|
+  bench("litecache multi-reads", count / 5) do |i|
     idx = i * 5
     payload = []
-    5.times {|j| payload << random_keys[idx+j]}
+    5.times { |j| payload << random_keys[idx + j] }
     cache.get_multi(*payload)
   end
 
-  bench("Redis multi-reads", count/5) do |i|
+  bench("Redis multi-reads", count / 5) do |i|
     idx = i * 5
     payload = []
-    5.times {|j| payload << random_keys[idx+j]}
+    5.times { |j| payload << random_keys[idx + j] }
     redis.mget(*payload)
   end
 

data/bench/bench_jobs_rails.rb

@@ -19,9 +19,9 @@ require "./rails_job"
 RailsJob.queue_adapter = :sidekiq
 t = Time.now.to_f
 puts "Make sure sidekiq is started with -c ./rails_job.rb"
-bench("enqueuing sidekiq jobs", count) do
-  RailsJob.perform_later(count, t)
-end
+#bench("enqueuing sidekiq jobs", count) do
+#  RailsJob.perform_later(count, t)
+#end
 
 puts "Don't forget to check the sidekiq log for processing time conclusion"
 

data/bench/bench_jobs_raw.rb

@@ -18,9 +18,9 @@ require "./skjob"
 
 t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
 puts "make sure sidekiq is started with skjob.rb as the job"
-bench("enqueuing sidekiq jobs", count) do |i|
-  SidekiqJob.perform_async(count, t, delay)
-end
+#bench("enqueuing sidekiq jobs", count) do |i|
+#  SidekiqJob.perform_async(count, t, delay)
+#end
 
 puts "Don't forget to check the sidekiq log for processing time conclusion"
 

data/bin/liteboard

@@ -2,20 +2,23 @@
 # frozen_string_literal: true
 
 require 'optparse'
+require 'erb'
+require 'yaml'
+require 'rackup'
 require_relative '../lib/litestack/liteboard/liteboard'
 DEFAULTS = {
   config_path: Litemetric::DEFAULT_OPTIONS[:config_path],
   path: Litemetric::DEFAULT_OPTIONS[:path]
 }
 
-options = { 
+options = {
   config_path: nil,
   path: nil,
-  deamonize: false,
+  daemonize: false,
   Port: 9292,
   Host: 'localhost',
   environment: 'production',
-  pid: 'tmp/pids/liteboard.pid',
+  pid: './liteboard.pid',
   quiet: false
 }
 
@@ -26,9 +29,9 @@ OptionParser.new do |parser|
   parser.on("-s", "--server SERVER", "use SERVER (e.g. puma/falcon/iodine)") { |v| options[:port] = v }
   parser.on("-H", "--host HOST", "listen on HOST (default: #{options[:Host]})") { |v| options[:Host] = v }
   parser.on("-p", "--port PORT", "use PORT (default: #{options[:Port]})") { |v| options[:Port] = v.to_i rescue options[:Port] }
-  parser.on("-D", "--daemonize", "run in the background") { |v| options[:daemonize] = true }  
-  parser.on("-E", "--env ENVIRONMENT", "which environment to use (default: #{options[:environment]})") { |v| options[:environment] = v }  
-  parser.on("-q", "--quiet", "turn off logging") { |v| options[:quiet] = true }  
+  parser.on("-D", "--daemonize", "run in the background") { |v| options[:daemonize] = true }
+  parser.on("-E", "--env ENVIRONMENT", "which environment to use (default: #{options[:environment]})") { |v| options[:environment] = v }
+  parser.on("-q", "--quiet", "turn off logging") { |v| options[:quiet] = true }
   parser.on("-h", "--help", "print this message") do
     puts parser
     exit
@@ -36,40 +39,39 @@ OptionParser.new do |parser|
 end.parse!
 
 def check_database(path)
-  unless File.exist?(path) 
+  unless File.exist?(path)
     puts "liteboard: missing database file, please ensure the db path is correct"
     puts "liteboard: exiting"
     exit
   end
 end
 
-check_database(options[:path]) if options[:path] 
+check_database(options[:path]) if options[:path]
 
 # if there is a config file then we need to check it
 config = nil
 if options[:config_path]
   begin
-    config = Yaml.load_file(options[:config_path]) 
+    config = YAML.load(ERB.new(File.read(options[:config_path])).result)
   rescue
     puts "liteboard: missing or bad config file, please ensure the config file path is correct"
     puts "liteboard: exiting"
     exit
-  end 
+  end
 else # no config path! use the default
-  config = Yaml.load_file(DEFAULTS[:config_path]) rescue nil 
+  config = YAML.load(ERB.new(File.read(DEFAULTS[:config_path])).result) rescue nil 
 end
 
 if config
   if options[:path].nil?
     path = config['path'] || config[options[:environment]]['path']
     options[:path] = path
-  end  
+  end
 end
 
 # if still no path we assume a default db path
 options[:path] = DEFAULTS[:path] if options[:path].nil?
 
-
 # check the validity of the path before starting the server
 check_database(options[:path])
 Litemetric.options = options
@@ -79,4 +81,4 @@ options[:app] = Liteboard.app
 require_relative '../lib/litestack'
 puts "Starting Liteboard version #{Litestack::VERSION}"
 
-Rack::Server.start(options)
+Rackup::Server.start(options)

data/gemfiles/rails70.gemfile

@@ -0,0 +1,5 @@
+source "https://rubygems.org"
+
+gemspec path: "../"
+
+gem "rails", "~> 7.0"

data/gemfiles/rails71.gemfile

@@ -0,0 +1,5 @@
+source "https://rubygems.org"
+
+gemspec path: "../"
+
+gem "rails", "~> 7.1"