litestack 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 025c78f9b04863bfce9f09dae2181b735187d0ff8852ff0a474ce59fc4f3b7fc
+  data.tar.gz: f4d950cbfef0e9b0e6642196b2efa0953dd8d988de54168ff9934bb257d6c729
+SHA512:
+  metadata.gz: 29bb82387bac7b66cf2e335fc072ff79613de006ead064cbbf01746b1c57687b47c5bf875af7e173273ca8ba961dfc561595dfb9fd1527f92989444e294be4ea
+  data.tar.gz: ae523a47ebeed583670541bec755c39559e6a24230aaf7fa074e262a7410e64dd6e5c4f82af9b9d40dc8ef39748bac6bc38656a00a2a0066957a10af02573a9e

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## [Unreleased]
+
+## [0.1.0] - 2022-02-26
+
+- Initial release

data/Gemfile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in ultralite.gemspec
+gemspec
+
+

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2022 TODO: Write your name
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,166 @@
+![litestack](https://github.com/oldmoe/litestack/blob/master/assets/litestack_logo_teal_large.png?raw=true)
+
+
+litestack is a revolutionary gem for Ruby and Ruby on Rails that provides an all-in-one solution for web application development. It exploits the power and embeddedness of SQLite to include a full-fledged SQL database, a fast cache, a robust job queue, and a simple yet performant full-text search all in a single package.
+
+Compared to conventional approaches that require separate servers and databases, LiteStack offers superior performance, efficiency, ease of use, and cost savings. Its embedded database and cache reduce memory and CPU usage, while its simple interface streamlines the development process. Overall, LiteStack sets a new standard for web application development and is an excellent choice for those who demand speed, efficiency, and simplicity.
+
+litestack provides integration with popular libraries, including:
+
+- Rack
+- Sequel
+- Rails
+- ActiveRecord
+- ActiveSupport::Cache
+- ActiveJob
+
+With litestack you only need to add a single gem to your app which would replace a host of other gems and services, for example, a typical Rails app using litestack will no longer need the following services:
+
+- PostgreSQL
+- Redis
+- Sidekiq
+
+To make it even more efficient, litestack will detect the presence of Fiber based IO frameworks like Async (e.g. when you use the Falcon web server) or Polyphony. It will then switch its background workers for caches and queues to fibers (using the semantics of the existing framework). This is done transparently and will generally lead to lower CPU and memory utilization.
+
+Litestack is still pretty young and under heavy development, but you are welcome to give it a try today!.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'litestack'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install litestack
+
+## Usage
+
+litestack currently offers three main components
+
+- litedb
+- litecache
+- litejob
+
+> ![litedb](https://github.com/oldmoe/litestack/blob/master/assets/litedb_logo_teal.png?raw=true)
+
+litedb is a wrapper around SQLite3, offering a better default configuration that is tuned for concurrency and performance. Out of the box, litedb works seamlessly between multiple processes without database locking errors. lite db can be used in multiple ways, including:
+
+#### Direct litedb usage
+
+litedb can be used exactly as the SQLite3 gem, since litedb iherits from SQLite3
+
+```ruby
+require 'litestack'
+db = Litedb.new(path_to_db)
+db.execute("create table users(id integer primary key, name text)")
+db.execute("insert into users(name) values (?)", "Hamada")
+db.get_first_value("select count(*) from users") # => 1
+```
+
+#### ActiveRecord
+
+litesd provides tight Rails/ActiveRecord integration and can be configured as follows
+
+In database.yml
+
+```yaml
+adapter: litedb
+# normal sqlite3 configuration follows
+```
+
+#### Sequel
+
+litedb offers integration with the Sequel database toolkit and can be configured as follows
+
+```ruby
+DB = Sequel.conncet("litedb://path_to_db_file")    
+```
+
+
+> ![litecache](https://github.com/oldmoe/litestack/blob/master/assets/litecache_logo_teal.png?raw=true)
+
+litecache is a high speed, low overhead caching library that uses SQLite as its backend. litecache can be accessed from multiple processes on the same machine seamlessly. It also has features like key expiry, LRU based eviction and increment/decrement of integer values.
+
+#### Direct litecache usage
+
+```ruby
+require 'litestack'
+cache = Litecache.new(path: "path_to_file")
+cache.set("key", "value")
+cache.get("key") #=> "value"
+```
+
+#### ActiveResource::Cache
+
+In your desired environment file (e.g. production.rb)
+
+```ruby
+config.cache_store = :litecache, {path: './path/to/your/cache/file'}
+```
+This provides a transparent integration that uses the Rails caching interface 
+
+litecache spawns a background thread for cleanup purposes. In case it detects that the current environment has *Fiber::Scheduler* or *Polyphony* loaded it will spawn a fiber instead, saving on both memory and CPU cycles.
+
+> ![litejob](https://github.com/oldmoe/litestack/blob/master/assets/litejob_logo_teal.png?raw=true)
+
+litejob is a fast and very efficient job queue processor for Ruby applications. It builds on top of SQLite as well, which provides transactional guarantees, persistence and exceptional performance. 
+
+#### Direct litejob usage
+```ruby
+require 'litestack'
+# define your job class
+class MyJob
+  include ::litejob
+      
+  queue = :default
+      
+  # must implement perform, with any number of params
+  def perform(params)
+    # do stuff
+  end
+end
+    
+#schedule a job asynchronusly
+MyJob.perform_async(params)
+    
+#schedule a job at a certain time
+MyJob.perform_at(time, params)
+    
+#schedule a job after a certain delay
+MyJob.perform_after(delay, params)
+```
+
+#### ActiveJob
+
+In your desired environment file (e.g. production.rb)
+
+```ruby
+config.active_job.queue_adapter = :litejob
+```
+#### Configuration file
+You can add more configuration in litejob.yml (or config/litejob.yml if you are integrating with Rails)
+
+```yaml
+queues:
+    - [default 1]
+    - [urgent 5]
+    - [critical 10 "spawn"]
+```
+
+The queues need to include a name and a priority (a number between 1 and 10) and can also optionally add the token "spawn", which means every job will run it its own concurrency context (thread or fiber)
+
+
+## Contributing
+
+Bug reports aree welcome on GitHub at https://github.com/oldmoe/litestack. Please note that this is not an open contribution project and that we don't accept pull requests.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/test_*.rb"]
+end
+
+task default: :test

data/WHYLITESTACK.md

@@ -0,0 +1,26 @@
+# Why Litestack?
+
+If you're developing a Ruby web application, you may be wondering which database, caching, and job queueing solution is right for you. While there are many options out there, Litestack stands out as a top choice for its performance benefits, ease of setup and administration, resource efficiency, and potential cost savings.
+
+## Standing on the shoulder of a (figurative) giant!
+At its core, Litestack is built on top of SQLite, a highly regarded open-source relational database engine. This means that Litestack is highly efficient and resource-friendly, making it an excellent choice for small to medium-sized web applications. Additionally, because SQLite is a file-based database, it is incredibly easy to set up and manage, requiring no separate server installation or configuration.
+
+## Performance and Effeciency
+One of the most significant advantages of Litestack is its performance benefits. SQLite has a small memory footprint and is highly optimized, meaning that it can deliver fast and reliable database access. Additionally, Litestack's job queueing and caching functionality provide additional performance benefits, enabling you to execute tasks asynchronously and store frequently accessed data in memory for faster access.
+
+## Dead simple Rails integration
+In addition to its performance benefits and ease of administration, Litestack also offers a remarkably simple setup process for Ruby on Rails applications. With just a single gem install and three lines of configuration changes, you can have a fully functional database, caching, and job queueing solution up and running in no time.
+
+## Simple by design
+This ease of setup and configuration is due to Litestack's design philosophy, which prioritizes simplicity and ease of use. Rather than requiring developers to make significant code changes or configure complex infrastructure, Litestack offers a turnkey solution that can be set up with minimal effort.
+
+This simplicity is a significant advantage for developers, as it allows them to focus on writing code and delivering features rather than dealing with the complexities of database and infrastructure management. By simplifying the setup process, Litestack can help reduce the time and cost required to get your application up and running, enabling you to focus on delivering value to your users.
+
+## Simple can be cutting edge
+Another benefit of Litestack is its deep integration with state of the art Ruby IO libraries like Async and Polyphony. This allows for even greater performance improvements and improved efficiency, as these libraries can help manage concurrency and parallelism in your web application.
+
+## DevOps? Think DevNops!
+Because Litestack is easy to set up and maintain, it can potentially reduce development costs and time to market. With fewer dependencies and simpler configuration requirements, developers can focus on writing code and delivering features rather than worrying about complex infrastructure and administration.
+
+## Conclusion
+In conclusion, if you're looking for a reliable, performant, and easy-to-use database, caching, and job queueing solution for your Ruby or Rails web application, Litestack is an excellent choice. With its built-in integration with SQLite and major IO libraries, Litestack can provide significant performance benefits while also reducing development costs and time to market.

data/bench/bench.rb

@@ -0,0 +1,23 @@
+require 'sqlite3'
+
+def bench(msg, iterations=1000)
+  GC.start
+  GC.compact
+  print "Starting #{iterations} iterations of #{msg} ... "
+  t1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+  iterations.times do |i|
+    yield i
+  end
+  t2 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+  time = ((t2 - t1)*1000).to_i.to_f / 1000 rescue 0
+  ips = ((iterations/(t2-t1))*100).to_i.to_f / 100 rescue "infinity?"
+  #{m: msg, t: time, ips: iteratinos/time, i: iterations}
+  puts "finished in #{time} seconds (#{ips} ips)"
+end
+
+@db = SQLite3::Database.new(":memory:") # sqlite database for fast random string generation
+
+def random_str(size)
+  @db.get_first_value("select hex(randomblob(?))", size)
+end 
+

data/bench/bench_cache_rails.rb

@@ -0,0 +1,67 @@
+require 'active_support'
+require_relative '../lib/litestack'
+require_relative './bench'
+
+cache = ActiveSupport::Cache::Litecache.new({path: '../db/rails_cache.db'})
+
+#can only use the lookup method when the gem is installed
+#cache = ActiveSupport::Cache.lookup_store(:litecache, {path: '../db/rails_cache.db'})
+
+redis = ActiveSupport::Cache.lookup_store(:redis_cache_store, {})
+
+values = []
+keys = []
+count = 1000
+
+[10, 100, 1000, 10000].each do |size|
+  count.times do
+    keys << random_str(10) 
+    values << random_str(size) 
+  end
+  
+  random_keys = keys.shuffle
+  puts "Benchmarks for values of size #{size} bytes"
+  puts "=========================================================="
+  puts "== Writes =="
+  bench("litecache writes", count) do |i|
+    cache.write(keys[i], values[i])
+  end
+
+  bench("Redis writes", count) do |i|
+    redis.write(keys[i], values[i])
+  end
+
+  puts "== Reads =="
+  bench("litecache reads", count) do |i|
+    cache.read(random_keys[i])
+  end
+
+  bench("Redis reads", count) do |i|
+    redis.read(random_keys[i])
+  end
+  puts "=========================================================="
+
+
+  keys = []
+  values = []
+end
+
+
+cache.write("somekey", 1, raw: true)
+
+redis.write("somekey", 1, raw: true)
+
+puts "Benchmarks for incrementing integer values"
+puts "=========================================================="
+
+bench("litecache increment", count) do
+  cache.increment("somekey", 1, raw: true)
+end
+
+bench("Redis increment", count) do
+  redis.increment("somekey", 1, raw: true )
+end
+
+cache.clear
+redis.clear
+

data/bench/bench_cache_raw.rb

@@ -0,0 +1,68 @@
+require 'redis'
+require 'sqlite3'
+require_relative './bench'
+
+#require 'polyphony'
+require 'async/scheduler'
+
+Fiber.set_scheduler Async::Scheduler.new
+Fiber.scheduler.run
+
+require_relative '../lib/litestack'
+
+
+cache = Litecache.new({path: '../db/cache.db'}) # default settings
+redis = Redis.new # default settings
+
+values = []
+keys = []
+count = 1000
+count.times { keys << random_str(10) }
+
+[10, 100, 1000, 10000].each do |size|
+  count.times do
+    values << random_str(size) 
+  end
+  
+  random_keys = keys.shuffle
+  puts "Benchmarks for values of size #{size} bytes"
+  puts "=========================================================="
+  puts "== Writes =="
+  bench("litecache writes", count) do |i|
+    cache.set(keys[i], values[i])
+  end
+
+  bench("Redis writes", count) do |i|
+    redis.set(keys[i], values[i])
+  end
+
+  puts "== Reads =="
+  bench("litecache reads", count) do |i|
+    cache.get(random_keys[i])
+  end
+
+  bench("Redis reads", count) do |i|
+    redis.get(random_keys[i])
+  end
+  puts "=========================================================="
+
+  values = []
+end
+
+
+cache.set("somekey", 1)
+redis.set("somekey", 1)
+
+bench("litecache increment") do
+  cache.increment("somekey", 1)
+end
+
+bench("Redis increment") do
+  redis.incr("somekey")
+end
+
+cache.clear
+redis.flushdb
+
+sleep
+

data/bench/bench_jobs_rails.rb

@@ -0,0 +1,38 @@
+require './bench'
+require 'async/scheduler'
+
+#ActiveJob::Base.logger = Logger.new(IO::NULL)
+
+
+Fiber.set_scheduler Async::Scheduler.new
+
+require_relative '../lib/active_job/queue_adapters/litejob_adapter'
+
+ActiveSupport::IsolatedExecutionState.isolation_level = :fiber
+
+require './rails_job.rb'
+
+
+puts Litesupport.environment
+
+count = 1000
+
+RailsJob.queue_adapter = :sidekiq
+t = Time.now.to_f
+puts "Make sure sidekiq is started with -c ./rails_job.rb"     
+bench("enqueuing sidekiq jobs", count) do 
+  RailsJob.perform_later(count, t)
+end
+
+puts "Don't forget to check the sidekiq log for processing time conclusion"
+
+RailsJob.queue_adapter = :litejob
+t = Time.now.to_f
+bench("enqueuing litejobs", count) do 
+  RailsJob.perform_later(count, t)
+end
+
+Fiber.scheduler.run
+
+
+sleep

data/bench/bench_jobs_raw.rb

@@ -0,0 +1,27 @@
+#require 'polyphony'
+require 'async/scheduler'
+require './bench'
+require './skjob.rb'
+require './uljob.rb'
+
+Fiber.set_scheduler Async::Scheduler.new
+
+count = 1000
+
+t = Time.now.to_f
+# make sure sidekiq is started with skjob.rb as the job-     
+bench("enqueuing sidekiq jobs", count) do |i|
+  SidekiqJob.perform_async(count, t)
+end
+
+puts "Don't forget to check the sidekiq log for processing time conclusion"
+
+t = Time.now.to_f
+bench("enqueuing litejobs", count) do |i|
+  MyJob.perform_async(count, t)
+end
+
+Fiber.scheduler.run
+
+sleep
+

data/bench/bench_queue.rb

@@ -0,0 +1,16 @@
+require './bench'
+require_relative '../lib/litestack'
+
+count = 1000
+
+q = Litequeue.new({path: '../db/queue.db' })
+
+bench("enqueue", count) do |i|
+  q.push i.to_s
+end
+
+bench("dequeue", count) do |i|
+  q.pop  
+end
+
+

data/bench/bench_rails.rb

@@ -0,0 +1,81 @@
+require 'ultralite'
+require 'active_support'
+require './bench'
+
+cache = ActiveSupport::Cache.lookup_store(:ultralite_cache_store, {})
+mem = ActiveSupport::Cache.lookup_store(:ultralite_cache_store, {path: ":memory:"})
+redis = ActiveSupport::Cache.lookup_store(:redis_cache_store, {})
+
+values = []
+keys = []
+count = 1000
+
+[10, 100, 1000, 10000].each do |size|
+  count.times do
+    keys << random_str(10) 
+    values << random_str(size) 
+  end
+  
+  random_keys = keys.shuffle
+  puts "Benchmarks for values of size #{size} bytes"
+  puts "=========================================================="
+  puts "== Writes =="
+  bench("Ultralite cache writes", count) do |i|
+    cache.write(keys[i], values[i])
+  end
+
+  bench("Ultralite memory cache writes", count) do |i|
+    mem.write(keys[i], values[i])
+  end
+
+  bench("Redis writes", count) do |i|
+    redis.write(keys[i], values[i])
+  end
+
+  puts "== Reads =="
+  bench("Ultralite cache reads", count) do |i|
+    cache.read(random_keys[i])
+  end
+
+  bench("Ultralite memory cache reads", count) do |i|
+    mem.read(random_keys[i])
+  end
+
+  bench("Redis reads", count) do |i|
+    redis.read(random_keys[i])
+  end
+  puts "=========================================================="
+
+
+  keys = []
+  values = []
+end
+
+
+cache.write("somekey", 1, raw: true)
+#puts cache.read("somekey", raw: true)
+
+mem.write("somekey", 1, raw: true)
+#puts mem.read("somekey", raw: true)
+
+redis.write("somekey", 1, raw: true)
+#puts redis.read("somekey", raw: true)
+
+puts "Benchmarks for incrementing integer values"
+puts "=========================================================="
+
+bench("Ultralite cache increment", count) do
+  cache.increment("somekey", 1, raw: true)
+end
+
+bench("Ultralite memory cache increment", count) do
+  mem.increment("somekey", 1, raw: true)
+end
+
+bench("Redis increment", count) do
+  redis.increment("somekey", 1, raw: true )
+end
+
+cache.clear
+redis.clear
+

data/bench/bench_raw.rb

@@ -0,0 +1,72 @@
+require 'ultralite'
+require './bench'
+require 'redis'
+require 'sqlite3'
+
+cache = Ultralite::Cache.new # default settings
+#mem = Ultralite::Cache.new(path: ":memory:") # default settings
+redis = Redis.new # default settings
+
+values = []
+keys = []
+count = 1000
+count.times { keys << random_str(10) }
+
+[10, 100, 1000, 10000].each do |size|
+  count.times do
+    values << random_str(size) 
+  end
+  
+  random_keys = keys.shuffle
+  puts "Benchmarks for values of size #{size} bytes"
+  puts "=========================================================="
+  puts "== Writes =="
+  bench("Ultralite cache writes", count) do |i|
+    cache.set(keys[i], values[i])
+  end
+
+ # bench("Ultralite memory cache writes", count) do |i|
+ #   mem.set(keys[i], values[i])
+ # end
+
+  bench("Redis writes", count) do |i|
+    redis.set(keys[i], values[i])
+  end
+
+  puts "== Reads =="
+  bench("Ultralite cache reads", count) do |i|
+    cache.get(random_keys[i])
+  end
+
+ # bench("Ultralite memory cache reads", count) do |i|
+ #   cache.get(random_keys[i])
+ # end
+
+  bench("Redis reads", count) do |i|
+    redis.get(random_keys[i])
+  end
+  puts "=========================================================="
+
+  values = []
+end
+
+
+cache.set("somekey", 1)
+#mem.set("somekey", 1)
+redis.set("somekey", 1)
+
+bench("Ultralite cache increment") do
+  cache.increment("somekey", 1)
+end
+
+#bench("Ultralite memory cache increment") do
+#  mem.increment("somekey", 1)
+#end
+
+bench("Redis increment") do
+  redis.incr("somekey")
+end
+
+cache.clear
+redis.flushdb
+

data/bench/rails_job.rb

@@ -0,0 +1,18 @@
+require 'active_job'
+
+class RailsJob < ActiveJob::Base
+  
+  queue_as :default
+
+  @@count = 0
+  
+  def perform(count, time)
+    #sleep 1
+    @@count += 1 
+    if @@count == count  
+      puts "[litejob] Finished in #{Time.now.to_f - time} seconds (#{count / (Time.now.to_f - time)} jps)"
+      @@count = 0
+    end
+  end
+
+end