litestack 0.1.7 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/BENCHMARKS.md +1 -1
- data/CHANGELOG.md +20 -3
- data/README.md +28 -1
- data/assets/litecable_logo_teal.png +0 -0
- data/bench/bench_cache_raw.rb +18 -2
- data/bench/bench_jobs_rails.rb +20 -14
- data/bench/bench_jobs_raw.rb +0 -2
- data/lib/action_cable/subscription_adapter/litecable.rb +36 -0
- data/lib/active_job/queue_adapters/litejob_adapter.rb +14 -10
- data/lib/litestack/litecable.rb +138 -0
- data/lib/litestack/litecable.sql.yml +24 -0
- data/lib/litestack/litecache.rb +56 -62
- data/lib/litestack/litecache.sql.yml +28 -0
- data/lib/litestack/litecache.yml +7 -0
- data/lib/litestack/litejob.rb +20 -11
- data/lib/litestack/litejobqueue.rb +122 -44
- data/lib/litestack/litemetric.rb +228 -0
- data/lib/litestack/litemetric.sql.yml +69 -0
- data/lib/litestack/litequeue.rb +57 -29
- data/lib/litestack/litequeue.sql.yml +34 -0
- data/lib/litestack/litesupport.rb +155 -6
- data/lib/litestack/metrics_app.rb +5 -0
- data/lib/litestack/version.rb +1 -1
- data/lib/litestack.rb +19 -10
- metadata +13 -6
- data/bench/bench_rails.rb +0 -81
- data/bench/bench_raw.rb +0 -72
- data/lib/active_job/queue_adapters/ultralite_adapter.rb +0 -49
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: afdf01934662a90b8c67455dd46e1649972ee62eb732cbf08746f8ac5f509dae
|
4
|
+
data.tar.gz: 734300372c194639072fdcfe4b35f630b4fb173bf453e14e9813f6c7569a6877
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: b0b46de8074b05b6bdab5438e932f53212bd3b98b2d534e02b8b81f1330b41869b2271055e7eeaa3643eeee0df11ef1c3884ce880ff235db063d2e05183f3ad7
|
7
|
+
data.tar.gz: 3bf959fa43c538140724114c56836bcaa5c88adc6d1ca5ec4bf0db9cfdd0c6feb98675f2afba01632352dcc94696a3c85d8b0de5537db0f4aa84d2e00829acd1
|
data/BENCHMARKS.md
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,19 +1,36 @@
|
|
1
1
|
## [Unreleased]
|
2
2
|
|
3
|
-
## [0.
|
3
|
+
## [0.2.0] - 2023-05-08
|
4
|
+
|
5
|
+
- Litecable, a SQLite driver for ActionCable
|
6
|
+
- Litemetric for metrics collection support (experimental, disabled by default)
|
7
|
+
- New schema for Litejob, old jobs are auto-migrated
|
8
|
+
- Code refactoring, extraction of SQL statements to external files
|
9
|
+
- Graceful shutdown support working properly
|
10
|
+
- Fork resilience
|
11
|
+
|
12
|
+
## [0.1.8] - 2023-03-08
|
13
|
+
|
14
|
+
- More code cleanups, more test coverage
|
15
|
+
- Retry support for jobs in Litejob
|
16
|
+
- Job storage and garbage collection for failed jobs
|
17
|
+
- Initial graceful shutdown support for Litejob (incomplete)
|
18
|
+
- More configuration options for Litejob
|
19
|
+
|
20
|
+
## [0.1.7] - 2023-03-05
|
4
21
|
|
5
22
|
- Code cleanup, removal of references to older name
|
6
23
|
- Fix for the litedb rake tasks (thanks: netmute)
|
7
24
|
- More fixes for the new concurrency model
|
8
25
|
- Introduced a logger for the Litejobqueue (doesn't work with Polyphony, fix should come soon)
|
9
26
|
|
10
|
-
## [0.1.6] -
|
27
|
+
## [0.1.6] - 2023-03-03
|
11
28
|
|
12
29
|
- Revamped the locking model, more robust, minimal performance hit
|
13
30
|
- Introduced a new resource pooling class
|
14
31
|
- Litecache and Litejob now use the resource pool
|
15
32
|
- Much less memory usage for Litecache and Litejob
|
16
33
|
|
17
|
-
## [0.1.0] -
|
34
|
+
## [0.1.0] - 2023-02-26
|
18
35
|
|
19
36
|
- Initial release
|
data/README.md
CHANGED
@@ -16,12 +16,14 @@ litestack provides integration with popular libraries, including:
|
|
16
16
|
- ActiveRecord
|
17
17
|
- ActiveSupport::Cache
|
18
18
|
- ActiveJob
|
19
|
+
- ActionCable
|
19
20
|
|
20
21
|
With litestack you only need to add a single gem to your app which would replace a host of other gems and services, for example, a typical Rails app using litestack will no longer need the following services:
|
21
22
|
|
22
23
|
- Database Server (e.g. PostgreSQL, MySQL)
|
23
24
|
- Cache Server (e.g. Redis, Memcached)
|
24
25
|
- Job Processor (e.g. Sidekiq, Goodjob)
|
26
|
+
- Pubsub Server (e.g. Redis, PostgreSQL)
|
25
27
|
|
26
28
|
To make it even more efficient, litestack will detect the presence of Fiber based IO frameworks like Async (e.g. when you use the Falcon web server) or Polyphony. It will then switch its background workers for caches and queues to fibers (using the semantics of the existing framework). This is done transparently and will generally lead to lower CPU and memory utilization.
|
27
29
|
|
@@ -50,6 +52,7 @@ litestack currently offers three main components
|
|
50
52
|
- litedb
|
51
53
|
- litecache
|
52
54
|
- litejob
|
55
|
+
- litecable
|
53
56
|
|
54
57
|
> ![litedb](https://github.com/oldmoe/litestack/blob/master/assets/litedb_logo_teal.png?raw=true)
|
55
58
|
|
@@ -113,6 +116,8 @@ litecache spawns a background thread for cleanup purposes. In case it detects th
|
|
113
116
|
|
114
117
|
> ![litejob](https://github.com/oldmoe/litestack/blob/master/assets/litejob_logo_teal.png?raw=true)
|
115
118
|
|
119
|
+
More info about Litejob can be found in the [litejob guide](https://github.com/oldmoe/litestack/wiki/Litejob-guide)
|
120
|
+
|
116
121
|
litejob is a fast and very efficient job queue processor for Ruby applications. It builds on top of SQLite as well, which provides transactional guarantees, persistence and exceptional performance.
|
117
122
|
|
118
123
|
#### Direct litejob usage
|
@@ -120,7 +125,7 @@ litejob is a fast and very efficient job queue processor for Ruby applications.
|
|
120
125
|
require 'litestack'
|
121
126
|
# define your job class
|
122
127
|
class MyJob
|
123
|
-
include ::
|
128
|
+
include ::Litejob
|
124
129
|
|
125
130
|
queue = :default
|
126
131
|
|
@@ -159,6 +164,28 @@ queues:
|
|
159
164
|
|
160
165
|
The queues need to include a name and a priority (a number between 1 and 10) and can also optionally add the token "spawn", which means every job will run it its own concurrency context (thread or fiber)
|
161
166
|
|
167
|
+
> ![litecable](https://github.com/oldmoe/litestack/blob/master/assets/litecable_logo_teal.png?raw=true)
|
168
|
+
|
169
|
+
#### ActionCable
|
170
|
+
|
171
|
+
This is a drop in replacement adapter for actioncable that replaces `async` and other production adapters (e.g. PostgreSQL, Redis). This adapter is currently only tested in local (inline) mode.
|
172
|
+
|
173
|
+
Getting up and running with litecable requires configuring your cable.yaml file under the config/ directory
|
174
|
+
|
175
|
+
cable.yaml
|
176
|
+
```yaml
|
177
|
+
development:
|
178
|
+
adapter: litecable
|
179
|
+
|
180
|
+
test:
|
181
|
+
adapter: test
|
182
|
+
|
183
|
+
staging:
|
184
|
+
adapter: litecable
|
185
|
+
|
186
|
+
production:
|
187
|
+
adapter: litecable
|
188
|
+
```
|
162
189
|
|
163
190
|
## Contributing
|
164
191
|
|
Binary file
|
data/bench/bench_cache_raw.rb
CHANGED
@@ -8,7 +8,7 @@ require 'async/scheduler'
|
|
8
8
|
Fiber.set_scheduler Async::Scheduler.new
|
9
9
|
Fiber.scheduler.run
|
10
10
|
|
11
|
-
require_relative '../lib/litestack'
|
11
|
+
require_relative '../lib/litestack/litecache'
|
12
12
|
#require 'litestack'
|
13
13
|
|
14
14
|
cache = Litecache.new({path: '../db/cache.db'}) # default settings
|
@@ -25,6 +25,9 @@ count.times { keys << random_str(10) }
|
|
25
25
|
end
|
26
26
|
|
27
27
|
random_keys = keys.shuffle
|
28
|
+
|
29
|
+
GC.compact
|
30
|
+
|
28
31
|
puts "Benchmarks for values of size #{size} bytes"
|
29
32
|
puts "=========================================================="
|
30
33
|
puts "== Writes =="
|
@@ -32,6 +35,13 @@ count.times { keys << random_str(10) }
|
|
32
35
|
cache.set(keys[i], values[i])
|
33
36
|
end
|
34
37
|
|
38
|
+
#bench("file writes", count) do |i|
|
39
|
+
# f = File.open("../files/#{keys[i]}.data", 'w+')
|
40
|
+
# f.write(values[i])
|
41
|
+
# f.close
|
42
|
+
#end
|
43
|
+
|
44
|
+
|
35
45
|
bench("Redis writes", count) do |i|
|
36
46
|
redis.set(keys[i], values[i])
|
37
47
|
end
|
@@ -41,12 +51,18 @@ count.times { keys << random_str(10) }
|
|
41
51
|
cache.get(random_keys[i])
|
42
52
|
end
|
43
53
|
|
54
|
+
#bench("file reads", count) do |i|
|
55
|
+
# data = File.read("../files/#{keys[i]}.data")
|
56
|
+
#end
|
57
|
+
|
44
58
|
bench("Redis reads", count) do |i|
|
45
59
|
redis.get(random_keys[i])
|
46
60
|
end
|
47
61
|
puts "=========================================================="
|
48
62
|
|
49
63
|
values = []
|
64
|
+
|
65
|
+
|
50
66
|
end
|
51
67
|
|
52
68
|
|
@@ -64,5 +80,5 @@ end
|
|
64
80
|
cache.clear
|
65
81
|
redis.flushdb
|
66
82
|
|
67
|
-
sleep
|
83
|
+
#sleep
|
68
84
|
|
data/bench/bench_jobs_rails.rb
CHANGED
@@ -1,22 +1,14 @@
|
|
1
1
|
require './bench'
|
2
|
-
require 'async/scheduler'
|
3
|
-
|
4
|
-
#ActiveJob::Base.logger = Logger.new(IO::NULL)
|
5
2
|
|
3
|
+
count = ARGV[0].to_i rescue 1000
|
4
|
+
env = ARGV[1] || "t"
|
5
|
+
delay = ARGV[2].to_f rescue 0
|
6
6
|
|
7
|
-
Fiber.set_scheduler Async::Scheduler.new
|
8
7
|
|
9
|
-
|
10
|
-
|
11
|
-
ActiveSupport::IsolatedExecutionState.isolation_level = :fiber
|
8
|
+
#ActiveJob::Base.logger = Logger.new(IO::NULL)
|
12
9
|
|
13
10
|
require './rails_job.rb'
|
14
11
|
|
15
|
-
|
16
|
-
puts Litesupport.environment
|
17
|
-
|
18
|
-
count = 1000
|
19
|
-
|
20
12
|
RailsJob.queue_adapter = :sidekiq
|
21
13
|
t = Time.now.to_f
|
22
14
|
puts "Make sure sidekiq is started with -c ./rails_job.rb"
|
@@ -26,13 +18,27 @@ end
|
|
26
18
|
|
27
19
|
puts "Don't forget to check the sidekiq log for processing time conclusion"
|
28
20
|
|
21
|
+
|
22
|
+
# Litejob bench
|
23
|
+
###############
|
24
|
+
|
25
|
+
if env == "a" # threaded
|
26
|
+
require 'async/scheduler'
|
27
|
+
Fiber.set_scheduler Async::Scheduler.new
|
28
|
+
ActiveSupport::IsolatedExecutionState.isolation_level = :fiber
|
29
|
+
end
|
30
|
+
|
31
|
+
require_relative '../lib/active_job/queue_adapters/litejob_adapter'
|
32
|
+
puts Litesupport.environment
|
33
|
+
|
29
34
|
RailsJob.queue_adapter = :litejob
|
30
35
|
t = Time.now.to_f
|
31
36
|
bench("enqueuing litejobs", count) do
|
32
37
|
RailsJob.perform_later(count, t)
|
33
38
|
end
|
34
39
|
|
35
|
-
|
36
|
-
|
40
|
+
if env == "a" # threaded
|
41
|
+
Fiber.scheduler.run
|
42
|
+
end
|
37
43
|
|
38
44
|
sleep
|
data/bench/bench_jobs_raw.rb
CHANGED
@@ -0,0 +1,36 @@
|
|
1
|
+
# frozen_stringe_literal: true
|
2
|
+
|
3
|
+
require_relative '../../litestack/litecable'
|
4
|
+
|
5
|
+
module ActionCable
|
6
|
+
module SubscriptionAdapter
|
7
|
+
class Litecable < ::Litecable# :nodoc:
|
8
|
+
|
9
|
+
attr_reader :logger, :server
|
10
|
+
|
11
|
+
prepend ChannelPrefix
|
12
|
+
|
13
|
+
DEFAULT_OPTIONS = {
|
14
|
+
config_path: "./config/litecable.yml",
|
15
|
+
path: "./db/cable.db",
|
16
|
+
sync: 0, # no need to sync at all
|
17
|
+
mmap_size: 16 * 1024 * 1024, # 16MB of memory hold hot messages
|
18
|
+
expire_after: 10, # remove messages older than 10 seconds
|
19
|
+
listen_interval: 0.005, # check new messages every 5 milliseconds
|
20
|
+
metrics: false
|
21
|
+
}
|
22
|
+
|
23
|
+
def initialize(server, logger=nil)
|
24
|
+
@server = server
|
25
|
+
@logger = server.logger
|
26
|
+
super(DEFAULT_OPTIONS.dup)
|
27
|
+
end
|
28
|
+
|
29
|
+
def shutdown
|
30
|
+
close
|
31
|
+
end
|
32
|
+
|
33
|
+
end
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
@@ -7,21 +7,16 @@ require "active_job"
|
|
7
7
|
|
8
8
|
module ActiveJob
|
9
9
|
module QueueAdapters
|
10
|
-
# ==
|
10
|
+
# == Litestack adapter for Active Job
|
11
11
|
#
|
12
12
|
#
|
13
13
|
# Rails.application.config.active_job.queue_adapter = :litejob
|
14
14
|
class LitejobAdapter
|
15
|
-
|
16
|
-
DEFAULT_OPTIONS = {
|
17
|
-
config_path: "./config/litejob.yml",
|
18
|
-
path: "../db/queue.db",
|
19
|
-
queues: [["default", 1, "spawn"]],
|
20
|
-
workers: 1
|
21
|
-
}
|
22
|
-
|
15
|
+
|
23
16
|
def initialize(options={})
|
24
|
-
|
17
|
+
# we currently don't honour individual options per job class
|
18
|
+
# possible in the future?
|
19
|
+
# Job.options = DEFAULT_OPTIONS.merge(options)
|
25
20
|
end
|
26
21
|
|
27
22
|
def enqueue(job) # :nodoc:
|
@@ -36,6 +31,15 @@ module ActiveJob
|
|
36
31
|
|
37
32
|
class Job # :nodoc:
|
38
33
|
|
34
|
+
DEFAULT_OPTIONS = {
|
35
|
+
config_path: "./config/litejob.yml",
|
36
|
+
path: "../db/queue.db",
|
37
|
+
queues: [["default", 1]],
|
38
|
+
logger: nil, # Rails performs its logging already
|
39
|
+
retries: 5, # It is recommended to stop retries at the Rails level
|
40
|
+
workers: 5
|
41
|
+
}
|
42
|
+
|
39
43
|
include ::Litejob
|
40
44
|
|
41
45
|
def perform(job_data)
|
@@ -0,0 +1,138 @@
|
|
1
|
+
# frozen_stringe_literal: true
|
2
|
+
|
3
|
+
# all components should require the support module
|
4
|
+
require_relative 'litesupport'
|
5
|
+
require_relative 'litemetric'
|
6
|
+
|
7
|
+
require 'base64'
|
8
|
+
require 'oj'
|
9
|
+
|
10
|
+
class Litecable
|
11
|
+
|
12
|
+
include Litesupport::Liteconnection
|
13
|
+
include Litemetric::Measurable
|
14
|
+
|
15
|
+
|
16
|
+
DEFAULT_OPTIONS = {
|
17
|
+
config_path: "./litecable.yml",
|
18
|
+
path: "./cable.db",
|
19
|
+
sync: 0,
|
20
|
+
mmap_size: 16 * 1024 * 1024, # 16MB
|
21
|
+
expire_after: 5, # remove messages older than 5 seconds
|
22
|
+
listen_interval: 0.05, # check new messages every 50 milliseconds
|
23
|
+
metrics: false
|
24
|
+
}
|
25
|
+
|
26
|
+
def initialize(options = {})
|
27
|
+
init(options)
|
28
|
+
@messages = []
|
29
|
+
end
|
30
|
+
|
31
|
+
# broadcast a message to a specific channel
|
32
|
+
def broadcast(channel, payload=nil)
|
33
|
+
# group meesages and only do broadcast every 10 ms
|
34
|
+
#run_stmt(:publish, channel.to_s, Oj.dump(payload), @pid)
|
35
|
+
# but broadcast locally normally
|
36
|
+
@mutex.synchronize{ @messages << [channel.to_s, Oj.dump(payload)] }
|
37
|
+
local_broadcast(channel, payload)
|
38
|
+
end
|
39
|
+
|
40
|
+
# subscribe to a channel, optionally providing a success callback proc
|
41
|
+
def subscribe(channel, subscriber, success_callback = nil)
|
42
|
+
@mutex.synchronize do
|
43
|
+
@subscribers[channel] = {} unless @subscribers[channel]
|
44
|
+
@subscribers[channel][subscriber] = true
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
# unsubscribe from a channel
|
49
|
+
def unsubscribe(channel, subscriber)
|
50
|
+
@mutex.synchronize do
|
51
|
+
@subscribers[channel].delete(subscriber) rescue nil
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
private
|
56
|
+
|
57
|
+
def local_broadcast(channel, payload=nil)
|
58
|
+
return unless @subscribers[channel]
|
59
|
+
subscribers = []
|
60
|
+
@mutex.synchronize do
|
61
|
+
subscribers = @subscribers[channel].keys
|
62
|
+
end
|
63
|
+
subscribers.each do |subscriber|
|
64
|
+
subscriber.call(payload)
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
68
|
+
def setup
|
69
|
+
super # create connection
|
70
|
+
@pid = Process.pid
|
71
|
+
@subscribers = {}
|
72
|
+
@mutex = Litesupport::Mutex.new
|
73
|
+
@running = true
|
74
|
+
@listener = create_listener
|
75
|
+
@pruner = create_pruner
|
76
|
+
@broadcaster = create_broadcaster
|
77
|
+
@last_fetched_id = nil
|
78
|
+
end
|
79
|
+
|
80
|
+
def create_broadcaster
|
81
|
+
Litesupport.spawn do
|
82
|
+
while @running do
|
83
|
+
@mutex.synchronize do
|
84
|
+
if @messages.length > 0
|
85
|
+
run_sql("BEGIN IMMEDIATE")
|
86
|
+
while msg = @messages.shift
|
87
|
+
run_stmt(:publish, msg[0], msg[1], @pid)
|
88
|
+
end
|
89
|
+
run_sql("END")
|
90
|
+
end
|
91
|
+
end
|
92
|
+
sleep 0.02
|
93
|
+
end
|
94
|
+
end
|
95
|
+
end
|
96
|
+
|
97
|
+
def create_pruner
|
98
|
+
Litesupport.spawn do
|
99
|
+
while @running do
|
100
|
+
run_stmt(:prune, @options[:expire_after])
|
101
|
+
sleep @options[:expire_after]
|
102
|
+
end
|
103
|
+
end
|
104
|
+
end
|
105
|
+
|
106
|
+
def create_listener
|
107
|
+
Litesupport.spawn do
|
108
|
+
while @running do
|
109
|
+
@last_fetched_id ||= (run_stmt(:last_id)[0][0] || 0)
|
110
|
+
@logger.info @last_fetched_id
|
111
|
+
run_stmt(:fetch, @last_fetched_id, @pid).to_a.each do |msg|
|
112
|
+
@logger.info "RECEIVED #{msg}"
|
113
|
+
@last_fetched_id = msg[0]
|
114
|
+
local_broadcast(msg[1], Oj.load(msg[2]))
|
115
|
+
end
|
116
|
+
sleep @options[:listen_interval]
|
117
|
+
end
|
118
|
+
end
|
119
|
+
end
|
120
|
+
|
121
|
+
def create_connection
|
122
|
+
conn = super
|
123
|
+
conn.wal_autocheckpoint = 10000
|
124
|
+
sql = YAML.load_file("#{__dir__}/litecable.sql.yml")
|
125
|
+
version = conn.get_first_value("PRAGMA user_version")
|
126
|
+
sql["schema"].each_pair do |v, obj|
|
127
|
+
if v > version
|
128
|
+
conn.transaction do
|
129
|
+
obj.each{|k, s| conn.execute(s)}
|
130
|
+
conn.user_version = v
|
131
|
+
end
|
132
|
+
end
|
133
|
+
end
|
134
|
+
sql["stmts"].each { |k, v| conn.stmts[k.to_sym] = conn.prepare(v) }
|
135
|
+
conn
|
136
|
+
end
|
137
|
+
|
138
|
+
end
|
@@ -0,0 +1,24 @@
|
|
1
|
+
schema:
|
2
|
+
1:
|
3
|
+
create_table_messages: >
|
4
|
+
CREATE TABLE IF NOT EXISTS messages(
|
5
|
+
id INTEGER PRIMARY KEY autoincrement,
|
6
|
+
channel TEXT NOT NULL,
|
7
|
+
value TEXT NOT NULL,
|
8
|
+
pid INTEGER,
|
9
|
+
created_at INTEGER NOT NULL ON CONFLICT REPLACE DEFAULT(unixepoch())
|
10
|
+
);
|
11
|
+
create_index_messages_by_date: >
|
12
|
+
CREATE INDEX IF NOT EXISTS messages_by_date ON messages(created_at);
|
13
|
+
|
14
|
+
stmts:
|
15
|
+
|
16
|
+
publish: INSERT INTO messages(channel, value, pid) VALUES ($1, $2, $3)
|
17
|
+
|
18
|
+
last_id: SELECT max(id) FROM messages
|
19
|
+
|
20
|
+
fetch: SELECT id, channel, value FROM messages WHERE id > $1 and pid != $2
|
21
|
+
|
22
|
+
prune: DELETE FROM messages WHERE created_at < (unixepoch() - $1)
|
23
|
+
|
24
|
+
check_prune: SELECT count(*) FROM messages WHERE created_at < (unixepoch() - $1)
|