litestack 0.4.2 → 0.4.4
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.standard.yml +3 -1
- data/BENCHMARKS.md +3 -3
- data/CAVEATS.md +20 -0
- data/CHANGELOG.md +40 -1
- data/FILESYSTEMS.md +55 -0
- data/Gemfile +2 -0
- data/README.md +8 -4
- data/ROADMAP.md +6 -6
- data/assets/litestack_advantage.png +0 -0
- data/bench/bench.rb +2 -0
- data/bench/bench_cache_rails.rb +33 -2
- data/bench/bench_cache_raw.rb +36 -12
- data/bench/bench_jobs_rails.rb +3 -3
- data/bench/bench_jobs_raw.rb +3 -3
- data/bin/liteboard +16 -13
- data/gemfiles/rails70.gemfile +5 -0
- data/gemfiles/rails71.gemfile +5 -0
- data/gemfiles/rails71.gemfile.lock +264 -0
- data/lib/active_job/queue_adapters/litejob_adapter.rb +11 -3
- data/lib/active_record/connection_adapters/litedb_adapter.rb +8 -0
- data/lib/active_support/cache/litecache.rb +40 -7
- data/lib/generators/litestack/install/install_generator.rb +2 -2
- data/lib/generators/litestack/install/templates/cable.yml +0 -3
- data/lib/litestack/liteboard/liteboard.rb +15 -19
- data/lib/litestack/liteboard/views/litecable.erb +1 -1
- data/lib/litestack/litecable.rb +1 -1
- data/lib/litestack/litecache.rb +51 -19
- data/lib/litestack/litecache.sql.yml +7 -5
- data/lib/litestack/litedb.rb +5 -1
- data/lib/litestack/litejob.rb +1 -1
- data/lib/litestack/litejobqueue.rb +24 -14
- data/lib/litestack/litemetric.rb +7 -6
- data/lib/litestack/litemetric.sql.yml +1 -1
- data/lib/litestack/litemetric_collector.sql.yml +1 -1
- data/lib/litestack/litequeue.rb +17 -2
- data/lib/litestack/litequeue.sql.yml +38 -5
- data/lib/litestack/litescheduler.rb +9 -4
- data/lib/litestack/litesearch/index.rb +11 -10
- data/lib/litestack/litesearch/model.rb +61 -3
- data/lib/litestack/litesearch/schema.rb +7 -2
- data/lib/litestack/litesearch/schema_adapters/backed_adapter.rb +69 -25
- data/lib/litestack/litesearch/schema_adapters.rb +4 -4
- data/lib/litestack/litesearch.rb +2 -2
- data/lib/litestack/litesupport.rb +9 -7
- data/lib/litestack/railtie.rb +4 -2
- data/lib/litestack/version.rb +1 -1
- data/lib/litestack.rb +15 -15
- data/lib/railties/rails/commands/dbconsole.rb +5 -5
- data/lib/sequel/adapters/litedb.rb +9 -1
- data/lib/sequel/adapters/shared/litedb.rb +2 -2
- data/scripts/build_metrics.rb +2 -2
- data/scripts/test_cable.rb +1 -1
- metadata +105 -56
- data/Gemfile.lock +0 -92
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ddcc1a0b4d5a5547b3e3deec7ce5abec68e31ffce348913867368513276ac69a
|
4
|
+
data.tar.gz: 743a21e710d5351131a2cbe50a94de13653a034b88d08b82e470bb3aac2bdfbb
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f54bdc732995e8083791c6b214ec6058bb439548ffdcbb14f33fbbeadbca870f4941cff558cdcbfa26212e53080534c12bfacfdfa2ba8ca689f380679ef0504a
|
7
|
+
data.tar.gz: f0a1fe1c95a592e69175e55ad212e351d5bca907b43c4215a4eca919d8dc2dfda68d80fa0ceb74da5909bf822adf0b2ea45e9108fcf0cd995a0a4b337a908ad0
|
data/.standard.yml
CHANGED
data/BENCHMARKS.md
CHANGED
@@ -125,11 +125,11 @@ On average, Litecable is quite faster than the Redis based version and offers be
|
|
125
125
|
|
126
126
|
> ![litesearch](https://github.com/oldmoe/litestack/blob/master/assets/litesearch_logo_teal.png?raw=true)
|
127
127
|
|
128
|
-
Litesearch was benchmarked against
|
128
|
+
Litesearch was benchmarked against Meilisearch, both using their respective ActiveRecord integrations. Meilisearch was running on the same machine as the benchmark script and was using the default configuration options. The dataset used for testing was the infamous Enron email corpus. Redisearch was not benchmarked due to the clients being not Rails 7.1 compatible (yet), will probably bench Redisearch when they are.
|
129
129
|
|
130
130
|
### Building the index
|
131
131
|
|
132
|
-
||
|
132
|
+
||Meilisearch|Litesearch|
|
133
133
|
|-:|-:|-:|
|
134
134
|
|Time to insert 10K docs|265.42 seconds|29.06 seconds|
|
135
135
|
|Inserted docs/second|38|344|
|
@@ -137,4 +137,4 @@ Litesearch was benchmarked against Mielsearch, both using their respective Activ
|
|
137
137
|
|Searches/second|133|19608|
|
138
138
|
|Index rebuild|0.822|0.626|
|
139
139
|
|
140
|
-
We only limited the test to 10K documents becuause
|
140
|
+
We only limited the test to 10K documents becuause Meilisearch was taking a long time to index, so we decided to stop at a reasonable sample size. The search numbers for litesearch were double checked, event against a 100K document set and they remained virtually the same. It is clear that litesearch is much faster than Meilisearch in both indexing and searching, this could be partially attributed to litesearch being a simpler text search engine, but still, the difference is huge! For rebuilding the index though, Litesearch is not that much faster than Meilisearch.
|
data/CAVEATS.md
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
# CAVEATS
|
2
|
+
|
3
|
+
This is a document that lists some caveats that you need to consider when dealing with Litestack and SQLite applications
|
4
|
+
|
5
|
+
## Single host (server) only
|
6
|
+
|
7
|
+
By design, SQLite can be accessed only (in a performant and safe way) from a single host machine. That means that you cannot have things like auto-scaling of app servers since there is but a single one of them. This app server can be a huge one, with many processes, but only one app server at a time.
|
8
|
+
|
9
|
+
## Containerization becomes a bit tricky
|
10
|
+
|
11
|
+
Container technology was originally designed for stateless applications. Based on this, one of the core aspects of containers is that they are immutable, deploying an app to container means destroying one container and creating another. With statefull applications, like database servers and similarly SQLite based applications, this will potentially mean a down time will be perceived, which in some approaches is mitigate but not perfectly (yet). Expect some quirks when dealing with SQLite and containers.
|
12
|
+
|
13
|
+
## Single writer / multi reader
|
14
|
+
|
15
|
+
In its default configuration, Litestack allows one writer at a time to write to the database file, while simultaneously allowing any number of readers to access the database. Generally SQLite is pretty fast in writes, but you should be aware that a long running write operation will prevent any other writers from proceeding, for example creating an index on a very large table will stop other writers until the index creation is finished. There is no concurrent index creation facility in SQLite, yet.
|
16
|
+
|
17
|
+
## Litestack does not release the GVL
|
18
|
+
|
19
|
+
This also applies to the Ruby SQLite3 gem, when it is performing a query, it will not allow any other threads in the Ruby process to resume until it returns. Care should be taken when writing low latency multi-threaded applications with Litestack, it is generally recommended to use fibers over threads in that case, since everything is serialized anyway it makes sense to avoid the ovreheads of mult-threaded environments
|
20
|
+
|
data/CHANGELOG.md
CHANGED
@@ -1,28 +1,59 @@
|
|
1
1
|
## [Unreleased]
|
2
2
|
|
3
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.4...master)
|
4
|
+
|
5
|
+
## [0.4.4] - 2024-05-21
|
6
|
+
|
7
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.3...v0.4.4)
|
8
|
+
- Fix a table not defined bug for Litesearch
|
9
|
+
- Add conditional mapping of external fields to Litesearch (backed index)
|
10
|
+
- As a consequence of the above, support AR polymorphic associations
|
11
|
+
- As a consequence of the above, support indexing and searching for ActionText attributes
|
12
|
+
- Remove 'hanami-router' as a dependency, rely on vanilla Rack for Liteboard
|
13
|
+
- All configuratin .yml files now support ERB rendering
|
14
|
+
- Fix adapter registration with Rails 7.2
|
15
|
+
- Ground work for more durable job persistence in case of a process crash
|
16
|
+
- Many minor fixes
|
17
|
+
|
18
|
+
## [0.4.3] - 2024-02-15
|
19
|
+
|
20
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.2...v0.4.3)
|
21
|
+
- Add "sequel" as a development dependency
|
22
|
+
- Diff links in CHANGELOG (thanks Weston Ganger)
|
23
|
+
- Fix daemonize type in liteboard (thanks Julian Rubisch)
|
24
|
+
- Better Litecache schema (streamlined numeric value support)
|
25
|
+
- Support for set_multi and get_multi in Litecache (read_multi and write_multi support for Rails Cache store)
|
26
|
+
- More tests written for Litecache and Rails Litecache store
|
27
|
+
- Experimenting with removing the Rails LocalCache as it doesn't show enough improvement in performance to compensate for the memory overhead
|
28
|
+
- Switch Litecache to a FIFO eviction model vs LRU (thanks Julian Rubisch and Stephen Margheim)
|
29
|
+
|
3
30
|
## [0.4.2] - 2023-11-11
|
4
31
|
|
32
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.1...v0.4.2)
|
5
33
|
- Add similarity search support for Litesearch (works best for non-trigram indexes)
|
6
34
|
- Enable similarity search for ActiveRecord and Sequel models
|
7
35
|
- Fix Litesearch tests
|
8
36
|
- Suppress chatty Litejob exit detector when there are no jobs in flight
|
9
37
|
- Tidy up the test folder
|
10
|
-
- [#41](https://github.com/oldmoe/litestack/pull/41) - Fix bug in Litecable where the `connected` event was not getting
|
38
|
+
- [#41](https://github.com/oldmoe/litestack/pull/41) - Fix bug in Litecable where the `connected` event was not getting propagated
|
11
39
|
- Add Litemetric and Liteboard info to README.doc
|
12
40
|
- Fix the testing rake task
|
13
41
|
|
14
42
|
## [0.4.1] - 2023-10-11
|
15
43
|
|
44
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.0...v0.4.1)
|
16
45
|
- Add missing Litesearch::Model dependency
|
17
46
|
|
18
47
|
## [0.4.0] - 2023-10-11
|
19
48
|
|
49
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.3.0...v0.4.0)
|
20
50
|
- Introduced Litesearch, dynamic & fast full text search capability for Litedb
|
21
51
|
- ActiveRecord and Sequel integration for Litesearch
|
22
52
|
- Slight improvement to the Sequel Litedb adapter for better Litesearch integration
|
23
53
|
|
24
54
|
## [0.3.0] - 2023-08-13
|
25
55
|
|
56
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.2.6...v0.3.0)
|
26
57
|
- Reworked the Litecable thread safety model
|
27
58
|
- Fixed multiple litejob bugs (thanks Stephen Margheim)
|
28
59
|
- Fixed Railtie dependency (thanks Marco Roth)
|
@@ -32,6 +63,7 @@
|
|
32
63
|
|
33
64
|
## [0.2.6] - 2023-07-16
|
34
65
|
|
66
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.2.3...v0.2.6)
|
35
67
|
- Much improved database location setting (thanks Brad Gessler)
|
36
68
|
- A Rails generator for better Rails Litestack defaults (thanks Brad Gessler)
|
37
69
|
- Revamped Litemetric, now much faster and more accurate (still experimental)
|
@@ -39,19 +71,23 @@
|
|
39
71
|
|
40
72
|
## [0.2.3] - 2023-05-20
|
41
73
|
|
74
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.2.2...v0.2.3)
|
42
75
|
- Cut back on options defined in the Litejob Rails adapter
|
43
76
|
|
44
77
|
## [0.2.2] - 2023-05-18
|
45
78
|
|
79
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.2.1...v0.2.2)
|
46
80
|
- Fix default queue location in Litejob
|
47
81
|
|
48
82
|
|
49
83
|
## [0.2.1] - 2023-05-08
|
50
84
|
|
85
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.2.0...v0.2.1)
|
51
86
|
- Fix a race condition in Litecable
|
52
87
|
|
53
88
|
## [0.2.0] - 2023-05-08
|
54
89
|
|
90
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.1.8...v0.2.0)
|
55
91
|
- Litecable, a SQLite driver for ActionCable
|
56
92
|
- Litemetric for metrics collection support (experimental, disabled by default)
|
57
93
|
- New schema for Litejob, old jobs are auto-migrated
|
@@ -61,6 +97,7 @@
|
|
61
97
|
|
62
98
|
## [0.1.8] - 2023-03-08
|
63
99
|
|
100
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.1.7...v0.1.8)
|
64
101
|
- More code cleanups, more test coverage
|
65
102
|
- Retry support for jobs in Litejob
|
66
103
|
- Job storage and garbage collection for failed jobs
|
@@ -69,6 +106,7 @@
|
|
69
106
|
|
70
107
|
## [0.1.7] - 2023-03-05
|
71
108
|
|
109
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.1.6...v0.1.7)
|
72
110
|
- Code cleanup, removal of references to older name
|
73
111
|
- Fix for the litedb rake tasks (thanks: netmute)
|
74
112
|
- More fixes for the new concurrency model
|
@@ -76,6 +114,7 @@
|
|
76
114
|
|
77
115
|
## [0.1.6] - 2023-03-03
|
78
116
|
|
117
|
+
- [View Diff](https://github.com/oldmoe/litestack/compare/v0.1.0...v0.1.6)
|
79
118
|
- Revamped the locking model, more robust, minimal performance hit
|
80
119
|
- Introduced a new resource pooling class
|
81
120
|
- Litecache and Litejob now use the resource pool
|
data/FILESYSTEMS.md
ADDED
@@ -0,0 +1,55 @@
|
|
1
|
+
# Filesystems & Litestack
|
2
|
+
|
3
|
+
In the containerized world we live in, many layers of the hardware/software stacks are far abstracted that we no longer know they exist. For the filesystem enthusiasts out there this is a quick overview of how Litestack (and hence SQLite) can benefit from different filesystem
|
4
|
+
|
5
|
+
## XFS
|
6
|
+
|
7
|
+
A very stable and trusted filesystem with excellent performance characteristics
|
8
|
+
|
9
|
+
- Fast reads / writes
|
10
|
+
|
11
|
+
## EXT4
|
12
|
+
|
13
|
+
Another stable and performant filesystem
|
14
|
+
|
15
|
+
- Fast reads / writes
|
16
|
+
|
17
|
+
## F2FS
|
18
|
+
|
19
|
+
Specially built for solid state storage, has an atomic write mode that is supported by SQLite
|
20
|
+
|
21
|
+
- Fast reads
|
22
|
+
- Reasonably fast durable writes in synchronous mode
|
23
|
+
- Compression (not very useful)
|
24
|
+
|
25
|
+
## ZFS
|
26
|
+
|
27
|
+
Copy-on-write filesystem with a nifty set of features, very fast snapshotting but only for the FS as a whole, can send incremental snapshots to another host for backup. Naturally a lot more pages are written on write operations thus it is a bit slower in writes.
|
28
|
+
|
29
|
+
- Fast reads
|
30
|
+
- Slower writes
|
31
|
+
- FS Snapshotting with send/recv
|
32
|
+
- fast device cache
|
33
|
+
- Compression
|
34
|
+
|
35
|
+
## Btrfs
|
36
|
+
|
37
|
+
Another CoW filesystem that delivers snapshotting and incremental send/recv at a much granular level.
|
38
|
+
|
39
|
+
- Fast reads
|
40
|
+
- Slower writes
|
41
|
+
- Fast copies (free backups)
|
42
|
+
- Sub volume Snapshotting with send/recv
|
43
|
+
- Compression
|
44
|
+
|
45
|
+
## Bcachefs
|
46
|
+
|
47
|
+
A new CoW filesystem built on the foundations of the bcache module. Improving rapidly but still lacks a few of the common CoW fs features
|
48
|
+
|
49
|
+
- Fast reads
|
50
|
+
- Slower writes
|
51
|
+
- fast device cache
|
52
|
+
|
53
|
+
|
54
|
+
|
55
|
+
|
data/Gemfile
CHANGED
data/README.md
CHANGED
@@ -1,5 +1,9 @@
|
|
1
1
|
![litestack](https://github.com/oldmoe/litestack/blob/master/assets/litestack_logo_teal_large.png?raw=true)
|
2
2
|
|
3
|
+
<a href="https://badge.fury.io/rb/litestack" target="_blank"><img height="21" style='border:0px;height:21px;' border='0' src="https://badge.fury.io/rb/litestack.svg" alt="Gem Version"></a>
|
4
|
+
<a href='https://github.com/oldmoe/litestack/actions' target='_blank'><img src="https://github.com/oldmoe/litestack/actions/workflows/ruby.yml/badge.svg?branch=master" style="max-width:100%;" height='21' style='border:0px;height:21px;' border='0' alt="CI Status"></a>
|
5
|
+
<a href='https://rubygems.org/gems/litestack' target='_blank'><img height='21' style='border:0px;height:21px;' src='https://img.shields.io/gem/dt/litestack?color=brightgreen&label=Rubygems%20Downloads' border='0' alt='RubyGems Downloads' /></a>
|
6
|
+
|
3
7
|
All your data infrastructure, in a gem!
|
4
8
|
|
5
9
|
Litestack is a Ruby gem that provides both Ruby and Ruby on Rails applications an all-in-one solution for web application data infrastructure. It exploits the power and embeddedness of SQLite to deliver a full-fledged SQL database, a fast cache , a robust job queue, a reliable message broker, a full text search engine and a metrics platform all in a single package.
|
@@ -24,7 +28,7 @@ With litestack you only need to add a single gem to your app which would replace
|
|
24
28
|
- Cache Server (e.g. Redis, Memcached)
|
25
29
|
- Job Processor (e.g. Sidekiq, Goodjob)
|
26
30
|
- Pubsub Server (e.g. Redis, PostgreSQL)
|
27
|
-
- Fulltext Search Server (e.g. Elasticsearch,
|
31
|
+
- Fulltext Search Server (e.g. Elasticsearch, Meilisearch)
|
28
32
|
|
29
33
|
To make it even more efficient, litestack will detect the presence of Fiber based IO frameworks like Async (e.g. when you use the Falcon web server) or Polyphony. It will then switch its background workers for caches and queues to fibers (using the semantics of the existing framework). This is done transparently and will generally lead to lower CPU and memory utilization.
|
30
34
|
|
@@ -55,7 +59,7 @@ litedb is a wrapper around SQLite3, offering a better default configuration that
|
|
55
59
|
|
56
60
|
#### Direct litedb usage
|
57
61
|
|
58
|
-
litedb can be used exactly as the SQLite3 gem, since litedb
|
62
|
+
litedb can be used exactly as the SQLite3 gem, since litedb inherits from SQLite3
|
59
63
|
|
60
64
|
```ruby
|
61
65
|
require 'litestack'
|
@@ -205,7 +209,7 @@ idx.search('kamal')
|
|
205
209
|
idx.search('subject: awa')
|
206
210
|
```
|
207
211
|
|
208
|
-
Litesearch integrates tightly with ActiveRecord
|
212
|
+
Litesearch integrates tightly with ActiveRecord and Sequel, here are integration examples
|
209
213
|
|
210
214
|
#### ActiveRecord
|
211
215
|
|
@@ -258,7 +262,7 @@ Book.search('author: writer').limit(1).all
|
|
258
262
|
> ![litemetric](https://github.com/oldmoe/litestack/blob/master/assets/litemetric_logo_teal.png?raw=true)
|
259
263
|
|
260
264
|
### Litemetric
|
261
|
-
Litestack comes with a
|
265
|
+
Litestack comes with a module that can collect useful metrics for its different components, in each component, you need to add the following to the respective .yml file (database.yml in case of Litedb)
|
262
266
|
```yml
|
263
267
|
metrics: true # default is false
|
264
268
|
```
|
data/ROADMAP.md
CHANGED
@@ -21,25 +21,25 @@ This is a list of functional/non-functional features that litestack targets to i
|
|
21
21
|
- [ ] Automated testing matrix
|
22
22
|
- [ ] Automated gem builds
|
23
23
|
- [ ] Litedb improvements
|
24
|
-
- [ ] ActiveRecord Sqlite3 Adapter compatibility mode
|
24
|
+
- [ ] ActiveRecord Sqlite3 Adapter compatibility mode
|
25
25
|
- [ ] Sequel Sqlite3 Adapter compatibility mode
|
26
|
-
- [ ] Extension bundling/loading
|
26
|
+
- [ ] Extension bundling/loading
|
27
27
|
- [ ] Liteiob improvements
|
28
28
|
- [ ] Persist jobs even during execution
|
29
29
|
- [ ] Zombie job detection
|
30
|
-
- [ ] Better process exit handling
|
30
|
+
- [ ] Better process exit handling
|
31
31
|
- [ ] Faster job dispatch for the no delay case
|
32
32
|
- [ ] Database maintenance scripts
|
33
33
|
- [ ] Online backup
|
34
34
|
- [ ] Restore
|
35
35
|
- [ ] Litestream integration
|
36
|
-
- [ ] CoW
|
36
|
+
- [ ] CoW filesystem-specific backup path
|
37
37
|
- [ ] Zero downtime deployment scripts
|
38
38
|
- [ ] Rails
|
39
|
-
- [ ]
|
39
|
+
- [ ] Generic Rack Applications
|
40
40
|
- [ ] Litemetric improvements
|
41
41
|
- [ ] Rails performance module
|
42
42
|
- [ ] Ruby Memory/GC module
|
43
43
|
- [ ] Restore generic module
|
44
44
|
- [ ] Better HTML/CSS
|
45
|
-
- [ ] Kredis replacement implementation?
|
45
|
+
- [ ] Kredis replacement implementation?
|
Binary file
|
data/bench/bench.rb
CHANGED
data/bench/bench_cache_rails.rb
CHANGED
@@ -1,8 +1,8 @@
|
|
1
1
|
require "active_support"
|
2
2
|
require_relative "../lib/litestack"
|
3
|
-
require_relative "
|
3
|
+
require_relative "bench"
|
4
4
|
|
5
|
-
cache = ActiveSupport::Cache::Litecache.new
|
5
|
+
cache = ActiveSupport::Cache::Litecache.new
|
6
6
|
|
7
7
|
# can only use the lookup method when the gem is installed
|
8
8
|
# cache = ActiveSupport::Cache.lookup_store(:litecache, {path: '../db/rails_cache.db'})
|
@@ -31,6 +31,21 @@ count = 1000
|
|
31
31
|
redis.write(keys[i], values[i])
|
32
32
|
end
|
33
33
|
|
34
|
+
puts "== Multi Writes =="
|
35
|
+
bench("litecache multi-writes", count / 5) do |i|
|
36
|
+
idx = i * 5
|
37
|
+
payload = {}
|
38
|
+
5.times { |j| payload[keys[idx + j]] = values[idx + j] }
|
39
|
+
cache.write_multi(payload)
|
40
|
+
end
|
41
|
+
|
42
|
+
bench("Redis multi-writes", count / 5) do |i|
|
43
|
+
idx = i * 5
|
44
|
+
payload = {}
|
45
|
+
5.times { |j| payload[keys[idx + j]] = values[idx + j] }
|
46
|
+
redis.write_multi(payload)
|
47
|
+
end
|
48
|
+
|
34
49
|
puts "== Reads =="
|
35
50
|
bench("litecache reads", count) do |i|
|
36
51
|
cache.read(random_keys[i])
|
@@ -39,6 +54,22 @@ count = 1000
|
|
39
54
|
bench("Redis reads", count) do |i|
|
40
55
|
redis.read(random_keys[i])
|
41
56
|
end
|
57
|
+
|
58
|
+
puts "== Multi Reads =="
|
59
|
+
bench("litecache multi-reads", count / 5) do |i|
|
60
|
+
idx = i * 5
|
61
|
+
payload = []
|
62
|
+
5.times { |j| payload << random_keys[idx + j] }
|
63
|
+
cache.read_multi(*payload)
|
64
|
+
end
|
65
|
+
|
66
|
+
bench("Redis multi-reads", count / 5) do |i|
|
67
|
+
idx = i * 5
|
68
|
+
payload = []
|
69
|
+
5.times { |j| payload << random_keys[idx + j] }
|
70
|
+
redis.read_multi(*payload)
|
71
|
+
end
|
72
|
+
|
42
73
|
puts "=========================================================="
|
43
74
|
|
44
75
|
keys = []
|
data/bench/bench_cache_raw.rb
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
require "redis"
|
2
2
|
require "sqlite3"
|
3
|
-
require_relative "
|
3
|
+
require_relative "bench"
|
4
4
|
|
5
5
|
# require 'polyphony'
|
6
6
|
require "async/scheduler"
|
@@ -11,7 +11,7 @@ Fiber.scheduler.run
|
|
11
11
|
require_relative "../lib/litestack/litecache"
|
12
12
|
# require 'litestack'
|
13
13
|
|
14
|
-
cache = Litecache.new({path: "../db/cache.db"}) # default settings
|
14
|
+
cache = Litecache.new # ({path: "../db/cache.db"}) # default settings
|
15
15
|
redis = Redis.new # default settings
|
16
16
|
|
17
17
|
values = []
|
@@ -35,28 +35,52 @@ count.times { keys << random_str(10) }
|
|
35
35
|
cache.set(keys[i], values[i])
|
36
36
|
end
|
37
37
|
|
38
|
-
# bench("file writes", count) do |i|
|
39
|
-
# f = File.open("../files/#{keys[i]}.data", 'w+')
|
40
|
-
# f.write(values[i])
|
41
|
-
# f.close
|
42
|
-
# end
|
43
|
-
|
44
38
|
bench("Redis writes", count) do |i|
|
45
39
|
redis.set(keys[i], values[i])
|
46
40
|
end
|
47
41
|
|
42
|
+
puts "== Multi Writes =="
|
43
|
+
bench("litecache multi-writes", count / 5) do |i|
|
44
|
+
idx = i * 5
|
45
|
+
payload = {}
|
46
|
+
5.times { |j| payload[keys[idx + j]] = values[idx + j] }
|
47
|
+
cache.set_multi(payload)
|
48
|
+
end
|
49
|
+
|
50
|
+
bench("Redis multi-writes", count / 5) do |i|
|
51
|
+
idx = i * 5
|
52
|
+
payload = []
|
53
|
+
5.times { |j|
|
54
|
+
payload << keys[idx + j]
|
55
|
+
payload << values[idx + j]
|
56
|
+
}
|
57
|
+
redis.mset(*payload)
|
58
|
+
end
|
59
|
+
|
48
60
|
puts "== Reads =="
|
49
61
|
bench("litecache reads", count) do |i|
|
50
62
|
cache.get(random_keys[i])
|
51
63
|
end
|
52
64
|
|
53
|
-
# bench("file reads", count) do |i|
|
54
|
-
# data = File.read("../files/#{keys[i]}.data")
|
55
|
-
# end
|
56
|
-
|
57
65
|
bench("Redis reads", count) do |i|
|
58
66
|
redis.get(random_keys[i])
|
59
67
|
end
|
68
|
+
|
69
|
+
puts "== Multi Reads =="
|
70
|
+
bench("litecache multi-reads", count / 5) do |i|
|
71
|
+
idx = i * 5
|
72
|
+
payload = []
|
73
|
+
5.times { |j| payload << random_keys[idx + j] }
|
74
|
+
cache.get_multi(*payload)
|
75
|
+
end
|
76
|
+
|
77
|
+
bench("Redis multi-reads", count / 5) do |i|
|
78
|
+
idx = i * 5
|
79
|
+
payload = []
|
80
|
+
5.times { |j| payload << random_keys[idx + j] }
|
81
|
+
redis.mget(*payload)
|
82
|
+
end
|
83
|
+
|
60
84
|
puts "=========================================================="
|
61
85
|
|
62
86
|
values = []
|
data/bench/bench_jobs_rails.rb
CHANGED
@@ -19,9 +19,9 @@ require "./rails_job"
|
|
19
19
|
RailsJob.queue_adapter = :sidekiq
|
20
20
|
t = Time.now.to_f
|
21
21
|
puts "Make sure sidekiq is started with -c ./rails_job.rb"
|
22
|
-
bench("enqueuing sidekiq jobs", count) do
|
23
|
-
RailsJob.perform_later(count, t)
|
24
|
-
end
|
22
|
+
#bench("enqueuing sidekiq jobs", count) do
|
23
|
+
# RailsJob.perform_later(count, t)
|
24
|
+
#end
|
25
25
|
|
26
26
|
puts "Don't forget to check the sidekiq log for processing time conclusion"
|
27
27
|
|
data/bench/bench_jobs_raw.rb
CHANGED
@@ -18,9 +18,9 @@ require "./skjob"
|
|
18
18
|
|
19
19
|
t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
20
20
|
puts "make sure sidekiq is started with skjob.rb as the job"
|
21
|
-
bench("enqueuing sidekiq jobs", count) do |i|
|
22
|
-
SidekiqJob.perform_async(count, t, delay)
|
23
|
-
end
|
21
|
+
#bench("enqueuing sidekiq jobs", count) do |i|
|
22
|
+
# SidekiqJob.perform_async(count, t, delay)
|
23
|
+
#end
|
24
24
|
|
25
25
|
puts "Don't forget to check the sidekiq log for processing time conclusion"
|
26
26
|
|
data/bin/liteboard
CHANGED
@@ -2,19 +2,23 @@
|
|
2
2
|
# frozen_string_literal: true
|
3
3
|
|
4
4
|
require 'optparse'
|
5
|
+
require 'erb'
|
6
|
+
require 'yaml'
|
7
|
+
require 'rackup'
|
5
8
|
require_relative '../lib/litestack/liteboard/liteboard'
|
6
9
|
DEFAULTS = {
|
7
10
|
config_path: Litemetric::DEFAULT_OPTIONS[:config_path],
|
8
11
|
path: Litemetric::DEFAULT_OPTIONS[:path]
|
9
12
|
}
|
10
13
|
|
11
|
-
options = {
|
14
|
+
options = {
|
12
15
|
config_path: nil,
|
13
16
|
path: nil,
|
14
|
-
|
17
|
+
daemonize: false,
|
15
18
|
Port: 9292,
|
16
19
|
Host: 'localhost',
|
17
20
|
environment: 'production',
|
21
|
+
pid: './liteboard.pid',
|
18
22
|
quiet: false
|
19
23
|
}
|
20
24
|
|
@@ -25,9 +29,9 @@ OptionParser.new do |parser|
|
|
25
29
|
parser.on("-s", "--server SERVER", "use SERVER (e.g. puma/falcon/iodine)") { |v| options[:port] = v }
|
26
30
|
parser.on("-H", "--host HOST", "listen on HOST (default: #{options[:Host]})") { |v| options[:Host] = v }
|
27
31
|
parser.on("-p", "--port PORT", "use PORT (default: #{options[:Port]})") { |v| options[:Port] = v.to_i rescue options[:Port] }
|
28
|
-
parser.on("-D", "--
|
29
|
-
parser.on("-E", "--env ENVIRONMENT", "which environment to use (default: #{options[:environment]})") { |v| options[:environment] = v }
|
30
|
-
parser.on("-q", "--quiet", "turn off logging") { |v| options[:quiet] = true }
|
32
|
+
parser.on("-D", "--daemonize", "run in the background") { |v| options[:daemonize] = true }
|
33
|
+
parser.on("-E", "--env ENVIRONMENT", "which environment to use (default: #{options[:environment]})") { |v| options[:environment] = v }
|
34
|
+
parser.on("-q", "--quiet", "turn off logging") { |v| options[:quiet] = true }
|
31
35
|
parser.on("-h", "--help", "print this message") do
|
32
36
|
puts parser
|
33
37
|
exit
|
@@ -35,40 +39,39 @@ OptionParser.new do |parser|
|
|
35
39
|
end.parse!
|
36
40
|
|
37
41
|
def check_database(path)
|
38
|
-
unless File.exist?(path)
|
42
|
+
unless File.exist?(path)
|
39
43
|
puts "liteboard: missing database file, please ensure the db path is correct"
|
40
44
|
puts "liteboard: exiting"
|
41
45
|
exit
|
42
46
|
end
|
43
47
|
end
|
44
48
|
|
45
|
-
check_database(options[:path]) if options[:path]
|
49
|
+
check_database(options[:path]) if options[:path]
|
46
50
|
|
47
51
|
# if there is a config file then we need to check it
|
48
52
|
config = nil
|
49
53
|
if options[:config_path]
|
50
54
|
begin
|
51
|
-
config =
|
55
|
+
config = YAML.load(ERB.new(File.read(options[:config_path])).result)
|
52
56
|
rescue
|
53
57
|
puts "liteboard: missing or bad config file, please ensure the config file path is correct"
|
54
58
|
puts "liteboard: exiting"
|
55
59
|
exit
|
56
|
-
end
|
60
|
+
end
|
57
61
|
else # no config path! use the default
|
58
|
-
config =
|
62
|
+
config = YAML.load(ERB.new(File.read(DEFAULTS[:config_path])).result) rescue nil
|
59
63
|
end
|
60
64
|
|
61
65
|
if config
|
62
66
|
if options[:path].nil?
|
63
67
|
path = config['path'] || config[options[:environment]]['path']
|
64
68
|
options[:path] = path
|
65
|
-
end
|
69
|
+
end
|
66
70
|
end
|
67
71
|
|
68
72
|
# if still no path we assume a default db path
|
69
73
|
options[:path] = DEFAULTS[:path] if options[:path].nil?
|
70
74
|
|
71
|
-
|
72
75
|
# check the validity of the path before starting the server
|
73
76
|
check_database(options[:path])
|
74
77
|
Litemetric.options = options
|
@@ -78,4 +81,4 @@ options[:app] = Liteboard.app
|
|
78
81
|
require_relative '../lib/litestack'
|
79
82
|
puts "Starting Liteboard version #{Litestack::VERSION}"
|
80
83
|
|
81
|
-
|
84
|
+
Rackup::Server.start(options)
|