solid_cache 1.0.1 → 1.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0ef3fc7a701fc5af744a290377eae445564a040ccb938e2ea64caf1bab5e90da
-  data.tar.gz: dffd9ee1ab8ea5f6f96620dfcaba35092bd7d5fd9ff5bed210271666e2b839fe
+  metadata.gz: 557f25039474c31f96924767eaa99756e7171355f531239fb8452770d7ceb8fd
+  data.tar.gz: 2b719db3b62c8b2711961e4ab5ce7a9543fdb03f33739a2e48ead12c80af284a
 SHA512:
-  metadata.gz: 22441a9d3132510beff5347788844f6bb05a7da369220ccdc2228587aabb8c2cd51dfb81eecff7ffc0febc0d8b1dea2f1ed852e6393e2a7bfa0af01418913d2a
-  data.tar.gz: 83b296209289d1399ac3b2b3857b6b2740d7d1da6e51f43ef9873e77179681ee65e05f6ca50afd9c10f0605279df637fa8ef8ba7dedd0f4b6a19349aa086c553
+  metadata.gz: 25187adeb93d8e1edfdfc4e754a9c064b61e128a3d5ca96a3c7b8d523b61aca3cb689878be0079c6cd387baf91cd5de8c57ae331a6efa82d087fe0123587e1bd
+  data.tar.gz: 55ed9ccacea8c5de554147d838a6ccb4c64970d1c8479c226114179bfd93e2680f2b1d77196b2b86393047bd2052e11df44310c4f24b6002735ad5bba5c0dcd3

data/README.md

@@ -1,105 +1,18 @@
 # Solid Cache
 
-**Upgrading from v0.3.0 or earlier? Please see [upgrading to version v0.4.x and beyond](upgrading_to_version_0.4.x.md)**
-
-Solid Cache is a database-backed Active Support cache store implementation.
-
-Using SQL databases backed by SSDs we can have caches that are much larger and cheaper than traditional memory-only Redis or Memcached backed caches.
-
-## Introduction
-
-Solid Cache is a FIFO (first in, first out) cache. While this is not as efficient as an LRU cache, it is mitigated by the longer cache lifespan.
-
-A FIFO cache is much easier to manage:
-1. We don't need to track when items are read.
-2. We can estimate and control the cache size by comparing the maximum and minimum IDs.
-3. By deleting from one end of the table and adding at the other end we can avoid fragmentation (on MySQL at least).
+Solid Cache is a database-backed Active Support cache store that let's you keep a much larger cache than is typically possible with traditional memory-only Redis or Memcached stores. This is thanks to the speed of modern SSD drives, which make the access-time penalty of using disk vs RAM insignificant for most caching purposes. Simply put, you're now usually better off keeping a huge cache on disk rather than a small cache in memory.
 
 ## Installation
-Add this line to your application's `Gemfile`:
-
-```ruby
-gem "solid_cache"
-```
-
-And then execute:
-```bash
-$ bundle
-```
-
-Or install it yourself as:
-```bash
-$ gem install solid_cache
-```
 
-#### Cache database configuration
+Solid Cache is configured by default in new Rails 8 applications. But if you're running an earlier version, you can add it manually following these steps:
 
-The default installation of Solid Cache expects a database named `cache` in `database.yml`. It should
-have it's own connection pool to avoid mixing cache queries in other transactions.
-
-You can use the primary database for your cache like this:
-
-```yaml
-# config/database.yml
-production:
-  primary: &production_primary
-    ...
-  cache:
-    <<: *production_primary
-```
-
-Or a separate database like this:
-
-```yaml
-production:
-  primary:
-    ...
-  cache:
-    database: cache_development
-    host: 127.0.0.1
-    migrations_paths: "db/cache/migrate"
-```
+1. `bundle add solid_cache`
+2. `bin/rails solid_cache:install`
+3. `bin/rails db:migrate`
 
-#### Install Solid Cache
+This will configure Solid Queue as the production cache store, create `config/solid_cache.yml`, and alter `config/database.yml` to include the new cache database.
 
-Now, you need to install the necessary migrations and configure the cache store. You can do both at once using the provided generator:
-
-```bash
-# If using the primary database
-$ bin/rails generate solid_cache:install
-
-# Or if using a dedicated database
-$ DATABASE=cache bin/rails generate solid_cache:install
-```
-
-This will set solid_cache as the cache store in production, and will copy the optional configuration file and the required migration over to your app.
-
-Alternatively, you can add only the migration to your app:
-
-```bash
-# If using the primary database
-$ bin/rails generate solid_cache:install:migrations
-
-# Or if using a dedicated database
-$ DATABASE=cache bin/rails generate solid_cache:install:migrations
-```
-
-And set Solid Cache as your application's cache store backend manually, in your environment config:
-
-```ruby
-# config/environments/production.rb
-config.cache_store = :solid_cache_store
-```
-
-#### Run migrations
-
-Finally, you need to run the migrations:
-
-```bash
-$ bin/rails db:migrate
-```
-
-### Configuration
+## Configuration
 
 Configuration will be read from `config/solid_cache.yml`. You can change the location of the config file by setting the `SOLID_CACHE_CONFIG` env variable.
 
@@ -127,7 +40,7 @@ production: &production
 
 For the full list of keys for `store_options` see [Cache configuration](#cache-configuration). Any options passed to the cache lookup will overwrite those specified here.
 
-#### Connection configuration
+### Connection configuration
 
 You can set one of `database`, `databases` and `connects_to` in the config file. They will be used to configure the cache databases in `SolidCache::Record#connects_to`.
 
@@ -145,10 +58,9 @@ SolidCache::Record.connects_to shards: { cache_db1: { writing: :cache_db1 },  ca
 
 If `connects_to` is set, it will be passed directly.
 
-If none of these are set, Solid Cache will use the `ActiveRecord::Base` connection pool. This means that cache reads and writes will be part of any wrapping
-database transaction.
+If none of these are set, Solid Cache will use the `ActiveRecord::Base` connection pool. This means that cache reads and writes will be part of any wrapping database transaction.
 
-#### Engine configuration
+### Engine configuration
 
 There are three options that can be set on the engine:
 
@@ -166,11 +78,11 @@ Rails.application.configure do
 end
 ```
 
-#### Cache configuration
+### Cache configuration
 
 Solid Cache supports these options in addition to the standard `ActiveSupport::Cache::Store` options:
 
-- `error_handler` - a Proc to call to handle any `ActiveRecord::ActiveRecordError`s that are raises (default: log errors as warnings)
+- `error_handler` - a Proc to call to handle any transient database errors that are raised (default: log errors as warnings)
 - `expiry_batch_size` - the batch size to use when deleting old records (default: `100`)
 - `expiry_method` - what expiry method to use `thread` or `job` (default: `thread`)
 - `expiry_queue` - which queue to add expiry jobs to (default: `default`)
@@ -186,7 +98,7 @@ Solid Cache supports these options in addition to the standard `ActiveSupport::C
 
 For more information on cache clusters, see [Sharding the cache](#sharding-the-cache)
 
-### Cache expiry
+## Cache expiry
 
 Solid Cache tracks writes to the cache. For every write it increments a counter by 1. Once the counter reaches 50% of the `expiry_batch_size` it adds a task to run on a background thread. That task will:
 
@@ -202,7 +114,7 @@ Only triggering expiry when we write means that if the cache is idle, the backgr
 
 If you want the cache expiry to be run in a background job instead of a thread, you can set `expiry_method` to `:job`. This will enqueue a `SolidCache::ExpiryJob`.
 
-### Sharding the cache
+## Sharding the cache
 
 Solid Cache uses the [Maglev](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf) consistent hashing scheme to shard the cache across multiple databases.
 
@@ -233,7 +145,7 @@ production:
   databases: [cache_shard1, cache_shard2, cache_shard3]
 ```
 
-### Enabling encryption
+## Enabling encryption
 
 To encrypt the cache values, you can add set the encrypt property.
 
@@ -271,7 +183,7 @@ config.solid_cache.encryption_context_properties = {
 Encryption currently does not work for PostgreSQL, as Rails does not yet support encrypting binary columns for it.
 See https://github.com/rails/rails/pull/52650.
 
-### Index size limits
+## Index size limits
 The Solid Cache migrations try to create an index with 1024 byte entries. If that is too big for your database, you should:
 
 1. Edit the index size in the migration.
@@ -322,5 +234,19 @@ $ appraisal update
 
 This ensures that all the Rails versions dependencies are updated.
 
+## Implementation
+
+Solid Cache is a FIFO (first in, first out) cache. While this is not as efficient as an LRU cache, it is mitigated by the longer cache lifespan.
+
+A FIFO cache is much easier to manage:
+1. We don't need to track when items are read.
+2. We can estimate and control the cache size by comparing the maximum and minimum IDs.
+3. By deleting from one end of the table and adding at the other end we can avoid fragmentation (on MySQL at least).
+
+
+## Upgrading
+
+**Upgrading from v0.3.0 or earlier? Please see [upgrading to version v0.4.x and beyond](upgrading_to_version_0.4.x.md)**
+
 ## License
 Solid Cache is licensed under MIT.

data/Rakefile

@@ -36,7 +36,7 @@ end
 configs.each do |config|
   namespace :test do
     task config do
-      if config.to_s.start_with?("encrypted") && ENV["TARGET_DB"] == "postgres"
+      if config.to_s.start_with?("encrypted") && ENV["TARGET_DB"] == "postgres" && Rails::VERSION::MAJOR <= 7
         puts "Skipping encrypted tests on PostgreSQL as binary encrypted columns are not supported by Rails yet"
         next
       end

data/app/models/solid_cache/entry.rb

@@ -33,7 +33,9 @@ module SolidCache
 
       def read_multi(keys)
         without_query_cache do
-          find_by_sql([select_sql(keys), *key_hashes_for(keys)]).pluck(:key, :value).to_h
+          query = Arel.sql(select_sql(keys), *key_hashes_for(keys))
+
+          connection.select_all(query, "SolidCache::Entry Load").cast_values(attribute_types).to_h
         end
       end
 
@@ -84,13 +86,24 @@ module SolidCache
           connection.supports_insert_conflict_target? ? :key_hash : nil
         end
 
+        # This constructs and caches a SQL query for a given number of keys.
+        #
+        # The query is constructed with two bind parameters to generate an IN (...) condition,
+        # which is then replaced with the correct amount based on the number of keys. The
+        # parameters are filled later when executing the query. This is done through Active Record
+        # to ensure the field and table names are properly quoted and escaped based on the used database adapter.
+
+        # For example: The query for 4 keys will be transformed from:
+        # > SELECT "key", "value" FROM "solid_cache_entries" WHERE "key_hash" IN (1111, 2222)
+        # into:
+        # > SELECT "key", "value" FROM "solid_cache_entries" WHERE "key_hash" IN (?, ?, ?, ?)
         def select_sql(keys)
-          @get_sql ||= {}
-          @get_sql[keys.count] ||= \
-            where(key_hash: [ "1111", "2222" ])
+          @select_sql ||= {}
+          @select_sql[keys.count] ||= \
+            where(key_hash: [ 1111, 2222 ])
               .select(:key, :value)
               .to_sql
-              .gsub("1111, 2222", (["?"] * keys.count).join(", "))
+              .gsub("1111, 2222", Array.new(keys.count, "?").join(", "))
         end
 
         def key_hash_for(key)

data/lib/generators/solid_cache/install/install_generator.rb

@@ -3,22 +3,70 @@
 class SolidCache::InstallGenerator < Rails::Generators::Base
   source_root File.expand_path("templates", __dir__)
 
-  class_option :skip_migrations,    type: :boolean, default: nil,
-                                    desc: "Skip migrations"
-
   def add_rails_cache
-    if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
-      gsub_file env_config, /(# )?config\.cache_store = (:.*)/, "config.cache_store = :solid_cache_store"
-    end
+    gsub_file app_root.join("config/environments/production.rb"),
+      /(# )?config\.cache_store = (:.*)/, "config.cache_store = :solid_cache_store"
   end
 
   def create_config_solid_cache_yml
     template "config/solid_cache.yml"
   end
 
-  def create_migrations
-    unless options[:skip_migrations]
-      rails_command "railties:install:migrations FROM=solid_cache", inline: true
+  def add_cache_db_to_database_yml
+    if app_is_using_sqlite?
+      gsub_file database_yml, /production:\s*<<: \*default.*/m, sqlite_database_config_with_cache
+    else
+      gsub_file database_yml, /production:\s*<<: \*default.*/m, generic_database_config_with_cache
     end
   end
+
+  def add_solid_cache_db_schema
+    template "db/cache_schema.rb"
+  end
+
+  private
+    def app_root
+      Pathname.new(destination_root)
+    end
+
+    def database_yml
+      app_root.join("config/database.yml")
+    end
+
+    def app_is_using_sqlite?
+      database_yml.read.match?(/production:.*sqlite3/m)
+    end
+
+    def sqlite_database_config_with_cache
+      <<~YAML
+        production:
+          primary:
+            <<: *default
+            database: storage/production.sqlite3
+          cache:
+            <<: *default
+            database: storage/production_cache.sqlite3
+            migrations_paths: db/cache_migrate
+      YAML
+    end
+
+    def app_name_from_production_database_name
+      database_yml.read.scan(/database: (\w+)_production/).flatten.first
+    end
+
+    def generic_database_config_with_cache
+      app_name = app_name_from_production_database_name
+
+      <<~YAML
+production:
+  primary: &production_primary
+    <<: *default
+    database: #{app_name}_production
+    username: #{app_name}
+    password: <%= ENV["#{app_name.upcase}_DATABASE_PASSWORD"] %>
+  cache:
+    <<: *production_primary
+    database: #{app_name}_production_cache
+      YAML
+    end
 end

data/lib/generators/solid_cache/install/templates/config/solid_cache.yml.tt

@@ -1,6 +1,8 @@
 default: &default
   database: <%= ENV.fetch("DATABASE", "cache") %>
   store_options:
+    # Cap age of oldest cache entry to fulfill retention policies
+    # max_age: <%%= 60.days.to_i %>
     max_size: <%%= 256.megabytes %>
     namespace: <%%= Rails.env %>
 

data/lib/generators/solid_cache/install/templates/db/cache_schema.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+ActiveRecord::Schema[8.0].define(version: 1) do
+  create_table "solid_cache_entries", force: :cascade do |t|
+    t.binary "key", limit: 1024, null: false
+    t.binary "value", limit: 536870912, null: false
+    t.datetime "created_at", null: false
+    t.integer "key_hash", limit: 8, null: false
+    t.integer "byte_size", limit: 4, null: false
+    t.index ["byte_size"], name: "index_solid_cache_entries_on_byte_size"
+    t.index ["key_hash", "byte_size"], name: "index_solid_cache_entries_on_key_hash_and_byte_size"
+    t.index ["key_hash"], name: "index_solid_cache_entries_on_key_hash", unique: true
+  end
+end

data/lib/solid_cache/engine.rb

@@ -37,9 +37,9 @@ module SolidCache
     end
 
     config.after_initialize do
-      if SolidCache.configuration.encrypt? && SolidCache::Record.connection.adapter_name == "PostgreSQL"
+      if SolidCache.configuration.encrypt? && Record.connection.adapter_name == "PostgreSQL" && Rails::VERSION::MAJOR <= 7
         raise \
-          "Cannot enable encryption for Solid Cache: Active Record Encryption does not currently support " \
+          "Cannot enable encryption for Solid Cache: in Rails 7, Active Record Encryption does not support " \
           "encrypting binary columns on PostgreSQL"
       end
     end

data/lib/solid_cache/store/api.rb

@@ -15,11 +15,21 @@ module SolidCache
       end
 
       def increment(name, amount = 1, options = nil)
-        adjust(name, amount, options)
+        options = merged_options(options)
+        key = normalize_key(name, options)
+
+        instrument :increment, key, amount: amount do
+          adjust(name, amount, options)
+        end
       end
 
       def decrement(name, amount = 1, options = nil)
-        adjust(name, -amount, options)
+        options = merged_options(options)
+        key = normalize_key(name, options)
+
+        instrument :decrement, key, amount: amount do
+          adjust(name, -amount, options)
+        end
       end
 
       def cleanup(options = nil)

data/lib/solid_cache/store/failsafe.rb

@@ -3,6 +3,16 @@
 module SolidCache
   class Store
     module Failsafe
+      TRANSIENT_ACTIVE_RECORD_ERRORS = [
+        ActiveRecord::AdapterTimeout,
+        ActiveRecord::ConnectionNotEstablished,
+        ActiveRecord::ConnectionTimeoutError,
+        ActiveRecord::Deadlocked,
+        ActiveRecord::LockWaitTimeout,
+        ActiveRecord::QueryCanceled,
+        ActiveRecord::StatementTimeout
+      ]
+
       DEFAULT_ERROR_HANDLER = ->(method:, returning:, exception:) do
         if Store.logger
           Store.logger.error { "SolidCacheStore: #{method} failed, returned #{returning.inspect}: #{exception.class}: #{exception.message}" }
@@ -20,7 +30,7 @@ module SolidCache
 
         def failsafe(method, returning: nil)
           yield
-        rescue ActiveRecord::ActiveRecordError => error
+        rescue *TRANSIENT_ACTIVE_RECORD_ERRORS => error
           ActiveSupport.error_reporter&.report(error, handled: true, severity: :warning)
           error_handler&.call(method: method, exception: error, returning: returning)
           returning

data/lib/solid_cache/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SolidCache
-  VERSION = "1.0.1"
+  VERSION = "1.0.3"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_cache
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 1.0.3
 platform: ruby
 authors:
 - Donal McBreen
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-08-22 00:00:00.000000000 Z
+date: 2024-09-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -112,11 +112,11 @@ files:
 - app/models/solid_cache/entry/size/estimate.rb
 - app/models/solid_cache/entry/size/moving_average_estimate.rb
 - app/models/solid_cache/record.rb
-- db/migrate/20240820123641_create_solid_cache_entries.rb
 - lib/active_support/cache/solid_cache_store.rb
 - lib/generators/solid_cache/install/USAGE
 - lib/generators/solid_cache/install/install_generator.rb
 - lib/generators/solid_cache/install/templates/config/solid_cache.yml.tt
+- lib/generators/solid_cache/install/templates/db/cache_schema.rb
 - lib/solid_cache.rb
 - lib/solid_cache/configuration.rb
 - lib/solid_cache/connections.rb
@@ -158,7 +158,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.11
+rubygems_version: 3.5.17
 signing_key:
 specification_version: 4
 summary: A database backed ActiveSupport::Cache::Store

data/db/migrate/20240820123641_create_solid_cache_entries.rb

@@ -1,29 +0,0 @@
-class CreateSolidCacheEntries < ActiveRecord::Migration[7.2]
-  def up
-    create_table :solid_cache_entries, if_not_exists: true do |t|
-      t.binary   :key,        null: false,   limit: 1024
-      t.binary   :value,      null: false,   limit: 512.megabytes
-      t.datetime :created_at, null: false
-      t.integer :key_hash,    null: false,    limit: 8
-      t.integer :byte_size,   null: false,    limit: 4
-
-      t.index  :key_hash, unique: true
-      t.index  [:key_hash, :byte_size]
-      t.index  :byte_size
-    end
-
-    raise "column \"key_hash\" does not exist" unless column_exists? :solid_cache_entries, :key_hash
-  rescue => e
-    if e.message =~ /(column "key_hash" does not exist|no such column: key_hash)/
-      raise \
-        "Could not find key_hash column on solid_cache_entries, if upgrading from v0.3 or earlier, have you followed " \
-        "the steps in https://github.com/rails/solid_cache/blob/main/upgrading_to_version_0.4.x.md?"
-    else
-      raise
-    end
-  end
-
-  def down
-    drop_table :solid_cache_entries
-  end
-end