solid_cache 1.0.2 → 1.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d8807bd93a8e16495e40542655163d95bcfac3b9a0d96de0ccbea687899ba600
-  data.tar.gz: 15107cc9ed202a7c8cf01d962c7ec8182eed76c3677e86155f104fac07eae6b2
+  metadata.gz: 4166bf227d59acde335e056a12f2a382863fe469fa8417827e19da72bc12fb7b
+  data.tar.gz: af1066a13d699e9ca3716c7b8b12f0b50e379b5cc5940757f613c51a08b9114a
 SHA512:
-  metadata.gz: 2c1bdc5f7ae9313d017388821a1792228b9ff59d9570cc46129f2b0127648aeda3e103b3f387758c37b4119010929f48452def1638dad43a90cb46d201db2274
-  data.tar.gz: a0352691be83a3ee2227c501235bfd7c8639ea08b717c0bcbfcf002a3901b367c142d430677afd30778cafef0918192ed4afc987b9082987524dbdc327d6e42b
+  metadata.gz: 8fe05aebd5d0279edb79654b8cd432efc78232d5f998b661d8026b23dcfb7ab73de74c407ce2594a15d01809a377711fb7bb8ffb060823619341ac0020564973
+  data.tar.gz: 78995f9165d9f0c3da70bfe8499d89cf4fefb17cd7a5c8d0971815e21688ff71303b9da040a301aed7cf6bac46d6a6245c32657d170e4b1631278327715d110f

data/README.md

@@ -1,105 +1,18 @@
 # Solid Cache
 
-**Upgrading from v0.3.0 or earlier? Please see [upgrading to version v0.4.x and beyond](upgrading_to_version_0.4.x.md)**
-
-Solid Cache is a database-backed Active Support cache store implementation.
-
-Using SQL databases backed by SSDs we can have caches that are much larger and cheaper than traditional memory-only Redis or Memcached backed caches.
-
-## Introduction
-
-Solid Cache is a FIFO (first in, first out) cache. While this is not as efficient as an LRU cache, it is mitigated by the longer cache lifespan.
-
-A FIFO cache is much easier to manage:
-1. We don't need to track when items are read.
-2. We can estimate and control the cache size by comparing the maximum and minimum IDs.
-3. By deleting from one end of the table and adding at the other end we can avoid fragmentation (on MySQL at least).
+Solid Cache is a database-backed Active Support cache store that let's you keep a much larger cache than is typically possible with traditional memory-only Redis or Memcached stores. This is thanks to the speed of modern SSD drives, which make the access-time penalty of using disk vs RAM insignificant for most caching purposes. Simply put, you're now usually better off keeping a huge cache on disk rather than a small cache in memory.
 
 ## Installation
-Add this line to your application's `Gemfile`:
-
-```ruby
-gem "solid_cache"
-```
-
-And then execute:
-```bash
-$ bundle
-```
-
-Or install it yourself as:
-```bash
-$ gem install solid_cache
-```
 
-#### Cache database configuration
+Solid Cache is configured by default in new Rails 8 applications. But if you're running an earlier version, you can add it manually following these steps:
 
-The default installation of Solid Cache expects a database named `cache` in `database.yml`. It should
-have it's own connection pool to avoid mixing cache queries in other transactions.
-
-You can use the primary database for your cache like this:
-
-```yaml
-# config/database.yml
-production:
-  primary: &production_primary
-    ...
-  cache:
-    <<: *production_primary
-```
-
-Or a separate database like this:
-
-```yaml
-production:
-  primary:
-    ...
-  cache:
-    database: cache_development
-    host: 127.0.0.1
-    migrations_paths: "db/cache/migrate"
-```
+1. `bundle add solid_cache`
+2. `bin/rails solid_cache:install`
+3. `bin/rails db:migrate`
 
-#### Install Solid Cache
+This will configure Solid Queue as the production cache store, create `config/solid_cache.yml`, and alter `config/database.yml` to include the new cache database.
 
-Now, you need to install the necessary migrations and configure the cache store. You can do both at once using the provided generator:
-
-```bash
-# If using the primary database
-$ bin/rails generate solid_cache:install
-
-# Or if using a dedicated database
-$ DATABASE=cache bin/rails generate solid_cache:install
-```
-
-This will set solid_cache as the cache store in production, and will copy the optional configuration file and the required migration over to your app.
-
-Alternatively, you can add only the migration to your app:
-
-```bash
-# If using the primary database
-$ bin/rails generate solid_cache:install:migrations
-
-# Or if using a dedicated database
-$ DATABASE=cache bin/rails generate solid_cache:install:migrations
-```
-
-And set Solid Cache as your application's cache store backend manually, in your environment config:
-
-```ruby
-# config/environments/production.rb
-config.cache_store = :solid_cache_store
-```
-
-#### Run migrations
-
-Finally, you need to run the migrations:
-
-```bash
-$ bin/rails db:migrate
-```
-
-### Configuration
+## Configuration
 
 Configuration will be read from `config/solid_cache.yml`. You can change the location of the config file by setting the `SOLID_CACHE_CONFIG` env variable.
 
@@ -127,7 +40,7 @@ production: &production
 
 For the full list of keys for `store_options` see [Cache configuration](#cache-configuration). Any options passed to the cache lookup will overwrite those specified here.
 
-#### Connection configuration
+### Connection configuration
 
 You can set one of `database`, `databases` and `connects_to` in the config file. They will be used to configure the cache databases in `SolidCache::Record#connects_to`.
 
@@ -145,10 +58,9 @@ SolidCache::Record.connects_to shards: { cache_db1: { writing: :cache_db1 },  ca
 
 If `connects_to` is set, it will be passed directly.
 
-If none of these are set, Solid Cache will use the `ActiveRecord::Base` connection pool. This means that cache reads and writes will be part of any wrapping
-database transaction.
+If none of these are set, Solid Cache will use the `ActiveRecord::Base` connection pool. This means that cache reads and writes will be part of any wrapping database transaction.
 
-#### Engine configuration
+### Engine configuration
 
 There are three options that can be set on the engine:
 
@@ -166,11 +78,11 @@ Rails.application.configure do
 end
 ```
 
-#### Cache configuration
+### Cache configuration
 
 Solid Cache supports these options in addition to the standard `ActiveSupport::Cache::Store` options:
 
-- `error_handler` - a Proc to call to handle any `ActiveRecord::ActiveRecordError`s that are raises (default: log errors as warnings)
+- `error_handler` - a Proc to call to handle any transient database errors that are raised (default: log errors as warnings)
 - `expiry_batch_size` - the batch size to use when deleting old records (default: `100`)
 - `expiry_method` - what expiry method to use `thread` or `job` (default: `thread`)
 - `expiry_queue` - which queue to add expiry jobs to (default: `default`)
@@ -186,7 +98,7 @@ Solid Cache supports these options in addition to the standard `ActiveSupport::C
 
 For more information on cache clusters, see [Sharding the cache](#sharding-the-cache)
 
-### Cache expiry
+## Cache expiry
 
 Solid Cache tracks writes to the cache. For every write it increments a counter by 1. Once the counter reaches 50% of the `expiry_batch_size` it adds a task to run on a background thread. That task will:
 
@@ -202,7 +114,7 @@ Only triggering expiry when we write means that if the cache is idle, the backgr
 
 If you want the cache expiry to be run in a background job instead of a thread, you can set `expiry_method` to `:job`. This will enqueue a `SolidCache::ExpiryJob`.
 
-### Sharding the cache
+## Sharding the cache
 
 Solid Cache uses the [Maglev](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf) consistent hashing scheme to shard the cache across multiple databases.
 
@@ -233,7 +145,7 @@ production:
   databases: [cache_shard1, cache_shard2, cache_shard3]
 ```
 
-### Enabling encryption
+## Enabling encryption
 
 To encrypt the cache values, you can add set the encrypt property.
 
@@ -271,7 +183,7 @@ config.solid_cache.encryption_context_properties = {
 Encryption currently does not work for PostgreSQL, as Rails does not yet support encrypting binary columns for it.
 See https://github.com/rails/rails/pull/52650.
 
-### Index size limits
+## Index size limits
 The Solid Cache migrations try to create an index with 1024 byte entries. If that is too big for your database, you should:
 
 1. Edit the index size in the migration.
@@ -322,5 +234,19 @@ $ appraisal update
 
 This ensures that all the Rails versions dependencies are updated.
 
+## Implementation
+
+Solid Cache is a FIFO (first in, first out) cache. While this is not as efficient as an LRU cache, it is mitigated by the longer cache lifespan.
+
+A FIFO cache is much easier to manage:
+1. We don't need to track when items are read.
+2. We can estimate and control the cache size by comparing the maximum and minimum IDs.
+3. By deleting from one end of the table and adding at the other end we can avoid fragmentation (on MySQL at least).
+
+
+## Upgrading
+
+**Upgrading from v0.3.0 or earlier? Please see [upgrading to version v0.4.x and beyond](upgrading_to_version_0.4.x.md)**
+
 ## License
 Solid Cache is licensed under MIT.

data/lib/generators/solid_cache/install/install_generator.rb

@@ -3,22 +3,70 @@
 class SolidCache::InstallGenerator < Rails::Generators::Base
   source_root File.expand_path("templates", __dir__)
 
-  class_option :skip_migrations,    type: :boolean, default: nil,
-                                    desc: "Skip migrations"
-
   def add_rails_cache
-    if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
-      gsub_file env_config, /(# )?config\.cache_store = (:.*)/, "config.cache_store = :solid_cache_store"
-    end
+    gsub_file app_root.join("config/environments/production.rb"),
+      /(# )?config\.cache_store = (:.*)/, "config.cache_store = :solid_cache_store"
   end
 
   def create_config_solid_cache_yml
     template "config/solid_cache.yml"
   end
 
-  def create_migrations
-    unless options[:skip_migrations]
-      rails_command "railties:install:migrations FROM=solid_cache", inline: true
+  def add_cache_db_to_database_yml
+    if app_is_using_sqlite?
+      gsub_file database_yml, /production:\s*<<: \*default.*/m, sqlite_database_config_with_cache
+    else
+      gsub_file database_yml, /production:\s*<<: \*default.*/m, generic_database_config_with_cache
     end
   end
+
+  def add_solid_cache_db_schema
+    template "db/cache_schema.rb"
+  end
+
+  private
+    def app_root
+      Pathname.new(destination_root)
+    end
+
+    def database_yml
+      app_root.join("config/database.yml")
+    end
+
+    def app_is_using_sqlite?
+      database_yml.read.match?(/production:.*sqlite3/m)
+    end
+
+    def sqlite_database_config_with_cache
+      <<~YAML
+        production:
+          primary:
+            <<: *default
+            database: storage/production.sqlite3
+          cache:
+            <<: *default
+            database: storage/production_cache.sqlite3
+            migrations_paths: db/cache_migrate
+      YAML
+    end
+
+    def app_name_from_production_database_name
+      database_yml.read.scan(/database: (\w+)_production/).flatten.first
+    end
+
+    def generic_database_config_with_cache
+      app_name = app_name_from_production_database_name
+
+      <<~YAML
+production:
+  primary: &production_primary
+    <<: *default
+    database: #{app_name}_production
+    username: #{app_name}
+    password: <%= ENV["#{app_name.upcase}_DATABASE_PASSWORD"] %>
+  cache:
+    <<: *production_primary
+    database: #{app_name}_production_cache
+      YAML
+    end
 end

data/lib/generators/solid_cache/install/templates/config/solid_cache.yml.tt

@@ -1,6 +1,8 @@
 default: &default
   database: <%= ENV.fetch("DATABASE", "cache") %>
   store_options:
+    # Cap age of oldest cache entry to fulfill retention policies
+    # max_age: <%%= 60.days.to_i %>
     max_size: <%%= 256.megabytes %>
     namespace: <%%= Rails.env %>
 

data/lib/generators/solid_cache/install/templates/db/cache_schema.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+ActiveRecord::Schema[8.0].define(version: 1) do
+  create_table "solid_cache_entries", force: :cascade do |t|
+    t.binary "key", limit: 1024, null: false
+    t.binary "value", limit: 536870912, null: false
+    t.datetime "created_at", null: false
+    t.integer "key_hash", limit: 8, null: false
+    t.integer "byte_size", limit: 4, null: false
+    t.index ["byte_size"], name: "index_solid_cache_entries_on_byte_size"
+    t.index ["key_hash", "byte_size"], name: "index_solid_cache_entries_on_key_hash_and_byte_size"
+    t.index ["key_hash"], name: "index_solid_cache_entries_on_key_hash", unique: true
+  end
+end

data/lib/solid_cache/engine.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "active_support"
+require "active_record"
 
 module SolidCache
   class Engine < ::Rails::Engine

data/lib/solid_cache/store/failsafe.rb

@@ -3,6 +3,16 @@
 module SolidCache
   class Store
     module Failsafe
+      TRANSIENT_ACTIVE_RECORD_ERRORS = [
+        ActiveRecord::AdapterTimeout,
+        ActiveRecord::ConnectionNotEstablished,
+        ActiveRecord::ConnectionTimeoutError,
+        ActiveRecord::Deadlocked,
+        ActiveRecord::LockWaitTimeout,
+        ActiveRecord::QueryCanceled,
+        ActiveRecord::StatementTimeout
+      ]
+
       DEFAULT_ERROR_HANDLER = ->(method:, returning:, exception:) do
         if Store.logger
           Store.logger.error { "SolidCacheStore: #{method} failed, returned #{returning.inspect}: #{exception.class}: #{exception.message}" }
@@ -20,7 +30,7 @@ module SolidCache
 
         def failsafe(method, returning: nil)
           yield
-        rescue ActiveRecord::ActiveRecordError => error
+        rescue *TRANSIENT_ACTIVE_RECORD_ERRORS => error
           ActiveSupport.error_reporter&.report(error, handled: true, severity: :warning)
           error_handler&.call(method: method, exception: error, returning: returning)
           returning

data/lib/solid_cache/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SolidCache
-  VERSION = "1.0.2"
+  VERSION = "1.0.4"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_cache
 version: !ruby/object:Gem::Version
-  version: 1.0.2
+  version: 1.0.4
 platform: ruby
 authors:
 - Donal McBreen
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-09-02 00:00:00.000000000 Z
+date: 2024-09-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -112,11 +112,11 @@ files:
 - app/models/solid_cache/entry/size/estimate.rb
 - app/models/solid_cache/entry/size/moving_average_estimate.rb
 - app/models/solid_cache/record.rb
-- db/migrate/20240820123641_create_solid_cache_entries.rb
 - lib/active_support/cache/solid_cache_store.rb
 - lib/generators/solid_cache/install/USAGE
 - lib/generators/solid_cache/install/install_generator.rb
 - lib/generators/solid_cache/install/templates/config/solid_cache.yml.tt
+- lib/generators/solid_cache/install/templates/db/cache_schema.rb
 - lib/solid_cache.rb
 - lib/solid_cache/configuration.rb
 - lib/solid_cache/connections.rb

data/db/migrate/20240820123641_create_solid_cache_entries.rb

@@ -1,29 +0,0 @@
-class CreateSolidCacheEntries < ActiveRecord::Migration[7.2]
-  def up
-    create_table :solid_cache_entries, if_not_exists: true do |t|
-      t.binary   :key,        null: false,   limit: 1024
-      t.binary   :value,      null: false,   limit: 512.megabytes
-      t.datetime :created_at, null: false
-      t.integer :key_hash,    null: false,    limit: 8
-      t.integer :byte_size,   null: false,    limit: 4
-
-      t.index  :key_hash, unique: true
-      t.index  [:key_hash, :byte_size]
-      t.index  :byte_size
-    end
-
-    raise "column \"key_hash\" does not exist" unless column_exists? :solid_cache_entries, :key_hash
-  rescue => e
-    if e.message =~ /(column "key_hash" does not exist|no such column: key_hash)/
-      raise \
-        "Could not find key_hash column on solid_cache_entries, if upgrading from v0.3 or earlier, have you followed " \
-        "the steps in https://github.com/rails/solid_cache/blob/main/upgrading_to_version_0.4.x.md?"
-    else
-      raise
-    end
-  end
-
-  def down
-    drop_table :solid_cache_entries
-  end
-end