readyset 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: a7eff87ba2d4dfc95a623d57486cefc986da1ef725d7fd4193b3611aa86b72be
+  data.tar.gz: fd8ce3ee3520d4943f759c716877d5ec82e185101403316529031346aec01d7d
+SHA512:
+  metadata.gz: bf03429429fec9e83f2d7a82dd9985302865f724f59fe93612b862a78447a5a210f024dbca1235e11ac1ae28edb277b6ba0b712a770ed09a2905b78b5a4525e9
+  data.tar.gz: 6979425963dc743d812707ce5d2e89c871ba779c2c85de035afa8c285c68a188b25fe43991752937aff6edb01829097edbe2a46eb380c0c058b031f814be56a2

data/.rspec

@@ -0,0 +1 @@
+--require rails_helper

data/.rubocop.yml

@@ -0,0 +1,15 @@
+inherit_from:
+  - .rubocop_airbnb.yml
+
+AllCops:
+  TargetRubyVersion: 3.2
+  NewCops: disable
+
+Style/StringLiterals:
+  EnforcedStyle: single_quotes
+  Enabled: true
+
+Style/StringLiteralsInInterpolation:
+  EnforcedStyle: single_quotes
+  Enabled: true
+

data/.rubocop_airbnb.yml

@@ -0,0 +1,2 @@
+require:
+  - rubocop-airbnb

data/.ruby-version

@@ -0,0 +1 @@
+3.2.1

data/Gemfile

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in readyset.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 ReadySet Technology Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,383 @@
+# ReadySet Rails
+
+A gem for caching with [ReadySet](https://readyset.io) within Rails applications.
+
+![Build status](https://github.com/readysettech/readyset-rails/actions/workflows/rspec.yml/badge.svg)
+[![Number of GitHub issues that are open](https://img.shields.io/github/issues/readysettech/readyset-rails)](https://github.com/readysettech/readyset-rails/issues)
+![Number of GitHub closed issues](https://img.shields.io/github/issues-closed/readysettech/readyset-rails)
+![Number of GitHub pull requests that are open](https://img.shields.io/github/issues-pr-raw/readysettech/readyset-rails)
+![GitHub release; latest by date](https://img.shields.io/github/v/release/readysettech/readyset-rails)
+[![Slack](https://img.shields.io/badge/Join%20Slack-gray?logo=slack&logoColor=white)](https://join.slack.com/t/readysetcommunity/shared_invite/zt-2272gtiz4-0024xeRJUPGWlRETQrGkFw)
+[![Follow us on X, formerly Twitter](https://img.shields.io/twitter/follow/ReadySet?style=social)](https://twitter.com/readysetio)
+
+:star: If you find this gem useful, please consider giving us a star on GitHub! Your support helps us continue to innovate and deliver exciting new features.
+
+## Table of Contents
+- [What is ReadySet?](#what-is-readyset)
+- [What does this gem do?](#what-does-this-gem-do)
+- [Installing](#installing)
+- [Quickstart](#quickstart)
+- [Usage](#usage)
+    - [Getting Started with Caching](#getting-started-with-caching)
+        - [Query Routing in Controllers](#query-routing-in-controllers)
+        - [Query Routing in Models](#query-routing-in-models)
+    - [Cache Migrations](#cache-migrations)
+    - [Automatic Failover](#automatic-failover)
+- [Configuration Options](#configuration-options)
+- [License](#license)
+
+## What is ReadySet?
+
+[ReadySet](https://readyset.io) is a database acceleration engine that acts as
+a read replica and implements a novel type of query caching that automatically 
+keeps your caches up-to-date by watching your database's replication stream.
+ReadySet helps you scale your database by shedding load and reducing query
+latency.
+
+## What does this gem do?
+
+This gem makes it easy to use ReadySet from within your Rails application by
+allowing you to selectively route queries to ReadySet. The high-level features
+of this gem include:
+
+- A `Readyset.route` method that takes a block and routes to ReadySet any
+  queries within that block
+- A controller extension that allows you to route to ReadySet all queries
+  that occur within specific controller actions
+- A model extension that allows you to define queries to be routed to ReadySet
+  within the context of an existing model
+- Rake tasks that allow you to easily manage cache migration on ReadySet,
+  ensuring that a consistent set of caches exists across all of your
+  environments
+- Automatic failover back to your primary database in the event that your
+  ReadySet instance is unreachable (disabled by default)
+
+**Note that ReadySet only guarantees support for PostgreSQL right now, so this
+gem only supports PostgreSQL.**
+
+## Installing
+
+If you run into any trouble with the below steps, please feel free to reach
+out via our [community Slack](https://join.slack.com/t/readysetcommunity/shared_invite/zt-2272gtiz4-0024xeRJUPGWlRETQrGkFw)!
+
+1. Follow the instructions [here](https://docs.readyset.io/get-started/install-rs/postgres)
+   to install and run ReadySet
+2. Add the following line to your Gemfile and run `bundle install`:
+   ```sh
+   gem 'readyset'
+   ```
+   The ReadySet Rails gem currenly supports Ruby versions >= 3.0 and Rails
+   versions >= 6.1.
+3. Add a section to your `config/database.yml` file with ReadySet's connection
+   information. If you currently connect to only one database, you'll need to
+   move your primary database connection information to be nested under a new
+   key named `primary`.
+   ```yaml
+   development:
+     primary:
+       # This is the connection information for your primary database
+       database: testdb
+       username: postgres
+       password: readyset
+       adapter: postgresql
+       port: 5432
+     readyset:
+       # This is the connection information for ReadySet
+       database: testdb
+       username: postgres
+       password: readyset
+       adapter: readyset
+       host: "127.0.0.1"
+       port: 5433
+       # This setting tells Rails that there's no need to run migrations or other database
+       # administration tasks against ReadySet
+       database_tasks: false
+   ```
+   You can verify that ReadySet is up and your application is connected by
+   running `rails readyset:status`:
+   ```sh
+   $ rails readyset:status
+   +----------------------------+------------------------+
+   | Database Connection        | Connected              |
+   | Connection Count           | 1                      |
+   | Snapshot Status            | Completed              |
+   | Maximum Replication Offset | (0/6DBBD78, 0/6DBBFF0) |
+   | Minimum Replication Offset | (0/6DBBD78, 0/6DBBFF0) |
+   | Last started Controller    | 2024-01-17 18:49:02    |
+   | Last completed snapshot    | 2024-01-19 15:18:02    |
+   | Last started replication   | 2024-01-19 15:18:02    |
+   +----------------------------+------------------------+
+   ```
+   You can also view the tables that ReadySet knows about and their statuses by
+   running `rails readyset:tables`:
+   ```sh
+   $ rails readyset:tables
+   +---------------------------------+-------------+-------------+
+   | table                           | status      | description |
+   +---------------------------------+-------------+-------------+
+   | "public"."posts"                | Snapshotted |             |
+   +---------------------------------+-------------+-------------+
+   ```
+4. Run `Readyset.configure` wherever you configure other gems in your
+   application, and set any desired configuration options:
+   ```ruby
+   Readyset.configure do |config|
+     # Set your config options here
+   end
+   ```
+   The list of available configuration options can be found
+   [here](#configuration-options).
+5. FOR RAILS 7 USERS: Enable Rails's query log tags features by setting
+   `config.active_record.query_log_tags_enabled = true` wherever you configure
+   your ActiveRecord settings. This will append information to your
+   ActiveRecord query logs that tells you where the given query was routed (e.g.
+   to ReadySet or to your primary database)
+
+## Quickstart
+
+1. Follow the instructions above to set up ReadySet and install the gem
+2. Route a query to ReadySet in your application like so:
+   ```ruby
+   Readyset.route do
+     Post.where(user_id: user_id)
+   end
+   ```
+3. Start up your application and drive traffic through the part of your
+   application that invokes the query you routed in the previous step
+4. If you are running Rails 7 and enabled query log tags as explained in step 5
+   [above](#installing), you should see an annotation next to the query in your
+   application logs that denotes where the query was routed. If the
+   `destination` tag has the value `readyset`, the query was routed to
+   ReadySet.
+
+   If you are running Rails 6, you can validate that the query was routed to
+   ReadySet by running `rails readyset:proxied_queries`. A "proxied" query is
+   one that was served by ReadySet but was proxied to your primary database,
+   since a cache for the query does not yet exist:
+   ```sh
+   $ rails readyset:proxied_queries
+   +--------------------+-------------------------------------------------------+-------------+-------+
+   | id                 | text                                                  | supported   | count |
+   +--------------------+-------------------------------------------------------+-------------+-------+
+   | q_281c5f9b8e4013bb | SELECT                                                | yes         | 1     |
+   |                    |   *                                                   |             |       |
+   |                    | FROM                                                  |             |       |
+   |                    |   "posts"                                             |             |       |
+   |                    | WHERE                                                 |             |       |
+   |                    |   ("user_id" = $1)                                    |             |       |
+   +--------------------+-------------------------------------------------------+-------------+-------+
+   ```
+5. Create a cache for the query by running
+   `rails readyset:proxied_queries:cache_all_supported`. This will create caches for
+   all of the queries proxied by ReadySet that are supported to be cached. You
+   can verify that the expected caches were created by running
+   `rails readyset:caches`:
+   ```sh
+   $ rails readyset:caches
+   +--------------------+--------------------+-------------------------------------+--------+-------+
+   | id                 | name               | text                                | always | count |
+   +--------------------+--------------------+-------------------------------------+--------+-------+
+   | q_281c5f9b8e4013bb | q_281c5f9b8e4013bb | SELECT                              | false  | 0     |
+   |                    |                    |   "public"."posts"."user_id"        |        |       |
+   |                    |                    | FROM                                |        |       |
+   |                    |                    |   "public"."posts"                  |        |       |
+   |                    |                    | WHERE                               |        |       |
+   |                    |                    |   ("public"."posts"."user_id" = $1) |        |       |
+   +--------------------+--------------------+-------------------------------------+--------+-------+
+   ```
+6. Drive traffic through the part of your application that invokes your cached
+   query. The first invocation of the query will be a cache miss, but the
+   second will be served from the cache. You can verify that the cache was
+   successfully used by looking at the `count` column in the output of
+   `rails readyset:caches`:
+   ```sh
+   $ rails readyset:caches
+   +--------------------+--------------------+-------------------------------------+--------+-------+
+   | id                 | name               | text                                | always | count |
+   +--------------------+--------------------+-------------------------------------+--------+-------+
+   | q_281c5f9b8e4013bb | q_281c5f9b8e4013bb | SELECT                              | false  | 1     |
+   |                    |                    |   "public"."posts"."user_id"        |        |       |
+   |                    |                    | FROM                                |        |       |
+   |                    |                    |   "public"."posts"                  |        |       |
+   |                    |                    | WHERE                               |        |       |
+   |                    |                    |   ("public"."posts"."user_id" = $1) |        |       |
+   +--------------------+--------------------+-------------------------------------+--------+-------+
+   ```
+
+## Usage
+
+### Getting Started with Caching
+
+Queries in arbitrary code blocks can be routed to ReadySet using the
+`Readyset.route` method like so:
+
+```ruby
+Readyset.route do
+  Post.where(user_id: user_id)
+end
+```
+
+Any queries invoked in the given block will be routed to the ReadySet instance
+configured in your `config/database.yml` file; however, until a cache is created
+for a particular query, invocations of that query against ReadySet will be proxied
+to your database. To create a cache for a specific query, you have a few options:
+
+- Invoke `.create_readyset_cache` directly on an ActiveRecord query in the
+  Rails console:
+  ```ruby
+  Post.where(user_id: 1).create_readyset_cache!
+  ```
+- Create caches for all of the queries supported by ReadySet that ReadySet has
+  seen and proxied to your database since it last started up using the provided
+  Rake task:
+  ```sh
+  rails readyset:proxied_queries:cache_all_supported
+  ```
+  **Note:** If you route a query to ReadySet, decide you no longer want to cache
+  that query, and stop routing that query to ReadySet, that query will still
+  exist in ReadySet's list of queries that it has proxied to your database.
+  This means that running the above Rake task will still create a cache for that
+  query **even though it is no longer annotated to be routed to ReadySet in your
+  application code**. The list of queries ReadySet has proxied can be cleared by
+  restarting ReadySet or by running `rails readyset:proxied_queries:drop_all`.
+- View the list of queries that ReadySet has proxied by running the following
+  in a Rails console:
+  ```ruby
+  Readyset::Query::ProxiedQuery.all
+  ```
+  You can invoke `#cache!` on the queries in this list for which you'd like to
+  create caches on ReadySet.
+- View the list of queries that ReadySet has proxied *and* that are supported
+  by ReadySet to be cached by running the following:
+  ```sh
+  rails readyset:proxied_queries:supported
+  ```
+  Pick a query from the list that you'd like to cache, and pass the ID to the
+  `rails readyset:create_cache` command like so:
+  ```sh
+  rails 'readyset:create_cache[your_query_id]'
+  ```
+
+Once a cache has been created for a particular query, it will persist on
+ReadySet across restarts (although any in-memory cached data will be lost when
+ReadySet goes down). You can view the list of existing caches using the provided
+Rake task:
+```sh
+rails readyset:caches
+```
+To drop a given cache in the list printed by the above command, you can pass the
+name of the cache to the `readyset:caches:drop` Rake task like so:
+```sh
+rails 'readyset:caches:drop[my_cache]'
+```
+You can also view the list of existing caches in an interactive form via the
+Rails console:
+```ruby
+Readyset::Query::CachedQuery.all
+```
+You can invoke `#drop!` on any of the caches in this list to remove the cache
+from ReadySet.
+
+#### Query Routing in Controllers
+
+The gem includes an extension to `ActionController` that allows you to route to
+ReadySet all of the queries that occur within the context of a given controller
+action:
+```ruby
+class PostsController < ActionController
+  route_to_readyset only: :show
+
+  def show
+    @post = Post.where(id: params[:id])
+  end
+end
+```
+`route_to_readyset` takes the same parameters as Rails's
+[`around_filters`](https://guides.rubyonrails.org/action_controller_overview.html#after-filters-and-around-filters).
+
+#### Query Routing in Models
+
+The gem also includes an extension that allows you to define queries in your
+model that will be routed to ReadySet:
+```ruby
+class Post < ApplicationRecord
+    readyset_query :posts_for_user, ->(user_id) { where(user_id: user_id) }
+end
+```
+The above example will define a `.posts_for_user` class method on the `Post`
+model that invokes the query `Post.where(user_id: user_id)` against ReadySet.
+**Note that other invocations of this query outside of the context of the
+`.posts_for_user` method will not be routed to ReadySet.**
+
+This feature allows you to specify which queries should be routed to ReadySet
+in a centralized location and prevents the need to use `Readyset.route`
+everywhere a cached query is invoked.
+
+### Cache Migrations
+
+Once you have a set of caches you are happy with in your development
+environment, you'll need a way to easily reproduce the same set of caches in
+other environments (e.g. staging, production, etc.). To facilitate this, the
+gem includes a "migration" feature, that allows you to dump the current set of
+caches to a "migration file" and re-create these caches using the same
+migration file.
+
+The following Rake task dumps the current set of caches to the
+`db/readyset_caches.rb` file:
+```sh
+rails readyset:caches:dump
+```
+This file should be checked into version control with your application code. To
+update a ReadySet instance so that its set of caches matches the caches in your
+migration file:
+```sh
+rails readyset:caches:migrate
+```
+This command 1) drops any caches that exist on ReadySet that are not present in
+the migration file and 2) creates any caches that are present in the migration
+file that do not exist on ReadySet. To run the command for a particular Rails
+environment, you can set the `RAILS_ENV` environment variable.
+
+### Automatic Failover
+
+To handle situations where ReadySet is unreachable for any reason, the gem
+includes an automatic failover feature. The gem tracks the number of ReadySet
+connection failures over a configurable window of time, and if the number of
+errors exceeds the configured threshold, any queries previously being routed to
+ReadySet will be routed to the primary database. A background task is started up
+that periodically attempts to establish a connection to ReadySet and check its
+status. When the task confirms that ReadySet is available again, it allows
+queries to be routed to ReadySet again.
+
+This feature is disabled by default. To enable it, set the
+`config.failover.enabled` configuration option to `true`. You can read about the
+other available configuration options [here](#configuration-options).
+
+## Configuration Options
+
+The gem's configuration options can be set by passing a block to
+`Readyset.configure` and setting options on the yielded
+`Readyset::Configuration` object. The available options are documented below.
+The values below are the default values for each of the options.
+
+```ruby
+Readyset.configure do |config|
+  # Whether the gem's automatic failover feature should be enabled.
+  config.failover.enabled = false
+  # Sets the interval upon which the background task will check
+  # ReadySet's availability after failover has occurred.
+  config.failover.healthcheck_interval = 5.seconds
+  # Sets the time window over which connection errors are counted
+  # when determining whether failover should occur.
+  config.failover.error_window_period = 1.minute
+  # Sets the number of errors that must occur within the configured
+  # error window period in order for failover to be triggered.
+  config.failover.error_window_size = 10
+  # The file in which cache migrations should be stored.
+  config.migration_path = File.join(Rails.root, 'db/readyset_caches.rb')
+end
+```
+
+## License
+
+[MIT License](LICENSE)

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+require 'rubocop/rake_task'
+
+RuboCop::RakeTask.new
+
+task default: %i(spec rubocop)

data/config.ru

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+require 'rubygems'
+require 'bundler'
+
+Bundler.require :default, :development
+
+Combustion.initialize! :all
+run Combustion::Application

data/docker-compose.yml

@@ -0,0 +1,54 @@
+services:
+  readyset:
+    image: public.ecr.aws/readyset/readyset:latest-stable
+    platform: linux/amd64
+    ports:
+      # The ReadySet Adapter listen port, i.e. what your application / SQL shell connects to
+      - "5433:5433"
+      # ReadySet Prometheus metrics available at http://localhost:6034/metrics
+      # e.g. curl -X GET http://localhost:6034/metrics
+      - "6034:6034"
+    environment:
+      DEPLOYMENT_ENV: readyset_rails_test
+      DB_DIR: /state
+      QUERY_CACHING: explicit
+      STANDALONE: 'true'
+      DEPLOYMENT: docker_compose_deployment
+      LISTEN_ADDRESS: 0.0.0.0:5433
+      UPSTREAM_DB_URL: postgresql://postgres:readyset@postgres/testdb
+      CONTROLLER_ADDRESS: 0.0.0.0
+    volumes:
+      - "readyset:/state"
+    healthcheck:
+      test: [ "CMD", "curl", "--fail", "127.0.0.1:6034/health" ]
+      interval: 2s
+      timeout: 1s
+      retries: 5
+      start_period: 5s
+    depends_on:
+      postgres:
+        condition: service_healthy
+  postgres:
+    image: postgres:14
+    environment:
+      - POSTGRES_PASSWORD=readyset
+      - POSTGRES_DB=testdb
+    ports:
+      # The ReadySet Adapter listen port, i.e. what your application / SQL shell connects to
+      - "5432:5432"
+    expose:
+      - 5432
+    command:
+      - "postgres"
+      - "-c"
+      - "wal_level=logical"
+    healthcheck:
+      test: ["CMD", "pg_isready", "-U", "postgres"]
+      interval: 5s
+      timeout: 5s
+      retries: 12
+    volumes:
+      - postgres:/var/lib/postgresql/data
+volumes:
+  postgres: ~
+  readyset: ~

data/lib/active_record/connection_adapters/readyset_adapter.rb

@@ -0,0 +1,52 @@
+require 'active_record'
+require 'active_record/connection_adapters/abstract_adapter'
+require 'active_record/connection_adapters/postgresql_adapter'
+require 'readyset/error'
+
+module ActiveRecord
+  module ConnectionAdapters
+    # The ReadySet adapter is a proxy object that delegates all its methods to an inner
+    # PostgreSQLAdapter instance.
+    class ReadysetAdapter
+      ADAPTER_NAME = 'Readyset'.freeze
+
+      # Finds the root cause of the given error and includes the Readyset::Error module in that
+      # error's singleton class if the root cause was a `PG::Error`. This allows us to invoke
+      # `#is_a?` on the error to determine if the error came from a connection to ReadySet.
+      #
+      # @param e [Exception] the error whose cause should be annotated
+      # @return [void]
+      def self.annotate_error(e)
+        if e.cause
+          annotate_error(e.cause)
+        else
+          if e.is_a?(::PG::Error)
+            e.singleton_class.instance_eval do
+              include ::Readyset::Error
+            end
+          end
+        end
+
+        nil
+      end
+
+      def self.method_missing(...)
+        PostgreSQLAdapter.send(...)
+      rescue => e
+        annotate_error(e)
+        raise e
+      end
+
+      def initialize(pg_conn)
+        @inner = pg_conn
+      end
+
+      def method_missing(...)
+        @inner.send(...)
+      rescue => e
+        self.class.annotate_error(e)
+        raise e
+      end
+    end
+  end
+end

data/lib/active_record/readyset_connection_handling.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module ActiveRecord
+  # The methods in these modules are required for Rails to recognize our custom adapter
+  module ReadysetConnectionHandling
+    def readyset_adapter_class
+      ConnectionAdapters::ReadysetAdapter
+    end
+
+    def readyset_connection(config) # :nodoc:
+      pg_conn = postgresql_connection(config)
+      readyset_adapter_class.new(pg_conn)
+    rescue => e
+      readyset_adapter_class.annotate_error(e)
+      raise e
+    end
+  end
+end

data/lib/readyset/caches.rb

@@ -0,0 +1,20 @@
+module Readyset
+  # Defines the DSL used in the gem's "migration" files. The DSL should be used by inheriting
+  # from this class and invoking the `.cache` class method to define new caches.
+  class Caches
+    class << self
+      attr_reader :caches
+    end
+
+    def self.cache(always: false)
+      @caches ||= Set.new
+
+      query = yield
+
+      @caches << Query::CachedQuery.new(
+        text: query.strip,
+        always: always,
+      )
+    end
+  end
+end

data/lib/readyset/configuration.rb

@@ -0,0 +1,30 @@
+# lib/readyset/configuration.rb
+require 'active_record'
+
+module Readyset
+  class Configuration
+    attr_accessor :migration_path, :shard
+
+    def initialize
+      @migration_path = File.join(Rails.root, 'db/readyset_caches.rb')
+      @shard = :readyset
+    end
+
+    def failover
+      if @failover
+        @failover
+      else
+        inner = ActiveSupport::OrderedOptions.new
+        inner.enabled = false
+        inner.healthcheck_interval = 5.seconds
+        inner.error_window_period = 1.minute
+        inner.error_window_size = 10
+        @failover = inner
+      end
+    end
+
+    def hostname
+      ActiveRecord::Base.configurations.configs_for(name: shard.to_s).configuration_hash[:host]
+    end
+  end
+end