makara 0.3.10 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 4dc4867024ed5527a9b0a499e4daff0c9c4bac5e
-  data.tar.gz: 170743d2e7f735b39fbcb477657643918c66cd32
+  metadata.gz: 942671e3f14754da500b275f0da2354cef9d7d0e
+  data.tar.gz: bfcd3be83e4101f627d10c0718afe41e5dae5b44
 SHA512:
-  metadata.gz: 8da212514ba19441a154323990544bb18ce0a3a39ff133f227bab84790e48b747f254b53d9a59c2aa93cfad0f0367fd03a44d3bcbe94a17874e7f2b084bec468
-  data.tar.gz: 731c52cd786a65d0a124edf94e1f721b69ec78cee0549d499b2cc35a5ac2210aa7ae87d8b22a3734829271ea538ed6b3cfb197fd1ebc2d4ce0f961d16e86afda
+  metadata.gz: f51e87da7797bbb503ebe60e1e719d498f2cc8020e09578a6b9d2ed2c770c7d879ca2f32d7b7ab4b224c76f0c5ba00c8dc48bc91cf806dd24be6299e5424e209
+  data.tar.gz: b1cacfad8294f8371f0c28ec6a76f0608ce9c0c23361cb256e9ba0b083bde43459ad2c18e575a166c6b3f258edbb562fc1ec7520f4653b5c30bcefdb9cef59ce

data/CHANGELOG.md

@@ -1,8 +1,15 @@
 # Change Log
 All notable changes to this project will be documented in this file.
 
-## v0.3.9 - 2013-03-20
-[Full Changelog](https://github.com/taskrabbit/makara/compare/v0.3.9...v0.3.10) 
+## v0.4.0 - 2018-04-01
+[Full Changelog](https://github.com/taskrabbit/makara/compare/v0.3.10...v0.4.0)
+
+This release is a major change to how we remember state between requests. A redis store is no longer needed. Everything is in the cookies.
+ - Implement stickiness for the duration of `master_ttl` via cookies [#194](https://github.com/taskrabbit/makara/pull/194) Rosa Gutierrez
+
+
+## v0.3.10 - 2018-03-20
+[Full Changelog](https://github.com/taskrabbit/makara/compare/v0.3.9...v0.3.10)
 
 Fixed
 - Send nextval queries to master and show queries to replicas for Postgres [#173](https://github.com/taskrabbit/makara/pull/173) Andrew Kane
@@ -18,14 +25,14 @@ Documentation and Test
 - Travis Upgrade [#199](https://github.com/taskrabbit/makara/pull/199) Brian Leonard
 
 ## v0.3.9 - 2017-08-14
-[Full Changelog](https://github.com/taskrabbit/makara/compare/v0.3.8...v0.3.9) 
+[Full Changelog](https://github.com/taskrabbit/makara/compare/v0.3.8...v0.3.9)
 
 Changed
 - Add postgis support [#118](https://github.com/taskrabbit/makara/pull/118) Kevin Bacha
 
 ## v0.3.8 - 2017-07-11
 
-[Full Changelog](https://github.com/taskrabbit/makara/compare/v0.3.7...v0.3.8) 
+[Full Changelog](https://github.com/taskrabbit/makara/compare/v0.3.7...v0.3.8)
 
 Changed
 - Rails 5.1 compatibility [#150](https://github.com/taskrabbit/makara/pull/150) Jeremy Daer

data/README.md

@@ -61,40 +61,44 @@ This implementation will send any request not like "SELECT..." to a master conne
 
 Makara comes with a config parser which will handle providing subconfigs to the `connection_for` method. Check out the ActiveRecord database.yml example below for more info.
 
-### Context
+### Stickiness Context
 
-Makara handles stickyness by keeping track of a context (sha). In a multi-instance environment it persists a context in a cache. If Rails is present it will automatically use Rails.cache. You can provide any kind of store as long as it responds to the methods required in [lib/makara/cache.rb](lib/makara/cache.rb).
+Makara handles stickiness by keeping track of which proxies are stuck at any given moment. The context is basically a mapping of proxy ids to the timestamp until which they are stuck.
 
-```ruby
-Makara::Cache.store = MyRedisCacheStore.new
-```
+To handle persistence of context across requests in a Rack app, makara provides a middleware. It lays a cookie named `_mkra_stck` which contains the current context. If the next request is executed before the cookie expires, that given context will be used. If something occurs which naturally requires master on the second request, the context is updated and stored again.
 
-To handle persistence of context across requests in a Rack app, makara provides a middleware. It lays a cookie named `_mkra_ctxt` which contains the current master context. If the next request is executed before the cookie expires, master will be used. If something occurs which naturally requires master on the second request, the context is changed and stored again.
+#### Releasing stuck connections (clearing context)
 
-#### Changing Context
+If you need to clear the current context, releasing any stuck connections, all you have to do is:
 
-If you need to change the makara context, releasing any stuck connections, all you have to do is:
+```ruby
+Makara::Context.release_all
+```
 
+You can also clear stuck connections for a specific proxy:
 ```ruby
-ctx = Makara::Context.generate # or any unique sha
-Makara::Context.set_current ctx
+Makara::Context.release(proxy_id)
+Makara::Context.release('mysql_main')
+Makara::Context.release('redis')
+...
 ```
 
 A context is local to the curent thread of execution. This will allow you to stick to master safely in a single thread
 in systems such as sidekiq, for instance.
 
 
-### Forcing Master
+#### Forcing Master
 
 If you need to force master in your app then you can simply invoke stick_to_master! on your connection:
 
 ```ruby
-write_to_cache = true # or false
-proxy.stick_to_master!(write_to_cache)
+persist = true # or false, it's true by default
+proxy.stick_to_master!(persist)
 ```
 
+It'll keep the proxy stuck to master for the current request, and if `persist = true` (default), it'll be also stored in the context for subsequent requests, keeping the proxy stuck up to the duration of `master_ttl` configured for the proxy.
 
-### Skipping the Stickiness
+#### Skipping the Stickiness
 
 If you're using the `sticky: true` configuration and you find yourself in a situation where you need to write information through the proxy but you don't want the context to be stuck to master, you should use a `without_sticking` block:
 
@@ -145,6 +149,8 @@ production:
   # add a makara subconfig
   makara:
 
+    # optional id to identify the proxy with this configuration for stickiness
+    id: mysql
     # the following are default values
     blacklist_duration: 5
     master_ttl: 5
@@ -167,6 +173,7 @@ Let's break this down a little bit. At the top level of your config you have the
 Following the adapter choice is all the standard configurations (host, port, retry, database, username, password, etc). With all the standard configurations provided, you can now provide the makara subconfig.
 
 The makara subconfig sets up the proxy with a few of its own options, then provides the connection list. The makara options are:
+* id - an identifier for the proxy, used for sticky behaviour and context. The default is to use a MD5 hash of the configuration contents, so if you are setting `sticky` to true, it's a good idea to also set an `id`. Otherwise any stuck connections will be cleared if the configuration changes (as the default MD5 hash id would change as well)
 * blacklist_duration - the number of seconds a node is blacklisted when a connection failure occurs
 * disable_blacklist - do not blacklist node at any error, useful in case of one master
 * sticky - if a node should be stuck to once it's used during a specific context
@@ -275,16 +282,7 @@ def handle_request_after_third_party_record_creation
 end
 ```
 
-Similarly, if you have a third party service which will conduct a generic request against your Rack app, you can force master via a query param:
-
-```ruby
-def send_url_to_third_party
-  context = Makara::Context.get_current
-  ThirdParty.read_from_here!("http://mysite.com/path/to/resource?_mkra_ctxt=#{context}")
-end
-```
-
 ## Todo
 
-* Cookie based cache store?
+* Support for providing context as query param
 * More real world examples

data/gemfiles/ar50.gemfile

@@ -9,7 +9,7 @@ gem 'activerecord', '~> 5.0.0'
 gem 'rspec'
 gem 'rack'
 gem 'timecop'
-gem 'mysql2', :platform => :ruby
+gem 'mysql2', '~> 0.4.10', :platform => :ruby
 gem 'activerecord-jdbcmysql-adapter', :platform => :jruby
 gem 'pg', '0.21.0', :platform => :ruby
 gem 'activerecord-jdbcpostgresql-adapter', :platform => :jruby

data/gemfiles/ar51.gemfile

@@ -9,7 +9,7 @@ gem 'activerecord', '~> 5.1.0'
 gem 'rspec'
 gem 'rack'
 gem 'timecop'
-gem 'mysql2', :platform => :ruby
+gem 'mysql2', '~> 0.4.10', :platform => :ruby
 gem 'activerecord-jdbcmysql-adapter', :platform => :jruby
 gem 'pg', '0.21.0', :platform => :ruby
 gem 'activerecord-jdbcpostgresql-adapter', :platform => :jruby

data/lib/makara.rb

@@ -6,6 +6,7 @@ module Makara
   autoload :ConfigParser,       'makara/config_parser'
   autoload :ConnectionWrapper,  'makara/connection_wrapper'
   autoload :Context,            'makara/context'
+  autoload :Cookie,             'makara/cookie'
   autoload :ErrorHandler,       'makara/error_handler'
   autoload :Middleware,         'makara/middleware'
   autoload :Pool,               'makara/pool'

data/lib/makara/cache.rb

@@ -1,53 +1,17 @@
-require 'active_support/core_ext/object/try'
-
-# The Makara Cache should have access to your centralized cache store.
-# It serves the purpose of storing the Makara::Context across requests, servers, etc.
-
 module Makara
   module Cache
 
-    autoload :MemoryStore, 'makara/cache/memory_store'
-    autoload :NoopStore,   'makara/cache/noop_store'
-
     class << self
 
       def store=(store)
-        @store = store
+        Makara::Logging::Logger.log deprecation_warning, :warn
       end
 
-      def read(key)
-        store.try(:read, key)
-      end
+      private
 
-      def write(key, value, ttl)
-        store.try(:write, key, value, :expires_in => ttl.to_i)
+      def deprecation_warning
+        "Makara's context is no longer persisted in a backend cache, a cookie store is used by default.\nSetting the Makara::Cache.store won't have any effects."
       end
-
-      protected
-
-      def store
-        case @store
-        when :noop, :null
-          @store = Makara::Cache::NoopStore.new
-        when :memory
-          @store = Makara::Cache::MemoryStore.new
-        else
-          if defined?(Rails)
-
-            # in AR3 RAILS_CACHE may not be loaded if the full env is not present
-            # Rails.cache will throw an error because of it.
-            if ActiveRecord::VERSION::MAJOR < 4
-              @store ||= Rails.cache if defined?(RAILS_CACHE)
-            else
-              @store ||= Rails.cache if defined?(Rails)
-            end
-          end
-        end
-
-        @store
-      end
-
     end
-
   end
 end

data/lib/makara/config_parser.rb

@@ -127,7 +127,7 @@ module Makara
     # NOTE: Does not use ENV['DATABASE_URL']
     def self.merge_and_resolve_default_url_config(config)
       if ENV['DATABASE_URL']
-        Logging::Logger.log "Please rename DATABASE_URL to use in the database.yml", :warn
+        Makara::Logging::Logger.log "Please rename DATABASE_URL to use in the database.yml", :warn
       end
       return config unless config.key?(:url)
       url = config[:url]
@@ -144,7 +144,7 @@ module Makara
       @config = config.symbolize_keys
       @makara_config = DEFAULTS.merge(@config[:makara] || {})
       @makara_config = @makara_config.symbolize_keys
-      @id = @makara_config[:id]
+      @id = sanitize_id(@makara_config[:id])
     end
 
 
@@ -197,5 +197,15 @@ module Makara
 
     end
 
+
+    def sanitize_id(id)
+      return if id.nil? || id.empty?
+
+      id.gsub(/[\|:]/, '').tap do |sanitized_id|
+        if sanitized_id.size != id.size
+          Makara::Logging::Logger.log "Proxy id '#{id}' changed to '#{sanitized_id}'", :warn
+        end
+      end
+    end
   end
 end

data/lib/makara/context.rb

@@ -1,63 +1,134 @@
 require 'digest/md5'
 
-# Keeps track of the current and previous context (hexdigests)
-# If a new context is needed it can be generated via Makara::Context.generate
-
+# Keeps track of the current stickiness state for different Makara proxies
 module Makara
   class Context
-    class << self
+    attr_accessor :stored_data, :staged_data
+    attr_reader :current_timestamp
+
+    def initialize(context_data)
+      @stored_data = context_data
+      @staged_data = {}
+      @dirty = @was_dirty = false
+
+      freeze_time
+    end
 
-      def generate(seed = nil)
-        seed ||= "#{Time.now.to_i}#{Thread.current.object_id}#{rand(99999)}"
-        Digest::MD5.hexdigest(seed)
+    def stage(proxy_id, ttl)
+      staged_data[proxy_id] = [staged_data[proxy_id].to_f, ttl.to_f].max
+    end
+
+    def stuck?(proxy_id)
+      stored_data[proxy_id] && !expired?(stored_data[proxy_id])
+    end
+
+    def staged?(proxy_id)
+      staged_data.key?(proxy_id)
+    end
+
+    def release(proxy_id)
+      @dirty ||= !!stored_data.delete(proxy_id)
+      staged_data.delete(proxy_id)
+    end
+
+    def release_all
+      if self.stored_data.any?
+        self.stored_data = {}
+        # We need to track a change made to the current stored data
+        # so we can commit it later
+        @dirty = true
       end
+      self.staged_data = {}
+    end
+
+    # Stores the staged data with an expiration time based on the current time,
+    # and clears any expired entries. Returns true if any changes were made to
+    # the current store
+    def commit
+      freeze_time
+      release_expired
+      store_staged_data
+      clean
 
-      def get_previous
-        fetch(:makara_context_previous) { generate }
+      was_dirty?
+    end
+
+    private
+
+    def freeze_time
+      @current_timestamp = Time.now.to_f
+    end
+
+    # Indicates whether there have been changes to the context that need
+    # to be persisted when the request finishes
+    def dirty?
+      @dirty
+    end
+
+    def was_dirty?
+      @was_dirty
+    end
+
+    def expired?(timestamp)
+      timestamp <= current_timestamp
+    end
+
+    def release_expired
+      previous_size = stored_data.size
+      stored_data.delete_if { |_, timestamp| expired?(timestamp) }
+      @dirty ||= previous_size != stored_data.size
+    end
+
+    def store_staged_data
+      staged_data.each do |proxy_id, ttl|
+        if ttl > 0 && self.stored_data[proxy_id].to_f < current_timestamp + ttl
+          self.stored_data[proxy_id] = current_timestamp + ttl
+          @dirty = true
+        end
       end
+    end
 
-      def set_previous(context)
-        previously_sticky.clear
-        set(:makara_context_previous,context)
+    def clean
+      @was_dirty = dirty?
+      @dirty = false
+      @staged_data = {}
+    end
+
+    class << self
+      def set_current(context_data)
+        set(:makara_current_context, new(context_data))
       end
 
-      def get_current
-        fetch(:makara_context_current) { generate }
+      # Called by `Proxy#stick_to_master!` to use master in subsequent requests
+      def stick(proxy_id, ttl)
+        current.stage(proxy_id, ttl)
       end
 
-      def set_current(context)
-        set(:makara_context_current,context)
+      def stuck?(proxy_id)
+        current.staged?(proxy_id) || current.stuck?(proxy_id)
       end
 
-      def previously_stuck?(config_id)
-        previously_sticky.fetch(config_id) do
-          stuck?(Makara::Context.get_previous, config_id)
+      def next
+        if current.commit
+          current.stored_data
         end
       end
 
-      # Called by `Proxy#stick_to_master!` to stick subsequent requests to
-      # master. They'll see the current context as their previous context
-      # when they're asking whether they should be stuck to master.
-      def stick(context, config_id, ttl)
-        Makara::Cache.write(cache_key_for(context, config_id), '1', ttl)
+      def release(proxy_id)
+        current.release(proxy_id)
       end
 
-      def stuck?(context, config_id)
-        !!Makara::Cache.read(cache_key_for(context, config_id))
+      def release_all
+        current.release_all
       end
 
       protected
-
-      def previously_sticky
-        fetch(:makara_previously_sticky) { Hash.new }
-      end
-
-      def cache_key_for(context, config_id)
-        "makara::#{context}-#{config_id}"
+      def current
+        fetch(:makara_current_context) { new({}) }
       end
 
       def fetch(key)
-        get(key) || set(key,yield)
+        get(key) || set(key, yield)
       end
 
       if Thread.current.respond_to?(:thread_variable_get)
@@ -65,19 +136,18 @@ module Makara
           Thread.current.thread_variable_get(key)
         end
 
-        def set(key,value)
-          Thread.current.thread_variable_set(key,value)
+        def set(key, value)
+          Thread.current.thread_variable_set(key, value)
         end
       else
         def get(key)
           Thread.current[key]
         end
 
-        def set(key,value)
+        def set(key, value)
           Thread.current[key]=value
         end
       end
-
     end
   end
 end