makara 0.3.5

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 43573a40ca103839cfe03e0194c500ab656e03a6
+  data.tar.gz: e2baf81e7328bfba4d363992aee0e3308df53705
+SHA512:
+  metadata.gz: 27dabc4695a384d29c6c41d410e1604020d8c157cd32341a329bbe9a2208efeb5e4b30708398b97cea37df9d990b602da103f5931e22d34aa73c0922f6a9961e
+  data.tar.gz: d8f28626cc7d690d160a86dbc9846a99f6112fa0dba32114e1020103ed0dd1c1276fff2d900c7ca57a0c1b278feac6bed02a604addb83e3e41a70d66800a809d

data/.gitignore

@@ -0,0 +1,19 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+.DS_Store
+.byebug_history
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format documentation

data/.ruby-gemset

@@ -0,0 +1 @@
+makara

data/.ruby-version

@@ -0,0 +1 @@
+2.1.5

data/.travis.yml

@@ -0,0 +1,28 @@
+language: ruby
+
+rvm:
+  - 2.0.0
+  - 2.1.1
+  - jruby
+
+gemfile:
+  - gemfiles/ar30.gemfile
+  - gemfiles/ar31.gemfile
+  - gemfiles/ar32.gemfile
+  - gemfiles/ar40.gemfile
+  - gemfiles/ar41.gemfile
+  - gemfiles/ar42.gemfile
+
+sudo: false
+
+services:
+  - mysql
+  - postgresql
+
+before_script:
+  - mysql -e 'create database makara_test;'
+  - psql -c 'create database makara_test;' -U postgres
+
+before_install:
+  - gem update --system 2.1.11
+  - gem --version

data/CHANGELOG.md

@@ -0,0 +1,27 @@
+# Change Log
+All notable changes to this project will be documented in this file.
+
+## 0.3.0rc2 - 2014-08-05
+### Added
+- allow bypassing of stickiness
+
+## 0.3.0rc2 - 2014-08-05
+### Added
+- add postgres specific tests.
+
+### Changed
+- change using methods for matchers to be able to monkey patch them
+- follow AR naming conventions for adapter naming
+
+## 0.3.0rc1 - 2014-08-05
+### Removed
+- removed initial connection logic. If a connection can't be made on startup, an error will be thrown rather than the node getting blacklisted.
+
+
+## 0.2.2 - 2014-04-03
+### Added
+- add logging of makara operations via the Makara::Logger
+
+### Changed
+- begin tracing the series of errors associated with blacklisting rather than just the last. This becomes apparent in error messages.
+- fix Rails.cache usage when full environment is not loaded.

data/Gemfile

@@ -0,0 +1,18 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in makara.gemspec
+gemspec
+
+gem 'rake'
+gem 'rspec'
+gem 'timecop'
+gem 'byebug', :platform => :ruby
+gem 'ruby-debug', :platform => :jruby
+gem 'rack', '1.6.0'
+
+gem 'mysql2', :platform => :ruby
+gem 'pg', :platform => :ruby
+
+gem 'activerecord-jdbcmysql-adapter', :platform => :jruby
+gem 'activerecord-jdbcpostgresql-adapter', :platform => :jruby
+

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Mike Nelson
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,278 @@
+# Makara
+
+[![Build Status](https://travis-ci.org/taskrabbit/makara.png?branch=master)](https://travis-ci.org/taskrabbit/makara)
+[![Code Climate](https://codeclimate.com/repos/526886a7f3ea00679b00cae6/badges/7905f7a000492a1078f7/gpa.png)](https://codeclimate.com/repos/526886a7f3ea00679b00cae6/feed)
+
+
+Makara is generic master/slave proxy. It handles the heavy lifting of managing, choosing, blacklisting, and cycling through connections. It comes with an ActiveRecord database adapter implementation.
+
+#### Warning:
+
+There is a potential performance issue when used alongside certain versions of
+[newrelic/rpm](https://github.com/newrelic/rpm), [Issue #59](https://github.com/taskrabbit/makara/issues/59).
+
+> Any newrelic_rpm `< 3.11.2` and `>= 3.7.2` will have the regression.
+> I'd recommend upgrading newrelic_rpm to avoid the problem.
+
+## Installation
+
+```ruby
+gem 'makara', github: 'taskrabbit/makara', tag: 'v0.3.x'
+```
+
+## Basic Usage
+
+If you're only interested in the ActiveRecord database adapter... [here you go.](#activerecord-database-adapter)
+
+Makara provides a base proxy class which you should inherit from. Your proxy connection class should implement a `connection_for` instance method which will be provided with an individual configuration and expect a real connection back.
+
+```ruby
+class MyAwesomeSqlProxy < ::Makara::Proxy
+  def connection_for(config)
+    ::Sql::Client.new(config)
+  end
+end
+```
+
+Next, you need to decide which methods are proxied and which methods should be sent to all underlying connections:
+
+```ruby
+  # within MyAwesomeSqlProxy
+  hijack_method :select, :ping
+  send_to_all :connect, :reconnect, :disconnect, :clear_cache
+```
+
+Assuming you don't need to split requests between a master and a slave, you're done. If you do need to, implement the `needs_master?` method:
+
+```ruby
+  # within MyAwesomeSqlProxy
+  def needs_master?(method_name, args)
+    return false if args.empty?
+    sql = args.first
+    sql !~ /^select/i
+  end
+```
+
+This implementation will send any request not like "SELECT..." to a master connection. There are more methods you can override and more control over blacklisting - check out the [makara database adapter](lib/active_record/connection_adapters/makara_abstract_adapter.rb) for examples of advanced usage.
+
+### Config Parsing
+
+Makara comes with a config parser which will handle providing subconfigs to the `connection_for` method. Check out the ActiveRecord database.yml example below for more info.
+
+### Context
+
+Makara handles stickyness by keeping track of a context (sha). In a multi-instance environment it persists a context in a cache. If Rails is present it will automatically use Rails.cache. You can provide any kind of store as long as it responds to the methods required in [lib/makara/cache.rb](lib/makara/cache.rb).
+
+```ruby
+Makara::Cache.store = MyRedisCacheStore.new
+```
+
+To handle persistence of context across requests in a Rack app, makara provides a middleware. It lays a cookie named `_mkra_ctxt` which contains the current master context. If the next request is executed before the cookie expires, master will be used. If something occurs which naturally requires master on the second request, the context is changed and stored again.
+
+#### Changing Context
+
+If you need to change the makara context, releasing any stuck connections, all you have to do is:
+
+```ruby
+ctx = Makara::Context.generate # or any unique sha
+Makara::Context.set_current ctx
+```
+
+A context is local to the curent thread of execution. This will allow you to stick to master safely in a single thread
+in systems such as sidekiq, for instance.
+
+
+### Forcing Master
+
+If you need to force master in your app then you can simply invoke stick_to_master! on your connection:
+
+```ruby
+write_to_cache = true # or false
+proxy.stick_to_master!(write_to_cache)
+```
+
+
+### Skipping the Stickiness
+
+If you're using the `sticky: true` configuration and you find yourself in a situation where you need to write information through the proxy but you don't want the context to be stuck to master, you should use a `without_sticking` block:
+
+```ruby
+proxy.without_sticking do
+  # do my stuff that would normally cause the proxy to stick to master
+end
+```
+
+### Logging
+
+You can set a logger instance to ::Makara::Logging::Logger.logger and Makara will log how it handles errors at the Proxy level.
+
+```ruby
+Makara::Logging::Logger.logger = ::Logger.new(STDOUT)
+```
+
+## ActiveRecord Database Adapter
+
+So you've found yourself with an ActiveRecord-based project which is starting to get some traffic and you realize 95% of you DB load is from reads. Well you've come to the right spot. Makara is a great solution to break up that load not only between master and slave but potentially multiple masters and/or multiple slaves.
+
+By creating a makara database adapter which simply acts as a proxy we avoid any major complexity surrounding specific database implementations. The makara adapter doesn't care if the underlying connection is mysql, postgresql, etc it simply cares about the sql string being executed.
+
+### What goes where?
+
+Any `SELECT` statements will execute against your slave(s), anything else will go to master. The only edge case is `SET` operations which are sent to all connections. Execution of specific methods such as `connect!`, `disconnect!`, and `clear_cache!` are invoked on all underlying connections.
+
+### Errors / blacklisting
+
+Whenever a node fails an operation due to a connection issue, it is blacklisted for the amount of time specified in your database.yml. After that amount of time has passed, the connection will begin receiving queries again. If all slave nodes are blacklisted, the master connection will begin receiving read queries as if it were a slave. Once all nodes are blacklisted the error is raised to the application and all nodes are whitelisted.
+
+### Database.yml
+
+Your database.yml should contain the following structure:
+
+```yml
+production:
+  adapter: 'mysql2_makara'
+  database: 'MyAppProduction'
+  # any other standard AR configurations
+
+  # add a makara subconfig
+  makara:
+
+    # the following are default values
+    blacklist_duration: 5
+    master_ttl: 5
+    sticky: true
+
+    # list your connections with the override values (they're merged into the top-level config)
+    # be sure to provide the role if master, role is assumed to be a slave if not provided
+    connections:
+      - role: master
+        host: master.sql.host
+      - role: slave
+        host: slave1.sql.host
+      - role: slave
+        host: slave2.sql.host
+```
+
+Let's break this down a little bit. At the top level of your config you have the standard `adapter` choice. Currently the available adapters are listed in [lib/active_record/connection_adapters/](lib/active_record/connection_adapters/). They are in the form of `#{db_type}_makara` where db_type is mysql, postgresql, etc.
+
+Following the adapter choice is all the standard configurations (host, port, retry, database, username, password, etc). With all the standard configurations provided, you can now provide the makara subconfig.
+
+The makara subconfig sets up the proxy with a few of its own options, then provides the connection list. The makara options are:
+* blacklist_duration - the number of seconds a node is blacklisted when a connection failure occurs
+* sticky - if a node should be stuck to once it's used during a specific context
+* master_ttl - how long the master context is persisted. generally, this needs to be longer than any replication lag
+* connection_error_matchers - array of custom error matchers you want to be handled gracefully by Makara (as in, errors matching these regexes will result in blacklisting the connection as opposed to raising directly).
+
+Connection definitions contain any extra node-specific configurations. If the node should behave as a master you must provide `role: master`. Any previous configurations can be overridden within a specific node's config. Nodes can also contain weights if you'd like to balance usage based on hardware specifications. Optionally, you can provide a name attribute which will be used in sql logging.
+
+```yml
+connections:
+  - role: master
+    host: mymaster.sql.host
+    blacklist_duration: 0
+
+  # implicit role: slave
+  - host: mybigslave.sql.host
+    weight: 8
+    name: Big Slave
+  - host: mysmallslave.sql.host
+    weight: 2
+    name: Small Slave
+```
+
+In the previous config the "Big Slave" would receive ~80% of traffic.
+
+#### DATABASE_URL
+
+Connections may specify a `url` parameter in place of host, username, password, etc.
+
+```yml
+connections:
+  - role: master
+    blacklist_duration: 0
+    url: 'mysql2://db_username:db_password@localhost:3306/db_name'
+```
+
+We recommend, if using environmental variables, to interpolate them via ERb.
+
+```yml
+connections:
+  - role: master
+    blacklist_duration: 0
+    url: <%= ENV['DATABASE_URL_MASTER'] %>
+  - role: slave
+    url: <%= ENV['DATABASE_URL_SLAVE'] %>
+```
+
+**Important**: *Do NOT use `ENV['DATABASE_URL']`*, as it inteferes with the the database configuration
+initialization and may cause Makara not to complete the configuration.  For the moment, it is easier
+to use a different ENV variable than to hook into the database initialization in all the supported
+Rails.
+
+For more information on url parsing, as used in
+[ConfigParser](https://github.com/taskrabbit/makara/blob/master/lib/makara/config_parser.rb), see:
+
+- 3.0
+    [ActiveRecord::Base::ConnectionSpecification.new](https://github.com/rails/rails/blob/3-0-stable/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb#L3-L7)
+- 3.0
+    [ActiveRecord::Base::ConnectionSpecification.new](https://github.com/rails/rails/blob/3-1-stable/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb#L3-L7)
+- 3.2
+  [ActiveRecord::Base::ConnectionSpecification::Resolver.send(:connection_url_to_hash, url_config[:url])](https://github.com/rails/rails/blob/3-2-stable/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb#L60-L77)
+- 4.0
+ [ActiveRecord::ConnectionAdapters::ConnectionSpecification::Resolver.send(:connection_url_to_hash, url_config[:url])](https://github.com/rails/rails/blob/4-0-stable/activerecord/lib/active_record/connection_adapters/connection_specification.rb#L68-L92)
+  - [ActiveRecord::ConnectionHandling::MergeAndResolveDefaultUrlConfig](https://github.com/rails/rails/blob/4-0-stable/activerecord/lib/active_record/connection_handling.rb)
+- 4.1
+ [ActiveRecord::ConnectionAdapters::ConnectionSpecification::ConnectionUrlResolver.new(url).to_hash](https://github.com/rails/rails/blob/4-1-stable/activerecord/lib/active_record/connection_adapters/connection_specification.rb#L17-L121)
+  - ActiveRecord::ConnectionHandling::MergeAndResolveDefaultUrlConfig.new(url_config).resolve
+- 4.2
+ [ActiveRecord::ConnectionAdapters::ConnectionSpecification::ConnectionUrlResolver.new(url).to_hash](https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/connection_handling.rb#L60-L81)
+- master
+ [ActiveRecord::ConnectionAdapters::ConnectionSpecification::ConnectionUrlResolver.new(url).to_hash](https://github.com/rails/rails/blob/97b980b4e61aea3cee429bdee4b2eae2329905cd/activerecord/lib/active_record/connection_handling.rb#L60-L81)
+
+
+
+## Custom error matchers:
+
+To enable Makara to catch and handle custom errors gracefully (blacklist the connection instead of raising directly), you must add your custom matchers to the `connection_error_matchers` setting of your config file, for example:
+
+```yml
+production:
+  adapter: 'mysql2_makara'
+
+  makara:
+    blacklist_duration: 5
+    connection_error_matchers:
+      - !ruby/regexp '/^ActiveRecord::StatementInvalid: Mysql2::Error: Unknown command:/'
+      - '/Sql Server Has Gone Away/'
+      - 'Mysql2::Error: Duplicate entry'
+```
+
+You can provide strings or regexes.  In the case of strings, if they start with `/` and end with `/` they will be converted to regexes when evaluated. Strings that don't start and end with `/` will get evaluated with standard comparison.
+
+## Common Problems / Solutions
+
+On occasion your app may deal with a situation where makara is not present during a write but a read should use master. In the generic proxy details above you are encouraged to use `stick_to_master!` to accomplish this. Here's an example:
+
+```ruby
+# some third party creates a resource in your db, slave replication may not have completed yet
+# ...
+# then your app is told to read the resource.
+def handle_request_after_third_party_record_creation
+  CreatedResourceClass.connection.stick_to_master!
+  CreatedResourceClass.find(params[:id]) # will go to master
+end
+```
+
+Similarly, if you have a third party service which will conduct a generic request against your Rack app, you can force master via a query param:
+
+```ruby
+def send_url_to_third_party
+  context = Makara::Context.get_current
+  ThirdParty.read_from_here!("http://mysite.com/path/to/resource?_mkra_ctxt=#{context}")
+end
+```
+
+## Todo
+
+* Cookie based cache store?
+* More real world examples

data/Rakefile

@@ -0,0 +1,9 @@
+#!/usr/bin/env rake
+require 'rake'
+require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+end
+task :default  => :spec

data/gemfiles/ar30.gemfile

@@ -0,0 +1,30 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in makara.gemspec
+gemspec :path => '../'
+
+
+gem 'rake'
+gem 'activerecord', '3.0.17'
+gem 'rspec'
+gem 'rack', '1.6.0'
+gem 'i18n', '~> 0.5.0'
+gem 'mysql2', '0.2.11', :platform => :ruby
+gem 'activerecord-jdbcmysql-adapter', :platform => :jruby
+gem 'activerecord-jdbcpostgresql-adapter', :platform => :jruby
+
+
+rmajor, rminor, rpatch = RUBY_VERSION.split(/[^\d]/)[0..2].map(&:to_i)
+
+if rmajor == 1 && rminor == 8
+  gem 'timecop', '0.5.9'
+else
+  gem 'timecop'
+end
+
+# 1.8 - 1.9.2
+if rmajor == 1 && (rminor == 8 || (rminor == 9 && rpatch < 3))
+  gem 'pg', '0.17.1', :platform => :ruby
+else
+  gem 'pg', :platform => :ruby
+end