elasticsearch-rails-ha 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: be1cae2a83f408a6cf97871d0cd9fd2d284c2474
+  data.tar.gz: 6faa05ee8cb145ad8c484a85634224bb1fc3369f
+SHA512:
+  metadata.gz: df5d68a32704dd00d819c1c7335d5a8bef8d7e64c86fdc61e8d903d661d79006384efbeb56d8e6fc2c888222930fe0c258842cb965156e95272266199bac5559
+  data.tar.gz: adb0cc7f73748e22fd62d663a59179430f4bf1f4bc282767254ed36f404206ebdcca31c87d342a9ff66458a9171a4d73732b33cfd263774823357fb3aedef265

data/CONTRIBUTING.md

@@ -0,0 +1,15 @@
+## Welcome!
+
+We're so glad you're thinking about contributing to an 18F open source project! If you're unsure or afraid of anything, just ask or submit the issue or pull request anyways. The worst that can happen is that you'll be politely asked to change something. We appreciate any sort of contribution, and don't want a wall of rules to get in the way of that.
+
+Before contributing, we encourage you to read our CONTRIBUTING policy (you are here), our LICENSE, and our README, all of which should be in this repository. If you have any questions, or want to read more about our underlying policies, you can consult the 18F Open Source Policy GitHub repository at https://github.com/18f/open-source-policy, or just shoot us an email/official government letterhead note to [18f@gsa.gov](mailto:18f@gsa.gov).
+
+## Public domain
+
+This project is in the public domain within the United States, and
+copyright and related rights in the work worldwide are waived through
+the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+
+All contributions to this project will be released under the CC0
+dedication. By submitting a pull request, you are agreeing to comply
+with this waiver of copyright interest.

data/LICENSE.md

@@ -0,0 +1,31 @@
+As a work of the United States Government, this project is in the
+public domain within the United States.
+
+Additionally, we waive copyright and related rights in the work
+worldwide through the CC0 1.0 Universal public domain dedication.
+
+## CC0 1.0 Universal Summary
+
+This is a human-readable summary of the [Legal Code (read the full text)](https://creativecommons.org/publicdomain/zero/1.0/legalcode).
+
+### No Copyright
+
+The person who associated a work with this deed has dedicated the work to
+the public domain by waiving all of his or her rights to the work worldwide
+under copyright law, including all related and neighboring rights, to the
+extent allowed by law.
+
+You can copy, modify, distribute and perform the work, even for commercial
+purposes, all without asking permission.
+
+### Other Information
+
+In no way are the patent or trademark rights of any person affected by CC0,
+nor are the rights that other persons may have in the work or in how the
+work is used, such as publicity or privacy rights.
+
+Unless expressly stated otherwise, the person who associated a work with
+this deed makes no warranties about the work, and disclaims liability for
+all uses of the work, to the fullest extent permitted by applicable law.
+When using or citing the work, you should not imply endorsement by the
+author or the affirmer.

data/README.md

@@ -0,0 +1,50 @@
+# elasticsearch-rails-ha RubyGem
+
+[![Build Status](https://travis-ci.org/18F/elasticsearch-rails-ha-gem.svg?branch=master)](https://travis-ci.org/18F/elasticsearch-rails-ha-gem)
+
+Elasticsearch for Rails, high availability extensions.
+
+See also:
+
+* [elasticsearch-rails](https://github.com/elastic/elasticsearch-rails)
+
+## Examples
+
+Add the high availability tasks to your Rake task file `lib/tasks/elasticsearch.rake`:
+
+```
+require 'elasticsearch/rails/ha/tasks'
+```
+
+Import all the Articles on a machine with 4 cores available:
+
+```
+% bundle exec rake environment elasticsearch:ha:import NPROCS=4 CLASS='Article'
+```
+
+Stage an index alongside your live index, but do not make it live yet:
+
+```
+% bundle exec rake environment elasticsearch:ha:stage NPROCS=4 CLASS='Article'
+```
+
+Promote your staged index:
+
+```
+% bundle exec rake environment elasticsearch:ha:promote
+```
+
+## Acknowledgements
+
+Thanks to [Pop Up Archive](http://popuparchive.com/) for
+contributing the [original version of this code](https://github.com/popuparchive/pop-up-archive/blob/master/lib/tasks/search.rake) to the public domain.
+
+## Public domain
+
+This project is in the worldwide [public domain](LICENSE.md). As stated in [CONTRIBUTING](CONTRIBUTING.md):
+
+> This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+>
+> All contributions to this project will be released under the CC0
+> dedication. By submitting a pull request, you are agreeing to comply
+> with this waiver of copyright interest.

data/lib/elasticsearch/rails/ha.rb

@@ -0,0 +1,3 @@
+require_relative 'ha/version'
+require_relative 'ha/parallel_indexer'
+require_relative 'ha/index_stager'

data/lib/elasticsearch/rails/ha/index_stager.rb

@@ -0,0 +1,103 @@
+module Elasticsearch
+  module Rails
+    module HA
+      class IndexStager
+        attr_reader :klass, :live_index_name
+
+        def initialize(klass)
+          @klass = klass.constantize
+        end
+
+        def stage_index_name
+          if klass.respond_to?(:stage_index_name)
+            klass.stage_index_name
+          else
+            klass.index_name + "_staged"
+          end
+        end
+
+        def tmp_index_name
+          @_suffix ||= Time.now.strftime('%Y%m%d%H%M%S') + '-' + SecureRandom.hex[0..7]
+          "#{klass.index_name}_#{@_suffix}"
+        end
+
+        def alias_stage_to_tmp_index
+          es_client.indices.delete index: stage_index_name rescue false
+          es_client.indices.update_aliases body: {
+            actions: [
+              { add: { index: tmp_index_name, alias: stage_index_name } }
+            ]
+          }
+        end
+
+        def promote(live_index_name=klass.index_name)
+          @live_index_name = live_index_name
+
+          # the renaming actions (performed atomically by ES)
+          rename_actions = [
+            { remove: { index: stage_aliased_to, alias: stage_index_name } },
+            {    add: { index: stage_index_name, alias: live_index_name } }
+          ]
+
+          # zap any existing index known as index_name,
+          # but do it conditionally since it is reasonable that it does not exist.
+          to_delete = []
+          existing_live_index = es_client.indices.get_aliases(index: live_index_name)
+          existing_live_index.each do |k,v|
+
+            # if the index is merely aliased, remove its alias as part of the aliasing transaction.
+            if k != klass.index_name
+              rename_actions.unshift({ remove: { index: k, alias: live_index_name } })
+
+              # mark it for deletion when we've successfully updated aliases
+              to_delete.push k
+
+            else
+              # this is a real, unaliased index with this name, so it must be deleted.
+              # (This usually happens the first time we implement the aliasing scheme against
+              # an existing installation.)
+              es_client.indices.delete index: live_index_name rescue false
+            end
+          end
+
+          # re-alias
+          es_client.indices.update_aliases body: { actions: rename_actions }
+
+          # clean up
+          to_delete.each do |idxname|
+            es_client.indices.delete index: idxname rescue false
+          end
+        end
+
+        private
+
+        def tmp_index_pattern
+          /#{klass.index_name}_(\d{14})-\w{8}$/
+        end
+
+        def es_client
+          klass.__elasticsearch__.client
+        end
+
+        def stage_aliased_to
+          # find the newest tmp index to which staged is aliased.
+          # we need this because we want to re-alias it.
+          aliased_to = nil
+          stage_aliases = es_client.indices.get_aliases(index: stage_index_name)
+          stage_aliases.each do |k,v|
+            aliased_to ||= k
+            stage_tstamp = aliased_to.match(tmp_index_pattern)[1]
+            k_tstamp = k.match(tmp_index_pattern)[1]
+            if Time.parse(stage_tstamp) < Time.parse(k_tstamp)
+              aliased_to = k
+            end
+          end
+          if !aliased_to
+            raise "Cannot identify index aliased to by '#{stage_index_name}'"
+          end
+          aliased_to
+        end
+      end
+    end
+  end
+end

data/lib/elasticsearch/rails/ha/parallel_indexer.rb

@@ -0,0 +1,152 @@
+require 'ansi'
+
+module Elasticsearch
+  module Rails
+    module HA
+      class ParallelIndexer
+
+        attr_reader :klass, :idx_name, :nprocs, :batch_size, :max, :force, :verbose, :scope
+
+        # leverage multiple cores to run indexing in parallel
+        def initialize(opts)
+          @klass    = opts[:klass] or fail "klass required"
+          @idx_name = opts[:idx_name] or fail "idx_name required"
+          @nprocs   = opts[:nprocs] or fail "nprocs required"
+          @batch_size = opts[:batch_size] or fail "batch_size required"
+          @max        = opts[:max]
+          @force      = opts[:force]
+          @verbose    = opts[:verbose]
+          @scope      = opts[:scope]
+
+          # calculate array of offsets based on nprocs
+          @total_expected = klass.count
+          @pool_size = (@total_expected / @nprocs.to_f).ceil
+        end
+
+        def run
+          # get all ids since we can't assume there are no holes in the PK sequencing
+          ids = klass.order('id ASC').pluck(:id)
+          offsets = []
+          ids.each_slice(@pool_size) do |chunk|
+            #puts "chunk: size=#{chunk.size} #{chunk.first}..#{chunk.last}"
+            offsets.push( chunk.first )
+          end
+          if @verbose
+            puts ::ANSI.blue{ "Parallel Indexer: index=#{@idx_name} total=#{@total_expected} nprocs=#{@nprocs} pool_size=#{@pool_size} offsets=#{offsets} " }
+          end
+
+          if @force
+            @verbose and puts ::ANSI.blue{ "Force creating new index" }
+            klass.__elasticsearch__.create_index! force: true, index: idx_name
+            klass.__elasticsearch__.refresh_index! index: idx_name
+          end
+
+          @current_db_config = ActiveRecord::Base.connection_config
+          # IMPORTANT before forks in offsets loop
+          ActiveRecord::Base.connection.disconnect!
+
+          child_pids = []
+          offsets.each do |start_at|
+            child_pid = fork do
+              run_child(start_at)
+            end
+            if child_pid
+              child_pids << child_pid
+            end
+          end
+
+          # reconnect in parent
+          ActiveRecord::Base.establish_connection(@current_db_config)
+
+          # Process.waitall seems to hang during tests. Do it manually.
+          child_results = []
+
+          child_pids.each do |pid|
+            Process.wait(pid)
+            child_results.push [pid, $?]
+          end
+
+          process_child_results(child_results)
+        end
+
+        def process_child_results(results)
+          # check exit status of each child so we know if we should throw exception
+          results.each do |pair|
+            pid = pair[0]
+            pstat = pair[1]
+            exit_ok = true
+            if pstat.exited?
+              @verbose and puts ::ANSI.blue{ "PID #{pid} exited with #{pstat.exitstatus}" }
+            end
+            if pstat.signaled?
+              puts ::ANSI.red{ " >> #{pid} exited with uncaught signal #{pstat.termsig}" }
+              exit_ok = false
+            end
+
+            if !pstat.success?
+              puts ::ANSI.red{ " >> #{pid} was not successful" }
+              exit_ok = false
+            end
+
+            if pair[1].exitstatus != 0
+              puts ::ANSI.red{ " >> #{pid} exited with non-zero status" }
+              exit_ok = false
+            end
+
+            if !exit_ok
+              raise ::ANSI.red{ "PID #{pair[0]} exited abnormally, so the whole reindex fails" }
+            end
+          end
+        end
+
+        def run_child(start_at)
+          # IMPORTANT after fork
+          ActiveRecord::Base.establish_connection(@current_db_config)
+
+          # IMPORTANT for tests to determine whether at_end should run
+          ENV["I_AM_HA_CHILD"] = "true"
+
+          completed = 0
+          errors    = []
+          @verbose and puts ::ANSI.blue{ "Start worker #{$$} at offset #{start_at}" }
+          pbar = ::ANSI::Progressbar.new("#{klass} [#{$$}]", @pool_size, STDOUT) rescue nil
+          checkpoint = false
+          if pbar
+            pbar.__send__ :show
+            pbar.bar_mark = '='
+          else
+            checkpoint = true
+          end
+
+          @klass.__elasticsearch__.import return: 'errors',
+            index: @idx_name,
+            start: start_at,
+            scope: @scope,
+            batch_size: @batch_size    do |resp|
+              # show errors immediately (rather than buffering them)
+              errors += resp['items'].select { |k, v| k.values.first['error'] }
+              completed += resp['items'].size
+              if pbar && @verbose
+                pbar.inc resp['items'].size
+              end
+              if checkpoint && @verbose
+                puts ::ANSI.blue{ "[#{$$}] #{Time.now.utc.iso8601} : #{completed} records completed" }
+              end
+              STDERR.flush
+              STDOUT.flush
+              if errors.size > 0
+                STDOUT.puts "ERRORS in #{$$}:"
+                STDOUT.puts pp(errors)
+              end
+              if completed >= @pool_size || (@max && @max.to_i == completed)
+                pbar.finish if pbar
+                @verbose and puts ::ANSI.blue{ "Worker #{$$} finished #{completed} records" }
+                exit!(true) # exit child worker
+              end
+            end # end do |resp| block
+        end
+
+      end
+    end
+  end
+end

data/lib/elasticsearch/rails/ha/tasks.rb

@@ -0,0 +1,61 @@
+# Rake tasks to make parallel indexing and high availability easier.
+
+require 'elasticsearch/rails/ha'
+
+namespace :elasticsearch do
+  namespace :ha do
+
+    desc "import records in parallel"
+    task :import do
+      nprocs     = ENV['NPROCS'] || 1
+      batch_size = ENV['BATCH']  || 100
+      max        = ENV['MAX']    || nil
+      klass      = ENV['CLASS'] or fail "CLASS required"
+
+      indexer = Elasticsearch::Rails::HA::ParallelIndexer.new(
+        klass: klass.constantize,
+        idx_name: (ENV['INDEX'] || klass.constantize.index_name),
+        nprocs: nprocs.to_i,
+        batch_size: batch_size.to_i,
+        max: max,
+        scope: ENV.fetch('SCOPE', nil),
+        force: ENV['FORCE'],
+        verbose: !ENV['QUIET']
+      )
+
+      indexer.run
+    end
+
+    desc "stage an index"
+    task :stage do
+      nprocs     = ENV['NPROCS'] || 1
+      batch_size = ENV['BATCH']  || 100
+      max        = ENV['MAX']    || nil
+      klass      = ENV['CLASS'] or fail "CLASS required"
+
+      stager = Elasticsearch::Rails::HA::IndexStager.new(klass)
+      indexer = Elasticsearch::Rails::HA::ParallelIndexer.new(
+        klass: stager.klass,
+        idx_name: (ENV['INDEX'] || stager.tmp_index_name),
+        nprocs: nprocs.to_i,
+        batch_size: batch_size.to_i,
+        max: max,
+        scope: ENV.fetch('SCOPE', nil),
+        force: true,
+        verbose: !ENV['QUIET']
+      )
+      indexer.run
+      stager.alias_stage_to_tmp_index
+      puts "[#{Time.now.utc.iso8601}] #{klass} index staged as #{stager.stage_index_name}"
+    end
+
+    desc "promote staged index to live"
+    task :promote do
+      klass = ENV['CLASS'] or fail "CLASS required"
+      stager = Elasticsearch::Rails::HA::IndexStager.new(klass)
+      stager.promote(ENV['INDEX'])
+      puts "[#{Time.now.utc.iso8601}] #{klass} promoted #{stage_index_name} to #{stager.live_index_name}"
+    end
+
+  end
+end

data/lib/elasticsearch/rails/ha/version.rb

@@ -0,0 +1,7 @@
+module Elasticsearch
+  module Rails
+    module HA
+      VERSION = '1.0.0'
+    end
+  end
+end

metadata

@@ -0,0 +1,193 @@
+--- !ruby/object:Gem::Specification
+name: elasticsearch-rails-ha
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Peter Karman
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-01-25 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: elasticsearch-model
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: elasticsearch-rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: ansi
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: about_yml
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: elasticsearch-extensions
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.1'
+description: High Availability extensions to the Elasticsearch::Rails gem
+email:
+- peter.karman@gsa.gov
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CONTRIBUTING.md
+- LICENSE.md
+- README.md
+- lib/elasticsearch/rails/ha.rb
+- lib/elasticsearch/rails/ha/index_stager.rb
+- lib/elasticsearch/rails/ha/parallel_indexer.rb
+- lib/elasticsearch/rails/ha/tasks.rb
+- lib/elasticsearch/rails/ha/version.rb
+homepage: https://github.com/18F/elasticsearch-rails-ha-gem
+licenses:
+- CC0
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 1.9.3
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5.1
+signing_key: 
+specification_version: 4
+summary: High Availability extensions to the Elasticsearch::Rails gem
+test_files: []
+has_rdoc: