elasticsearch-indexstager 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ee43a625d4aa2fb58e9217e24c8fad74a46650b0
+  data.tar.gz: e3c035fb1260b752efcdbe606dd06083c7a9c3e9
+SHA512:
+  metadata.gz: 90ed4e107ada5446a21e56b937b83130dddc28e34f1dd49a54f7178437efb764bfa30c3963e48ebd647f0cbcb95dbcf6745f0194e4937db53184c6e4a75ea355
+  data.tar.gz: 33c25980044d506eefa1c1a49036e5311911bf9b45900c51e1145965918ffd6f90de061f4964e821db89a7470c9fba06cbd421fcc7788d639883502b45e21095

data/CONTRIBUTING.md

@@ -0,0 +1,15 @@
+## Welcome!
+
+We're so glad you're thinking about contributing to an 18F open source project! If you're unsure or afraid of anything, just ask or submit the issue or pull request anyways. The worst that can happen is that you'll be politely asked to change something. We appreciate any sort of contribution, and don't want a wall of rules to get in the way of that.
+
+Before contributing, we encourage you to read our CONTRIBUTING policy (you are here), our LICENSE, and our README, all of which should be in this repository. If you have any questions, or want to read more about our underlying policies, you can consult the 18F Open Source Policy GitHub repository at https://github.com/18f/open-source-policy, or just shoot us an email/official government letterhead note to [18f@gsa.gov](mailto:18f@gsa.gov).
+
+## Public domain
+
+This project is in the public domain within the United States, and
+copyright and related rights in the work worldwide are waived through
+the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+
+All contributions to this project will be released under the CC0
+dedication. By submitting a pull request, you are agreeing to comply
+with this waiver of copyright interest.

data/LICENSE.md

@@ -0,0 +1,31 @@
+As a work of the United States Government, this project is in the
+public domain within the United States.
+
+Additionally, we waive copyright and related rights in the work
+worldwide through the CC0 1.0 Universal public domain dedication.
+
+## CC0 1.0 Universal Summary
+
+This is a human-readable summary of the [Legal Code (read the full text)](https://creativecommons.org/publicdomain/zero/1.0/legalcode).
+
+### No Copyright
+
+The person who associated a work with this deed has dedicated the work to
+the public domain by waiving all of his or her rights to the work worldwide
+under copyright law, including all related and neighboring rights, to the
+extent allowed by law.
+
+You can copy, modify, distribute and perform the work, even for commercial
+purposes, all without asking permission.
+
+### Other Information
+
+In no way are the patent or trademark rights of any person affected by CC0,
+nor are the rights that other persons may have in the work or in how the
+work is used, such as publicity or privacy rights.
+
+Unless expressly stated otherwise, the person who associated a work with
+this deed makes no warranties about the work, and disclaims liability for
+all uses of the work, to the fullest extent permitted by applicable law.
+When using or citing the work, you should not imply endorsement by the
+author or the affirmer.

data/README.md

@@ -0,0 +1,40 @@
+# elasticsearch-indexstager RubyGem
+
+[![Build Status](https://travis-ci.org/18F/elasticsearch-indexstager-gem.svg?branch=master)](https://travis-ci.org/18F/elasticsearch-indexstager-gem)
+
+Elasticsearch index management, for stage/promote pattern.
+
+See also:
+
+* [elasticsearch-rails-ha](https://github.com/18F/elasticsearch-rails-ha)
+
+## Examples
+
+A busy ES installation needs to stay up and serving requests. If you need to build a new index,
+you can "stage" it alongside the live index, and then "promote" the stage to be live. This allows
+for zero downtime cutovers of new indices.
+
+```ruby
+require 'elasticsearch'
+require 'elasticsearch/index_stager'
+
+client = Elasticsearch::Client.new log: true
+stager = Elasticsearch::IndexStager.new(index_name: 'foo', es_client: client)
+
+client.index(index: stager.tmp_index_name, type: 'article', id: 1, body: { title: 'Test' })
+stager.alias_stage_to_tmp_index
+
+results = client.search(index: stager.stage_index_name, body: { query: { match: { title: 'test' } } })
+stager.promote
+results = client.search(index: stager.index_name, body: { query: { match: { title: 'test' } } })
+```
+
+## Public domain
+
+This project is in the worldwide [public domain](LICENSE.md). As stated in [CONTRIBUTING](CONTRIBUTING.md):
+
+> This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+>
+> All contributions to this project will be released under the CC0
+> dedication. By submitting a pull request, you are agreeing to comply
+> with this waiver of copyright interest.

data/lib/elasticsearch/index_stager.rb

@@ -0,0 +1,100 @@
+module Elasticsearch
+  class IndexStager
+    VERSION = '1.0.0'
+
+    attr_reader :index_name, :es_client
+
+    def initialize(opts)
+      @index_name = opts[:index_name] or fail ":index_name required"
+      @es_client = opts[:es_client] or fail ":es_client required"
+    end
+
+    def stage_index_name
+      index_name + "_staged"
+    end 
+
+    def tmp_index_name
+      @_suffix ||= Time.now.strftime('%Y%m%d%H%M%S') + '-' + SecureRandom.hex[0..7]
+      "#{index_name}_#{@_suffix}"
+    end 
+
+    def alias_stage_to_tmp_index
+      es_client.indices.delete index: stage_index_name rescue false
+      es_client.indices.update_aliases body: {
+        actions: [
+          { add: { index: tmp_index_name, alias: stage_index_name } } 
+        ]   
+      }   
+    end
+
+    def promote(live_index_name=index_name)
+      @live_index_name = live_index_name || index_name
+
+      # the renaming actions (performed atomically by ES)
+      rename_actions = [ 
+        { remove: { index: stage_aliased_to, alias: stage_index_name } },
+        {    add: { index: stage_index_name, alias: @live_index_name } } 
+      ]   
+
+      # zap any existing index known as index_name,
+      # but do it conditionally since it is reasonable that it does not exist.
+      to_delete = []
+      existing_live_index = es_client.indices.get_aliases(index: @live_index_name)
+      existing_live_index.each do |k,v|
+
+        # if the index is merely aliased, remove its alias as part of the aliasing transaction.
+        if k != @live_index_name
+          rename_actions.unshift({ remove: { index: k, alias: @live_index_name } })
+
+          # mark it for deletion when we've successfully updated aliases
+          to_delete.push k
+
+        else
+          # this is a real, unaliased index with this name, so it must be deleted.
+          # (This usually happens the first time we implement the aliasing scheme against
+          # an existing installation.)
+          es_client.indices.delete index: @live_index_name rescue false
+        end
+      end
+
+      # re-alias
+      es_client.indices.update_aliases body: { actions: rename_actions }
+
+      # clean up
+      to_delete.each do |idxname|
+        es_client.indices.delete index: idxname rescue false
+      end
+    end
+
+    private
+
+    def tmp_index_pattern
+      /#{index_name}_(\d{14})-\w{8}$/
+    end
+
+    def stage_aliased_to
+      # find the newest tmp index to which staged is aliased.
+      # we need this because we want to re-alias it.
+      aliased_to = find_newest_alias_for(stage_index_name)
+    end
+
+    def find_newest_alias_for(the_index_name)
+      aliased_to = nil
+      aliases = es_client.indices.get_aliases(index: the_index_name)
+      aliases.each do |k,v|
+        next unless k.match(tmp_index_pattern)
+        aliased_to ||= k
+        alias_tstamp = aliased_to.match(tmp_index_pattern)[1]
+        k_tstamp = k.match(tmp_index_pattern)[1]
+        if Time.parse(alias_tstamp) < Time.parse(k_tstamp)
+          aliased_to = k
+        end
+      end
+      if !aliased_to
+        raise "Cannot identify index aliased to by '#{the_index_name}'"
+      end
+      aliased_to
+    end
+
+  end
+end

metadata

@@ -0,0 +1,133 @@
+--- !ruby/object:Gem::Specification
+name: elasticsearch-indexstager
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Peter Karman
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-02-25 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: elasticsearch
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: about_yml
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: elasticsearch-extensions
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: manage Elasticsearch indexes using staging pattern
+email:
+- peter.karman@gsa.gov
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CONTRIBUTING.md
+- LICENSE.md
+- README.md
+- lib/elasticsearch/index_stager.rb
+homepage: https://github.com/18F/elasticsearch-indexstager-gem
+licenses:
+- CC0
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 1.9.3
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5.1
+signing_key: 
+specification_version: 4
+summary: manage Elasticsearch indexes using staging pattern
+test_files: []
+has_rdoc: