consul_watcher 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 84e80ff2e7d93407b1fbf14774a8cb4d585962468da5a15e8c491a22408d3de9
+  data.tar.gz: 227c86658315bcbe920e6110b3818a3a7879b7899809c5f095293d085d7656e7
+SHA512:
+  metadata.gz: 992881948d116b0ef750143056cc2d0595ea25795c6866e69ff3c28910d9939383f8eaf468401146a6e403fd5cfa39c294261141d1716253471dd3b2c0c2ec20
+  data.tar.gz: 9c6eaf4fd1c6c1e98eba0e2f640d83cb602e064de618cdc57280bd50a5a0f0bf7d2a15c6b4415474f7a33adf99d3b65c6039d609a6919eebb0d3bf3921c0c372

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/Dockerfile

@@ -0,0 +1,13 @@
+FROM consul:1.4.0
+
+ARG gem_file
+
+RUN apk add --no-cache jq build-base libc-dev linux-headers postgresql-dev libxml2-dev libxslt-dev ruby ruby-dev ruby-rdoc ruby-irb
+
+COPY ./docker-entrypoint.sh /
+COPY pkg/$gem_file /consul_watcher/
+
+RUN cd /consul_watcher ; \
+    gem install $gem_file
+
+ENTRYPOINT ["/docker-entrypoint.sh"]

data/Gemfile

@@ -0,0 +1,8 @@
+source "https://rubygems.org"
+
+repo_name = '/fortman/consul_watcher'
+
+git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in consul_watcher.gemspec
+gemspec

data/README.md

@@ -0,0 +1,33 @@
+# NOTICE
+This gem is a work in progress.  Please see the [TODO](https://github.com/fortman/consul_watcher/blob/master/docs/TODO.md) documentation
+
+
+## Note on naming
+This project name is a little confusing in that it called `consul_watcher`.  It is wrapping the `consul watch` command line tool from Hashicorp.  To avoid confusion, this project in the context of both the docker image and ruby gem will be referred to as `consul_watcher`.  Any time that `consul watch` is used, that is a reference to the actualy Hashicorp command.
+
+## Basic functionality
+This git project produces a docker image.  The basic premise of this docker image is to
+* parse the json output of the [consul watch](https://www.consul.io/docs/commands/watch.html) command
+* compare the output to the previous consul watch output
+* send the diff to a destination
+The primary destination at this time is an AMQP topic exchange.  The underlying logic in the docker image is a ruby gem.  To make the gem modular and extensible, the functionality has been broken up into 3 sections; storage, watch_type, and destination.
+
+If we want to do a diff of consul watches, we need to store the previous consul watch json.  This is because consul watches kick off new executions on every change, so we can't store the previous run in memory.  Storage is accomplished with [storage classes](https://github.com/fortman/consul_watcher/blob/master/docs/storage/storage.md).  The purpose of the storage class is to just store and retrieve previous consul watch json.  At this time, backend storage is planned for both local filesystem, and consul kv storage.
+
+The next type of class, is the [watch_type class](https://github.com/fortman/consul_watcher/blob/master/docs/watch_type/watch_type.md).  This maps to the actual consul watch command `--type` flag.  There are two unfortunate things about the way consul watch commands work.  The first is that there is not one consul watch command that can watch for all changes.  Each different type of watch requires its own definition.  It would be nice to specify one watch that can monitor for changes in all consul areas.  This is not the case however.  The lack of a single definition bleeds into the next issue which is, each watch type has a very distinct json output format.  For the watch json output, instead of having meta data that describes what type of watch the output represents, it is expected that you already know the format because you have to pass the type to the watch command.  The entire purpose of the watch_type class is to handle the unique json output for each consul watch type.  For every option to the consul watch --type flag, there is a watch_type class.  The one exception to this is that `key` and `keyprefix` types share the same output.  There is just one `key` watch_type class to represent both.
+
+After the watch_type class handles the data, it will create a json diff between the previous run and the current.  This json diff is the [message](https://github.com/fortman/consul_watcher/blob/master/docs/messages_overview.md) that is sent to the [destinations classes](https://github.com/fortman/consul_watcher/blob/master/docs/destination/destination.md).  The initial implementation will support two destination classes; AMQP topic exchanges, and stdout through [jq](https://stedolan.github.io/jq/).
+
+## Quick start guide
+The quickest and easiest way to get started is to follow the [docker quickstart](https://github.com/fortman/consul_watcher/blob/master/docs/docker-quickstart.md).
+
+If you are planning to write your own extension for storage/watch_type/destination, then you can check out the [ruby quickstart](https://github.com/fortman/consul_watcher/blob/master/docs/ruby-quickstart.md).
+
+## Development
+Please read the [general design documentation](https://github.com/fortman/consul_watcher/blob/master/docs/general_design.md).  More detailed architectual documenation will be created in the future.  All development [build and test commands](https://github.com/fortman/consul_watcher/blob/master/docs/rake_tasks.md) are defined as rake tasks.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/fortman/consul_watcher.
+
+PS. I swear I'm done changing the project name :)

data/Rakefile

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'open3'
+
+spec_file = Gem::Specification.load('consul_watcher.gemspec')
+
+task default: :docker_build
+
+task :docker_tag, [:version, :docker_image_id] do |_task, args|
+  puts "Docker id #{args['docker_image_id']} => tag rfortman/consul_watcher:#{args['version']}"
+  tag_cmd = "docker tag #{args['docker_image_id']} rfortman/consul_watcher:#{args['version']}"
+  Open3.popen3(tag_cmd) do |_stdin, _stdout, stderr, wait_thr|
+    error = stderr.read
+    puts error unless wait_thr.value.success?
+  end
+end
+
+task docker_build: [:build] do
+  docker_image_id = nil
+  build_cmd = "docker build --build-arg gem_file=consul_watcher-#{spec_file.version}.gem ."
+  threads = []
+  Open3.popen3(build_cmd) do |_stdin, stdout, stderr, wait_thr|
+    { out: stdout, err: stderr }.each do |key, stream|
+      threads << Thread.new do
+        until (raw_line = stream.gets).nil?
+          match = raw_line.match(/Successfully built (.*)$/i)
+          docker_image_id = match.captures[0] if match
+          puts raw_line.to_s
+        end
+      end
+    end
+    threads.each(&:join)
+    if wait_thr.value.success?
+      Rake::Task['docker_tag'].invoke(spec_file.version, docker_image_id)
+      Rake::Task['docker_tag'].reenable
+      Rake::Task['docker_tag'].invoke('latest', docker_image_id)
+    end
+  end
+end
+
+task :test do
+  build_cmd = 'docker-compose --file test/docker-compose.yml up rabbitmq consul consul-watcher'
+  threads = []
+  Open3.popen3(build_cmd) do |_stdin, stdout, stderr, wait_thr|
+    { out: stdout, err: stderr }.each do |key, stream|
+      threads << Thread.new do
+        until (raw_line = stream.gets).nil?
+          puts raw_line.to_s
+        end
+      end
+    end
+    threads.each(&:join)
+  end
+end

data/VERSION

@@ -0,0 +1 @@
+0.0.1

data/bin/console_watcher

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'consul_watcher'
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require 'irb'
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/consul_watcher.gemspec

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+lib = File.expand_path('lib', '..')
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |spec|
+  spec.name          = 'consul_watcher'
+  spec.version       = IO.read('VERSION').chomp
+  spec.authors       = ['Ryan Fortman']
+  spec.email         = ['r.fortman.dev@gmail.com']
+
+  spec.summary       = 'Send consul watch events to an amqp.'
+  spec.homepage      = 'https://github.com/fortman/consul_watcher'
+
+  spec.files = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = 'exe'
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+
+  # spec.add_development_dependency 'bundler', '~> 2.0'
+  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_runtime_dependency 'bundler', '~> 2.0'
+  spec.add_dependency 'bunny', '~> 1.7.0'
+  spec.add_dependency 'hashdiff', '~> 0.3'
+  spec.add_dependency 'slop', '~> 4.6'
+end

data/docker-entrypoint.sh

@@ -0,0 +1,25 @@
+#!/bin/sh
+
+[ "x${CONSUL_HTTP_ADDR}" == "x" ] && export CONSUL_HTTP_ADDR="http://127.0.0.1:8500"
+[ "x${WATCH_SCRIPT}" == "x" ] && export WATCH_SCRIPT="/usr/bin/env jq"
+
+echo
+echo "CLI_ARGS: \"${@}\""
+echo "WATCH_ARGS: \"${WATCH_ARGS}\""
+echo "WATCH_SCRIPT: \"${WATCH_SCRIPT}\""
+echo "CONSUL_HTTP_ADDR: \"${CONSUL_HTTP_ADDR}\""
+echo "RUN_ONCE: \"${RUN_ONCE}\""
+if [[ ${#} != 0 ]]; then
+  exec $@
+elif [[ "x${WATCH_ARGS}" != "x" && "x${RUN_ONCE}" == "x" ]]; then
+  echo "consul watch ${WATCH_ARGS} ${WATCH_SCRIPT}"
+  echo
+  exec consul watch ${WATCH_ARGS} ${WATCH_SCRIPT}
+elif [[ "x${WATCH_ARGS}" != "x" && "x${RUN_ONCE}" != "x" ]]; then
+  echo "consul watch ${WATCH_ARGS} | ${WATCH_SCRIPT}"
+  echo
+  exec consul watch ${WATCH_ARGS} | ${WATCH_SCRIPT}
+else
+  echo "Don't know what to do, WATCH_ARGS not defined and no arguments passed in"
+  exit 1
+fi

data/docs/TODO.md

@@ -0,0 +1,46 @@
+
+# TODO:
+
+#### Status abbrevations
+
+| abbrevation | meaning                                      |
+| ----------- |:-------------------------------------------- |
+| COMP        | completed (per initial design)               |
+| FUNC        | functional (working, but will have updates)  |
+| IP          | in progress                                  |
+| PL          | planned                                      |
+| PROP        | proposed                                     |
+
+## High level tasks
+| task                               | status |
+|:---------------------------------- |:------ |
+| Create documentation               | IP     |
+| Publish docker image and ruby gem  | IP     |
+| Write unit tests                   | PL     |
+
+## Functional ruby classes implementation
+
+#### Storage class implementation status
+
+| storage class | status      |
+| ------------- |:----------- |
+| file system   | FUNC        |
+| consul        | PL          |
+
+#### Watch Type class implementation status
+
+| watch_type class | status      |
+| ---------------- |:----------- |
+| key/keyprefix    | FUNC        |
+| checks           | IP          |
+| services         | planned     |
+| service          | planned     |
+| nodes            | planned     |
+| event            | planned     |
+
+#### Destination class implementation status
+
+| destination class | status      |
+| ----------------- |:----------- |
+| AMQP              | FUNC        |
+| JQ                | FUNC        |

data/docs/destination/amqp.md

@@ -0,0 +1 @@
+# AMQP message destination

data/docs/destination/destination.md

@@ -0,0 +1,2 @@
+# Consul Watcher Destinations
+The basic premise of consul_watcher is to monitor consul via a consul watch command.  The json that is generated by consul watch, and then send a `diff`

data/docs/docker-quickstart.md

@@ -0,0 +1,30 @@
+# Docker quickstart
+
+The easiest way to get started with consul_watcher is with the docker image.  The docker entry point will run a consul watch and then pipe the output to the ruby consul_watcher program.  The behavior is mostly driven by passing environmental variables into docker.  Environmental variables, command line parameters and a configuration file are being tested to determine which is best way to configure consul_watch.  There is a docker compose file to test things out locally  The following steps should have you working with a local cluster fairly quick.  A rake task will be created to automate these steps at some point.
+
+From the root of the git repository execute the following<br/>
+:> `docker-compose --file test/docker-compose.yml up --no-start`<br/>
+:> `docker-compose --file test/docker-compose.yml start consul rabbitmq`<br/>
+Wait a bit of time for those services to start.  30 seconds should suffice.<br/>
+Login to rabbitmq locally http://localhost:15672. Use guest/guest as password.  <br/>
+Create a queue and bind it to the amq.topic exchange.  Use routing key `consul_watcher.key.#` for the bind.<br/>
+:> `docker-compose --file test/docker-compose.yml start consul-watcher`<br/>
+Login to consul locally http://localhost:8500.  You should be able to start creating, updating and deleting entries in the kv store.  Go back to rabbitmq and you should be seeing messages in the queue you created.<br/>
+
+# environmental variables used by docker image
+| environment variable | description                                                         | example value                                                                                               |
+| -------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
+| WATCH_ARGS           | arguments passed into the consul watch command                      | --type keyprefix --prefix /                                                                                 |
+| WATCH_SCRIPT         | the output of the consul watch is piped to this command             | /usr/bin/consul_watcher --config-file /etc/consul_watch/config.json --storage-name testing --watch-type key |
+| RUN_ONCE             | determins if the consul watch runs once, or multiple times          | set to any string to enable `run once` mode.  Unset/empty string means run continuously                     |
+| CONSUL_HTTP_ADDR     | used by consul, any consul environmental variables can be specified | http://localhost:8500                                                                                       |
+
+## consul_watcher command line arguments
+| command line argument | description | example value |
+| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------- |
+| --config-file (file)  | reference to the configuration file, need to mount in as volume to docker                                                                  | /etc/consul_watch/config.json           |  
+| --watch-type (type)   | must match the --watch-type value passed to consul, keyprefix is should just be specified as key                                           | key/services/nodes/service/checks/event | 
+| --storage-name (name) | just a unqiue name for this watch.  if you are storing multiple watches to the same backend, you should make each watch have a unique name | test-watch                              |
+
+## Configuration file
+

data/docs/messages_overview.md

@@ -0,0 +1,61 @@
+# Messages
+Messages are the json diffs that are sent to the destinations.  Each watch_type will be unique because each json watch type has it's own json formatting.  Please look the documentation for each watch type for an example of that types message format.  The following is an overview of the shared properties that each watch type shares.  As well as the actual json diff, there is a little bit of meta data included as well.
+
+Example of a key watch message
+```
+{
+  "watch_type": "key",
+  "id": "key.test",
+  "diff": [
+    "~",
+    [
+      "test",
+      "Value"
+    ],
+    "old_value",
+    "new_value"
+  ]
+}
+```
+#### watch_type
+The top level of the data structure is a hash with 3 elements.  `watch_type`, `id` and `diff`.  `watch_type` should be obvious.  It matches both the `--type` sent to the consul watch command, as well as the `--watch-type`.  The example output above was generated by the following command;
+
+`consul watch --type keyprefix --prefix / /usr/bin/consul_watcher --config-file /etc/consul_watch/config.json --storage-name testing --watch-type key`
+
+Note that the `--type keyprefix` and `--type key` specified to the consul watch command share the same json output format.  For these two, they both just use `--watch-type key` parameter for the consul_watcher ruby command.
+
+#### id
+This is a unique id that can be used to identify what change.  For a `key` watch, this will result in an id of `key_(path to key that change)`.
+
+#### diff
+The comparision between the previous json and the current json is handled by a ruby gem called [hashdiff](https://github.com/liufengyun/hashdiff#diff).  The `hashdiff` github page has a good explaination of the diff output.  Because arrays can be unordered in the consul watch output, this can lead to false positives in diff changes.  To avoid this, data manipulation is done in the consul_watcher logic to remove arrays.  The json diff format can be summed up as follows.
+
+The first element in the diff array is the operation. '+' refers to a new element. '-' refers to an element that was removed. '~' refers to an element that changed.
+
+The number of elements for diff arrays of '+' and '-' are both size 3.  The size for a '~' is size for.
+
+'+' operation
+The first element defines it as an addition operation, and will always be '+'
+The second element is an array that represents the nested path inside the json to the key that was added
+The third element is the value of the new key 
+
+'-' operation
+The first element defines it as a removal operation, and will always be '-'
+The second element is an array that represents the nested path inside the json to the key that was removed
+The third element is the value of the key that was removed
+
+'~' operation ( note the 4th array element )
+The first element defines it as a modification operation, and will always be '~'
+The second element is an array that represents the nested path inside the json to the key that was change
+The third element is the previous value of the key that was changed
+The fourth element is the new value of the key that was changed
+
+#### Wrapping up
+
+Summing up our example using this information:
+* There was a modification to the key located at 'test' in the consul kv store
+* This was represented in the consul watch json output as { "test": { "Value": "new value" } }
+* The old value was a string of 'old_value'
+* and the new value a string of 'new_value'
+
+This will enable consumers of the messages to have all the information they need; the type of operation, the location of what changed, and old/new value(s)

data/docs/storage/disk.md

@@ -0,0 +1,2 @@
+# Disk Storage for consul watch output
+storage_parent_dir

data/docs/storage/storage.md

@@ -0,0 +1,8 @@
+# Storage classes
+The storage classes are responsible for storing previous consul watch output.  Even though multiple backends are supported, you will only ever have one backend enabled and configured at a time.  This is needed so that future runs of consul watch can compare output with previous watch runs.  The entire purpose of the consul_watcher project is to create this diff and send it to a destination.
+
+## Disk storage class
+The most basic form of storage is the [disk storage class](https://github.com/fortman/consul_watcher/blob/master/lib/consul_watcher/storage/disk.rb).  The behavior of this class is driven by the configuration file.  Please see the detailed [disk class documentation](https://github.com/fortman/consul_watcher/blob/master/docs/storage/disk.md) for configuration options.
+
+## Consul kv storage class
+Since we know for sure that consul is installed and running, it is probably a good idea to enable consul kv as a backend storage type.  The main concern here is that we need to make sure we ignore the storage path from the consul watch, so we do not get into an infinite watch loop.  The [consul storage class](https://github.com/fortman/consul_watcher/blob/master/lib/consul_watcher/storage/consul.rb) is responsible for reading and writing previous json runs in the consul kv store.  Configuration for this is tricky as we need to drive the docker entrypoint with environmental variables.  The docker entrypoint is what is running the consul watch command, so it needs access to consul environmental variables.  The direction is to start using the configuration files for all things related to the ruby script.  For now the consul_watcher will only look at the configuration file for consul backend settings.  This will lead to duplicated settings (env vars and config file).  The behavior might change in the future (allow env var references in config file for example).

data/docs/watch_type/watch_type.md

@@ -0,0 +1,7 @@
+# watch_type classes
+The watch_type classes have a few different functions.  The first and main function is to create a json diff between consul watch runs.  This diff will later be sent to a [destination class](https://github.com/fortman/consul_watcher/blob/master/docs/destination/destination.md).  The second purpose of this class is to do some data translation.  The consul watch json output is not ideal for doing a diff.  A good example is that arrays in the output are un-ordered, and between runs can change position.  This causes things to appear to have changed, when really all they did was change their order in the array.  To get around this, the watch_type classes strive to obscure these non-changes and provide a good json format for diffs.
+
+## key and keyprefix
+Key and keyprefix have identical output format, so they will share a [watch_type class named key](https://github.com/fortman/consul_watcher/blob/master/lib/consul_watcher/watch_type/key.rb).  Configuration is driven by the consul_watcher configuration file and [key specific configuration](https://github.com/fortman/consul_watcher/blob/master/docs/watch_type/key.md).  
+
+## More to come later, there will need to be watch_type for every single `consul watch --type`

data/exe/consul_watcher

@@ -0,0 +1,24 @@
+#!/usr/bin/env ruby
+
+# frozen_string_literal: true
+
+require 'slop'
+require 'logger'
+require 'json'
+require 'consul_watcher'
+
+logger = Logger.new(STDOUT)
+
+opts = Slop.parse do |o|
+  o.string '--watch-type', required: true
+  o.string '--storage-name', required: true
+  o.string '--config-file', required: true
+  o.on '-h', '--help', 'print help' do
+    logger.warn("\n#{o}")
+    exit(false)
+  end
+end
+
+config = JSON.parse(File.read(opts['config-file']))
+
+ConsulWatcher.watch(opts['watch-type'], opts['storage-name'], config)

data/lib/consul_watcher/class_helper.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module ConsulWatcher
+  # Define methods to handle default initialization behavior
+  module ClassHelper
+    def populate_variables(config = {})
+      defaults.each_pair do |key, default_value|
+        key = key.to_s
+        if config[key]
+          instance_variable_set("@#{key}", config[key])
+        elsif ENV[key.upcase.to_s]
+          instance_variable_set("@#{key}", ENV[key.upcase.to_s])
+        else
+          instance_variable_set("@#{key}", default_value)
+        end
+      end
+    end
+  end
+end

data/lib/consul_watcher/destination/amqp.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require 'bunny'
+
+module ConsulWatcher
+  module Destination
+    # Send diff output to jq command line
+    class Amqp
+      def initialize(destination_config)
+        @conn = Bunny.new(host: destination_config['rabbitmq']['server'] || 'localhost',
+                          port: destination_config['rabbitmq']['port'] || '5672',
+                          vhost: destination_config['rabbitmq']['vhost'] || '/',
+                          username: destination_config['rabbitmq']['username'] || 'guest',
+                          password: destination_config['rabbitmq']['password'] || 'guest')
+        @conn.start
+        @ch = @conn.create_channel
+        @ex = Bunny::Exchange.new(@ch,
+                                  :topic,
+                                  destination_config['rabbitmq']['exchange'] || 'amq.topic',
+                                  durable: true)
+      end
+
+      def send(change)
+        puts 'publishing message'
+        @ex.publish(change, routing_key: "consul_watcher.#{change['id']}")
+      end
+    end
+  end
+end

data/lib/consul_watcher/destination/jq.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require 'open3'
+
+module ConsulWatcher
+  module Destination
+    # Send diff output to jq command line
+    class Jq
+      def initialize(destination_config)
+      end
+
+      def send(change)
+        change_json = JSON.pretty_generate(change)
+        Open3.popen3("/usr/bin/env jq '.'") do |stdin, stdout, stderr, wait_thr|
+          stdin.puts "#{change_json}\r\n"
+          stdin.close
+          error = stderr.read
+          stderr.close
+          puts stdout.read
+          stdout.close
+          puts error unless wait_thr.value.success?
+          puts
+        end
+      end
+    end
+  end
+end

data/lib/consul_watcher/diff.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require 'hashdiff'
+require 'logger'
+
+module ConsulWatcher
+  class Diff
+    def initialize(parser_config) end
+
+    def parse(previous_watch_data, current_watch_data)
+      diff = HashDiff.diff(sanitize_data(previous_watch_data), sanitize_data(current_watch_data), array_path: true)
+      diff.each do |change|
+        puts "change: #{change}"
+        change[1] = change[1]&.join('/')
+      end
+      puts "hashdiff: #{diff}"
+      diff
+    end
+
+    private
+
+    def sanitize_data(data)
+      data = '[]' if data.nil? || data == "null\n"
+      data
+    end
+  end
+end

data/lib/consul_watcher/storage/consul.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+# This will be a module to store previous consul watch json to compare with previous watch data
+module ConsulWatcher
+  module Storage
+    # Consul storage for previous watch data
+    class Disk
+      def initialize(storage_config)
+        @parent_dir = storage_config['storage_parent_dir'] || '/tmp'
+      end
+    end
+  end
+end

data/lib/consul_watcher/storage/disk.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module ConsulWatcher
+  module Storage
+    # Disk storage for previous watch data
+    class Disk
+      def initialize(storage_config)
+        @parent_dir = storage_config['storage_parent_dir'] || '/tmp'
+      end
+
+      def fetch(watch_name)
+        file = File.open(cache_file_name(watch_name), mode: 'r')
+        data = file.read
+        file.close
+        data
+      rescue Errno::ENOENT
+        '{}'
+      end
+
+      def push(watch_name, data)
+        file = File.open(cache_file_name(watch_name), mode: 'w')
+        file.write(data)
+        file.close
+      end
+
+      private
+
+      def cache_file_name(watch_name)
+        "#{@parent_dir}/#{watch_name}.json"
+      end
+    end
+  end
+end

data/lib/consul_watcher/watch_type/checks.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require 'open3'
+
+module ConsulWatcher
+  module WatchType
+    class Checks
+      def initialize(destination_config) end
+
+      def get_changes(previous_watch_json, current_watch_json)
+        HashDiff.diff(json_to_hash(previous_watch_json),
+                      json_to_hash(current_watch_json),
+                      array_path: true)
+      end
+
+      def id(change)
+        "key.#{change[1][0].tr('/', '.')}"
+      end
+
+      def json_to_hash(json)
+        json = '{}' if json.nil? || json == "null\n"
+        JSON.parse(json).map do |check|
+          { check['Node'] => { check['CheckID'] => check } }
+        end.reduce({}, :merge)
+      end
+    end
+  end
+end

data/lib/consul_watcher/watch_type/key.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+require 'base64'
+
+module ConsulWatcher
+  module WatchType
+    class Key
+      def initialize(destination_config) end
+
+      def get_changes(previous_watch_json, current_watch_json)
+        json_diff = HashDiff.diff(json_to_hash(previous_watch_json),
+                                  json_to_hash(current_watch_json),
+                                  array_path: true)
+        json_diff.each.collect do |change|
+          # change[1] = change[1].join('/')
+          change[2] = Base64.decode64(change[2]) if change[2].is_a? String
+          change[3] = Base64.decode64(change[3]) if change[3].is_a? String
+          { 
+            'watch_type' => 'key',
+            'id' => id(change),
+            'diff' => change
+          }
+        end
+        #json_diff.reject { |change| change[0] == '~' && change[1][-1] == 'ModifyIndex' }
+      end
+
+      def id(change)
+        "key.#{change[1][0].tr('/', '.')}"
+      end
+
+      def json_to_hash(json)
+        json = '{}' if json.nil? || json == "null\n"
+        JSON.parse(json).map do |kv|
+          { kv['Key'] => kv.reject { |key, _value| key == 'Key' } }
+        end.reduce({}, :merge)
+      end
+      
+    end
+  end
+end

data/lib/consul_watcher.rb

@@ -0,0 +1,47 @@
+#!/usr/bin/env ruby
+
+# frozen_string_literal: true
+
+require 'json'
+require 'hashdiff'
+
+# Top Level module to run watch logic
+module ConsulWatcher
+  def self.watch(watch_type, storage_name, config)
+    storage = get_storage(config['storage']) if config['storage']
+    watch_type = get_watch_type(config['watch_type']) if config['watch_type']
+    filters = get_filters(config['filters']) if config['filters']
+    destination = get_destination(config['destination']) if config['destination']
+
+    current_watch_json = $stdin.read
+    previous_watch_json = storage.fetch(storage_name)
+    changes = watch_type.get_changes(previous_watch_json, current_watch_json)
+    changes.each do |change|
+      destination.send(change)
+    end
+    storage.push(storage_name, current_watch_json)
+  end
+
+  def self.get_storage(storage_config)
+    classname = storage_config['classname'] || 'ConsulWatcher::Storage::Disk'
+    require classname_to_file(classname)
+    Object.const_get(classname).new(storage_config)
+  end
+
+  def self.get_watch_type(watch_type_config)
+    classname = watch_type_config['classname'] || 'ConsulWatcher::WatchType::Checks'
+    require classname_to_file(classname)
+    Object.const_get(classname).new(watch_type_config)
+  end
+
+  def self.get_destination(destination_config)
+    classname = destination_config['classname'] || 'ConsulWatcher::Destination::Jq'
+    require classname_to_file(classname)
+    Object.const_get(classname).new(destination_config)
+  end
+
+  # Dynamically require handler class from passed in handler class
+  def self.classname_to_file(classname)
+    classname.gsub('::', '/').gsub(/([a-zA-Z])([A-Z])/, '\1_\2').downcase
+  end
+end

metadata

@@ -0,0 +1,148 @@
+--- !ruby/object:Gem::Specification
+name: consul_watcher
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Ryan Fortman
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2019-04-14 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: bunny
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.7.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.7.0
+- !ruby/object:Gem::Dependency
+  name: hashdiff
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
+- !ruby/object:Gem::Dependency
+  name: slop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.6'
+description: 
+email:
+- r.fortman.dev@gmail.com
+executables:
+- consul_watcher
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Dockerfile
+- Gemfile
+- README.md
+- Rakefile
+- VERSION
+- bin/console_watcher
+- bin/setup
+- consul_watcher.gemspec
+- docker-entrypoint.sh
+- docs/TODO.md
+- docs/destination/amqp.md
+- docs/destination/destination.md
+- docs/destination/jq.md
+- docs/docker-quickstart.md
+- docs/general_design.md
+- docs/messages_overview.md
+- docs/rake_tasks.md
+- docs/ruby-quickstart.md
+- docs/storage/consul.md
+- docs/storage/disk.md
+- docs/storage/storage.md
+- docs/watch_type/key.md
+- docs/watch_type/watch_type.md
+- exe/consul_watcher
+- lib/consul_watcher.rb
+- lib/consul_watcher/class_helper.rb
+- lib/consul_watcher/destination/amqp.rb
+- lib/consul_watcher/destination/jq.rb
+- lib/consul_watcher/diff.rb
+- lib/consul_watcher/storage/consul.rb
+- lib/consul_watcher/storage/disk.rb
+- lib/consul_watcher/watch_type/checks.rb
+- lib/consul_watcher/watch_type/key.rb
+homepage: https://github.com/fortman/consul_watcher
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.8
+signing_key: 
+specification_version: 4
+summary: Send consul watch events to an amqp.
+test_files: []