consul-templaterb 1.0.8 → 1.0.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 49568fc4a6f043e390484820ea383914a31d35c99149669a4b77772bfeed0504
-  data.tar.gz: f56e6cd3882cb1154219c3c56da2ae9b484e9c6196c685980c296e713c6f9650
+  metadata.gz: 0d408d6a635426874369831aa3c7f12890ebec2c7b5bb7d571b76bdbef9da041
+  data.tar.gz: 93af5f9b70790a9c9665bea3d13ae3d6322f7a3b5ff703d132a890cdc37ed6f6
 SHA512:
-  metadata.gz: 7e241be76b837635ef83dff4ec76191bba016cdb7e040b4c17325458465d6b10ae5a8430527484be30a2b0bf05d0a86350e970742b570766d14578a621f817fb
-  data.tar.gz: 9f4ccd3d0222a28abf6bcb89b46e101c627d736c73b6451a39f9bb4507e68713b1a88abfaf66d4e5a152de09d84b55ac486af2f2a692967a26762bb62129fd84
+  metadata.gz: 2237575069b92c50374adc5f91587f1737f0cf42a634acb9b469b6cc076175dbdb48e8d944df8de91ccf8b5ebaf089dca4cdb088d4cb46fe07fcb2dbebbabf54
+  data.tar.gz: 7ad73644600bc5130306c2a9190e092ae59ca08a529e75097b26da91907353a2a50d17fb249a040f8758856917463f035372fb5899728c8ae7f03630b1ee6895

data/CHANGELOG.md

@@ -1,5 +1,19 @@
+# CHANGELOG
+
 ## (UNRELEASED)
 
+IMPROVEMENTS:
+
+## 1.0.8 (March 18, 2018)
+
+IMPROVEMENTS:
+
+* Added [samples/sample_keys.html.erb](samples/sample_keys.html.erb)
+* When `get_value_json` is called, allow to catch silently errors
+* Nicer error messages when templates are invalid with exact line of error
+* Added optional params_file to `--template` flag to create parameters for templates
+* Documentation improvements
+
 ## 1.0.8 (March 18, 2018)
 
 IMPROVEMENTS:
@@ -10,7 +24,7 @@ IMPROVEMENTS:
 * Added optional parameters to sub-template in `render_file`
 * Added `param(name, default_value)` to retrieve parameter from sub-template
 * Use `CONSUL_HTTP_TOKEN` if present in environment variables to get the token
-* Added [TemplateAPI.md](TemplateAPI.md] for documenting functions
+* Added [TemplateAPI.md](TemplateAPI.md) for documenting functions
 
 ## 1.0.7 (March 16, 2018)
 
@@ -37,7 +51,7 @@ IMPROVEMENTS:
 
 BUG FIXES:
 
-* fixed default value for parameter -c to http://localhost:8500
+* fixed default value for parameter -c to `http://localhost:8500`
 
 ## 1.0.4 (March 12, 2018)
 

data/README.md

@@ -1,4 +1,8 @@
-# consul-templaterb [![Build Status](https://api.travis-ci.org/criteo/consul-templaterb.svg?branch=master)](https://travis-ci.org/criteo/consul-templaterb)
+# consul-templaterb
+
+[![Build Status](https://api.travis-ci.org/criteo/consul-templaterb.svg?branch=master)](https://travis-ci.org/criteo/consul-templaterb)
+[![Gem Version](https://badge.fury.io/rb/consul-templaterb.svg)](http://github.com/criteo/consul-templaterb/releases)
+[![License](https://img.shields.io/badge/license-ApacheV2-yellowgreen.svg)](#license)
 
 The ruby GEM [consul-templaterb](https://rubygems.org/gems/consul-templaterb)
 is both a library and an executable that allows to generate files
@@ -58,7 +62,7 @@ Compared to consul-template, consul-templaterb offers the following features:
 The executable supports semantics and command line flags and options similar to
 HashiCorp's Consul-template, so many flags you might use in consul-template will
 work in a similar way. It also supports the same environment variable
-`CONSUL_HTTP_ADDR` to find the Consul Agent to query and 'CONSUL_HTTP_TOKEN' to
+`CONSUL_HTTP_ADDR` to find the Consul Agent to query and `CONSUL_HTTP_TOKEN` to
 get the token.
 
 ## Installation
@@ -128,7 +132,7 @@ templates or write your owns, it is very easy!
 
 ```shell
 $ consul-templaterb --help
-USAGE: bin/consul-templaterb [[options]]
+USAGE: consul-templaterb [[options]]
     -h, --help                       Show help
     -v, --version                    Show Version
     -c, --consul-addr=<address>      Address of Consul, eg: http://localhost:8500
@@ -136,12 +140,12 @@ USAGE: bin/consul-templaterb [[options]]
     -w, --wait=<min_duration>        Wait at least n seconds before each template generation
     -r, --retry-delay=<min_duration> Min Retry delay on Error/Missing Consul Index
     -k, --hot-reload=<behavior>      Control hot reload behaviour, one of :[die (kill daemon on hot reload failure), keep (on error, keep running), disable (hot reload disabled)]
-    -K, --sig-term=kill_signal       Signal to sent to next --exec command on kill, default=#{cur_sig_term}
-    -R, --sig-reload=reload_signal   Signal to sent to next --exec command on reload (NONE supported), default=#{cur_sig_reload}
+    -K, --sig-term=kill_signal       Signal to sent to next --exec command on kill, default=TERM
+    -R, --sig-reload=reload_signal   Signal to sent to next --exec command on reload (NONE supported), default=HUP
     -e, --exec=<command>             Execute the following command
     -d, --debug-network-usage        Debug the network usage
-    -t erb_file:[output]:[command],  Add a erb template, its output and optional reload command
-        --template
+    -t erb_file:[output]:[command]:[params_file],
+        --template                   Add a erb template, its output and optional reload command
         --once                       Do not run the process as a daemon
 ```
 
@@ -156,12 +160,15 @@ if the `--template my_file.ext.erb:myfile.ext` was used.
 ### Generate multiple templates
 
 In the same way as consul-template, consul-templaterb supports multiple templates and executing
-commands when the files do change. The parameter `--template <ERB>:<DEST>:[reload_command]` works
+commands when the files do change. The parameter `--template <ERB>:<DEST>:[reload_command]:params_file` works
 in the following way:
 
 * ERB : the ERB file to use as a template
 * DEST: the destination file
 * reload_command: optional shell command executed whenever the file has been modified
+* params_file: JSON or YAML file to load and to use as parameter for template (see
+  [param() function](TemplateAPI.md#paramparameter_name-default_value-nil) to retrieve
+  the values)
 
 The argument can be specified multiple times, ex:
 

data/TemplateAPI.md

@@ -1,9 +1,10 @@
-# Template API for writting templates
+# Template API for writing templates
 
 Here are the various functions you might use in your templates.
 
-For each function, mandatory arguments are specified at the beginning while optional ones are marked with `[]`.
-Most of them support the optional dc attribute to access data from another datacenter. If the `dc`
+For each function, documentation specifies mandatory arguments at the beginning while optional ones are
+annoted with `[]`.
+Most of them support the optional dc attribute to use data from another datacenter. If the `dc`
 attribute is not specified, the function will output data from the current datacenter.
 
 To ease template development, `consul-templaterb` supports HOT reload of templates, thus it is possible to
@@ -43,11 +44,78 @@ name or its ID. If DC is specified, will lookup for given node in another datace
 
 [Find all the checks](https://www.consul.io/api/health.html#list-checks-for-service) of a given service.
 
-## kv(name = nil, dc: nil, keys: nil, recurse: false)
+## kv(name, [dc: nil], [keys: false], [recurse: false])
 
 [Read keys from KV Store](https://www.consul.io/api/kv.html#read-key). It can be used for both listing the keys and
 getting the values. See the file in samples [keys.html.erb](samples/keys.html.erb) for a working example.
 
+Variants:
+
+* no additional parameter: will only retrieve the key you asked for
+* `keys: true` : will retrieve the hierarchy of keys, but without the values, useful if values might be large, in
+  order to perform simple listings
+* `recurse: true`: will retrieve all hierarchy of keys with their values
+
+Please have a look at [samples/sample_keys.html.erb](samples/sample_keys.html.erb) for examples on how using it.
+
+### Using the result of kv
+
+Since KV has several modes, it depends whether you asked for one or several keys.
+
+Thus, we recommend using a helper to get the value (while you might use it directly).
+
+In order to ease the use, 3 helpers are available and all have a optional argument `path`. When path is specified
+and the call is retrieving several keys, it allows to select a specific one.The available helpers are the following:
+
+* `get_value( [path] )` : Get a raw value
+* `get_value_decoded( [path] )` : Get the value decoded from Base64
+* `get_value_json( [path], [catch_errors: true] )` : when your payload is JSON,
+  decode Base64 first and then decode the JSON. If catch_errors is set to true,
+  if will not throw an error during rendering of template and return nil. Otherwise
+  an Error will be thrown and have to be catch if you are unsure if the value is valid
+  JSON
+
+#### Get the result of a single value
+
+The easiest, use the helpers to retrieve the values in the following formats:
+
+* `kv('/my/path/to/value').get_value` : get the raw value of a single key in KV
+* `kv('/my/path/to/value').get_decoded` : get the decoded value of a single key in KV
+* `kv('/my/path/to/value').get_value_json` : get the base64 decoded value and try decoding it as JSON
+
+#### Iterate over values
+
+If you want to iterate amongst all values, you might to it that way:
+
+```erb
+<%
+kv('/my/multiple/values', recurse: true).each do |tuple|
+  key = tuple['Key']
+  value_decoded = Base64.decode64(tuple['Value'])
+%>
+  <div>Decoded value: <%= value_decoded %>
+  <div>JSON value: <%= JSON.parse( value_decoded ) %>
+<%
+end
+%>
+```
+
+#### Fetch all values at once, but interrested only by a few
+
+When using `kv('/my/multiple/values', recurse: true)`, only a single call is performed, thus,
+it is far more efficient to retrive multiple values from the KV under the same root. Thus,
+in order to display several discreet values, it is possible to do the following:
+
+```erb
+result = kv('/my/multiple/values', recurse: true)
+value1 : <%= result.get_decoded('/my/multiple/values/value1') %>
+value42 : <%= result.get_decoded('/my/multiple/values/value42') %>
+value123 : <%= result.get_decoded('/my/multiple/values/value123') %>
+```
+
+Since `kv('/my/multiple/values', recurse: true)` will retrieve all values at once, it might be more
+efficient in some cases than retrieving all values one by one.
+
 ## agent_metrics()
 
 [Get the metrics of Consul Agent](https://www.consul.io/api/agent.html#view-metrics). Since this endpoint does
@@ -81,4 +149,27 @@ the template using `param('title', 'My default Value')` in the `header.html.erb`
 Can be used within a template to access a parameter. Parameters can be specified with `render_file`
 directive. Optional value `default_value` allow to get a value if parameter has not been set.
 
-See [samples/common/header.html.erb](samples/common/header.html.erb) for example of usage.
+It allows to create re-usable sub-templates that might be used in several places with several types
+of parameters. Thus, templates can be called like kind of functions and generate output based on
+the parameters provided.
+
+Example:
+
+```erb
+render_file('show_service.html.erb', {service_name: 'service1',  title: 'My Nice Service'})
+[...]
+render_file('show_service.html.erb', {service_name: 'service2',  title: 'My Nicer Service'})
+```
+
+Note that you can also add parameters into a top level service using the command line:
+
+```sh
+consul-templaterb --template "source.html.erb:dest.html:reload_command:params.yaml"
+[...]
+```
+
+In that case, it would load the content of params.yaml and inject it as params when evaluating
+template `source.html.erb`. Injection of params using 4th parameter of `--template` supports
+YAML as well as JSON format. Those parameter files are NOT automatically reloaded however.
+
+See [samples/common/header.html.erb](samples/common/header.html.erb) for example of usage.

data/bin/consul-templaterb

@@ -154,7 +154,7 @@ optparse = OptionParser.new do |opts|
     end
   end
 
-  opts.on('-t', '--template erb_file:[output]:[command]', String, 'Add a erb template, its output and optional reload command') do |tpl|
+  opts.on('-t', '--template erb_file:[output]:[command]:[params_file]', String, 'Add a erb template, its output and optional reload command') do |tpl|
     splitted = tpl.split(':')
     source = splitted[0]
     dest = splitted[1]
@@ -164,7 +164,10 @@ optparse = OptionParser.new do |opts|
     end
     raise "Source and destination cannot be the same in #{tpl}" if source == dest || dest.empty?
     command = splitted[2]
-    consul_engine.add_template(source, dest)
+    parameter_file = splitted[3]
+    params = {}
+    params = Consul::Async::Utilities.load_parameters_from_file(parameter_file) if parameter_file
+    consul_engine.add_template(source, dest, params)
 
     if command
       consul_engine.add_template_callback do |_all_ready, _template_manager, results|

data/consul-templaterb.gemspec

@@ -26,7 +26,7 @@ Gem::Specification.new do |spec|
   end
   spec.bindir        = 'bin'
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
-  spec.extra_rdoc_files = ['README.md', 'TemplateAPI.md']
+  spec.extra_rdoc_files = ['README.md', 'CHANGELOG.md', 'TemplateAPI.md']
   spec.require_paths = ['lib']
 
   spec.require_paths = ['lib']

data/lib/consul/async/consul_template.rb

@@ -12,6 +12,13 @@ module Consul
       end
     end
 
+    class SyntaxErrorInTemplate < InvalidTemplateException
+      attr_reader :cause
+      def initialize(cause)
+        @cause = cause
+      end
+    end
+
     class ConsulEndPointsManager
       attr_reader :conf, :net_info, :start_time
       def initialize(consul_configuration)
@@ -126,6 +133,14 @@ module Consul
         render(File.read(new_path), new_path, params)
       end
 
+      def find_line(e)
+        return e.message.dup[5..-1] if e.message.start_with? '(erb):'
+        e.backtrace.each do |line|
+          return line[5..-1] if line.start_with? '(erb):'
+        end
+        nil
+      end
+
       def render(tpl, tpl_file_path, params = {})
         # Ugly, but allow to use render_file well to support stack of calls
         old_value = @context
@@ -138,11 +153,14 @@ module Consul
         result
       rescue StandardError => e
         e2 = InvalidTemplateException.new e
-        raise e2, "Template contains errors: #{e.message}", e.backtrace
+        raise e2, "[TEMPLATE EVALUATION ERROR] #{tpl_file_path}#{find_line(e)}: #{e.message}"
+      rescue SyntaxError => e
+        e2 = SyntaxErrorInTemplate.new e
+        raise e2, "[TEMPLATE SYNTAX ERROR] #{tpl_file_path}#{find_line(e)}: #{e.message}"
       end
 
-      def write(file, tpl, last_result, tpl_file_path)
-        data = render(tpl, tpl_file_path)
+      def write(file, tpl, last_result, tpl_file_path, params = {})
+        data = render(tpl, tpl_file_path, params)
         not_ready = []
         ready = 0
         @iteration = Time.now.utc - @start_time
@@ -329,10 +347,15 @@ module Consul
       end
 
       # Helper to get the value decoded as JSON
-      def get_value_json(name = root)
+      def get_value_json(name = root, catch_errors: true)
         x = get_value_decoded(name)
         return nil unless x
-        JSON.parse(x)
+        begin
+          JSON.parse(x)
+        rescue JSON::ParserError => e
+          return nil if catch_errors
+          raise StandardError.new(e), "get_value_json() cannot deserialize kv(#{name}) as JSON: #{e.message}", e.backtrace
+        end
       end
     end
   end

data/lib/consul/async/consul_template_engine.rb

@@ -21,17 +21,18 @@ module Consul
         @template_callbacks << block
       end
 
-      def add_template(source, dest)
-        @templates.push([source, dest])
+      def add_template(source, dest, params = {})
+        @templates.push([source, dest, params])
       end
 
       def run(template_manager)
         @template_manager = template_manager
         EventMachine.run do
           template_renders = []
-          @templates.each do |template_file, output_file|
+          @templates.each do |template_file, output_file, params|
             template_renders << Consul::Async::ConsulTemplateRender.new(template_manager, template_file, output_file,
-                                                                        hot_reload_failure: hot_reload_failure)
+                                                                        hot_reload_failure: hot_reload_failure,
+                                                                        params: params)
           end
           EventMachine.add_periodic_timer(1) do
             begin
@@ -49,6 +50,10 @@ module Consul
                 STDERR.puts "[ERROR] callback error: #{cbk_error.inspect}"
                 raise cbk_error
               end
+            rescue Consul::Async::InvalidTemplateException => e
+              STDERR.puts "[FATAL]#{e}"
+              template_manager.terminate
+              EventMachine.stop
             rescue StandardError => e
               STDERR.puts "[ERROR] Fatal error occured: #{e.inspect} - #{e.backtrace}"
               template_manager.terminate

data/lib/consul/async/consul_template_render.rb

@@ -20,8 +20,8 @@ module Consul
       end
     end
     class ConsulTemplateRender
-      attr_reader :template_file, :output_file, :template_file_ctime, :hot_reload_failure
-      def initialize(template_manager, template_file, output_file, hot_reload_failure: 'die')
+      attr_reader :template_file, :output_file, :template_file_ctime, :hot_reload_failure, :params
+      def initialize(template_manager, template_file, output_file, hot_reload_failure: 'die', params: {})
         @hot_reload_failure = hot_reload_failure
         @template_file = template_file
         @output_file = output_file
@@ -29,10 +29,11 @@ module Consul
         @last_result = ''
         @last_result = File.read(output_file) if File.exist? output_file
         @template = load_template
+        @params = params
       end
 
       def render(tpl = @template)
-        @template_manager.render(tpl, template_file)
+        @template_manager.render(tpl, template_file, params)
       end
 
       def run
@@ -58,7 +59,7 @@ module Consul
       end
 
       def write
-        success, modified, last_res = @template_manager.write(@output_file, @template, @last_result, template_file)
+        success, modified, last_res = @template_manager.write(@output_file, @template, @last_result, template_file, params)
         @last_result = last_res if last_res
         [success, modified, @last_result]
       end
@@ -70,7 +71,7 @@ module Consul
             @template_file_ctime = new_time
             return update_template(load_template)
           rescue Consul::Async::InvalidTemplateException => e
-            STDERR.puts "****\n[ERROR] HOT Reload of template #{template_file} did fail due to #{e}\n****\n"
+            STDERR.puts "****\n[ERROR] HOT Reload of template #{template_file} did fail due to:\n #{e}\n****\n"
             raise e unless hot_reload_failure == 'keep'
             STDERR.puts "[WARN] Hot reload of #{template_file} was not taken into account, keep running with previous version"
           end

data/lib/consul/async/utilities.rb

@@ -1,3 +1,6 @@
+require 'tempfile'
+require 'yaml'
+require 'json'
 module Consul
   module Async
     class Utilities
@@ -12,6 +15,18 @@ module Consul
           "#{(bytes / 1_073_741_824.0).round(2)} Gb"
         end
       end
+
+      # Loads parameters from a file, supports JSON and YAML
+      def self.load_parameters_from_file(parameters_file)
+        raise "Parameters file #{parameters_file} does not exists" unless File.exist? parameters_file
+        if parameters_file.downcase.end_with?('.yaml', '.yml')
+          YAML.load_file(parameters_file)
+        elsif parameters_file.downcase.end_with?('.json')
+          JSON.parse(File.read(parameters_file))
+        else
+          raise "Don't know how to load parameters file #{parameters_file}: JSON and YAML supported"
+        end
+      end
     end
   end
 end

data/lib/consul/async/version.rb

@@ -1,5 +1,5 @@
 module Consul
   module Async
-    VERSION = '1.0.8'.freeze
+    VERSION = '1.0.9'.freeze
   end
 end

data/samples/sample_keys.html.erb

@@ -0,0 +1,96 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8"/>
+    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"/>
+    <meta name="description" content="Display Consul information"/>
+    <meta name="author" content="Criteo"/>
+    <meta http-equiv="refresh" content="<%= param('refresh', ENV['REFRESH'] || '10') %>"/>
+    <title>This page displays random K/V every 10 seconds %></title>
+    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
+    <style>
+      body {
+        padding-top: 5rem;
+      }
+      .check {
+        transition: opacity 1s ease-out;
+      }
+    </style>
+    <style id="css-states">
+    </style>
+  </head>
+  <body>
+<main class="container">
+<h1>Keys example</h1>
+<div>
+This example shows how to use kv() results. It requires at least one key if the Consul's KV in order to display something.
+</div>
+<%
+  result = kv('', keys:true)
+  num_keys = result.count
+  # We refresh key every 10 seconds only
+  key_index = (Time.now.to_i / 10)
+  prng = Random.new
+  if num_keys > 0
+    key_to_display = result[key_index % num_keys]
+  end
+%>
+<div>
+<h2>Using a single key</h2>
+<div>Found <%= num_keys %> keys in KV/Store</div>
+<%
+if key_to_display
+%>
+  <h1>Display a random single key using helpers</h1>
+  <div>
+    <h3>kv('<%= key_to_display %>').get_value :</h3>
+    <pre><%= kv(key_to_display).get_value %></pre>
+  </div>
+  <div>
+    <h3>kv('<%= key_to_display %>').get_value_decoded</h3>
+    <pre><%= 
+  ERB::Util.html_escape(kv(key_to_display).get_value_decoded)
+  %></pre>
+  </div>
+  <div>
+    <h3>kv('<%= key_to_display %>').get_value_json(catch_errors: true)</h3>
+    <code><% 
+  # catch_errors: true will return nil if data cannot be converted to JSON
+  json = kv(key_to_display).get_value_json(catch_errors: true)
+  %><%= (json ? jERB::Util.html_escape(json) : 'No Valid JSON Data') %></code>
+  </div>
+<%
+end
+%>
+</div>
+<div>
+<h2>Listing all keys (without their values)</h2>
+<ul>
+<%
+result.each do |key_name|
+%>
+  <li><%= key_name %></li>
+<%
+end
+%>
+</ul>
+</div>
+<div>
+<h2>Listing 10 first keys (with their values)</h2>
+<ol>
+<% kv_values = kv('', recurse: true)
+   index = 0
+   kv_values.each do |tuple|
+    index += 1
+    break if index > 10
+    key_name = tuple['Key']
+%>
+    <li><%= key_name %>: <code><%= kv_values.get_value_decoded(key_name) %></code></li>
+<%
+end
+%>
+</ol>
+</div>
+</main>
+</body>
+</html>

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: consul-templaterb
 version: !ruby/object:Gem::Version
-  version: 1.0.8
+  version: 1.0.9
 platform: ruby
 authors:
 - SRE Core Services
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-03-18 00:00:00.000000000 Z
+date: 2018-03-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: em-http-request
@@ -131,6 +131,7 @@ executables:
 extensions: []
 extra_rdoc_files:
 - README.md
+- CHANGELOG.md
 - TemplateAPI.md
 files:
 - ".gitignore"
@@ -166,6 +167,7 @@ files:
 - samples/ha_proxy.cfg.erb
 - samples/keys.html.erb
 - samples/nodes.html.erb
+- samples/sample_keys.html.erb
 - samples/services.html.erb
 homepage: https://rubygems.org/gems/consul-templaterb
 licenses: