salthiera 0.3 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 --- 
 SHA1: 
-  metadata.gz: 5461ff49c54a09e4177915e17099fddc9cebadc9
-  data.tar.gz: 4d8b101c4a9253b7b00e783c1c83b386a5bffa63
+  data.tar.gz: 9bd3abfbe7c895386dfe61fd6811503276f7c2f1
+  metadata.gz: 6d925891be35b8f2cbac96e83c8adbf9c5cbeeb6
 SHA512: 
-  metadata.gz: 799aa3067dc9a909e53149325fa3dab93b31b3be9422e3bf31d750bb9d4801cf8f355e93ad3a55f3f324ef1c2a285cc2fb523d0980b1b0305aea41d6acffa722
-  data.tar.gz: c38b356619db6e625920a99deaeedfc5f673cf18934f68d6240f8c23827a9d39d9a562a54884ebd0aef52474bc7b3a695e870b5be12af23a63090f0357a230c1
+  data.tar.gz: e249fd728cd2c32c1acdea9d2c9bd9cbf6ee5ce95f107634c0bdb0d2a6d9de9b107d59922403c8cc2f12479f44a888eacb00a67a68a0a5cdc6ab9d9329185566
+  metadata.gz: 89cfaa09056892f404cccc8474b5d8400a40f13cc6b54e895cf8e430246f3c7c9457fc1b362c6e34c92d22a513a8c8745852c29c47d392eadac3195d0ab034d3

data/README.md

@@ -1,8 +1,6 @@
 SaltHiera
 =========
 
-[![Build Status](https://travis-ci.org/Gtmtechltd/salthiera.png?branch=master)](https://travis-ci.org/gtmtechltd/salthiera)
-
 salthiera is a hiera-like tool which is designed to work with SaltStack Pillars via the ext_pillar option.
 
 Background
@@ -17,25 +15,172 @@ Initially this project is written in Ruby as a gem, however I intend to port to
 Setup
 -----
 
-### Installing salthiera
+#### Installing salthiera
+
+    (on saltmaster only)
 
     $ gem install salthiera
 
-### Define a salthiera.yaml configuration file
 
-    $ chown -R puppet:puppet /etc/puppet/secure/keys
-    $ chmod -R 0500 /etc/puppet/secure/keys
-    $ chmod 0400 /etc/puppet/secure/keys/*.pem
-    $ ls -lha /etc/puppet/secure/keys
-    -r-------- 1 puppet puppet 1.7K Sep 24 16:24 private_key.pkcs7.pem
-    -r-------- 1 puppet puppet 1.1K Sep 24 16:24 public_key.pkcs7.pem
+#### Define a /etc/salt/salthiera.yaml configuration file
+
+    ---
+    hierarchy:
+      - yaml:/srv/salt/environments/%{saltenvironment}/*.yaml.%{id}   # e.g. production/haproxy.yaml.dmzserver01
+      - yaml:/srv/salt/environments/%{saltenvironment}/*.yaml         # e.g. production/haproxy.yaml
+      - yaml:/srv/salt/environments/common/*.yaml                     # e.g. common/haproxy.yaml
+      - files:/srv/salt/environments/%{saltenvironment}/files/**/*
+      - files:/srv/salt/environments/common/files/**/*
+      - eyaml:/srv/salt/environments/%{saltenvironment}/**/*.eyaml
+      - eyaml:/srv/salt/environments/common/**/*.eyaml
+      - efiles:/srv/salt/environments/%{saltenvironment}/efiles/**/*
+      - efiles:/srv/salt/environments/common/efiles/**/*
+
+Items that appear earlier in the hierarchy override items that appear later in the hierarchy. 
+
+In the above hierarchy (defined in salthiera.yaml), the specific environment YAML (parameterised as %{saltenvironment} always overrides the common environment YAML, because it is referenced earlier in the hierarchy section. 
+
+The hierarchy supports YAML files (prefixed with yaml:), EYAML files which are encrypted yaml files (prefixed with eyaml:), and raw files (which might contain public SSL certs, prefixed with files:), and encrypted raw files (which might contain private SSL keys, prefixed with efiles:). 
+
+#### Put some example YAML files together (representing your environment-based data)
+
+/srv/salt/environments/production/example.yaml
+
+    ---
+    my_string: I am the production value
+    
+/srv/salt/environments/common/example.yaml
+
+    ---
+    my_string: I am the common value
+    default_string: I am the common default value
+
+#### Use salthiera on the saltmaster to lookup data on the commandline!
+
+(saltmaster)
+
+    $ salthiera -c /etc/salt/salthiera.yaml key1=value1 key2=value
+
+e.g.
+
+    $ salthiera -c /etc/salt/salthiera.yaml saltenvironment=production id=dmzserver01
+
+(the keys and values you supply relate directly to the %{} delimeters in your salthiera.yaml config file (above)
+
+In reality when salt runs, it will run the salthiera command, and supply a whole bunch of grain data as key-value pairs to the salthiera command. You get things like %{id} for free - (which is the hostname of the minion), but for the saltenvironment, we need to add this grain data to the salt minion itself.
+
+#### Configure salt to do this automatically as part of its external pillar processes:
+
+/etc/salt/master configuration file for saltmaster
+
+    ...
+    ext_pillar:
+      - salthiera: /etc/salt/salthiera.yaml
+    ...
+
+    #pillar_roots:  (comment this out unless you want to use pillar in conjunction with salthiera)
+
+#### Copy the salthiera.py file in this repos salt/pillar directory into salts pymodules (will depend on your python version and install location)
+
+    $ cp salt/pillar/salthiera.py /usr/share/pyshared/salt/pillar/salthiera.py
+    $ ln -s /usr/share/pyshared/salt/pillar/salthiera.py /usr/lib/pymodules/python2.7/salt/pillar/salthiera.py
+
+#### Configure a salt minion with its environment
+
+    # /etc/salt/minion config file
+
+    ...
+    grains:
+      saltenvironment: production    # different on different environments
+    ...
+
+#### Test on the minion
+
+    $ salt-call pillar.get my_string
+    > I am the production value
+
+    $ salt-call pillar.get default_string
+    > I am the common default value
+
+In this way you can manage key-value pairs, using YAML, or entire files which makes it easier to manage keys, binary files etc.
+
+Encryption
+==========
+
+Salthiera support encrypting using the PKCS7 encryption format. Salthiera has a "sister" project for puppet called hiera-eyaml, which we will use on our local machine to manage the encryption. Hiera-eyaml does not need to be installed anywhere else.
+
+#### Generate keys for encrypting stuff LOCALLY (if you dont want to do this, then bypass this section)
+
+    (on your local desktop)
+
+    $ gem install hiera-eyaml
+    $ cd /tmp
+    $ eyaml createkeys         # keys will be created in a keys/ subdirectory
+    $ ls keys/
+      private_key.pkcs7.pem  public_key.pkcs7.pem
+
+#### Encrypt a sensitive password like this
+
+    $ eyaml encrypt --pkcs7-public-key keys/public_key.pkcs7.pem -s "SOME STRING TO ENCRYPT"
+      ENC[PKCS7,..........]
+   
+ Copy this ENC string into any .eyaml file in your salt git repo (environments data)
+
+  /srv/salt/environments/production/database.eyaml
+
+    ---
+    database_password: ENC[PKCS7,............]
+
+#### EYAML files
+
+Eyaml files in your data hierarchy are exactly the same as YAML files, they just have the ENC[PKCS7,.....] blocks in place of values. Note that you can mix and match ENC[] blocks with plaintext values in the same .eyaml file, that's up to you. But salthiera will not decrypt ENC[] blocks, unless there is a matching eyaml: line in the hierarchy section of salthiera.yaml config file
+
+#### EFILES
+
+Salthiera also supports encrypted entire files, which is useful for sensitive binary files (e.g. .DER private keys), or even "plaintext" .PEM private keys that need encrypting.
+
+To encrypt an entire file:
+
+    $ eyaml encrypt --pkcs7-public-key keys/public_key.pkcs7.pem -f file -o string > encryptedfile
+
+In the above salthiera.yaml config example, you would then copy encryptedfile into the "efiles" section of your data hierarchy
+
+#### Encryption Notes 
+
+You would typically generate different keys for different environments and use the correct environments keys to generate encrypted data for that environment. 
+
+Hiera-eyaml toolset allows you to do a lot of things, for example you can interactively edit an .eyaml file using vi, replacing the encrypted tokens in realtime. For further information see the hiera-eyaml project to see all the possibilities available to you
+
+When done, copy the public_key.pkcs7.pem into your salt git repo somewhere nice (you can publish this to your dev team)
+
+#### Set up the salt-master to decrypt the values
+ 
+Copy the public and private keys you generated above into your saltmaster in a secure place:
+
+(on your saltmaster)
+
+    $ mkdir -p /etc/salt/salthiera/keys
+    $ chown -R root:root /etc/salt/salthiera/keys
+    $ chmod -R 0500 /etc/salt/salthiera/keys
+
+Edit /etc/salt/salthiera.yaml and add:
+
+    eyaml_public_key: /etc/salt/salthiera/keys/public_key.pkcs7.pem
+    eyaml_private_key: /etc/salt/salthiera/keys/private_key.pkcs7.pem
 
-### Use it to lookup data!
+Ensure keys directory exists
+       
+    $ chmod 0400 /etc/salt/salthiera/keys/*.pem
+    $ ls -lha /etc/salt/salthiera/keys
+    -r-------- 1 root   root   1.7K Sep 24 16:24 private_key.pkcs7.pem
+    -r-------- 1 root   root   1.1K Sep 24 16:24 public_key.pkcs7.pem
 
-    $ salthiera -c /etc/salthiera.yaml key1=value1 key2=value
+You need a public key and a private key to decrypt a value, but only a public to encrypt a value. In this way, you can distribute your public key to your dev team so they are able to encrypt values and commit them to your salt git repo, and only the saltmaster itself can decrypt the value. This allows you to commit production data alongside your vagrant/dev/staging data in the same repo in safe way.
 
-SaltHiera supports keys, values
+TROUBLESHOOTING
+---------------
 
+If for some reason salt isnt returning ANY values (either plaintext or encyrpted), but salthiera commandline lookups executed properly as explained above is, then the chances are that salt cannot find the "salthiera" binary that comes with this gem. If so, alter the salt.utils.which value in /usr/share/pyshared/salt/pillar/salthiera.py that refers to the salthiera binary and supply the fully qualified path to the salthiera binary (and likewise lower in the code too). This will fix this problem.
 
 Authors
 -------

data/lib/salt_hiera/plugins/efiles.rb

@@ -0,0 +1,59 @@
+module SaltHiera
+  module Plugins
+    class Efiles
+
+      def self.process_file file
+
+        key = file.split("/").last
+        value = File.read file
+        returnhash = { key => value }
+        returnhash = self.recurse returnhash
+        returnhash
+
+      end
+
+      def self.recurse obj
+        if obj.is_a? Array
+          obj.each.with_index do |element, index|
+            obj[index] = self.recurse element
+          end
+        elsif obj.is_a? Hash
+          obj.each do |k, v|
+            obj[k] = self.recurse v
+          end
+        elsif obj.is_a? String
+          obj = obj.gsub(/ENC\[PKCS7,(.*?)\]/) {|x| self.decrypt($1) }
+        else
+          obj
+        end
+      end
+
+      def self.decrypt cipherbinary
+
+        public_key = Configuration.get "eyaml_public_key"
+        private_key = Configuration.get "eyaml_private_key"
+
+        raise StandardError, "pkcs7_public_key is not defined" unless public_key
+        raise StandardError, "pkcs7_private_key is not defined" unless private_key
+
+        private_key_pem = File.read private_key
+
+        private_key_rsa = OpenSSL::PKey::RSA.new( private_key_pem )
+
+        public_key_pem = File.read public_key
+
+        public_key_x509 = OpenSSL::X509::Certificate.new( public_key_pem )
+
+        ciphertext = Base64.decode64(cipherbinary) 
+        pkcs7 = OpenSSL::PKCS7.new( ciphertext )
+
+        pkcs7.decrypt(private_key_rsa, public_key_x509)
+
+      end
+
+
+      end
+
+    end
+  end
+end

data/lib/salt_hiera/plugins/eyaml.rb

@@ -26,8 +26,7 @@ module SaltHiera
             obj[k] = self.recurse v
           end
         elsif obj.is_a? String
-          obj = obj.gsub(/ENC\[PKCS7,(.*?)\]/) {|x| puts "*******" + $1
-            self.decrypt($1) }
+          obj = obj.gsub(/ENC\[PKCS7,(.*?)\]/) {|x| self.decrypt($1) }
         else
           obj
         end

data/lib/salt_hiera/salt_hiera.rb

@@ -6,7 +6,7 @@ require 'salt_hiera/plugins/no_plugin'
 module SaltHiera
 class SaltHiera
 
-  VERSION="0.3"
+  VERSION="0.3.1"
 
   def initialize attributes
     raise "Config error (#{params})" unless attributes[:config_file]

data/salt/pillar/salthiera.py

@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+'''
+Take in a salthiera configuration file location and execute it.
+Adds the salthiera data to pillar
+'''
+
+# Import python libs
+import logging
+
+# Import salt libs
+import salt.utils
+from salt._compat import string_types
+
+# Import third party libs
+import yaml
+
+
+# Set up logging
+log = logging.getLogger(__name__)
+
+
+def __virtual__():
+    '''
+    Only return if salthiera is installed
+    '''
+    return 'salthiera' if salt.utils.which('salthiera') else False
+
+
+def ext_pillar(minion_id, pillar, conf):
+    '''
+    Execute salthiera and return the data
+    '''
+    cmd = 'salthiera -c {0}'.format(conf)
+    for key, val in __grains__.items():
+        if isinstance(val, string_types):
+            cmd += ' {0}=\'{1}\''.format(key, val)
+    try:
+        data = yaml.safe_load(__salt__['cmd.run'](cmd))
+    except Exception:
+        log.critical(
+                'SaltHiera YAML data failed to parse from conf {0}'.format(conf)
+                )
+        return {}
+    return data

data/tools/regem.sh

@@ -5,4 +5,4 @@ RAKE_OUT=`rake build`
 echo ${RAKE_OUT}
 VERSION=`echo ${RAKE_OUT} | awk '{print $2}'`
 echo Installing version: ${VERSION} ...
-gem install pkg/salthiera-${VERSION}.gem
+gem install pkg/salthiera-${VERSION}.gem --no-rdoc --no-ri

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: salthiera
 version: !ruby/object:Gem::Version 
-  version: "0.3"
+  version: 0.3.1
 platform: ruby
 authors: 
 - Geoff Meakin
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2014-04-28 00:00:00 Z
+date: 2014-04-30 00:00:00 Z
 dependencies: []
 
 description: Hiera like tool for use as an external pillar with SaltStack
@@ -36,11 +36,13 @@ files:
 - lib/salt_hiera/CLI.rb
 - lib/salt_hiera/configuration.rb
 - lib/salt_hiera/logger.rb
+- lib/salt_hiera/plugins/efiles.rb
 - lib/salt_hiera/plugins/eyaml.rb
 - lib/salt_hiera/plugins/files.rb
 - lib/salt_hiera/plugins/no_plugin.rb
 - lib/salt_hiera/plugins/yaml.rb
 - lib/salt_hiera/salt_hiera.rb
+- salt/pillar/salthiera.py
 - salthiera.gemspec
 - tools/regem.sh
 homepage: http://github.com/gtmtechltd/salthiera