hiera-mysql-json-backend 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 1c4ac218cf64261c1fd9fc92bc3a2c134f69043a
+  data.tar.gz: 59d4fb9ab2446375b3cc3299a399344b643cc716
+SHA512:
+  metadata.gz: d726e49677179bcdac9dacb35cecdddad977df4900071abf32c7ed9478f868c0087def3a5a7f36242fe02c0e8f680df681cff34ca16417c3f477b74e8a632571
+  data.tar.gz: 16229d5f6d694e2d1139946e91b0ae711b3a8331ca18fd31a1556109bc11ad642d649a26840f7b598e6ff47afee5e3d0ac83fb93397f3d2abfdd3c1b54d0d115

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+coverage
+InstalledFiles
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+
+# YARD artifacts
+.yardoc
+_yardoc
+doc/

data/CHANGELOG.md

@@ -0,0 +1,17 @@
+### 0.0.1
+- Initial Release
+
+### 0.0.2 - 0.0.3
+- Minor Updates
+
+### 0.0.4
+- Connection strings can now be defined per hierarchy file instead of being global
+
+### 0.0.5
+- Added port as one of the settings that can be defined, thanks [Dave Seff](https://github.com/daveseff)
+
+### 0.0.6
+
+- Made improvements to the connection Hash
+- Fixed an issue where if no port was defined nill would be passed
+- Added defaults for hostname and port, hostname will default to `localhost` port to `3306`

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in hiera-mysql-backend.gemspec
+gemspec

data/Gemfile.lock

@@ -0,0 +1,22 @@
+PATH
+  remote: .
+  specs:
+    hiera-mysql-backend (0.0.4)
+      hiera
+      mysql2
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    hiera (1.3.1)
+      json_pure
+    json_pure (1.8.1)
+    mysql2 (0.3.14)
+    rake (10.1.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  hiera-mysql-backend!
+  rake

data/LICENSE

@@ -0,0 +1,20 @@
+The MIT License (MIT)
+
+Copyright (c) 2013 Roberto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,98 @@
+[![Gem Version](https://badge.fury.io/rb/hiera-mysql-json-backend.png)](http://badge.fury.io/rb/hiera-mysql-json-backend)
+
+## hiera-mysql-backend
+
+Alternate MySQL backend for Hiera
+
+This is a MySQL backend for Hiera inspired by [hiera-mysql](https://github.com/crayfishx/hiera-mysql). Unfortunately no work has been done to that backend for the past 9 months and it was missing a couple of features I needed so I decided to pick up the torch and implement them myself.
+
+### What is different from hiera-mysql
+
+In [hiera-mysql](https://github.com/crayfishx/hiera-mysql) you define the queries in the hiera.yaml file. I felt this was too restricting so instead hiera-mysql-backend uses, poorly named, sql files. This sql files follow the Hiera hierarchy.
+
+[hiera-mysql](https://github.com/crayfishx/hiera-mysql) would also return the last matching query not the first one which I felt it was confusing.
+
+[hiera-mysql](https://github.com/crayfishx/hiera-mysql) used the mysql gem, I am partial to [mysql2](https://github.com/brianmario/mysql2)
+
+Exception handling. hiera-mysql would cause a puppet run to fail if one of the queries was incorrect. For example a fact that you are distributing with a module is needed for the query to return its data but that fact is not available outside the module having a `SELECT * from %{custom_fact}` would make puppet runs fail.
+
+### What goes into the sql files.
+
+The poorly named sql files are really yaml files where the key is the lookup key and the value is the SQL statement (it accepts interpolation)
+
+As of version 0.0.4 you can also add connection information to these sql files, this allows you to connect to different databases. This is optional if no connection information is found it will use the default defined in your hiera.yaml config.
+
+Lets assume your _datadir_ is `/etc/puppet/hieradata/` and your hierarchy for hiera just have a common. hiera-mysql-backend would look for /etc/puppet/hieradata/common.sql the common.sql would look like:
+
+```yaml
+---
+# This is optional, if not present it will use the default connection info from hiera.yaml
+:dbconfig:
+  :host: database.example.com
+  :user: hieratest
+  :pass: sekret
+  :database: testhieradb
+  :port: 44445
+
+applications: SELECT * FROM applications WHERE host='%{fqdn}';
+coats: SELECT cut,name,type FROM coats WHERE color='brown';
+```
+
+If `host` is not defined it will use `localhost` as default.
+
+If `port` is not defined it will use the default `3306` mysql port
+
+running `hiera applications` would run the query against the configured database.
+
+
+### Using
+
+`gem install hiera-mysql-backend`
+
+
+### Configuring Hiera
+
+Hiera configuration is pretty simple
+
+```yaml
+---
+:backends:
+  - yaml
+  - mysql2
+
+:yaml:
+  :datadir: /etc/puppet/hieradata
+
+:mysql2:
+  :datadir: /etc/puppet/hieradata
+  :host: hostname
+  :user: username
+  :pass: password
+  :database: database
+  :port: 3306
+
+:hierarchy:
+  - "%{::clientcert}"
+  - "%{::custom_location}"
+  - common
+
+:logger: console
+```
+
+If `host` is not defined it will use `localhost` as default.
+
+If `port` is not defined it will use the default `3306` mysql port
+
+## Known issues
+
+1. It always return an Array of hashes regardless of the number of items returned. (I did this on purpose because it is what I needed but I may be persuaded to do otherwise)
+2. This README is poorly written.
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/hiera-mysql-json-backend-puppetserver.gemspec

@@ -0,0 +1,19 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |gem|
+  gem.name          = "hiera-mysql-json-backend-jruby"
+  gem.version       = "1.0.0"
+  gem.authors       = ["Hostnet"]
+  gem.email         = ["opensource@hostnet.nl"]
+  gem.description   = %q{Alternative MySQL backend with json support for hiera}
+  gem.summary       = %q{Alternative MySQL backend with json support for hiera}
+  gem.homepage      = "https://github.com/hostnet/hiera-mysql-json-backend"
+  gem.license       = "MIT"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+  gem.add_dependency('jdbc-mysql')
+  gem.add_development_dependency('rake')
+end

data/hiera-mysql-json-backend.gemspec

@@ -0,0 +1,20 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |gem|
+  gem.name          = "hiera-mysql-json-backend"
+  gem.version       = "1.0.0"
+  gem.authors       = ["Hostnet"]
+  gem.email         = ["opensource@hostnet.nl"]
+  gem.description   = %q{Alternative MySQL backend with json support for hiera}
+  gem.summary       = %q{Alternative MySQL backend with json support for hiera}
+  gem.homepage      = "https://github.com/hostnet/hiera-mysql-json-backend"
+  gem.license       = "MIT"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+  gem.add_dependency('mysql2')
+  gem.add_dependency('hiera')
+  gem.add_development_dependency('rake')
+end

data/lib/hiera/backend/mysql_json_backend.rb

@@ -0,0 +1,152 @@
+class Hiera
+  module Backend
+    class Mysql_json_backend
+
+      def initialize(cache=nil)
+        require 'json'
+        if defined?(JRUBY_VERSION)
+          require 'jdbc/mysql'
+          require 'java'
+        else
+          begin
+            require 'mysql2'
+          rescue LoadError
+            require 'rubygems'
+            require 'mysql2'
+          end
+        end
+
+        @cache = cache || Filecache.new
+
+        Hiera.debug("Hiera mysql_json initialized")
+      end
+
+      def lookup(key, scope, order_override, resolution_type)
+        # default answer is set to nil otherwise the lookup ends up returning
+        # an Array of nils and causing a Puppet::Parser::AST::Resource failed with error ArgumentError
+        # for any other lookup because their default value is overwriten by [nil,nil,nil,nil]
+        # so hiera('myvalue', 'test1') returns [nil,nil,nil,nil]
+        results = nil
+
+        Hiera.debug("looking up #{key} in MySQL2 Backend")
+        Hiera.debug("resolution type is #{resolution_type}")
+
+        Backend.datasources(scope, order_override) do |source|
+          Hiera.debug("Looking for data source #{source}")
+          sqlfile = Backend.datafile(:mysql2, scope, source, "sql") || next
+
+          next unless File.exist?(sqlfile)
+          data = @cache.read(sqlfile, Hash, {}) do |datafile|
+            YAML.load(datafile)
+          end
+
+          mysql_config = data.fetch(:dbconfig, {})
+          mysql_host = mysql_config.fetch(:host, nil) || Config[:mysql2][:host] || 'localhost'
+          mysql_user = mysql_config.fetch(:user, nil) || Config[:mysql2][:user]
+          mysql_pass = mysql_config.fetch(:pass, nil) || Config[:mysql2][:pass]
+          mysql_port = mysql_config.fetch(:port, nil) || Config[:mysql2][:port] || '3306'
+          mysql_database = mysql_config.fetch(:database, nil) || Config[:mysql2][:database]
+
+          connection_hash = {
+            :host => mysql_host,
+            :username => mysql_user,
+            :password => mysql_pass,
+            :database => mysql_database,
+            :port => mysql_port,
+            :reconnect => true}
+
+
+          Hiera.debug("data #{data.inspect}")
+          next if data.empty?
+          next unless data.include?(key)
+
+          Hiera.debug("Found #{key} in #{source}")
+
+          new_answer = Backend.parse_answer(data[key], scope)
+
+          results = query(connection_hash, new_answer)
+
+          next if results.length != 1
+          begin
+            new_answer = JSON.parse(results[0]['value'])
+          rescue
+            Hiera.debug("Miserable failure while looking for #{key}.")
+            next
+          end
+
+          case resolution_type.is_a?(Hash) ? :hash : resolution_type
+          when :array
+            raise Exception, "Hiera type mismatch for key '#{key}': expected Array and got #{new_answer.class}" unless new_answer.kind_of? Array or new_answer.kind_of? String
+            results ||= []
+            results << new_answer
+          when :hash
+            raise Exception, "Hiera type mismatch for key '#{key}': expected Hash and got #{new_answer.class}" unless new_answer.kind_of? Hash
+            results ||= {}
+            results = Backend.merge_answer(new_answer, results, resolution_type)
+          else
+            results = new_answer
+            break
+          end
+        end
+        return results
+      end
+
+
+      def query(connection_hash, query)
+        Hiera.debug("Executing SQL Query: #{query}")
+
+        data=[]
+        mysql_host=connection_hash[:host]
+        mysql_user=connection_hash[:username]
+        mysql_pass=connection_hash[:password]
+        mysql_database=connection_hash[:database]
+        mysql_port=connection_hash[:port]
+
+        if defined?(JRUBY_VERSION)
+          Jdbc::MySQL.load_driver
+          url = "jdbc:mysql://#{mysql_host}:#{mysql_port}/#{mysql_database}"
+          props = java.util.Properties.new
+          props.set_property :user, mysql_user
+          props.set_property :password, mysql_pass
+
+          conn = com.mysql.jdbc.Driver.new.connect(url,props)
+          stmt = conn.create_statement
+
+          res = stmt.execute_query(query)
+          md = res.getMetaData
+          numcols = md.getColumnCount
+
+          Hiera.debug("Mysql Query returned #{numcols} rows")
+
+          while ( res.next ) do
+            if numcols < 2
+              Hiera.debug("Mysql value : #{res.getString(1)}")
+              data << res.getString(1)
+            else
+              row = {}
+              (1..numcols).each do |c|
+                row[md.getColumnName(c)] = res.getString(c)
+              end
+              data << row
+            end
+          end
+
+        else
+          client = Mysql2::Client.new(connection_hash)
+          begin
+            data = client.query(query).to_a
+            Hiera.debug("Mysql Query returned #{data.size} rows")
+          rescue => e
+            Hiera.debug e.message
+            data = nil
+          ensure
+            client.close
+          end
+        end
+
+        return data
+
+      end
+    end
+  end
+end

metadata

@@ -0,0 +1,96 @@
+--- !ruby/object:Gem::Specification
+name: hiera-mysql-json-backend
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Hostnet
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2017-07-14 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: mysql2
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: hiera
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Alternative MySQL backend with json support for hiera
+email:
+- opensource@hostnet.nl
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- CHANGELOG.md
+- Gemfile
+- Gemfile.lock
+- LICENSE
+- README.md
+- Rakefile
+- hiera-mysql-json-backend-puppetserver.gemspec
+- hiera-mysql-json-backend.gemspec
+- lib/hiera/backend/mysql_json_backend.rb
+homepage: https://github.com/hostnet/hiera-mysql-json-backend
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.6.11
+signing_key: 
+specification_version: 4
+summary: Alternative MySQL backend with json support for hiera
+test_files: []