ncedit 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 509d160cbfff794ee5d1880dbde293aa6fedb7dd
-  data.tar.gz: cccf9de5724dc7e43397625985d2cdf9a9eb625e
+  metadata.gz: 326ad90c579b6ab122bc1aca4e92f2ba9409f556
+  data.tar.gz: 58fb80c7453318166436e8fac0d84a3d57b52c56
 SHA512:
-  metadata.gz: b195209858825b5924aa7e975525760622caa1ba60a1fea8f12c6a4ac7f28318ea2e82f29849276433d7ff465cae8fc1719cdf88a03a29a6f934f6acad537a60
-  data.tar.gz: bbc67d80b0b6d7c6ed5f00dc646dd03cfc06ce0bff54db67fabc48025fd3d09b26f6e5d9c3d6f7072f4db921ad7dec6a4ba5d243877f06fc15d6a61ec24c09cc
+  metadata.gz: a3053204bc285ae301a4848391e7bf52d46baf3feb4c40c8beac48998710df92888927e4b1fe4072affc4415823b28874bb9cdac81f8631765de60a44567a9ef
+  data.tar.gz: 0d0c2c813e97aad9b2d41a1bf3ae8c9cce8b34ac310c3b8f3ab0b1e98b2ac4a1f8148ac45f8394e21e46e6fdbac4c85d0f17beb24accb7ddd25db3792dd388cb

data/.gitignore

@@ -3,10 +3,10 @@
 /Gemfile.lock
 /_yardoc/
 /coverage/
-/doc/
 /pkg/
 /spec/reports/
 /tmp/
 
 # rspec failure tracking
 .rspec_status
+*.gem

data/README.md

@@ -1,36 +1,131 @@
 [![Build Status](https://travis-ci.org/GeoffWilliams/ncedit.svg?branch=master)](https://travis-ci.org/GeoffWilliams/ncedit)
-# Ncedit
+# ncedit - Puppet Node Classifier CLI editor
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/ncedit`. To experiment with that code, run `bin/console` for an interactive prompt.
+ncedit is a small utility program that lets you edit the Puppet Enterprise Node Classifier rules from the commandline.
 
-TODO: Delete this and the text above, and describe your gem
+Why would you want to do this given that we have the excellent [node_manager] (https://forge.puppet.com/WhatsARanjit/node_manager) module already on the forge?  Well... lots of reasons.  First off, using puppet code to drive the Node Classifier means that you have to have the `node_manager` module alread installed, which means that you must already have your [classification rules](https://docs.puppet.com/pe/latest/console_classes_groups_getting_started.html) in to reference the module through [Code Manager](https://docs.puppet.com/pe/latest/code_mgr.html).  You could in-theory use Puppet Enterprise's new idempotent installer (just reinstall puppet over the top of itself) to fix this exact issue but then you still have the problem of how to classify your master in order to activate any other new rules (eg node-ttl) you want to use, which are associated with a new [role](https://docs.puppet.com/pe/latest/r_n_p_intro.html) for the Puppet Master.
+
+That's where this tool comes in since all you need is root shell on the Puppet Master and a YAML or JSON file with the changes you want to make...
+
+## Features
+* Create node groups
+* Add or remove classes
+* Add or remove class parameters
+* Add or remove rules
+
+All from the convenience of the CLI.  This also allows this tool to be called from scripts and other systems in order to setup Puppet Enterprise the way you want and with the minimum of effort.
+
+For the moment, the edits to be carried out need to be placed into either a JSON or YAML file for bulk processing.  If there is interest, the tool will be enhanced to allow the above operations to be specified individually on the command line so to avoid the need to write YAML/JSON.
 
 ## Installation
+Install this tool on your Puppet Master (Master-of-Masters)
 
-Add this line to your application's Gemfile:
+```shell
+$ sudo /opt/puppetlabs/puppet/bin/gem install ncedit
+```
+This will install the gem into the Puppet Master's vendored ruby.  You could instead use your OS ruby and it should work, just make sure you have a recent-ish version of ruby.
+
+## Usage
 
-```ruby
-gem 'ncedit'
+### Node Classifier API access
+ncedit is intended to be run on the Puppet Master as root.  Doing so avoids having to deal with networks, firewalls, whitelists and the Puppet Enterprise RBAC API.  While it would be cool to expand the tool to deal with these issues, its just a whole lot simpler to just run from the Puppet Master so right now that's all thats supported.
+
+### Making batch changes
+To avoid the need to repeatedly invoke this tool using say, a bash script, ncedit natively supports reading a file of batch changes to make that is written in either YAML or JSON.
+
+#### Batch data file
+* Since ncedit internally represents all from the Node Classifier API as hashes, its easy to support both YAML and JSON since they both resolve to this format
+* It's possible to add multiple groups at a time, you just need another `NAME_OF_GROUP` stanza
+
+In each case the file needs to be ordered as follows:
+
+##### YAML
+```yaml
+"NAME_OF_GROUP":
+  # hash of classes to edit/create
+  "classes":
+    "CLASS_TO_ENSURE":
+      "OPTIONAL_PARAM_NAME": "VALUE_TO_SET"
+  # Array of classes to delete
+  "delete_classes":
+      - "CLASS_TO_DELETE"
+  # Hash classes + Array of parameter names to delete
+  "delete_params":
+    "CLASS_TO_PROCESS":
+      - "PARAMETER_TO_DELETE"
+  # Rules to append to group
+  "append_rules":
+    - "CONDITIONAL" # 'and'/'or'
+    - - "="         # rule tuple 0 - comparator, eg '='
+      - "VARIABLE"  # rule tuple 1 - variable, eg 'fqdn'
+      - "VALUE"     # rule tuple 2 - value to match, eg 'pupper.puppet.com'
 ```
 
-And then execute:
+* [Worked example](doc/example/batch.yaml)
+
+##### JSON
+```json
+{
+  "NAME_OF_GROUP": {
+    "classes": {
+      "CLASS_TO_ENSURE": {
+        "OPTIONAL_PARAM_NAME": "VALUE_TO_SET"
+      }
+    },
+    "delete_classes": [
+      "CLASS_TO_DELETE"
+    ],
+    "delete_params": {
+      "CLASS_TO_PROCESS": [
+        "PARAMETER_TO_DELETE"
+      ]
+    },
+    "append_rules": [
+      "CONDITIONAL",
+      [
+        "=",
+        "VARIABLE",
+        "VALUE"
+      ]
+    ]
+  }
+}
+```
 
-    $ bundle
+* JSON doesn't support comments natively so please see above YAML example for notes
+* [Worked example](doc/example/batch.json)
 
-Or install it yourself as:
+#### Ensuring changes
+* ncedit is idempotent so you may run the command as often as you like
 
-    $ gem install ncedit
+##### YAML
+```shell
+ncedit  batch --yaml-file /path/to/batch.yaml
+```
 
-## Usage
+##### JSON
+```shell
+ncedit  batch --json-file /path/to/batch.json
+```
 
-TODO: Write usage instructions here
+## Making per-item changes
+* coming? (soon?) -- anyone want this?
 
-## Development
+## Troubleshooting
+* If the ncedit fails, it will swallow stack traces by default, pass the `--verbosity debug` argument if you need to obtain one.  Note the position of the argument before the command name:
+```
+bundle exec ncedit --verbosity debug  batch --yaml-file /path/to/batch.yaml
+```
+* Ensure you are running as `root` to avoid permission errors
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+## Testing
+To run tests:
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+```shell
+bundle install
+bundle exec rake spec
+```
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/ncedit.
+Bug reports and pull requests are welcome on GitHub at https://github.com/GeoffWilliams/ncedit.

data/doc/example/batch.json

@@ -0,0 +1,34 @@
+{
+  "PE Master": {
+    "classes": {
+      "puppet_enterprise::profile::master": {
+        "code_manager_auto_configure": true,
+
+        "r10k_remote": "https://github.com/GeoffWilliams/r10k-control",
+        "r10k_private_key": "/etc/puppetlabs/puppetserver/ssh/id-control_repo.rsa"
+      }
+    },
+    "delete_classes": [
+      "puppet_enterprise::profile::masterbad"
+    ],
+    "delete_params": {
+      "puppet_enterprise::profile::redo:": [
+        "badparam"
+      ]
+    }
+  },
+  "Puppet Masters": {
+    "classes": {
+      "r_role::puppet::master_minimal": {
+      }
+    },
+    "append_rules": [
+      "or",
+      [
+        "=",
+        "name",
+        "pupper.puppet.com"
+      ]
+    ]
+  }
+}

data/{batch.yaml → doc/example/batch.yaml}

@@ -9,18 +9,19 @@
       "code_manager_auto_configure": true
       "r10k_remote": "https://github.com/GeoffWilliams/r10k-control"
       "r10k_private_key": "/etc/puppetlabs/puppetserver/ssh/id-control_repo.rsa"
-#
-#   "delete_classes":
-#     - "puppet_enterprise::profile::masterbad"
-#
-#   "delete_params":
-#     "puppet_enterprise::profile::redo:":
-#       - "badparam"
+
+  "delete_classes":
+    - "puppet_enterprise::profile::masterbad"
+
+  "delete_params":
+    "puppet_enterprise::profile::redo:":
+      - "badparam"
 
 "Puppet Masters":
   "classes":
     "r_role::puppet::master_minimal": {}
-  # "append_rules":
-  #   - - "="
-  #     - "name"
-  #     - "vmpump02.puppet.com"
+  "append_rules":
+    - "or"
+    - - "="
+      - "name"
+      - "pupper.puppet.com"

data/exe/ncedit

@@ -56,22 +56,25 @@ Escort::App.create do |app|
   # end
 
   app.command :batch do |command|
-    command.summary "Batch processing from YAML"
-    command.description "Process a YAML file to add/delete classes, parameters and rules"
+    command.summary "Batch processing from YAML/JSON"
+    command.description "Process a YAML/JSON file to add/delete classes, parameters and rules"
     command.options do |opts|
-      opts.opt(:filename,
+      opts.opt(:yaml_file,
         'YAML file',
-        :long => '--filename',
+        :long => '--yaml-file',
+        :type => :string,
+      )
+      opts.opt(:json_file,
+        'JSON file',
+        :long => '--json-file',
         :type => :string,
       )
     end
     command.action do |options, arguments|
-      filename = options[:global][:commands][:batch][:options][:filename]
-      if filename == nil
-        Escort::Logger.error.error "Filename is required for batch updates"
-      else
-        NCEdit::Cmd::batch(filename)
-      end
+      yaml_file = options[:global][:commands][:batch][:options][:yaml_file]
+      json_file = options[:global][:commands][:batch][:options][:json_file]
+
+      NCEdit::Cmd::batch(yaml_file: yaml_file, json_file: json_file)
     end
   end
 end

data/lib/ncedit/cmd.rb

@@ -1,9 +1,11 @@
 require 'puppetclassify'
 require 'yaml'
+require 'json'
 require 'escort'
 
 module NCEdit
   module Cmd
+    DEFAULT_RULE = "or"
 
     def self.init(puppetclassify = nil)
       if puppetclassify
@@ -133,6 +135,35 @@ module NCEdit
       end
     end
 
+    def self.read_batch_data(yaml_file: nil, json_file:nil)
+      if yaml_file == nil and json_file == nil
+        raise "YAML or JSON file must be specified for batch updates"
+      elsif yaml_file and json_file
+        raise "Cannot process both YAML and JSON at the same time"
+      elsif yaml_file
+        if File.exists?(yaml_file)
+          begin
+            data = YAML.load_file(yaml_file)
+          rescue Psych::SyntaxError
+            raise "syntax error parsing #{yaml_file}"
+          end
+        else
+          raise "YAML file not found: #{yaml_file}"
+        end
+      elsif json_file
+        if File.exists?(json_file)
+          begin
+            data = JSON.parse(IO.read(json_file))
+          rescue JSON::ParserError
+            raise "syntax error parsing #{json_file}"
+          end
+        else
+          raise "JSON file not found: #{json_file}"
+        end
+      end
+      data
+    end
+
     # Batch entry from YAML file, example file format:
     # 'PE Master':
     #   'classes':
@@ -154,44 +185,36 @@ module NCEdit
     #     - - "="
     #       - "name"
     #       - "vmpump02.puppet.com"
-    def self.batch(filename)
-      if File.exists?(filename)
-        begin
-          yaml = YAML.load_file(filename)
-
-          yaml.each { |group_name, data|
-            Escort::Logger.output.puts "Processing #{group_name}"
-
-            if data.has_key?("delete_classes")
-              if delete_classes(group_name, data["delete_classes"])
-                update_group(group_name, classes: data["delete_classes"])
-              end
-            end
+    def self.batch(yaml_file: nil, json_file: nil)
+      data = read_batch_data(yaml_file: yaml_file, json_file: json_file)
+      data.each { |group_name, data|
 
-            if data.has_key?("delete_params")
-              if delete_params(group_name, data["delete_params"])
-                update_group(group_name, classes: data["delete_params"])
-              end
-            end
+        Escort::Logger.output.puts "Processing #{group_name}"
 
-            if data.has_key?("classes")
-              if ensure_classes_and_params(group_name, data["classes"])
-                update_group(group_name, classes: data["classes"])
-              end
-            end
+        if data.has_key?("delete_classes")
+          if delete_classes(group_name, data["delete_classes"])
+            update_group(group_name, classes: data["delete_classes"])
+          end
+        end
 
-            if data.has_key?("append_rules")
-              if ensure_rules(group_name, data["append_rules"])
-                update_group(group_name, rule: data["append_rules"])
-              end
-            end
-          }
-        rescue Psych::SyntaxError
-          Escort::Logger.error.error "Syntax error found in #{filename}, please fix and retry"
+        if data.has_key?("delete_params")
+          if delete_params(group_name, data["delete_params"])
+            update_group(group_name, classes: data["delete_params"])
+          end
         end
-      else
-        Escort::Logger.error.error "File not found: #{filename}"
-      end
+
+        if data.has_key?("classes")
+          if ensure_classes_and_params(group_name, data["classes"])
+            update_group(group_name, classes: data["classes"])
+          end
+        end
+
+        if data.has_key?("append_rules")
+          if ensure_rules(group_name, data["append_rules"])
+            update_group(group_name, rule: data["append_rules"])
+          end
+        end
+      }
     end
 
     def self.delete_class(group, class_name)
@@ -279,16 +302,26 @@ module NCEdit
       updated
     end
 
+    # Ensure a partualar rule exists in the group["rule"] array
+    # This affects only the items in the chain, eg:
+    # [
+    #  "or",
+    #   [
+    #     <--- here!
+    #   ]
+    # ]
+    #
+    # Only the rule to be added in should be passed as the rule parameter, eg:
+    # ["=", "name", "bob"]
     def self.ensure_rule(group, rule)
       updated = false
-      if ! group["rules"]
-        # no rules yet - just add our new one
-        group["rules"] = []
-      end
 
       # see if rule already exists, if it doesn't, append it
       found = false
-      group["rules"].each {|system_rule|
+
+      # rules are nested like this, the "or" applies to the whole rule chain:
+      # "rule"=>["or", ["=", "name", "bob"], ["=", "name", "hello"]]
+      group["rule"][1].each {|system_rule|
         if  system_rule[0] == rule[0] and
             system_rule[1] == rule[1] and
             system_rule[2] == rule[2]
@@ -298,20 +331,45 @@ module NCEdit
       }
       if ! found
         Escort::Logger.output.puts "Appending rule: #{rule}"
-        group["rules"] << rule
+        group["rule"][1] << rule
         updated = true
       end
 
       updated
     end
 
+    # rules need to arrive like this:
+    # ["or", ["=", "name", "pupper.megacorp.com"], ["=", "name", "pupper.megacorp.com"]]
+    # since the rule conjunction "or" can only be specified once per rule chain
+    # we will replace whatever already exists in the rule with what the user
+    # specified
     def self.ensure_rules(group_name, rules)
       updated = false
-      rules.each { |rule|
-        updated |= ensure_rule(nc_group(group_name), rule)
+      group = nc_group(group_name)
+      if ! group["rule"] or group["rule"].empty?
+        # no rules yet - just add our new one
+        group["rule"] = [DEFAULT_RULE,[]]
+      end
+      updated |= ensure_rule_conjunction(group, rules[0])
+      rules[1].each { |rule|
+        updated |= ensure_rule(group, rule)
       }
 
       updated
     end
+
+    def self.ensure_rule_conjunction(group, op)
+      updated = false
+      if ["and", "or"].include?(op)
+        if group["rule"][0] != op
+          group["rule"][0] = op
+          updated = true
+        end
+      else
+        raise "Illegal rule conjunction #{op}, allowed: 'and', 'or'"
+      end
+
+      updated
+    end
   end
 end

data/lib/ncedit/version.rb

@@ -1,3 +1,3 @@
 module NCEdit
-  VERSION = "0.0.1"
+  VERSION = "0.1.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ncedit
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.1.0
 platform: ruby
 authors:
 - Geoff Williams
@@ -109,9 +109,10 @@ files:
 - LICENSE
 - README.md
 - Rakefile
-- batch.yaml
 - bin/console
 - bin/setup
+- doc/example/batch.json
+- doc/example/batch.yaml
 - exe/ncedit
 - lib/ncedit.rb
 - lib/ncedit/cmd.rb