ncedit 0.1.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 326ad90c579b6ab122bc1aca4e92f2ba9409f556
-  data.tar.gz: 58fb80c7453318166436e8fac0d84a3d57b52c56
+SHA256:
+  metadata.gz: e96cfedf083f77d881e84689e8126fcb2eacd5e965a1c78a8bfbbb3957468ccd
+  data.tar.gz: b2f81404318ea89f27d49bf7f2275c570aacb89816857ad76802479ba93e1fad
 SHA512:
-  metadata.gz: a3053204bc285ae301a4848391e7bf52d46baf3feb4c40c8beac48998710df92888927e4b1fe4072affc4415823b28874bb9cdac81f8631765de60a44567a9ef
-  data.tar.gz: 0d0c2c813e97aad9b2d41a1bf3ae8c9cce8b34ac310c3b8f3ab0b1e98b2ac4a1f8148ac45f8394e21e46e6fdbac4c85d0f17beb24accb7ddd25db3792dd388cb
+  metadata.gz: 47c9753e923f1c6410093a8cc55506b6344bc8c13b0ce89d6a1f295fdfd4b3a2b3d0f0ac82b9cce6566c754300954a9dd58c2b599c0b97237e470be5d4e144f1
+  data.tar.gz: a8d646edffb771c405fccc8726769612f66126868e07c05cafb852df35d14f6b6ebfca44f9ff02cbc255cea59d49503b6e7c7081537c1e6a50cd775d2f45156d

data/.gitignore

@@ -10,3 +10,4 @@
 # rspec failure tracking
 .rspec_status
 *.gem
+/.idea

data/.travis.yml

@@ -1,6 +1,6 @@
 sudo: false
 language: ruby
 rvm:
-  - 2.2.3
-before_install: gem install bundler -v 1.12.5
+  - 2.6.6
+before_install: gem install bundler -v 2.1.4
 

data/Makefile

@@ -0,0 +1,3 @@
+build:
+	bundle exec rake spec
+	gem build ncedit.gemspec

data/README.md

@@ -1,21 +1,23 @@
-[![Build Status](https://travis-ci.org/GeoffWilliams/ncedit.svg?branch=master)](https://travis-ci.org/GeoffWilliams/ncedit)
+[![Build Status](https://travis-ci.org/declarativesystems/ncedit.svg?branch=master)](https://travis-ci.org/declarativesystems/ncedit)
 # ncedit - Puppet Node Classifier CLI editor
 
-ncedit is a small utility program that lets you edit the Puppet Enterprise Node Classifier rules from the commandline.
+ncedit is a small utility program that lets you edit the Puppet Enterprise Node Classifier rules from the command line.
 
-Why would you want to do this given that we have the excellent [node_manager] (https://forge.puppet.com/WhatsARanjit/node_manager) module already on the forge?  Well... lots of reasons.  First off, using puppet code to drive the Node Classifier means that you have to have the `node_manager` module alread installed, which means that you must already have your [classification rules](https://docs.puppet.com/pe/latest/console_classes_groups_getting_started.html) in to reference the module through [Code Manager](https://docs.puppet.com/pe/latest/code_mgr.html).  You could in-theory use Puppet Enterprise's new idempotent installer (just reinstall puppet over the top of itself) to fix this exact issue but then you still have the problem of how to classify your master in order to activate any other new rules (eg node-ttl) you want to use, which are associated with a new [role](https://docs.puppet.com/pe/latest/r_n_p_intro.html) for the Puppet Master.
+Why would you want to do this given that we have the excellent [node_manager](https://forge.puppet.com/WhatsARanjit/node_manager) module already on the forge?  Well... lots of reasons.  First off, using puppet code to drive the Node Classifier means that you have to have the `node_manager` module already installed, which means that you must already have your [classification rules](https://docs.puppet.com/pe/latest/console_classes_groups_getting_started.html) in to reference the module through [Code Manager](https://docs.puppet.com/pe/latest/code_mgr.html).  You could in-theory use Puppet Enterprise's new idempotent installer (just reinstall puppet over the top of itself) to fix this exact issue but then you still have the problem of how to classify your master in order to activate any other new rules (eg node-ttl) you want to use, which are associated with a new [role](https://docs.puppet.com/pe/latest/r_n_p_intro.html) for the Puppet Master.
 
-That's where this tool comes in since all you need is root shell on the Puppet Master and a YAML or JSON file with the changes you want to make...
+That's where this tool comes in since all you need is root shell on the Puppet Master.
 
 ## Features
+You can:
 * Create node groups
 * Add or remove classes
 * Add or remove class parameters
 * Add or remove rules
+* Add or edit environment groups
 
-All from the convenience of the CLI.  This also allows this tool to be called from scripts and other systems in order to setup Puppet Enterprise the way you want and with the minimum of effort.
+...All from the convenience of the CLI.  This allows us to be called from scripts and other systems in order to setup Puppet Enterprise the way you want and with the minimum of effort.
 
-For the moment, the edits to be carried out need to be placed into either a JSON or YAML file for bulk processing.  If there is interest, the tool will be enhanced to allow the above operations to be specified individually on the command line so to avoid the need to write YAML/JSON.
+You have a choice of running `ncedit` for each change you wish to make or collecting all of your edits into either a JSON or YAML file for bulk processing.
 
 ## Installation
 Install this tool on your Puppet Master (Master-of-Masters)
@@ -28,7 +30,7 @@ This will install the gem into the Puppet Master's vendored ruby.  You could ins
 ## Usage
 
 ### Node Classifier API access
-ncedit is intended to be run on the Puppet Master as root.  Doing so avoids having to deal with networks, firewalls, whitelists and the Puppet Enterprise RBAC API.  While it would be cool to expand the tool to deal with these issues, its just a whole lot simpler to just run from the Puppet Master so right now that's all thats supported.
+ncedit is intended to be run on the Puppet Master as root.  Doing so avoids having to deal with networks, firewalls, certificate whitelists and the Puppet Enterprise RBAC API.  While it would be cool to expand the tool to deal with these issues, its just a whole lot simpler to just run from the Puppet Master so right now that's all thats supported.
 
 ### Making batch changes
 To avoid the need to repeatedly invoke this tool using say, a bash script, ncedit natively supports reading a file of batch changes to make that is written in either YAML or JSON.
@@ -48,7 +50,7 @@ In each case the file needs to be ordered as follows:
       "OPTIONAL_PARAM_NAME": "VALUE_TO_SET"
   # Array of classes to delete
   "delete_classes":
-      - "CLASS_TO_DELETE"
+    - "CLASS_TO_DELETE"
   # Hash classes + Array of parameter names to delete
   "delete_params":
     "CLASS_TO_PROCESS":
@@ -96,27 +98,159 @@ In each case the file needs to be ordered as follows:
 * [Worked example](doc/example/batch.json)
 
 #### Ensuring changes
-* ncedit is idempotent so you may run the command as often as you like
+* `ncedit` is idempotent so you may run the command as often as you like
+* Classes used in rules *must* already exist for classification to succeed, `--smart-update` will do a code redeploy for you if specified.
 
 ##### YAML
 ```shell
-ncedit  batch --yaml-file /path/to/batch.yaml
+ncedit batch --smart-update --yaml-file /path/to/batch.yaml
 ```
 
 ##### JSON
 ```shell
-ncedit  batch --json-file /path/to/batch.json
+ncedit batch --smart-update --json-file /path/to/batch.json
 ```
 
 ## Making per-item changes
-* coming? (soon?) -- anyone want this?
+If you don't want to go to the hassle of creating a batch file or you have small, dynamic edits you wish to perform, your able to perform individual edits on the command line:
+
+
+### Practical examples
+One of the main uses of this tool is to configure Puppet Code Manager, so here are the exact commands you need (alternatively make a `batch.yaml` file to do it all in one hit):
+
+#### Code Manager R10K checkout URL
+```shell
+ncedit classes --smart-update --group-name 'PE Master' \
+    --class-name puppet_enterprise::profile::master \
+    --param-name r10k_remote \
+    --param-value https://github.com/puppetlabs/control-repo
+```
+
+#### Code Manager SSH key
+```shell
+ncedit classes --smart-update --group-name 'PE Master' \
+    --class-name puppet_enterprise::profile::master \
+    --param-name r10k_ \
+    --param-value /etc/puppetlabs/puppetserver/ssh/id_rsa
+```
+
+#### Code Manager proxy server
+```shell
+ncedit classes --smart-update --group-name 'PE Master' \
+    --class-name puppet_enterprise::profile::master \
+    --param-name r10k_proxy \
+    --param-value http://proxy.megacorp.com:3128
+```
+
+
+#### Enable Code Manager
+```shell
+ncedit classes --smart-update --group-name 'PE Master' \
+    --class-name puppet_enterprise::profile::master \
+    --param-name code_manager_auto_configure \
+    --param-value true
+    
+ncedit classes --smart-update --group-name 'PE Master' \
+    --class-name puppet_enterprise::profile::master \
+    --param-name file_sync_enabled \
+    --param-value automatic
+```
+
+
+#### Add a rule to configure the puppet master with extra classes
+```shell
+ncedit classes --smart-update --group-name "Puppet Masters" \
+    --class-name role::my_puppet_master \
+    --rule '["and", ["=",["fact","fqdn"],"'$(facter fqdn)'"]]' \
+    --rule-mode replace
+```
+
+* Run puppet and then re-deploy puppet code
+* Create a new group called `Puppet Masters`
+* Make it load the role `role::my_puppet_master`
+* Have it match any host matching the FQDN of the host the command was run from and replace any existing rules for the group
+
+#### Refresh the list of classes in the console
+```shell
+ncedit update_classes
+```
+
+### Add a class to group
+```shell
+ncedit classes --group-name FOO --class-name BAR
+```
+* Group will be created if it doesn't exist
+* Class MUST exist AND puppet must be aware of it (defeat caching) for the request to be accepted
+* Pass `--smart-update` to run puppet and re-deploy classes if desired
+
+### Add a parameter to a class for a group
+```shell
+ncedit classes --group-name FOO --class-name BAR --param-name BAZ --param-value BAS
+```
+* Group will be created if it doesn't exist
+* Class will be added if it isn't already
+* Class MUST exist AND puppet must be aware of it (defeat caching) AND the parameter must exist on the class for the request to be accepted
+* Pass `--smart-update` to run puppet and re-deploy classes if desired
+
+### Delete a parameter from a class inside a group
+```shell
+ncedit classes --group-name FOO --class-name BAR --param-name BAZ --delete-param
+```
+* Group will be created if it doesn't exist
+* Class will be added if it isn't already
+* Ensures the parameter `BAZ` is not set
+
+### Delete a class inside a group
+```shell
+ncedit classes --group-name FOO --class-name BAR --delete-class
+```
+* Group will be created if it doesn't exist
+* Ensures the class `BAR` is not defined
+
+### Replace all rules for group
+```shell
+ncedit classes --group-name FOO --rule 'JSON_FRAGMENT' --rule-mode replace
+```
+* Group will be created if it doesn't exist
+* Completely replaces existing rules for this group
+* Example JSON_FRAGMENT: `["and",["=",["fact","ipaddress"],"192.168.1.1"]]`
+
+### Append a rule to group
+```shell
+ncedit classes --group-name FOO --rule 'JSON_FRAGMENT' --rule-mode append
+```
+* Group will be created if it doesn't exist
+* Appends supplied JSON_FRAGMENT to the group's rules
+* Example JSON_FRAGMENT: `["and",["=",["fact","osfamily"],"RedHat"]]`
+* Notice in our example JSON_FRAGMENT that we have *kept* the outer `and` rule.  If we supply a conjuctive different to the rule's current value we will change it for the whole rule
+
+### Add/edit an environment group (experimental)
+
+```shell
+ncedit groups --group-name "Agent-specified environment" --environment agent-specified --environment-trumps --rule '["~",["fact","fqdn"],".*"]' --rule-mode replace
+```
+* Group will be created if it doesn't exit
+* Will be nested under `All Environments` 
+
+## Smart updates
+The NC API will only accept rules for classes that currently exist on the system, so its likely that your rules will be rejected if they have not yet been deployed to the master.  To make this process as seamless as possible, the `--smart-update` option will perform this task for you as-and-when required.  This assumes you have already setup RBAC to grant access to Code Manager, this can be done in a single command using the [pe_rbac gem](https://github.com/declarativesystems/pe_rbac#setting-up-code-manager-on-the-command-line)
+
+Smart updates work as follows:
+
+* If any of the `r10k_*` or `code_manager_auto_configure` parameters are set, update them immediately
+* Run `puppet agent -t` then `puppet-code deploy --all --wait`
+* Process any remaining directives
 
 ## Troubleshooting
+* If you cannot install the `ncedit` gem and you are behind a corporate proxy, ensure that you have correctly set your `http_proxy` and `https_proxy` variables on the shell before running `gem install`.
+* Some corporate proxies will attempt to eavesdrop on all SSL connections which will cause downloads to fail.  This can be resolved by installing a CA bundle (be sure you understand the implications of doing so) or domain whitelisting on the proxy itself
 * If the ncedit fails, it will swallow stack traces by default, pass the `--verbosity debug` argument if you need to obtain one.  Note the position of the argument before the command name:
 ```
 bundle exec ncedit --verbosity debug  batch --yaml-file /path/to/batch.yaml
 ```
 * Ensure you are running as `root` to avoid permission errors
+* If your shell can't find `ncedit` after successful installation and you installed into Puppet Enterprise's vendored ruby, make sure that your `PATH` contains `/opt/puppetlabs/puppet/bin` or run ncedit directly: `/opt/puppetlabs/puppet/bin/ncedit`
+* Don't attempt to debug the NC API through the PE console by capturing traffic, it uses a completely different API all of its own (In particular, there is another layer of boolean logic around rules).  Instead, use [NCIO](https://github.com/jeffmccune/ncio) to dump the current state of the NC API or see [Puppet's NC API documentation](https://docs.puppet.com/pe/latest/nc_index.html)
 
 ## Testing
 To run tests:
@@ -128,4 +262,4 @@ bundle exec rake spec
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/GeoffWilliams/ncedit.
+Bug reports and pull requests are welcome on GitHub at https://github.com/declarativesystems/ncedit.

data/doc/example/batch.json

@@ -3,7 +3,6 @@
     "classes": {
       "puppet_enterprise::profile::master": {
         "code_manager_auto_configure": true,
-
         "r10k_remote": "https://github.com/GeoffWilliams/r10k-control",
         "r10k_private_key": "/etc/puppetlabs/puppetserver/ssh/id-control_repo.rsa"
       }
@@ -12,22 +11,26 @@
       "puppet_enterprise::profile::masterbad"
     ],
     "delete_params": {
-      "puppet_enterprise::profile::redo:": [
-        "badparam"
+      "puppet_enterprise::profile::mcollective::agent": [
+        "stomp_user"
       ]
     }
   },
   "Puppet Masters": {
     "classes": {
-      "r_role::puppet::master_minimal": {
+      "pe_r10k": {
+        "proxy": "proxy.megacorp.com:8080"
       }
     },
     "append_rules": [
       "or",
       [
         "=",
-        "name",
-        "pupper.puppet.com"
+        [
+          "fact",
+          "ipaddress"
+        ],
+        "192.168.0.252"
       ]
     ]
   }

data/doc/example/batch.yaml

@@ -14,14 +14,16 @@
     - "puppet_enterprise::profile::masterbad"
 
   "delete_params":
-    "puppet_enterprise::profile::redo:":
-      - "badparam"
+    "puppet_enterprise::profile::mcollective::agent":
+      - "stomp_user"
 
 "Puppet Masters":
   "classes":
-    "r_role::puppet::master_minimal": {}
+    "pe_r10k":
+      "proxy": "proxy.megacorp.com:8080"
   "append_rules":
     - "or"
     - - "="
-      - "name"
-      - "pupper.puppet.com"
+      - - "fact"
+        - "ipaddress"
+      - "192.168.0.252"

data/exe/ncedit

@@ -6,54 +6,74 @@ require 'ncedit/cmd'
 # display help if nothing specified
 ARGV.push('-h') if ARGV.empty?
 
+# Add the path to the puppet-code command as a fallback.  It's last to
+# allow user to override via the real PATH if necessary
+ENV['PATH'] = "#{ENV['PATH']}:/opt/puppetlabs/puppet/bin/:/opt/puppetlabs/client-tools/bin/"
+
+
 Escort::App.create do |app|
   app.version NCEdit::VERSION
   app.summary "ncedit"
   app.description "Edit PE node classification groups"
 
-  # not currently working...
-  # app.command :classes do |command|
-  #   command.summary "Install all known nodes"
-  #   command.description "Install Puppet Enterprise master and agents on all known nodes"
-  #   command.options do |opts|
-  #     opts.opt(:group_name,
-  #       'NC group name',
-  #       :long => '--group-name',
-  #       :type => :string,
-  #     )
-  #     opts.opt(:class_name,
-  #       'NC class name',
-  #       :long => '--class-name',
-  #       :type => :string,
-  #     )
-  #     opts.opt(:param_name,
-  #       'NC parameter name',
-  #       :long => '--param-name',
-  #       :type => :string,
-  #     )
-  #     opts.opt(:param_value,
-  #       'NC parameter value',
-  #       :long => '--param-value',
-  #       :type => :string,
-  #     )
-  #     opts.opt(:delete_class,
-  #       'delete this rule',
-  #       :long    => '--delete-class',
-  #       :type    => :boolean,
-  #       :default => false,
-  #     )
-  #     opts.opt(:delete_param,
-  #       'delete this rule',
-  #       :long    => '--delete-param',
-  #       :type    => :boolean,
-  #       :default => false,
-  #     )
-  #     opts.dependency :param_value, :on => :param_name
-  #   end
-  #   command.action do |options, arguments|
-  #     NCEdit::Cmd::classes(options, arguments)
-  #   end
-  # end
+  app.command :classes do |command|
+    command.summary "Edit classes"
+    command.description "Create/edit/delete class in rule"
+    command.options do |opts|
+      opts.opt(:group_name,
+        'NC group name',
+        :long => '--group-name',
+        :type => :string,
+      )
+      opts.opt(:class_name,
+        'NC class name',
+        :long => '--class-name',
+        :type => :string,
+      )
+      opts.opt(:param_name,
+        'NC parameter name',
+        :long => '--param-name',
+        :type => :string,
+      )
+      opts.opt(:param_value,
+        'NC parameter value',
+        :long => '--param-value',
+        :type => :string,
+      )
+      opts.opt(:delete_class,
+        'Delete this class',
+        :long    => '--delete-class',
+        :type    => :boolean,
+        :default => false,
+      )
+      opts.opt(:delete_param,
+        'Delete this param',
+        :long    => '--delete-param',
+        :type    => :boolean,
+        :default => false,
+      )
+      opts.opt(:rule,
+        'Update the NC group with this rule (JSON fragment)',
+        :long    => '--rule',
+        :type    => :string,
+      )
+      opts.dependency :param_value, :on => :param_name
+
+      opts.opt(:rule_mode,
+        "Processing instruction for rule supplied with --rule (allowed: 'replace', 'append')",
+        :long    => '--rule-mode',
+        :type    => :string,
+      )
+      opts.opt(:smart_update,
+        'Smart update',
+        :long => '--smart-update',
+        :type => :boolean,
+      )
+    end
+    command.action do |options, arguments|
+      NCEdit::Cmd::classes(options[:global][:commands][:classes][:options])
+    end
+  end
 
   app.command :batch do |command|
     command.summary "Batch processing from YAML/JSON"
@@ -69,12 +89,63 @@ Escort::App.create do |app|
         :long => '--json-file',
         :type => :string,
       )
+      opts.opt(:smart_update,
+        'Smart update',
+        :long => '--smart-update',
+        :type => :boolean,
+      )
     end
     command.action do |options, arguments|
       yaml_file = options[:global][:commands][:batch][:options][:yaml_file]
       json_file = options[:global][:commands][:batch][:options][:json_file]
+      smart_update = options[:global][:commands][:batch][:options][:smart_update]
 
-      NCEdit::Cmd::batch(yaml_file: yaml_file, json_file: json_file)
+      NCEdit::Cmd::batch(yaml_file: yaml_file, json_file: json_file, smart_update: smart_update)
+    end
+  end
+
+  app.command :groups do |command|
+    command.summary "Create/Edit groups"
+    command.description "Create or edit groups (and set environment)"
+    command.options do |opts|
+      opts.opt(:group_name,
+               'NC group name',
+               :long => '--group-name',
+               :type => :string,
+               )
+      opts.opt(:environment,
+               'Group should use this environment',
+               :long => '--environment',
+               :type => :string,
+      )
+      opts.opt(:environment_trumps,
+               'Group environment overrides all others',
+               :long => '--environment-trumps',
+               :type => :boolean,
+               )
+      opts.opt(:rule,
+               'Update the NC group with this rule (JSON fragment)',
+               :long    => '--rule',
+               :type    => :string,
+               )
+
+      opts.opt(:rule_mode,
+               "Processing instruction for rule supplied with --rule (allowed: 'replace', 'append')",
+               :long    => '--rule-mode',
+               :type    => :string,
+               )
+    end
+    command.action do |options, arguments|
+      NCEdit::Cmd::groups(options[:global][:commands][:groups][:options])
+    end
+  end
+
+
+  app.command :update_classes do |command|
+    command.summary "Refresh the classes available in the console"
+    command.description "Invalidate class cache and ask puppet to re-scan classes"
+    command.action do |options, arguments|
+      NCEdit::Cmd::update_classes()
     end
   end
 end

data/lib/ncedit/cmd.rb

@@ -2,11 +2,21 @@ require 'puppetclassify'
 require 'yaml'
 require 'json'
 require 'escort'
+require 'puppet_https'
 
 module NCEdit
   module Cmd
     DEFAULT_RULE = "or"
 
+    R10K_SETTINGS_CLASS = "puppet_enterprise::profile::master"
+    R10K_SETTINGS_PARAMS = [
+        "code_manager_auto_configure",
+        "r10k_remote",
+        "r10k_private_key",
+        "r10k_proxy",
+        "r10k_postrun",
+    ]
+
     def self.init(puppetclassify = nil)
       if puppetclassify
         # use passed in puppetclassify if present - allows injection for easy
@@ -16,9 +26,12 @@ module NCEdit
         hostname = %x(facter fqdn).strip.downcase
         port = 4433
 
+
         # Define the url to the classifier API - we can't just do localhost because
         # the name has to match the SSL certificate
-        rest_api_url = "https://#{hostname}:#{port}/classifier-api"
+        base_url = "https://#{hostname}:"
+        @puppet_url = "#{base_url}8140"
+        @rest_api_url = "#{base_url}#{port}/classifier-api"
 
         # We need to authenticate against the REST API using a certificate
         # that is whitelisted in /etc/puppetlabs/console-services/rbac-certificate-whitelist.
@@ -54,12 +67,15 @@ module NCEdit
           end
         end
 
-        @puppetclassify = PuppetClassify.new(rest_api_url, auth_info)
+        @puppetclassify = PuppetClassify.new(@rest_api_url, auth_info)
+
+        # borrow the cool HTTPS requester built into puppetclassify
+        @puppet_https = PuppetHttps.new(auth_info)
       end
     end
 
     # Fetch a group by ID, make the group if it doesn't already exist
-    def self.nc_group_id(group_name)
+    def self.nc_group_id(group_name, parent_name: "All Nodes")
       if ! @puppetclassify
         init
       end
@@ -70,7 +86,7 @@ module NCEdit
         res = @puppetclassify.groups.create_group(
           {
             "name"    => group_name,
-            "parent"  => @puppetclassify.groups.get_group_id("All Nodes"),
+            "parent"  => @puppetclassify.groups.get_group_id(parent_name),
             "classes" => {},
           }
         )
@@ -85,21 +101,48 @@ module NCEdit
       group_id
     end
 
-    def self.nc_group(group_name)
+    def self.nc_group(group_name, parent_name:nil)
       if ! @puppetclassify
         init
       end
       # Get the wanted group from the API
       #   1. Get the id of the wanted group
       #   2. Use the id to fetch the group
-      group_id  = nc_group_id(group_name)
+      group_id = nc_group_id(group_name, parent_name: parent_name)
       Escort::Logger.output.puts "Group #{group_name} found, getting definition"
       group = @puppetclassify.groups.get_group(group_id)
 
       group
     end
 
-    def self.update_group(group_name, classes: nil, rule: nil)
+    # to see if our changes were saved or not we need to remove all nillified
+    # keys from both levels (class, parameter) of the class_delta array, since
+    # when we re-read from the NC our nillified data will be completely gone.  A
+    # naive comparison would then report a failure even though the operation
+    # succeeded.  On a practical level we must convert:
+    # {"puppet_enterprise"=>{"proxy"=>nil, "keep"=>"keep"}, "b"=>nil}
+    # ...to...
+    # {"puppet_enterprise"=>{"keep"=>"keep"}}
+    #
+    # @param nc_class The class hash as re-read from the NC API
+    # @param class_delta The class delta we originally requested (with nils for deletes)
+    def self.delta_saved?(nc_class, class_delta)
+      class_delta_reformatted = class_delta.map { |class_name, params|
+        if params == nil
+          # skip classes that are requested to be deleted for the moment since
+          # we will catch them on the outer pass
+      	  params_fixed = params
+        else
+          # remove all individual nullified parameters
+      	  params_fixed = params.reject{|param_name, param_value| param_value == nil}
+        end
+        [class_name,params_fixed]
+      }.to_h.reject { |class_name,params| params == nil}
+
+      nc_class == class_delta_reformatted
+    end
+
+    def self.update_group(group_name, classes: nil, rule: nil, environment: nil, environment_trumps: nil)
       # group_delta will actually replace all classes/rules with whatever is
       # specified, so we need to merge this with any existing definition if
       # one of these fields is not needed for a particular update otherwise
@@ -112,10 +155,20 @@ module NCEdit
         rule = nc_group(group_name)["rule"]
       end
 
+      if environment == nil
+        environment = nc_group(group_name)["environment"]
+      end
+
+      if ! environment_trumps
+        environment_trumps = nc_group(group_name)["environment_trumps"]
+      end
+
       group_delta = {
-        'id'      => nc_group_id(group_name),
-        'rule'    => rule,
-        'classes' => classes,
+        'id'                  => nc_group_id(group_name),
+        'rule'                => rule,
+        'classes'             => classes,
+        'environment'         => environment,
+        'environment_trumps'  => environment_trumps,
       }
       res = @puppetclassify.groups.update_group(group_delta)
 
@@ -126,11 +179,14 @@ module NCEdit
       # now present.  If there was an error, then the user should have
       # previously seen some output since puppetclassify prints some useful
       # debug output
-      saved_group = nc_group(group_name)
-      if saved_group["classes"] == classes and saved_group["rule"] == rule
+      re_read_group = nc_group(group_name)
+      if delta_saved?(re_read_group["classes"], classes) &&
+          re_read_group["rule"] == rule &&
+          re_read_group["environment"] == environment &&
+          re_read_group["environment_trumps"] == environment_trumps
         Escort::Logger.output.puts "changes saved"
       else
-        Escort::Logger.error.error "re-read #{group_name} results in #{saved_group} should have delta of #{group_delta}"
+        Escort::Logger.error.error "re-read #{group_name} results in #{re_read_group} should have delta of #{group_delta}"
         raise "Error saving #{group_name}"
       end
     end
@@ -179,66 +235,96 @@ module NCEdit
     #         'badparam'
     #
     # 'Puppet Masters':
-    #   'clases':
+    #   'classes':
     #     'role::puppet::master':
     #   'append_rules':
     #     - - "="
     #       - "name"
     #       - "vmpump02.puppet.com"
-    def self.batch(yaml_file: nil, json_file: nil)
+    def self.batch(yaml_file: nil, json_file: nil, smart_update: false)
       data = read_batch_data(yaml_file: yaml_file, json_file: json_file)
+
+      if smart_update
+        needs_reclassify = contains_r10k_settings(data)
+        if needs_reclassify
+          apply_r10k_settings_now(needs_reclassify)
+        end
+        puppet_code_deploy
+
+        Escort::Logger.output.puts "Sleep for 60 seconds to let classes finish their refresh..."
+        sleep(60)
+      end
+
       data.each { |group_name, data|
 
         Escort::Logger.output.puts "Processing #{group_name}"
+        group = nc_group(group_name)
 
+        #
+        # delete classes
+        #
         if data.has_key?("delete_classes")
-          if delete_classes(group_name, data["delete_classes"])
-            update_group(group_name, classes: data["delete_classes"])
+          changes = false
+
+          data["delete_classes"].each { |class_name|
+            changes |= ensure_class(group, class_name, delete:true)
+          }
+          if changes
+            update_group(group_name, classes: group["classes"])
           end
         end
 
+        #
+        # delete params
+        #
         if data.has_key?("delete_params")
-          if delete_params(group_name, data["delete_params"])
-            update_group(group_name, classes: data["delete_params"])
+          changes = false
+          data["delete_params"].each { |class_name, delete_params|
+            delete_params.each { | param_name|
+              changes |= ensure_class(group, class_name)
+              changes |= ensure_param(group, class_name, param_name, nil, delete:true)
+            }
+          }
+          if changes
+            update_group(group_name, classes: group["classes"])
           end
         end
 
+        #
+        # classes (and optionally params)
+        #
         if data.has_key?("classes")
-          if ensure_classes_and_params(group_name, data["classes"])
-            update_group(group_name, classes: data["classes"])
+          if ensure_classes_and_params(group, data["classes"])
+            update_group(group_name, classes: group["classes"])
           end
         end
 
+        #
+        # append rules
+        #
         if data.has_key?("append_rules")
-          if ensure_rules(group_name, data["append_rules"])
-            update_group(group_name, rule: data["append_rules"])
+          if ensure_rules(group, data["append_rules"])
+            update_group(group_name, rule: group["rule"])
           end
         end
       }
     end
 
-    def self.delete_class(group, class_name)
-      if group["classes"].delete(class_name)
-        changes = true
-      else
-        changes = false
-      end
-
-      changes
-    end
 
-    def self.delete_param(group, class_name, param_name)
-      if  group["classes"].has_key?(class_name) and
-          group["classes"][class_name].delete(param_name)
+    # Classes are only removed when they have their parameters nilled so we must
+    # formulate special json to allow delete
+    # @see https://docs.puppet.com/pe/latest/nc_groups.html#post-v1groupsid
+    #
+    # Updates `group` to ensure that it now contains `class_name` (or marks it
+    # for deletion).  To commit changes, need to pass the updated
+    # `group['class']` hash to `update_group`
+    def self.ensure_class(group, class_name, delete:false)
+      if group["classes"].has_key?(class_name) and delete
+        # delete class by nilling its parameters
+        group["classes"][class_name] = nil
         changes = true
-      else
-        changes = false
-      end
-      changes
-    end
-
-    def self.ensure_class(group, class_name)
-      if ! group["classes"].has_key?(class_name)
+      elsif ! group["classes"].has_key?(class_name) and ! delete
+        # create class because we are not deleting it and it doesn't exist yet
         group["classes"][class_name] = {}
         changes = true
       else
@@ -248,12 +334,21 @@ module NCEdit
       changes
     end
 
-    def self.ensure_param(group, class_name, param_name, param_value)
+    # Updates `group` to ensure that it now contains `param_name` set to
+    # `param_value` (or marks the parameter it for deletion).  To commit changes
+    # , need to pass the updated `group['class']` hash to `update_group`
+    def self.ensure_param(group, class_name, param_name, param_value, delete:false)
       # ensure parameter set if specified
-      if ! group["classes"][class_name].has_key?(param_name) or
-          group["classes"][class_name][param_name] != param_value
+      if ! delete and (
+            ! group["classes"][class_name].has_key?(param_name) or
+            group["classes"][class_name][param_name] != param_value
+          )
+        # update or add a new parameter
         group["classes"][class_name][param_name] = param_value
         changes = true
+      elsif delete and group["classes"][class_name].has_key?(param_name)
+        group["classes"][class_name][param_name] = nil
+        changes = true
       else
         changes = false
       end
@@ -261,40 +356,19 @@ module NCEdit
       changes
     end
 
-    def self.delete_classes(group_name, data)
-      updated = false
-      if data
-        data.each{ |class_name|
-          Escort::Logger.output.puts "Deleting class: #{group_name}->#{class_name}"
-          updated |= delete_class(nc_group(group_name), class_name)
-        }
-      end
-      updated
-    end
-
-    def self.delete_params(group_name, data)
-      updated = false
-      if data
-        data.each{ |class_name, param_names|
-          param_names.each { |param_name|
-            Escort::Logger.output.puts "Deleting param: #{group_name}->#{class_name}=>#{param_name}"
-            updated |= delete_param(nc_group(group_name), class_name, param_name)
-          }
-        }
-      end
-      updated
-    end
-
-    def self.ensure_classes_and_params(group_name, data)
+    # Updates `group` to ensure that it now contains classes and parameters as
+    # specified in the `data` paramater.  To commit changes, need to pass the
+    # updated `group['class']` hash to `update_group`
+    def self.ensure_classes_and_params(group, data)
       updated = false
       if data
         data.each{ |class_name, params|
-          Escort::Logger.output.puts "ensuring class: #{group_name}->#{class_name}"
-          updated |= ensure_class(nc_group(group_name), class_name)
+          Escort::Logger.output.puts "ensuring class: #{group['name']}->#{class_name}"
+          updated |= ensure_class(group, class_name)
           if params
             params.each { |param_name, param_value|
-              Escort::Logger.output.puts "ensuring param: #{group_name}->#{class_name}->#{param_name}=#{param_value}"
-              updated |= ensure_param(nc_group(group_name), class_name, param_name, param_value)
+              Escort::Logger.output.puts "ensuring param: #{group['name']}->#{class_name}->#{param_name}=#{param_value}"
+              updated |= ensure_param(group, class_name, param_name, param_value)
             }
           end
         }
@@ -302,7 +376,7 @@ module NCEdit
       updated
     end
 
-    # Ensure a partualar rule exists in the group["rule"] array
+    # Ensure a particular rule exists in the group["rule"] array
     # This affects only the items in the chain, eg:
     # [
     #  "or",
@@ -313,6 +387,8 @@ module NCEdit
     #
     # Only the rule to be added in should be passed as the rule parameter, eg:
     # ["=", "name", "bob"]
+    #
+    # To commit changes, need to pass the updated `group['rule']` hash to `update_group`
     def self.ensure_rule(group, rule)
       updated = false
 
@@ -321,7 +397,7 @@ module NCEdit
 
       # rules are nested like this, the "or" applies to the whole rule chain:
       # "rule"=>["or", ["=", "name", "bob"], ["=", "name", "hello"]]
-      group["rule"][1].each {|system_rule|
+      group["rule"].drop(1).each {|system_rule|
         if  system_rule[0] == rule[0] and
             system_rule[1] == rule[1] and
             system_rule[2] == rule[2]
@@ -331,33 +407,43 @@ module NCEdit
       }
       if ! found
         Escort::Logger.output.puts "Appending rule: #{rule}"
-        group["rule"][1] << rule
+        group["rule"].push(rule)
         updated = true
       end
 
       updated
     end
 
+    # Modify `group` to ensure the passed in `rules` exist. To commit changes,
+    # need to pass the updated `group['rule']` hash to `update_group`
+    #
     # rules need to arrive like this:
     # ["or", ["=", "name", "pupper.megacorp.com"], ["=", "name", "pupper.megacorp.com"]]
     # since the rule conjunction "or" can only be specified once per rule chain
     # we will replace whatever already exists in the rule with what the user
     # specified
-    def self.ensure_rules(group_name, rules)
+    def self.ensure_rules(group, rules)
       updated = false
-      group = nc_group(group_name)
+
       if ! group["rule"] or group["rule"].empty?
         # no rules yet - just add our new one
-        group["rule"] = [DEFAULT_RULE,[]]
+        group["rule"] = [DEFAULT_RULE]
       end
       updated |= ensure_rule_conjunction(group, rules[0])
-      rules[1].each { |rule|
+      rules.drop(1).each { |rule|
         updated |= ensure_rule(group, rule)
       }
 
       updated
     end
 
+    # Ensure the correct boolean conjunction ('and'/'or' - 'not' is not allowed)
+    # is being used for a given rule chain.  If user tried to append a rule with
+    # a different conjuction to the one currently in use we will change the
+    # conjuction used on the entire chain to match.
+    #
+    # Updates `group` in-place, To commit changes, need to pass the updated
+    # `group['rule']` hash to `update_group`
     def self.ensure_rule_conjunction(group, op)
       updated = false
       if ["and", "or"].include?(op)
@@ -371,5 +457,194 @@ module NCEdit
 
       updated
     end
+
+    # process any rule changes separately since they are valid for all actions
+    # returns true if changes were made
+    def self.rule_change(group, rule, rule_mode)
+      rule_change = false
+
+      rule_modes = ['replace', 'append']
+      if rule and (! rule_modes.include?(rule_mode))
+        raise "Invalid rule mode '#{rule_mode}'.  Allowed: #{rule_modes}"
+      end
+
+      if rule
+        begin
+          rule_json = JSON.parse(rule)
+        rescue JSON::ParserError
+          raise "Syntax error in data supplied to --rule (must be valid JSON)"
+        end
+
+        if rule_mode == 'replace'
+          if group['rule'] != rule_json
+            group['rule'] = rule_json
+            rule_change = true
+          end
+        else
+          rule_change = ensure_rules(group, rule_json)
+        end
+      end
+
+      rule_change
+    end
+
+    def self.classes(options)
+      group_name    = options[:group_name]
+      class_name    = options[:class_name]
+      param_name    = options[:param_name]
+      param_value   = options[:param_value]
+      delete_class  = options[:delete_class]
+      delete_param  = options[:delete_param]
+      rule          = options[:rule]
+      rule_mode     = options[:rule_mode]
+      smart_update  = options[:smart_update]
+
+      rule_change   = false
+      class_change  = false
+
+      if group_name
+        group = nc_group(group_name)
+      else
+        raise "All operations require a valid group_name"
+      end
+
+      if class_name and delete_class
+        # delete a class from a group
+        Escort::Logger.output.puts "Deleting class #{class_name} from #{group_name}"
+        class_change = ensure_class(group, class_name, delete:true)
+      elsif class_name and param_name and delete_param
+        # delete a parameter from a class
+        Escort::Logger.output.puts "Deleting parameter #{param_name} on #{class_name} from #{group_name}"
+        class_change = ensure_class(group, class_name)
+        class_change |= ensure_param(group, class_name, param_name, nil, delete:true)
+      elsif class_name and param_name and param_value
+        # set a value inside a class
+        if smart_update
+          if ! is_r10k_param(class_name, param_name)
+
+            # not an R10K parameter, do an immediate update to make sure any classes
+            # we need are in-place
+            puppet_code_deploy
+          end
+        end
+        Escort::Logger.output.puts "Setting parameter #{param_name} to #{param_value} on #{class_name} in #{group_name}"
+        class_change = ensure_class(group, class_name)
+        class_change |= ensure_param(group, class_name, param_name, param_value)
+      elsif class_name
+        if smart_update
+          puppet_code_deploy
+        end
+        Escort::Logger.output.puts "Adding #{class_name} to #{group_name}"
+        class_change = ensure_class(group, class_name)
+      end
+
+      # process any rule changes separately since they are valid for all actions
+      rule_change = rule_change(group, rule, rule_mode)
+
+      # save changes
+      if class_change or rule_change
+        update_group(group_name, classes: group["classes"], rule: group["rule"])
+      else
+        Escort::Logger.output.puts "Already up-to-date"
+      end
+    end
+
+    def self.update_classes
+      if ! @puppetclassify
+        init
+      end
+      @puppet_https.delete("#{@puppet_url}/puppet-admin-api/v1/environment-cache")
+      @puppet_https.post("#{@rest_api_url}/v1/update-classes")
+    end
+
+    def self.puppet_code_deploy
+      Escort::Logger.output.puts "Running puppet and deploying code..."
+
+      # if puppet run fails, just deploy the code anyway as the fix needed might
+      # be in the update...
+      system("puppet agent -t ; puppet-code deploy --all --wait")
+    end
+
+    # Extract the r10k settings ONLY from the passed in data has and update
+    # NCAPI with them immediately
+    def self.apply_r10k_settings_now(data)
+      data.each { |group_name,opts|
+        group = nc_group(group_name)
+
+        if opts.key?("classes")
+          Escort::Logger.output.puts "Setting up rules for #{group_name} immediately"
+          if ensure_classes_and_params(group, opts["classes"])
+            update_group(group_name, classes: group["classes"])
+          end
+        end
+      }
+    end
+
+    # Check if this is an r10k parameter or not
+    #
+    # @return true if this is to do with R10K otherwise false
+    def self.is_r10k_param(class_name, param_name)
+      class_name == R10K_SETTINGS_CLASS and R10K_SETTINGS_PARAMS.include?(param_name)
+    end
+
+    # Evaluate whether change instructions contain R10K settings or not
+    #
+    # @param data Hash of data settings to look at
+    # @return false if not found otherwise hash of with just the settings that
+    #   need to be applied immediately
+    def self.contains_r10k_settings(data)
+      found = {}
+      data.each { |group_name, opts|
+        if opts.key?("classes") and opts["classes"].key?(R10K_SETTINGS_CLASS)
+          opts["classes"][R10K_SETTINGS_CLASS].each { |param_name,param_value|
+            if is_r10k_param(R10K_SETTINGS_CLASS, param_name)
+              # make the hash structure we need
+              if ! found.key?(group_name)
+                found[group_name] = {}
+              end
+
+              if ! found[group_name].key?("classes")
+                found[group_name]["classes"] = {}
+              end
+
+              if ! found[group_name]["classes"].key?(R10K_SETTINGS_CLASS)
+                found[group_name]["classes"][R10K_SETTINGS_CLASS] = {}
+              end
+
+              found[group_name]["classes"][R10K_SETTINGS_CLASS][param_name] = param_value
+            end
+          }
+        end
+      }
+
+      # Return the found elements if there were any, otherwise simplify to false
+      ! found.empty? ? found : false
+    end
+
+    def self.groups(options)
+      group_name          = options[:group_name]
+      environment         = options[:environment]
+      environment_trumps  = options[:environment_trumps]
+      rule                = options[:rule]
+      rule_mode           = options[:rule_mode]
+
+      # step 1: create the group with the parent "All Environments"
+      nc_group(group_name, parent_name: "All Environments")
+
+      # step 2: set the environment + environment trumps
+      update_group(
+        group_name,
+        environment: environment,
+        environment_trumps: environment_trumps,
+      )
+
+      # step 3: set the rules - separate step because rule_mode needs special
+      # handling
+      group = nc_group(group_name)
+      rule_change = rule_change(group, rule, rule_mode)
+      if rule_change
+        update_group(group_name, rule: group["rule"])
+      end
+    end
   end
 end

data/lib/ncedit/version.rb

@@ -1,3 +1,3 @@
 module NCEdit
-  VERSION = "0.1.0"
+  VERSION = "0.5.0"
 end

data/ncedit.gemspec

@@ -6,12 +6,13 @@ require 'ncedit/version'
 Gem::Specification.new do |spec|
   spec.name          = "ncedit"
   spec.version       = NCEdit::VERSION
-  spec.authors       = ["Geoff Williams"]
-  spec.email         = ["geoff@geoffwilliams.me.uk"]
+  spec.authors       = ["Declarative Systems"]
+  spec.email         = ["sales@declarativesystems.com"]
+  spec.license       = "Apache-2.0"
 
   spec.summary       = %q{Edit Puppet Enterprise Node Classifier rules}
   spec.description   = %q{Use the puppet-classify gem to create/edit NC rules}
-  spec.homepage      = "https://github.com/GeoffWilliams/ncedit"
+  spec.homepage      = "https://github.com/declarativesystems/ncedit"
 
 
   spec.files         = `git ls-files -z`.split("\x0").reject do |f|
@@ -21,11 +22,11 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.add_development_dependency "bundler", "~> 1.14"
-  spec.add_development_dependency "rake", "~> 10.0"
-  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_development_dependency "bundler", "~> 2.1"
+  spec.add_development_dependency "rake", "~> 13.0"
+  spec.add_development_dependency "rspec", "~> 3.9"
 
   spec.add_runtime_dependency "escort", "0.4.0"
-  spec.add_runtime_dependency "json", "2.0.3"
-  spec.add_runtime_dependency "puppetclassify", "0.1.5"
+  spec.add_runtime_dependency "json_pure", "2.3.0"
+  spec.add_runtime_dependency "puppetclassify", "0.1.8"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ncedit
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.5.0
 platform: ruby
 authors:
-- Geoff Williams
+- Declarative Systems
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-03-15 00:00:00.000000000 Z
+date: 2020-06-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -16,42 +16,42 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.14'
+        version: '2.1'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.14'
+        version: '2.1'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '13.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '13.0'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '3.9'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '3.9'
 - !ruby/object:Gem::Dependency
   name: escort
   requirement: !ruby/object:Gem::Requirement
@@ -67,36 +67,36 @@ dependencies:
       - !ruby/object:Gem::Version
         version: 0.4.0
 - !ruby/object:Gem::Dependency
-  name: json
+  name: json_pure
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 2.0.3
+        version: 2.3.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 2.0.3
+        version: 2.3.0
 - !ruby/object:Gem::Dependency
   name: puppetclassify
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 0.1.5
+        version: 0.1.8
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: 0.1.5
+        version: 0.1.8
 description: Use the puppet-classify gem to create/edit NC rules
 email:
-- geoff@geoffwilliams.me.uk
+- sales@declarativesystems.com
 executables:
 - ncedit
 extensions: []
@@ -107,6 +107,7 @@ files:
 - ".travis.yml"
 - Gemfile
 - LICENSE
+- Makefile
 - README.md
 - Rakefile
 - bin/console
@@ -118,8 +119,9 @@ files:
 - lib/ncedit/cmd.rb
 - lib/ncedit/version.rb
 - ncedit.gemspec
-homepage: https://github.com/GeoffWilliams/ncedit
-licenses: []
+homepage: https://github.com/declarativesystems/ncedit
+licenses:
+- Apache-2.0
 metadata: {}
 post_install_message: 
 rdoc_options: []
@@ -136,8 +138,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.5.2
+rubygems_version: 3.0.3
 signing_key: 
 specification_version: 4
 summary: Edit Puppet Enterprise Node Classifier rules