noms-command 0.5.0

data/ROADMAP.rst

@@ -0,0 +1,127 @@
+noms-command Roadmap
+====================
+
+Roadmap
+-------
+
+These items are necessary and a fairly well-identified goal.
+
+Authentication
+~~~~~~~~~~~~~~
+
+Authentication needs to be handled in a rich and correct way. One of
+the original inspirations for replacing the **noms** v1 thick client
+was the daunting nature of adding authentication code to all the different
+client libraries for the NOMS_ components and then exposing an interface
+to dealing with them to the command-line.
+
+Instead, all the web plumbing and dealing with things like how to
+present persistent web sessions to a user are going to be handled by the
+general purpose web application client **noms**, and services like the
+NOMS CMDB, NCC-API, NagUI-API, etc. can provide their own command-line
+implementations. The NOMS_ components are likely to use standard Javascript
+libraries to present a consistent look, feel and implementation, but
+**noms** will provide the ability for a service provider to define a CLI
+that looks and feels any way they want.
+
+That means **noms** should be good at command-line web authentication. The
+old v1 client could not do more than Basic authentication, and this necessitated
+the storage of personal credentials in plaintext on the system, mixed with
+other configuration like the ReST enpoint URLs and usage conveniences like
+default values.
+
+Instead **noms** will handle many authentication flavors in a way that is
+as secure as possible:
+
+============================ =====================================================
+Authentication Type          Description
+============================ =====================================================
+Username/password (basic,    **noms** will prompt the user for credentials when
+digest)                      receives the authorization required HTTP status, and
+                             persist the result in an obfuscated, expiring form
+                             for that origin.
+---------------------------- -----------------------------------------------------
+Login URL with cookie-based  **noms** honors redirects to a login URL in the
+sessions                     same way as a web browser and stores cookies
+                             using proper cookie expiration. **noms** doesn't
+                             do HTML parsing, so authentication on the login
+                             URL must be of some other type or scriptable
+                             with Javascript.
+
+                             This implies that Javascript must be able to
+                             script the authentication dialog, which means
+                             there must be some way to do prompting; possibly
+                             with the ``window.prompt()`` interface.
+---------------------------- -----------------------------------------------------
+Login URL with special token **noms** will offer a way for javascripts to persist
+to be included in request    state across requests. This mechanism will allow
+headers                      a javascript to set special headers that it will
+                             have access to for later requests.
+---------------------------- -----------------------------------------------------
+Login URL with special token Similar to above, with a somewhat different request
+to be included in request    implementation.
+bodies
+---------------------------- -----------------------------------------------------
+OAuth                        **noms** will use httpclient's built-in OAuth
+                             and prompt the user to authenticate with OAuth, and
+                             use the OAuth token for subsequent requests.
+---------------------------- -----------------------------------------------------
+Client certificate           Prompt for passphrase or use authentication agent
+                             to provide client certificate.
+============================ =====================================================
+
+Storage
+~~~~~~~
+
+In order to persist state across requests, **noms** should probably
+have a `Web Storage`_ implementation.
+
+.. _`Web Storage`: http://dev.w3.org/html5/webstorage/
+
+Caching
+~~~~~~~
+
+HTTP provides a rich way to control caching of resources, and **noms** should
+honor these strictly for efficiency. It will honor ``Cache-control`` headers
+and use ``ETags`` and ``If-Modified`` appropriately to avoid loading
+application documents, scripts and even data unnecessarily. This should reduce
+many of the inefficiencies associated with serving an interface from the server.
+
+Crossroads
+----------
+
+This is not a roadmap, but a series of ideas of how **noms** could be enhanced as
+well as unanswered questions about how it should work.
+
+I/O
+~~~
+
+Its prototype does no local I/O. Outside of restricted situations like
+`Web Storage`_ this is probably desirable. It's extremely typical of CLIs, even
+those for "remote" data stores, to be able to do some I/O, and here are a couple
+of ideas how:
+
+* stdin - Right now there's no way to even read stdin. So if you want to make a
+  CLI for uploading batch data you can't even do it--your scripts only have access
+  to command-line arguments.
+
+  Use node.js-style `Readable Stream`_ implementation to have access to stdin.
+
+.. _`Readable Stream`: https://nodejs.org/api/stream.html
+
+Another thing is the possibility of doing I/O on select named files. **noms** could
+perhaps pre-parse the command line and add the files mentioned on the command line
+to an ACL which would allow downloaded scripts to access them. For example::
+
+  noms http://cmdb/cmdb.json generate-ansible-inventory --output=inventory.json
+
+**noms** could scan the command line and guess at what files the user intends the
+application to have access to, and allow the Javascript to open them using a node.js-like
+stream implementation. Exactly how to do this safely would be a challenge. Are only
+certain options allowed? Files that already exist (otherwise it would be easy to allow
+the script to write to unintended files). No files with '..'? What about -oFile.json
+vs. -onf?
+
+Another I/O-related subject is that **noms** is currently completely request/response-
+oriented. It might be nice to be able to stream output data, or input data for file-
+or batch-upload type operations, or wait for events on a websocket.

data/Rakefile

@@ -0,0 +1,49 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new
+
+task :test => [:spec, :testcmd]
+
+task :testcmd do
+    ENV['RUBYLIB'] = ['lib', ENV['RUBYLIB']].join(':')
+    ENV['PATH'] = ['bin', ENV['PATH']].join(':')
+    Dir.new('spec').each do |script|
+        next unless script =~ /\.sh$/
+        puts script
+        system(File.join('spec', script))
+    end
+end
+
+# Start the DNC application web server on port 8787
+task :start do
+    FileUtils.rm_r 'test' if File.directory? 'test'
+    system 'cp -R fixture test'
+    system("sh -c '#{RbConfig.ruby} test/dnc.rb >test/dnc.out 2>&1 &'")
+end
+
+task :status do
+    begin
+        pid = File.read('test/dnc.pid').to_i
+        Process.kill 0, File.read('test/dnc.pid').to_i
+        puts "Test server running (PID #{pid})"
+    rescue Errno::ESRCH
+        puts "Test server not running on PID #{pid}"
+    rescue Errno::ENOENT
+        puts "Test server not running (no pidfile)"
+    end
+end
+
+task :sync do
+    system 'cp -R fixture/* test'
+end
+
+task :stop do
+    Process.kill 'TERM', File.read('test/dnc.pid').to_i
+    FileUtils.rm 'test/dnc.pid'
+end
+
+task :clean do
+    rm_rf ['pkg', 'test']
+end
+

data/TODO.rst

@@ -0,0 +1,7 @@
+TODO
+====
+
+* Flesh out example application CLI.
+* Auth sessions
+* Caching
+* SSL handling, TOFU for SSL

data/bin/noms2

@@ -0,0 +1,20 @@
+#!/usr/bin/env ruby
+
+# Copyright 2015 Evernote Corporation, all rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+require 'noms/command'
+
+Process.exit NOMS::Command.run(ARGV)

data/fixture/dnc.rb

@@ -0,0 +1,120 @@
+require 'sinatra/base'
+require 'json'
+
+# Implement Do Not Call List Example REST application
+# and static file server
+class DNC < Sinatra::Application
+
+    set :port, 8787
+    set :root, File.expand_path("#{File.dirname(__FILE__)}")
+    enable :static
+
+    File.open(File.join(settings.root, 'dnc.pid'), 'w') {|f| f.puts Process.pid }
+
+    def load_data
+        JSON.load(File.open(File.join(settings.root, 'public', 'files', 'data.json')))
+    end
+
+    def write_data(data)
+        File.open(File.join(settings.root, 'public', 'files', 'data.json'), 'w') { |fh| fh << data.to_json }
+    end
+
+    helpers do
+        def require_auth
+            return if authorized?
+            headers['WWW-Authenticate'] = 'Basic realm="Authorization Required"'
+            halt 401, "Not authorized\n"
+        end
+
+        def authorized?
+            @auth ||=  Rack::Auth::Basic::Request.new(request.env)
+            @auth.provided? and @auth.basic? and @auth.credentials and @auth.credentials == ['testuser', 'testpass']
+        end
+    end
+
+    get '/readme' do
+        redirect 'https://raw.githubusercontent.com/en-jbrinkley/noms-command/master/README.rst', 'README'
+    end
+
+    get '/dnc' do
+        data = load_data
+        if request.query_string.empty?
+            [ 200, { 'Content-type' => 'application/json'},
+                JSON.pretty_generate(data) ]
+        else
+            [ 200, { 'Content-type' => 'application/json' },
+                JSON.pretty_generate(
+                                     data.select do |item|
+                                         params.keys.all? { |k| item[k.to_s] && item[k.to_s].to_s === params[k] }
+                                     end)
+            ]
+        end
+    end
+
+    get '/dnc/:id' do
+        data = load_data
+        object = data.find { |e| e['id'] == params[:id].to_i }
+        if object
+            [ 200, { 'Content-type' => 'application/json' },
+                JSON.pretty_generate(object) ]
+        else
+            404
+        end
+    end
+
+    post '/dnc' do
+        request.body.rewind
+        new_object = JSON.parse request.body.read
+
+        data = load_data
+        # How unsafe is this?
+        new_object['id'] = data.map { |e| e['id'] }.max + 1
+        data << new_object
+        write_data data
+
+        [ 201, { 'Content-type' => 'application/json' },
+            JSON.pretty_generate(new_object) ]
+    end
+
+    put '/dnc/:id' do
+        request.body.rewind
+        new_object = JSON.parse request.body.read
+
+        data = load_data
+        new_data = data.reject { |e| e['id'] == params[:id].to_i }
+        if new_data.size == data.size
+            404
+        else
+            new_object['id'] = params[:id].to_i
+            new_data << new_object
+            write_data new_data
+
+            [ 200, { 'Content-type' => 'application/json' },
+                JSON.pretty_generate(new_object) ]
+        end
+    end
+
+    delete '/dnc/:id' do
+        data = load_data
+        new_data = data.reject { |e| e['id'] == params[:id].to_i }
+
+        if new_data.size == data.size
+            404
+        else
+            write_data new_data
+            204
+        end
+    end
+
+    get '/alt/dnc.json' do
+        redirect to('/dnc.json')
+    end
+
+    get '/auth/dnc.json' do
+        require_auth
+        redirect to('/dnc.json')
+    end
+
+    run! if app_file = $0
+
+end

data/fixture/identity

@@ -0,0 +1,5 @@
+{ "id": "Authorization+Required=http://localhost:8787/",
+  "realm": "Authorization Required",
+  "domain": "http://localhost:8787/",
+  "username": "testuser",
+  "password": "testpass" }

data/fixture/public/dnc.json

@@ -0,0 +1,22 @@
+{ "$doctype": "noms-v2",
+  "$comment": [
+      "noms2 http://localhost:8787/dnc.json",
+      "noms2 http://localhost:8787/dnc.json list"
+  ],
+  "$script": [
+      { "$source": "https://rawgit.com/jfd/optparse-js/v1.0.5/lib/optparse.js",
+        "$comment": "Optparse.js 1.0.3 - https://github.com/jfd/optparse-js; via rawgit.com" },
+      { "$source": "https://rawgit.com/douglascrockford/JSON-js/master/json2.js",
+        "$comment": "JSON in JavaScript - https://github.com/douglascrockford/JSON-js" },
+      { "$source": "lib/noms-args.js" },
+      { "$source": "lib/dnc.js" }
+  ],
+  "$body": [
+      "Usage:",
+      "   noms dnc query <field>=<value>",
+      "   noms dnc add <field>=<value> [<field>=<value> [...]]",
+      "   noms dnc remove <id>",
+      "   noms dnc check { <phone> | <name> }",
+      "   noms dnc list"
+  ]
+}

data/fixture/public/echo.json

@@ -0,0 +1,7 @@
+{ "$doctype": "noms-v2",
+  "$comment": "noms2 http://localhost:8787/echo.json one two three",
+  "$body": [],
+  "$script": [
+      "if (document.argv.length > 1) { document.body = document.argv.slice(1).join(\" \"); } else { line1 = prompt('String to echo: '); line2 = prompt(\"Password to echo (don't use real one): \", false); document.body = [line1, line2]; }"
+  ]
+}

data/fixture/public/files/data.json

@@ -0,0 +1,12 @@
+[
+{"id":1,"name":"Manuela Irwin","street":"427 Maple Ln","city":"Arlington, TX  76010","phone":"(817) 555-0427"},
+{"id":2,"name":"Ronda Sheppard","street":"801 New First Rd","city":"Providence, RI  02940","phone":"(401) 555-0801"},
+{"id":3,"name":"Leonor Foreman","street":"428 Willow Rd","city":"Providence, RI  02940","phone":"(401) 555-0428"},
+{"id":4,"name":"Emma Roman","street":"589 Flanty Terr","city":"Anderson, IN  46018","phone":"(317) 555-0589"},
+{"id":5,"name":"Frieda English","street":"930 Stonehedge Blvd","city":"Chicago, IL  60607","phone":"(312) 555-0930"},
+{"id":6,"name":"Kitty Morton","street":"618 Manchester St","city":"Richmond, VA  23232","phone":"(804) 555-0618"},
+{"id":7,"name":"Kathy Mcleod","street":"52 Wommert Ln","city":"Binghamton, NY  13902","phone":"(607) 555-0052"},
+{"id":8,"name":"Bettie Wolfe","street":"523 Sharon Rd","city":"Coward, SC  29530","phone":"(843) 555-0523"},
+{"id":9,"name":"Vanessa Conway","street":"885 Old Pinbrick Dr","city":"Athens, GA  30601","phone":"(404) 555-0885"},
+{"id":10,"name":"Ian Welch","street":"555 Hamlet St","city":"Arlington, TX  76010","phone":"(817) 555-0555"}
+]

data/fixture/public/files/foo.json

@@ -0,0 +1,2 @@
+{ "$doctype": "noms-v2",
+  "$body": ["Test output for foo.json"] }

data/fixture/public/lib/dnc.js

@@ -0,0 +1,81 @@
+if (document.argv.length > 1) {
+    var argv = document.argv;
+    var me = argv.shift();
+    var command;
+    var format;
+    var xmlhttp = new XMLHttpRequest();
+
+    // document attributes can be set
+    // but are immutable
+    document.body = [ ];
+    var output = [ ];
+
+    var optspec = [
+        ["-J", "--json", "Display JSON"],
+        ["-Y", "--yaml", "Display YAML"],
+        ["-C", "--csv", "Display CSV"],
+        ["-v", "--verbose", "Enable verbose output"],
+        ["--nofeedback", "Don't print feedback"]
+    ];
+
+    var parser = new optparse.OptionParser(optspec);
+    var options = {
+        "feedback": true,
+        "format": "default",
+        "verbose": false
+    };
+    var args = [ ];
+
+    parser.on("verbose", function() { options["verbose"] = true });
+    parser.on("json", function() {
+        options["format"] = "json";
+        options["feedback"] = false;
+    });
+    parser.on("yaml", function() {
+        options["format"] = "yaml";
+        options["feedback"] = false;
+    });
+    parser.on("csv", function() {
+        options["format"] = "csv";
+        options["feedback"] = false;
+    });
+    parser.on("nofeedback", function() { options["feedback"] = false; });
+    parser.on(0, function(arg) { command = arg });
+    parser.on(function(arg) { args.push(arg); });
+
+    parser.parse(argv);
+
+    switch(command) {
+    case "list":
+        if (options["format"] === "default") {
+            format = "lines";
+        } else {
+            format = options["format"];
+        }
+        xmlhttp.open("GET", "/dnc", false);
+        xmlhttp.send();
+        var records = eval('(' + xmlhttp.responseText + ')');
+        output.push(
+            {
+                '$type': 'object-list',
+                '$format': format,
+                '$columns': [
+                    { 'field': 'id', 'width': 3, 'align': 'right' },
+                    { 'field': 'name', 'width': 20 },
+                    { 'field': 'phone', 'width': 20 }
+                ],
+                '$data': records
+            });
+        if (options["feedback"]) {
+            output.push(records.length + " objects");
+        }
+        break;
+    default:
+        document.exitcode = 8;
+        window.alert(
+            me + " error: Unknown command '" + command + "'"
+        );
+    }
+
+    document.body = output;
+}

data/fixture/public/lib/noms-args.js

@@ -0,0 +1,13 @@
+var nomsargs = { };
+
+(function (self) {
+    function kwargs() {
+        var result = { };
+        for (i = 0; i < arguments.length; i++) {
+            pair = arguments[i].split("=", 2);
+            result[pair[0]] = pair[1];
+        }
+        return result;
+    }
+
+})(nomsargs);

data/fixture/public/lib/showopt.js

@@ -0,0 +1,18 @@
+var optspec = [
+    ["-n", "--dry-run", "Don't change anything"],
+    ["-d", "--debug", "Enable debugging output"],
+    ["-c", "--config FILE", "Specify configuration file"]
+];
+
+var parser = new optparse.OptionParser(optspec);
+var options = { "dry-run": false,
+                "debug": false,
+                "config": null }
+
+parser.on("dry-run", function () { options["dry-run"] = true })
+parser.on("debug", function() { options["debug"] = true })
+parser.on("config", function(opt, file) { options["config"] = file })
+
+parser.parse(document.argv)
+
+document.body = options

data/fixture/public/location.json

@@ -0,0 +1,8 @@
+{ "$doctype": "noms-v2",
+  "$script": ["var out = { };",
+              "out.protocol = location.protocol;",
+              "out.host = location.host;",
+              "out.href = location.href;",
+              "document.body = out;"
+             ],
+  "$body": [] }

data/fixture/public/showopt.json

@@ -0,0 +1,15 @@
+{ "$doctype": "noms-v2",
+  "$comment": [
+      "noms2 http://localhost:8787/showopt.json",
+      "noms2 http://localhost:8787/showopt.json --debug --config /dev/null"
+  ],
+  "$body": [
+      "Usage:",
+      "  showopt [--dry-run] [--config <s>] [--verbose]"
+  ],
+  "$script": [
+      { "$source": "https://rawgit.com/jfd/optparse-js/v1.0.5/lib/optparse.js",
+        "$comment": "Optparse.js 1.0.3 - https://github.com/jfd/optparse-js; via rawgit.com" },
+      { "$source": "lib/showopt.js" }
+  ]
+}

data/fixture/rig2json

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+echo '['
+id=0
+
+while read name
+do
+    let id=id+1
+    read street
+    read city
+    read phone
+    number=$(echo $street | cut -f1 -d\ )
+    number=555-$(printf '%04d' $number)
+    phone=$(echo $phone | sed "s/xxx-xxxx/$number/")
+    echo -n "{'id':$id,'name':'$name','street':'$street','city':'$city','phone':'$phone'}" | tr "'" '"'
+    read blank || break
+    echo ,
+done
+
+echo
+echo ']'