noms-command 0.5.0 → 2.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 0bcefa4ec6a121ab3dc7f9be3c7242a87f58203c
-  data.tar.gz: f12c00d11f170af8cb53f4ed3349d20d9ee74505
+  metadata.gz: a2ac7a3782b3dc374c77d11d618d66ac99d37c45
+  data.tar.gz: 241d7122a0756c8f8f5229d6ebfabcbd66eedcba
 SHA512:
-  metadata.gz: f6d50e6bb50f761f49f766ffec47db3c8fbddadcc55cfa78223c99ffb2ceffcced639cc08e5a299d1bdcf3c11035cb337da761db5e3eb39532b40724bd465213
-  data.tar.gz: 5c4e2f8970d23764a8cfabe04448748c2f9f717b16bd8073808ad35ca1b377b6f6558e113a99fe487b2540ffd6b57067055545b477e63084d4edf3fca279faa8
+  metadata.gz: 760eb723b068038cdf9a65b082c22610a44ebcda16fe745ec2d7b415498d399bf36d89a1b162e18d026f6c80a7a5061a926a99691fde8c68857707eb2b389489
+  data.tar.gz: dcbd6336014c56f90e70d9e64957650fe5aa61c287ffb0c1be0f8b1b2892d620fd72204dcba9ebf5f9771f1df320382e26a1a5f489bcca6164b1b7a8d9fce559

data/README.rst

@@ -54,18 +54,18 @@ Syntax
 
 The basic way of invoking a **noms** command is as follows::
 
-  noms *url* *options* *arguments*
+  noms <url> <options> <arguments>
 
 **noms** invokes the app at *url* with the given options and
- arguments, displaying the results.
+arguments, displaying the results.
 
 **noms** has its own options, which can be given before the
-*url*, and are available when invoking ``noms --help`.
+*url*, and are available when invoking ``noms --help``.
 
 Bookmarks
 ~~~~~~~~~
 
-* ``noms *bookmark*[/arg] ...``
+* ``noms <bookmark>[/extra] ...``
 
 **noms** bookmarks are stored, by default, in ``~/.noms/bookmarks.json``,
 ``/usr/local/etc/noms/bookmarks.json``, and ``/etc/noms/bookmarks.json``
@@ -101,7 +101,8 @@ strictness policies may be configurable in a file, but the point of
 **noms** is to remove the distribution of thick client libraries that
 try to abstract away server-side interfaces and the configurations
 they require, so it would defeat part of its purpose to allow rich
-configuration of things like default values for objects and so forth.
+configuration of things like default values for objects, extra
+(application-level) connection or protocol information and so forth.
 
 Implementation
 --------------
@@ -119,28 +120,71 @@ according to `Dynamic Doctype`_, below. Otherwise, it is assumed to be
 structured list or object data, and **noms** will render the object or
 array using its default format (usually YAML).
 
+Otherwise, **noms** assumes the data may be binary and will send it
+to stdout as long as stdout is not a terminal.
+
 Authentication
 --------------
 
-*NOTE:* This is under active development, expect it to change
- frequently.
-
-**noms** supports Basic authentication. It will prompt the user for
-a username and password when required. A saved identity file can be
-provided with the ``--identity`` option to **noms** for non-interactive
-use.
+**noms** supports Basic authentication. It prompts the user for a
+username and password when authorization is required, and saves the
+authentication identity information in a "vault" to be used for future
+invocations (that is, you don't need to enter a password for every
+**noms** command run).
+
+If you are "idle" for an hour (have not used the saved authentication
+vault for any web transactions), **noms** will time out the vault and
+you will need to re-authenticate.
+
+You can "close" the vault immediately by using the ``--logout``
+option. I strongly suggest you call this in your ``~/.logout``
+mechanism so that it runs immediately when you have completed a
+login session.
+
+You can bypass the "vault" by running **noms** with the
+``--plaintext-identity`` option. This has the effect of caching these
+credentials permanently (well, until they are no longer good). You
+should only use this option to generate an identity file for service
+(non-interactive) accounts that need perpetual access to a web
+service. The file will appear in the ``~/.noms/identities`` directory
+in a file named for the SHA-1 hash of the munged authentication realm
+and domain.
+
+Such a saved identity file can be provided with the ``--identity`` option
+to **noms** for non-interactive use.
+
+Security
+~~~~~~~~
+
+**noms** uses strong cryptography to implement the "vault" metaphor, but
+this is only a metaphor and the security provided is limited. The vault
+is opened and closed by means of an on-disk key. Because you can easily
+remove it and because it times out when idle (when this happens, **noms**
+overwrites it the next time it tries to use it), it's significantly better
+than ``.netrc`` files or passwords in environment variables or on the
+command-line; and more ergonomic as well. In the future, this cryptographic
+service will be provided by an independent **noms-agent**, similar to
+ssh-agent or gpg-agent. These still have the same problem in that they
+reduce the security of the system to operating system mechanisms rather
+than cryptographic mechanisms, but it is still an improvement.
 
 Dynamic Doctype
-~~~~~~~~~~~~~~~
+---------------
 
 The dynamic doctype is the ``noms-v2`` type, which is an object with
 the following top-level attributes:
 
 ``$doctype``
-  Must be ``noms-v2``. In future, backwards-incompatible extensions may be implemented in ``noms-v3`` or higher doctypes.
+  Must be ``noms-v2``. In future, backwards-incompatible extensions
+  may be implemented in ``noms-v3`` or higher doctypes.
 
 ``$script``
-  An ordered array of scripts to fetch and evaluate; or Javascript text to evaluate directly.
+  An ordered array of script references to fetch and evaluate; or
+  Javascript strings to evaluate directly. A script reference
+  consists of object with a ``$source`` key, the value of which
+  is the URL of the script to load. Additional fields in the
+  object are ignored and can be used to document the origin
+  or license of the scripts.
 
 ``$argv``
   The arguments passed to the application. It's called ``$argv``
@@ -148,16 +192,18 @@ the following top-level attributes:
   invoked (that is, the bookmark or URL).
 
 ``$exitcode``
-  The unix process exit code with which **noms** will exit at the completion of the command.
+  The unix process exit code with which **noms** will exit at the
+  completion of the command.
 
 ``$body``
-  The body of the document is the data to display. See `Output Formatting`_ below.
+  The body of the document is the data to display. See `Output
+  Formatting`_ below.
 
 From the perspective of javascript executing within the application,
 these are accessible as properties of the global **document** object
 (e.g., ``document.argv`` is the array of arguments given on the **noms**
 command line; Javascript can set ``document.exitcode`` to determine
-**noms'** exit code.
+**noms'** exit code).
 
 Output Formatting
 ~~~~~~~~~~~~~~~~~
@@ -202,8 +248,6 @@ The following entities are allowed in the body of a **noms-v2** document:
 
     * **labels**: Default ``true``; whether to display header row with field labels
 
-    * **columns**: Field names, headings and widths
-
     * **data**: The objects to render
 
   * ``$type``: **object** An object has the following attributes:
@@ -251,7 +295,7 @@ Scripts have access to the following global objects:
   * **argv** - The arguments being invoked. The first element of this
     array is the first argument passed to **noms** itself (not the
     script it ultimately fetches, but how it's invoked, similar to
-    ``$1``.
+    ``$0``.
 
   * **exitcode** - The numeric exit code with which **noms** will
     exit. Initially 0.
@@ -264,20 +308,6 @@ Scripts have access to the following global objects:
 
 .. _`NOMS::Command::XMLHttpRequest`: http://www.rubydoc.info/gems/noms-command/NOMS/Command/XMLHttpRequest
 
-
-Web 1.0 vs Web 2.0
-------------------
-
-Like the "real web", **noms** commands can choose to do some
-calculation on the server and some on the client: **noms** doesn't
-care. You can use no ``$script`` tag at all and just calculate the
-entire document to be rendered in the client (though this currently
-doesn't allow for argument interpretation, in the future the
-arguments may be passed in request headers or **noms** may allow a way
-for them to show up in a query string or POST request--but **noms** is
-not really a command-line http client either). This is up to the
-application designer.
-
 Example Application
 -------------------
 
@@ -366,11 +396,48 @@ The example application is a very simple sinatra REST API to a data
 store consisting of a JSON file, and the static files comprising the
 Javascript source code and the **noms** application document.
 
-Running Examples
-----------------
+Hacking/Running Examples
+------------------------
+
+Use Ruby 1.9.3 or higher (e.g. if you need to set PATH
+so that Ruby 1.9 executables are found, do that:
+``export PATH=/usr/local/ruby1.9/bin:$PATH``.
 
 Use ``rake start`` to start the test webserver and run the
 example applications (see the comments inside the
 ``fixture/public/*.json`` files for syntax).
 
 Start with ``noms2 http://localhost:8787/echo.json hello world``.
+
+Workflow
+~~~~~~~~
+
+Set up your environment::
+
+  mkdir ~/.noms
+  echo '{ "dnc": "http://localhost:8787/dnc.json" }' >~/.noms/bookmarks.json
+  export PATH=`pwd`/bin:$PATH
+  expert RUBYLIB=lib
+  bundle install
+  noms2            # NOMS usage message
+  noms2 dnc        # dnc usage message
+
+Do ``rake start`` to start the webserver: web root is is ``test/``.
+
+Hack files in:
+
+* ``lib/`` - Ruby files for ``noms2`` command
+
+* ``fixture/dnc.rb`` - Sinatra app which is webserver for dnc app (serves
+  static files and implements rest interface).
+
+* ``fixture/public/dnc.json`` - App document for 'dnc' subcommand.
+
+* ``fixture/public/lib`` - Javascript files, ``dnc.js`` implements dnc
+  operations
+
+Do ``rake sync`` to sync over updated files from ``fixture`` and test
+(the webserver document root is ``test/public``).
+
+``noms2 -d`` produces debugging showing full stack traces for Javascript
+errors, ``console.log()`` output and web traffic.

data/TODO.rst

@@ -1,7 +1,7 @@
 TODO
 ====
 
-* Flesh out example application CLI.
-* Auth sessions
-* Caching
+* Warn for basic auth over http
+* Warn for plaintext identity
+* Caching - Consider switching to Typhoeus/libcurl
 * SSL handling, TOFU for SSL

data/fixture/dnc.rb

@@ -7,7 +7,7 @@ class DNC < Sinatra::Application
 
     set :port, 8787
     set :root, File.expand_path("#{File.dirname(__FILE__)}")
-    enable :static
+    enable :static, :sessions
 
     File.open(File.join(settings.root, 'dnc.pid'), 'w') {|f| f.puts Process.pid }
 
@@ -30,6 +30,41 @@ class DNC < Sinatra::Application
             @auth ||=  Rack::Auth::Basic::Request.new(request.env)
             @auth.provided? and @auth.basic? and @auth.credentials and @auth.credentials == ['testuser', 'testpass']
         end
+
+        def generated_body(h={})
+            JSON.pretty_generate({ 'generated' => Time.now.httpdate }.merge(h)) + "\n"
+        end
+
+        def require_cookie_auth
+            return if cookie_authorized?
+            redirect "/cookie/login?return_to=#{CGI.escape(request.path)}"
+        end
+
+        def cookie_authorized?
+            session[:userid] == 'testuser'
+        end
+    end
+
+    before do
+        content_type 'application/json'
+    end
+
+    get '/cookie/login' do
+        require_auth
+        session[:userid] = @auth.credentials.first
+        landing = params[:return_to] || '/cookie/home'
+        redirect landing
+    end
+
+    get '/cookie/home' do
+        require_cookie_auth
+        generated_body({'cookie_user' => session[:userid] })
+    end
+
+    get '/cookie/logout' do
+        old_userid = session[:userid]
+        session[:userid] = nil
+        generated_body({'message' => "#{old_userid} logged out"})
     end
 
     get '/readme' do
@@ -66,6 +101,8 @@ class DNC < Sinatra::Application
         request.body.rewind
         new_object = JSON.parse request.body.read
 
+        puts "POST for object: #{new_object.inspect}"
+
         data = load_data
         # How unsafe is this?
         new_object['id'] = data.map { |e| e['id'] }.max + 1
@@ -115,6 +152,78 @@ class DNC < Sinatra::Application
         redirect to('/dnc.json')
     end
 
+    get '/auth/ok' do
+        require_auth
+        "SUCCESS"
+    end
+
+    # Caching client should let sit in cache
+    # for 4s then refetch
+    get '/static/max-age-4' do
+        cache_control :max_age => 4
+        generated_body
+    end
+
+    # Caching client must always revalidate
+    # even within 4s
+    get '/static/must-revalidate' do
+        cache_control :must_revalidate, :max_age => 4
+        expires 4
+        etag "10"
+        generated_body
+    end
+
+    # Caching client must never cache
+    get '/static/no-cache' do
+        cache_control :no_cache
+        generated_body
+    end
+
+    # Caching client should let sit in cache
+    # for 4s then refetch
+    get '/static/expires-4' do
+        expires 4
+        generated_body
+    end
+
+    # Caching client should let sit in cache
+    # for 4s then revalidate using If-Modified-Since
+    get '/static/last-modified' do
+        expires 4
+        $static_time ||= Time.now
+        last_modified $static_time
+        generated_body
+    end
+
+    # Caching client should let sit in cache
+    # for 4s then revalidate using If-None-Match
+    get '/static/expires-4-changing' do
+        expires 4
+        etag Time.now.httpdate
+        generated_body
+    end
+
+    # Caching client should let sit in cache
+    # for 2s then revalidate using If-None-Match
+    get '/static/expires-2-constant' do
+        etag "10"
+        expires 2
+        generated_body
+    end
+
+    get '/static/long-cache' do
+        etag "11"
+        expires 100
+        generated_body
+    end
+
+    get '/auth/cacheable' do
+        require_auth
+        expires 100
+        etag "11"
+        generated_body
+    end
+
     run! if app_file = $0
 
 end

data/fixture/public/dnc.json

@@ -8,15 +8,16 @@
         "$comment": "Optparse.js 1.0.3 - https://github.com/jfd/optparse-js; via rawgit.com" },
       { "$source": "https://rawgit.com/douglascrockford/JSON-js/master/json2.js",
         "$comment": "JSON in JavaScript - https://github.com/douglascrockford/JSON-js" },
-      { "$source": "lib/noms-args.js" },
+      { "$source": "lib/nomsargs.js" },
       { "$source": "lib/dnc.js" }
   ],
   "$body": [
       "Usage:",
-      "   noms dnc query <field>=<value>",
+      "   noms dnc query [<field>=<value> [...]] [<field> [...]]",
+      "   noms dnc list [<field> [...]]",
+      "   noms dnc show id [<field> [...]]",
       "   noms dnc add <field>=<value> [<field>=<value> [...]]",
-      "   noms dnc remove <id>",
-      "   noms dnc check { <phone> | <name> }",
-      "   noms dnc list"
+      "   noms dnc set id <field>=<value> [<field>=<value> [...]]",
+      "   noms dnc remove <id>"
   ]
 }

data/fixture/public/lib/dnc.js

@@ -1,32 +1,42 @@
 if (document.argv.length > 1) {
     var argv = document.argv;
     var me = argv.shift();
-    var command;
     var format;
     var xmlhttp = new XMLHttpRequest();
 
-    // document attributes can be set
-    // but are immutable
-    document.body = [ ];
-    var output = [ ];
-
     var optspec = [
         ["-J", "--json", "Display JSON"],
         ["-Y", "--yaml", "Display YAML"],
         ["-C", "--csv", "Display CSV"],
         ["-v", "--verbose", "Enable verbose output"],
-        ["--nofeedback", "Don't print feedback"]
+        ["-q", "--terse", "Use terse output"],
+        ["--nofeedback", "Don't print feedback"],
+        ["--nolabel", "Don't print field names"],
+        ["--noheader", "Don't print column headings"]
     ];
 
+    // document attributes can be set
+    // but are immutable
+    document.body = [ ];
+    var output = [ ];
+
     var parser = new optparse.OptionParser(optspec);
     var options = {
-        "feedback": true,
         "format": "default",
-        "verbose": false
+        "verbose": false,
+        "feedback": true,
+        "label": true,
     };
-    var args = [ ];
 
-    parser.on("verbose", function() { options["verbose"] = true });
+    var field_config = {
+        'id':      { 'field': 'id', 'width': 3, 'align': 'right' },
+        'name':    { 'field': 'name', 'width': 20 },
+        'phone':   { 'field': 'phone', 'width': 20 },
+        'street':  { 'field': 'street', 'width': 40 },
+        'city':    { 'field': 'city', 'width': 31 }
+    }
+
+    parser.on("verbose", function() { options["verbose"] = true; });
     parser.on("json", function() {
         options["format"] = "json";
         options["feedback"] = false;
@@ -40,36 +50,173 @@ if (document.argv.length > 1) {
         options["feedback"] = false;
     });
     parser.on("nofeedback", function() { options["feedback"] = false; });
-    parser.on(0, function(arg) { command = arg });
-    parser.on(function(arg) { args.push(arg); });
+    parser.on("nolabel", function() { options["label"] = false; });
+    parser.on("terse", function() {
+        options["feedback"] = false;
+        options["label"] = false;
+    });
+    // parser.on(0, function(arg) { command = arg; });
+
+    var args = parser.parse(argv);
+    var command = args.shift();
 
-    parser.parse(argv);
+    var format, records, field_list;
 
     switch(command) {
-    case "list":
-        if (options["format"] === "default") {
-            format = "lines";
+
+    case "add":
+        var keywords = new nomsargs.NomsArgs(args);
+
+        xmlhttp.open("POST", "/dnc", false);
+        xmlhttp.setRequestHeader("Content-type", "application/json");
+        xmlhttp.send(JSON.stringify(keywords.assignment));
+
+        if (xmlhttp.status == 201) {
+            record = JSON.parse(xmlhttp.responseText);
+            output.push("Entry created with id " + record['id']);
+        } else {
+            alert("Error " + xmlhttp.status + " creating entry");
+            document.exitcode = 2;
+        }
+
+        break;
+
+    case "remove":
+        var id = args.shift();
+
+        if (id == undefined) {
+            alert("No id to remove");
+            document.exitcode = 1;
+        } else {
+            xmlhttp.open("DELETE", "/dnc/" + id, false);
+            xmlhttp.send();
+
+            if (xmlhttp.status == 404) {
+                alert("Entry " + id + " does not exist");
+            } else if (xmlhttp.status != 204) {
+                alert("Error deleting id " + id);
+                document.exitcode = 2;
+            }
+        }
+
+        break;
+
+    case "set":
+        // We are not doing upsert
+        var id = args.shift();
+        var keywords = new nomsargs.NomsArgs(args);
+
+        if (id == undefined) {
+            alert("No id to set");
+            document.exitcode = 1;
         } else {
-            format = options["format"];
+            xmlhttp.open("GET", "/dnc/" + id, false);
+            xmlhttp.send();
+
+            if (xmlhttp.status == 404) {
+                alert("Entry " + id + " does not exist (use add)");
+                document.exitcode = 2;
+            } else if (xmlhttp.status == 200) {
+                the_object = JSON.parse(xmlhttp.responseText);
+
+                // Update assigned fields in retrieved object
+                keywords.assignmentKeys().map(function (key) {
+                    the_object[key] = keywords.assignment[key];
+                });
+
+                xmlhttp.open("PUT", "/dnc/" + id, false);
+                xmlhttp.send(JSON.stringify(the_object));
+
+                if (xmlhttp.status == 200) {
+                    output.push("Entry " + id + " updated");
+                } else {
+                    alert("Error " + xmlhttp.status + " updating entry " + id);
+                    document.exitcode = 2;
+                }
+            }
+        }
+
+        break;
+
+    case "show":
+        var id = args.shift();
+        var field_list = args;
+        format = (options["format"] == "default" ? "record" : options["format"]);
+
+        if (id == undefined) {
+            alert("No id to show");
+            document.exitcode = 1;
+        } else {
+            xmlhttp.open("GET", "/dnc/" + id, false);
+            xmlhttp.send();
+
+            if (xmlhttp.status == 404) {
+                alert("Entry " + id + " not found");
+                document.exitcode = 2;
+            } else {
+                console.log("output for record");
+                record = JSON.parse(xmlhttp.responseText);
+
+                console.log(record);
+
+                output.push({
+                    '$type': 'object',
+                    '$format': format,
+                    '$labels': options["label"],
+                    '$fields': field_list,
+                    '$data': record
+                });
+            }
+        }
+
+        break;
+
+    case "query":
+        var keywords = new nomsargs.NomsArgs(args);
+        format = (options["format"] == "default" ? "lines" : options["format"]);
+        var query = keywords.query();
+        var field_list = keywords.extra;
+
+        if (field_list.length == 0) {
+            field_list = ['id', 'name', 'phone'];
         }
+        xmlhttp.open("GET", "/dnc?" + query, false);
+        xmlhttp.send();
+        records = JSON.parse(xmlhttp.responseText);
+
+        output.push(
+            {
+                '$type': 'object-list',
+                '$format': format,
+                '$labels': options["label"],
+                '$columns': field_list.map(function(item) { return field_config[item]; }),
+                '$data': records
+            });
+        if (options["feedback"]) {
+            output.push(records.length + " objects");
+        }
+        break;
+
+    case "list":
+        format = (options["format"] == "default" ? "lines" : options["format"]);
+
         xmlhttp.open("GET", "/dnc", false);
         xmlhttp.send();
-        var records = eval('(' + xmlhttp.responseText + ')');
+        records = JSON.parse(xmlhttp.responseText);
+        field_list = (args.length == 0 ? ['id', 'name', 'phone'] : args)
         output.push(
             {
                 '$type': 'object-list',
                 '$format': format,
-                '$columns': [
-                    { 'field': 'id', 'width': 3, 'align': 'right' },
-                    { 'field': 'name', 'width': 20 },
-                    { 'field': 'phone', 'width': 20 }
-                ],
+                '$labels': options["label"],
+                '$columns': field_list.map(function(item) { return field_config[item]; }),
                 '$data': records
             });
         if (options["feedback"]) {
             output.push(records.length + " objects");
         }
         break;
+
     default:
         document.exitcode = 8;
         window.alert(