cult 0.1.3.pre → 0.1.4.pre

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 255215974c24d700b973b431c1a6759b0f44c753
-  data.tar.gz: c8b95b6beedcd9da94bc634000917d7452bee86c
+  metadata.gz: 69eca6bb220e6610aabe66cfa4727881535bf8d0
+  data.tar.gz: 90b206c03fa2fb8b9f7c6f743b9edaaca0ce643e
 SHA512:
-  metadata.gz: 0d3618e471c1e966cb1c77b2643880eb0a133cace9769ca61553b215793228d8a129b2fb76ba490ebcc40ccb90656f1f5063946b084e657771fbbfc8fe322e22
-  data.tar.gz: ae2c2c7a96ebb863bffc3ca2a0bf32ece7cb35f455b0642b6a9a53b17bda38ac223f47537e00c7324d0aae5dd5f7ba2340a4c0d1b6ffff84b449208971cabd8d
+  metadata.gz: 43ca3f870a7cad5335fbaa4785e29973364cc0a3a0410586f33c0fd24f61b628123078d8d09f8cfd6565fba29f1cc550d59fac142048c463ec25f424295115e4
+  data.tar.gz: e1e79c67bdb9b968f6d19a1c62a4c198f6828d2449df4fd33c3d2dd3e25ca644a1e77b8e6ba853eb58cb4cd088499f1515c331a5a554a27995149c5f37dd9827

data/README.md

@@ -3,7 +3,6 @@
 
 
 ## Introduction
-
 Cult is a tool to manage fleets of servers. It tries to work in an obvious
 way, and doesn't intend on you learning a bunch of new metaphors, languages,
 and terminology.
@@ -56,7 +55,6 @@ Hopefully.
 
 
 ## Installation
-
 Cult is written in Ruby and available as a gem, but it is an application, not a
 library. It's not written in, and has no relation to Rails. It depends on
 Ruby 2.3 or greater. If you've got Ruby installed, via any means, the following
@@ -72,13 +70,12 @@ use the driver.
 Cult requires nothing to be installed on each node, other than an operating SSH
 server and Bourne Shell. If you know you've got Bash on the other end, feel
 free to write your tasks in Bash. If you want to write tasks in Ruby, Python,
-Node or Perl, etc, one of your firsts Tasks in the `all` or `bootstrap` role
+Node or Perl, etc, one of your firsts Tasks in the `base` or `bootstrap` role
 should be to `apt-get -y install {ruby,python,node}`. All subsequent tasks will
 have that interpreter available.
 
 
 ## General Theory
-
 I think a reasonable level of abstraction for cloud deployments is such:
 
   1. *Nodes*: Actual machines, virtual or otherwise, running somewhere.
@@ -87,11 +84,11 @@ I think a reasonable level of abstraction for cloud deployments is such:
   3. *Tasks*: Roles are made of tasks, which are basically scripts, called
      things like `install-postgres` or `configure-nginx`. They're written in
      your language of choice. When you being up a Node with a Role, all of the
-     Role's tasks are executed to get it up and running.
+     Role's tasks are executed to get it up and running, and later to update
+     configuration.
   4. *Providers*: Your VPS provider, e.g., Linode, DigitalOcean, etc. These
      allow you to spawn and destroy nodes, and typically charge you a few cents
-     per node per hour. I guess there's also *Drivers*, but a Provider is
-     really just an instance of a *Driver* with an API key configured.
+     per node per hour.
 
 We hate adding anything on top of this because it increases complexity.
 
@@ -100,11 +97,11 @@ hatches like ERB templating on shell scripts and JSON files to do weird stuff.
 
 
 ### Drivers
-
 Cult provides a handful of Drivers which can talk to common VPS providers,
-initially DigitalOcean, Linode, and Vultr. A driver is typically 200-300 lines
-of Ruby code, and is pretty well isolated from the rest of Cult, so feel free
-to open a PR to add your provider of choice. You can get a current list with:
+initially DigitalOcean, Linode, and Vultr, and a VirtualBox driver for local
+development. A driver is typically 200-300 lines of Ruby code, and is pretty
+well isolated from the rest of Cult, so feel free to open a PR to add your
+provider of choice. You can get a current list with:
 
     $ cult provider drivers
 
@@ -119,61 +116,79 @@ default bootstrap role indeed does exactly this.)
 
 
 ### Nodes
-
 A node is a physical instance running somewhere. A node has a name, like
-'web1', but also has a full description, that looks something like:
-
-  web1@ubuntu-16-01.2gb:digitalocean.nyc1
-
-A node named `web1` is an idea, but hasn't spawned. A node named with its full
-description should represent a real server somewhere, that is (hopefully)
-running.
+'web1-hr7sjdh8'.  You create a node with `cult node new -r some-role`.  
+Multiple roles may be specified.  You can check on all nodes with
+`cult node ping`.
 
 
 ### Roles
 A role is a collection of files (usually configuration files), Tasks (usually
-shell scripts), and a configuration (`role.json`). A role can include other
-roles via `includes:`.
-
-A Role's tasks are named like `000-a-descriptive-name`, because the only
-ordering Cult does is asciibetical.  Tasks named numerically are considered
-build tasks.  A Task named "sync" is built, shipped, and executed during
-`cult fleet sync`.  Other files are ignored, so you can symlink or whatever
-between them (if you want some sort of meta-role or something.)
+shell scripts), and a generated configuration (`role.json`). A role can include
+other roles via `includes:`.
 
-Every task, file, and even `role.json` is pre-processed with ERB before it gets
-processed by Cult or shipped to a node. This lets you customize behavior based
-on the node, the role, the provider, the project, /dev/urandom, or anything
-else you'd like.
+Tasks and Files, and role.json  are processed with ERB when used to create a
+node.  This lets you customize behavior based on the node, the role, the
+provider, the project, /dev/urandom, or anything else you'd like.
 
 
 #### Special Roles
 There are two Roles generated by default that you should think of as
 special-ish:
 
-  1. `all`: If a Role does not explicitly list an `includes` value, Cult will
-     act as if it specified `includes: ['all']`. In practice, this means tasks
-     in the `all` role are common to all nodes that haven't explicitly opted
-     out of it. A task can opt-out of including `all` with an explicit
+  1. `base`: If a Role does not explicitly list an `includes` value, Cult will
+     act as if it specified `includes: ['base']`. In practice, this means tasks
+     in the `base` role are common to all nodes that haven't explicitly opted
+     out of it. A task can opt-out of including `base` with an explicit
      `includes: []`.
   2. `bootstrap`: The generator creates this role to be the first one ran on a
      new node. Before it starts, we have a root account with an SSH key, when
      `bootstrap` finishes, we have a `cult` user on the node with `sudo` access
      who uses the same SSH key, and the root account is disabled. `bootstrap`
-     opts-out of `all`. The generator also installs a MOTD banner so you'll
+     opts-out of `base`. The generator also installs a MOTD banner so you'll
      know Cult was enabled on the server, has a demo script that sets the
      hostname.
 
+### Tasks
+Tasks are simple, but because they've been utilized in building real systems,
+there's quite a few special ways to use them.  A Role's tasks live in
+`roles/name/tasks`, and its type is inferred by its filename.
+
+#### Build Tasks
+A Role's build tasks are named like `000-a-descriptive-name`, because the only
+ordering Cult does is asciibetical. Build tasks are executed in order to build
+a node out of a role.
 
-## Usage
 
+#### Sync Tasks
+Sync tasks are meant to inform existing nodes of their state.  They are
+executed via `cult node sync`.  They are named `sync-description`.  By default,
+sync tasks are in "pass 0", which work a lot like build tasks.  If you have
+the need to place a sync task into a separate, later pass, you can specify it
+by naming the file something like `sync-P1-something`.  Note that
+`sync-something` and `sync-P0-something` mean the same thing.  All sync tasks
+of the same pass are executed in parallel.  To guarantee an order, place a sync
+task that must execute after another in a later pass.
+
+
+## Usage
 We're going to put together a complete usage guide, tutorial, and example repo
 once Cult has settled down a bit. It's still pre-1.0 software, and we still
 like breaking things to make it work better for us.
 
 
-### Spooky Secrets
+### Local VirtualBox-driver Development
+The 'virtual-box' driver sees your VMs as images, and clones one to start from.
+The guest must have the VirtualBox Guest Extensions installed and the first
+network adapter set up to "Bridged", so it acts like an actual machine.
+
+Cult expects to be able to login as "root" with the password "password", and
+immediately locks the root account after provisioning.  On ubuntu, you'll have
+to re-enabled the account with `sudo passwd root` and/or `sudo passwd -u root`.
 
+Note that VirtualBox is a little weird, and intended for development only.
+
+### Spooky Secrets
   * `cult console` is built to be really nice to use. If you're not afraid of
     Ruby, the method names are chosen to read almost like pseudo-code. It
     supports IRB, Pry, and Ripl with command-line flags.
@@ -187,18 +202,11 @@ like breaking things to make it work better for us.
     `nodes[/^dev/].with(role: /httpd/).with(something: /else/)`
   * The NamedArray stuff even works on the command-line with String arguments,
     and will convert strings that start with '/' to Regexps to search by name.
-  * Although Cult will only *generate* JSON, not having comments and other
-    stuff is a pain. If you don't care about JSON-readability of your
-    `node.json`s or `role.json`s, you can just rename it to `node.yaml` or
-    `node.yml` and it'll get parsed with a YAML parser. Cult keeps transient
-    state in separate files for this reason: so it doesn't overwrite your
-    long-lived YAML replacements with JSON equivalents.
 
 
 ## Development
 
 ### History
-
 Cult was developed to basically avoid Puppet, Chef, and Ansible. I know there
 are some really large, successful deployments of all of these. I just think
 they just do too much for me to feel safe with when shit goes crazy.
@@ -233,7 +241,6 @@ you feel a bit antsy.  Think of it more as jazz improv than an orchestra.
 
 
 ### Contributing
-
 We greatly appreciate bug reports, pull-requests, questions, and general
 commentary in the GitHub Issues. These are all *contributions*. However,
 before opening an issue demanding us to work on your feature that Cult *has to
@@ -253,7 +260,6 @@ the above items you away.
 
 
 ## License
-
 Cult is available as open source software under the terms of the
 [MIT License](http://opensource.org/licenses/MIT).
 

data/cult.gemspec

@@ -27,12 +27,13 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.required_ruby_version = '~> 2.3'
+  spec.required_ruby_version = '>= 2.2'
 
-  spec.add_dependency "cri",       "~> 2.7"
-  spec.add_dependency "net-ssh",   "~> 3.2"
-  spec.add_dependency "net-scp",   "~> 1.2"
-  spec.add_dependency "rainbow",   "~> 2.1"
+  spec.add_dependency "cri", "~> 2.7"
+  spec.add_dependency "net-ssh", "~> 3.2"
+  spec.add_dependency "net-scp", "~> 1.2"
+  spec.add_dependency "rainbow", "~> 2.1"
+  spec.add_dependency "erubis", "~> 2.7.0"
 
   spec.add_development_dependency "bundler", "~> 1.12"
   spec.add_development_dependency "rake", "~> 10.0"

data/exe/cult

@@ -37,6 +37,7 @@ cult = Cri::Command.define do
 
   flag :h, :help, 'Show this help' do |value, cmd|
     puts cmd.help
+    exit
   end
 
   flag :y, :yes, 'Answer "yes" to any questions' do
@@ -52,13 +53,27 @@ cult = Cri::Command.define do
     puts 'Copyright (C) 2016 Mike A. Owens, meter.md, and Contributors'
   end
 
-  run(arguments: 0) do |opts, args, cmd|
-    puts cmd.help
+  required :j, :jobs, 'Number of concurrent jobs.  Defaults to max' do |value|
+    Cult.concurrency = case value
+      when /^(\d+)$/
+         $1.to_i
+      when 'max'
+        :max
+      else
+        fail Cult::CLI::CLIError, "--jobs must be a number or 'max'"
+    end
+  end
+
+  run(arguments: none) do |opts, args, cmd|
+    if opts.empty? && args.empty?
+      puts cmd.help
+      exit
+    end
   end
 end
 
-Cult::CLI.commands.each do |m|
-  cult.add_command(m)
+Cult::CLI.commands.each do |root_command|
+  cult.add_command(root_command)
 end
 
 if (env = ENV['CULT_PROJECT'])

data/lib/cult/cli/common.rb

@@ -193,8 +193,7 @@ module Cult
 
 
     # Takes a list of keys and returns an array of objects that correspond
-    # to any of them.  If required is true, each key must correspond to at
-    # least one object.
+    # to any of them.
     def fetch_items(*keys, **kw)
       keys.flatten.map do |key|
         fetch_item(key, method: :all, **kw)

data/lib/cult/cli/console_cmd.rb

@@ -50,7 +50,7 @@ module Cult
     def console_cmd
       Cri::Command.define do
         name        'console'
-        summary     'Launch an REPL with you project loaded'
+        summary     'Launch a REPL with the project loaded'
         description <<~EOD.format_description
           The Cult console loads your project, and starts a Ruby REPL.  This can
           be useful for troubleshooting, or just poking around the project.
@@ -63,7 +63,7 @@ module Cult
         flag :p,  :pry,    'Pry'
         flag nil, :reexec, 'Console has been exec\'d for a reload'
 
-        run(arguments: 0) do |opts, args, cmd|
+        run(arguments: none) do |opts, args, cmd|
           context = ConsoleContext.new(Cult.project, ARGV)
 
           if opts[:reexec]

data/lib/cult/cli/cri_extensions.rb

@@ -8,6 +8,42 @@ module Cult
       end
     end
 
+    # This allows further -- options to be passed as literals instead of
+    # being stripped.
+    #   cult node ssh Something -- some-command -- something
+    module ArgumentArrayExtensions
+      attr_reader :explicit_tail
+
+      def initialize(raw_arguments)
+        @explicit_tail = []
+
+        super_super = Array.instance_method(:initialize).bind(self)
+        if (index = raw_arguments.index("--"))
+          @explicit_tail = raw_arguments[index + 1 .. -1]
+          processed = raw_arguments[0 ... index] + @explicit_tail
+          super_super.call(processed)
+        else
+          super_super.call(raw_arguments)
+        end
+        @raw_arguments = raw_arguments
+      end
+
+      ::Cri::ArgumentArray.prepend(self)
+    end
+
+    # This extension stops option processing at the first non-option bare-word.
+    # Without it, further arguments that look like options are treated as such.
+    # use-case:
+    #  cult node ssh SomeNode ls -l
+    module OptionParserExtensions
+      def run
+        peek = @unprocessed_arguments_and_options[0]
+        @no_more_options = true if peek && peek[0] != '-'
+        super
+      end
+      ::Cri::OptionParser.prepend(self)
+    end
+
 
     module CommandExtensions
       def project_required?
@@ -27,42 +63,44 @@ module Cult
       def block
         lambda do |opts, args, cmd|
           if project_required? && Cult.project.nil?
-            fail CLIError, "command '#{name}' requires a Cult project"
+            fail CLIError, "command '#{cmd.name}' requires a Cult project"
           end
 
-          check_argument_spec!(args, argument_spec) if argument_spec
+          check_argument_spec!(args, argument_spec, cmd) if argument_spec
 
           super.call(opts, args, cmd)
         end
       end
 
-
-      def check_argument_spec!(args, range)
+      def check_argument_spec!(args, range, cmd)
+        range = (0..range) if range == Float::INFINITY
         range = (range..range) if range.is_a?(Integer)
-        if range.end == -1
-          range = range.begin .. Float::INFINITY
-        end
 
         unless range.cover?(args.size)
           msg = case
             when range.size == 1 && range.begin == 0
               "accepts no arguments"
             when range.size == 1 && range.begin == 1
-              "accepts one argument"
+              "requires exactly one argument"
             when range.begin == range.end
-              "accepts exactly #{range.begin} arguments"
+              "requires exactly #{range.begin} arguments"
             else
               if range.end == Float::INFINITY
-                "requires #{range.begin}+ arguments"
+                words =%w(zero one two three)
+                if range.begin < words.size
+                  "requires #{words[range.begin]} or more arguments"
+                else
+                  "requires #{range.begin}+ arguments"
+                end
               else
                 "accepts #{range} arguments"
               end
           end
-          fail CLIError, "Command #{msg}"
+          fail CLIError, "command '#{cmd.name}' #{msg} (usage: #{cmd.usage})"
         end
       end
 
-      Cri::Command.prepend(self)
+      ::Cri::Command.prepend(self)
     end
 
 
@@ -72,12 +110,27 @@ module Cult
       end
 
 
+      # Lets us say run(arguments: 1 .. unlimited) instead of
+      #   run(arguments: 1 .. Float::INFINITY)
+      # or just outright:
+      #   run(arguments: unlimited)
+      def unlimited
+        Float::INFINITY
+      end
+
+      # Lets us say run(arguments: none)
+      def none
+        0
+      end
+
+      # This allows an explicit number of arguments to be passed to
+      # run, and halts with an error otherwise
       def run(arguments: nil, &block)
         @command.argument_spec = arguments if arguments
         super(&block)
       end
 
-      Cri::CommandDSL.prepend(self)
+      ::Cri::CommandDSL.prepend(self)
     end
   end
 end

data/lib/cult/cli/init_cmd.rb

@@ -1,4 +1,5 @@
 require 'cult/drivers/load'
+require 'json'
 
 module Cult
   module CLI
@@ -37,7 +38,7 @@ module Cult
           be accomplished by hand, so if you want to change anything later, it's
           not a big deal.
 
-          The project generated sets up a pretty common configuration: an 'all'
+          The project generated sets up a pretty common configuration: a `base`
           role, a 'bootstrap' role, and a demo task that puts a colorful banner
           in each node's MOTD.
         EOD
@@ -87,15 +88,13 @@ module Cult
             FileUtils.mkdir_p(provider_dir)
 
 
-            provider_file = File.join(provider_dir,
-                                      project.dump_name("provider"))
-            File.write(provider_file, project.dump_object(provider_conf))
+            provider_file = File.join(provider_dir, "provider.json")
+            File.write(provider_file, JSON.pretty_generate(provider_conf))
 
 
-            defaults_file = File.join(provider_dir,
-                                      project.dump_name("defaults"))
+            defaults_file = File.join(provider_dir, "defaults.json")
             defaults = Provider.generate_defaults(provider_conf)
-            File.write(defaults_file, project.dump_object(defaults))
+            File.write(defaults_file, JSON.pretty_generate(defaults))
           end
 
           if opts[:git]