cult 0.1.1.pre → 0.1.2.pre

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: cea2fc50c8b92985e08159d77c53f16cb71569ce
-  data.tar.gz: cba93e78130a741980b6f3be07038edb2a9035fc
+  metadata.gz: fb7c0f07f4c2efffce72e4a5d2ef466e08d081c2
+  data.tar.gz: 5036b7ec3e84fd0aaa30b63b06c706c9cec9f190
 SHA512:
-  metadata.gz: 70ccf020691c38b75dce07f9e408cce0f0849a1d54b06a96038de5b0f4c459be024822f6931b433c115e04fa318b461e05f937faad60f7b85ce492474efa72bc
-  data.tar.gz: 98370627f82c3c4aa5c240e66aa2dfce29c17f4d8368e509c205651dfd0fdcd97d4a0aef2f9711fe03f28c5d9e158206ee10771108bc04c5a6b01b8965c02525
+  metadata.gz: d42118fa748b4c7291c21700fc2d4ffb8a11bb343beec84d7d6f9093fec1610d36a10d00d721e62fb339b8ce749022ee18f2e83b6c8c663a5fc17a13a7964804
+  data.tar.gz: 6845470b42be10a33e6e9336b06a6fb6ab22871c242181481ade21470a0e98066c20f7d56f6368e51fe9bf3b92c6e3f2919c6f37bbbb0ef67ed1b7d0d1e3ef1d

data/README.md

@@ -1,5 +1,7 @@
+![Cult][logo]
 # Cult
 
+
 ## Introduction
 
 Cult is a tool to manage fleets of servers. It tries to work in an obvious
@@ -9,16 +11,17 @@ and terminology.
 Cult may be what you're looking for if:
 
   * You like the transparency of shell-script setup but have outgrown it, and
-    it's turning into a tangled mess
+    it's turning into a tangled mess.  You can create a more structured tangled
+    mess with Cult.
   * "Configuration Management Systems" and "Known Configuration State" stuff
-    rubs you the wrong way
+    rubs you the wrong way.
   * You're not worried about abstracting away Unix, as if you'll be deploying
-    on a herd of Amigas next year
+    on a herd of Amigas next year.
   * You're not looking to find a cloud-hosted meta-script to effectively
-    `apt-get -y install nginx`
+    `apt-get -y install nginx`.
   * You have no ceremony around spinning up and killing servers. Cult can
     manage real metal, but its default mindset is: if you fuck up too bad, you
-    can spin up a fresh instance
+    can spin up a fresh instance.
   * When you think of a forward migration, the first thing that comes to mind
     is `#!...`
   * You don't get why you need more than a working SSH to configure a server,
@@ -32,13 +35,13 @@ Cult's probably not your bourbon and ginger if:
 
   * You see value in "converging toward a configuration", as if you're guiding
     your precious children through the path of life, and helping them evolve
-    into better people.
+    into better servers.
   * You have one big-ass old server that's gets conservatively upgraded via
     efforts big enough to have a project name. Cult can help you *out* of
-    that, though.
+    that, though.  Maybe.
   * You're sold on image-in-a-container-in-a-KVM deployment. That's totally
     reasonable, but Cult doesn't really do anything particularly special
-    to help you if you're happy with the process you have building containers.
+    to help you if you're happy with the process you have building images.
   * You expect to have the same configuration abstractly work on a totally
     diverse set of platforms (e.g., one "formula" that works on Ubuntu,
     FreeBSD, and SCO OpenServer 5). Cult can manage these absolutely fine, but
@@ -48,7 +51,8 @@ Cult's probably not your bourbon and ginger if:
     worth of highly-tuned inputs into a Configuration Management System. Cult
     itself does less for you, but requires a lot less of you.
 
-But, what you gain via Cult is transparency, repeatability, and obviousness.
+But, what you gain via Cult is transparency, repeatability, and obviousness.  
+Hopefully.
 
 
 ## Installation
@@ -68,9 +72,9 @@ use the driver.
 Cult requires nothing to be installed on each node, other than an operating SSH
 server and Bourne Shell. If you know you've got Bash on the other end, feel
 free to write your tasks in Bash. If you want to write tasks in Ruby, Python,
-Node or Perl, etc, one of your firsts Tasks in the `all` role should be to
-`apt-get -y install {ruby,python,node}`. All subsequent tasks will have that
-interpreter available.
+Node or Perl, etc, one of your firsts Tasks in the `all` or `bootstrap` role
+should be to `apt-get -y install {ruby,python,node}`. All subsequent tasks will
+have that interpreter available.
 
 
 ## General Theory
@@ -113,6 +117,7 @@ you can connect to it. The `bootstrap` role can then kick in. Your `boostrap`
 role will typically create the `cult` user, disable the root account, etc (the
 default bootstrap role indeed does exactly this.)
 
+
 ### Nodes
 
 A node is a physical instance running somewhere. A node has a name, like
@@ -124,18 +129,23 @@ A node named `web1` is an idea, but hasn't spawned. A node named with its full
 description should represent a real server somewhere, that is (hopefully)
 running.
 
+
 ### Roles
 A role is a collection of files (usually configuration files), Tasks (usually
 shell scripts), and a configuration (`role.json`). A role can include other
-roles via `includes:`. A role can state it conflicts with another role via
-`conflicts:`.
+roles via `includes:`.
 
-A Role's tasks are named like `00000-a-descriptive-name`, because the only
-ordering Cult does is asciibetical.
+A Role's tasks are named like `000-a-descriptive-name`, because the only
+ordering Cult does is asciibetical.  Tasks named numerically are considered
+build tasks.  A Task named "sync" is built, shipped, and executed during
+`cult fleet sync`.  Other files are ignored, so you can symlink or whatever
+between them (if you want some sort of meta-role or something.)
 
 Every task, file, and even `role.json` is pre-processed with ERB before it gets
 processed by Cult or shipped to a node. This lets you customize behavior based
-on the node, the role, the provider, the project, or anything else you'd like.
+on the node, the role, the provider, the project, /dev/urandom, or anything
+else you'd like.
+
 
 #### Special Roles
 There are two Roles generated by default that you should think of as
@@ -154,12 +164,6 @@ special-ish:
      know Cult was enabled on the server, has a demo script that sets the
      hostname.
 
-## UI
-
-Even though you aren't required to use it, Cult includes the terminal-mode GUI
-packages (`$ cult ui`). Because it's 2016, and we're not worried about a few
-MB of extra packages hanging around anymore. We're steadily trying to improve
-this cool-ass tmux-based UI, but it's still new right now.
 
 ## Usage
 
@@ -167,7 +171,8 @@ We're going to put together a complete usage guide, tutorial, and example repo
 once Cult has settled down a bit. It's still pre-1.0 software, and we still
 like breaking things to make it work better for us.
 
-### 👻 Spooky Secrets
+
+### Spooky Secrets
 
   * `cult console` is built to be really nice to use. If you're not afraid of
     Ruby, the method names are chosen to read almost like pseudo-code. It
@@ -178,6 +183,8 @@ like breaking things to make it work better for us.
     first driver with `drivers[0]`, find it by name with `drivers['linode']`,
     or (*get ready for fancy stuff:*) look it up by a Regexp with
     `drivers[/ocean/i]`.
+  * Check this out on the console:
+    `nodes[/^dev/].with(role: /httpd/).with(something: /else/)`
   * The NamedArray stuff even works on the command-line with String arguments,
     and will convert strings that start with '/' to Regexps to search by name.
   * Although Cult will only *generate* JSON, not having comments and other
@@ -205,7 +212,8 @@ particular:
   1. Exercising the server bring-up process from bottom-up as the normal mode
      of operation. This is why Cult makes you bring up a node from provision/
      bootstrap each time, instead of using snapshots, a feature virtually every
-     VPS provider supports.
+     VPS provider supports.  (The snapshot thing is looking pretty tempting,
+     though.)
   2. Making it less reasonable to have nodes hanging around that have ended up
      in their current state via baby-step migrations for too long. Cult does
      this fine, but makes it easier to just test a new node with a clean build.
@@ -213,8 +221,16 @@ particular:
      conditional logic. If a node is not where you want, bring up another that
      is, and kill the old one.
 
-We can be convinced otherwise, but these are sort of core tenets of what Cult
-is about.
+We can be convinced otherwise, but these sort of feel like the core tenets of
+what Cult is about.
+
+Cult began and reached a useful state in a burst of exploratory hacking.  It
+accidentally turned out really useful for us.  It wasn't written with tests in
+lockstep. We'd love to have tests, but with our devel-branch style division, it
+hasn't been a priority yet (with other fires to put out.)  There wasn't, and
+probably won't be perfectly bisect-able single-feature commits.  This may make
+you feel a bit antsy.  Think of it more as jazz improv than an orchestra.
+
 
 ### Contributing
 
@@ -229,12 +245,16 @@ have to be taken seriously*, note:
      can be unpaid software contractors working on the demands of strangers on
      the internet.
   *  If your contribution consists of instructing the team how to run the
-     project: Thanks, but we've got that handled.
+     project or interface with the community: Thanks, but we've got that
+     handled.
 
 If you're contributing code, asking a question, or reporting a bug, don't let
 the above items you away.
 
+
 ## License
 
 Cult is available as open source software under the terms of the
 [MIT License](http://opensource.org/licenses/MIT).
+
+[logo]: ./doc/images/masthead@0.5x.png "Cult Logo"

data/cult

@@ -1 +1 @@
-./exe/cult
+exe/cult

data/cult.gemspec

@@ -28,10 +28,10 @@ Gem::Specification.new do |spec|
 
   spec.required_ruby_version = '~> 2.3'
 
-  spec.add_dependency "cri",       "~> 2.7.0"
-  spec.add_dependency "net-ssh",   "~> 3.2.0"
-  spec.add_dependency "net-scp",   "~> 1.2.1"
-  spec.add_dependency "rainbow",   "~> 2.1.0"
+  spec.add_dependency "cri",       "~> 2.7"
+  spec.add_dependency "net-ssh",   "~> 3.2"
+  spec.add_dependency "net-scp",   "~> 1.2"
+  spec.add_dependency "rainbow",   "~> 2.1"
 
   spec.add_development_dependency "bundler", "~> 1.12"
   spec.add_development_dependency "rake", "~> 10.0"

data/exe/cult

@@ -2,11 +2,8 @@
 $LOAD_PATH << File.expand_path(File.join(__dir__, '../lib'))
 
 require 'cri'
-
-require 'cult/version'
-require 'cult/config'
-require 'cult/project'
-require 'cult/node'
+require 'cult'
+require 'cult/cli/common'
 require 'cult/cli/load'
 require 'cult/drivers/load'
 

data/lib/cult/artifact.rb

@@ -1,11 +1,12 @@
 require 'cult/transferable'
-require 'cult/named_array'
+require 'cult/singleton_instances'
 
 module Cult
   # I'd love to just call this "File", but the ambiguity with ::File would
   # make it a pain.
   class Artifact
     include Transferable
+    include SingletonInstances
 
     def self.collection_name
       "files"

data/lib/cult/bundle.rb

@@ -0,0 +1,26 @@
+require 'rubygems/package'
+require 'rubygems/package/tar_writer'
+
+module Cult
+  class Bundle
+    attr_reader :tar
+    def initialize(io, &block)
+      @tar = Gem::Package::TarWriter.new(io)
+      if block_given?
+        begin
+          yield self
+        ensure
+          @tar.close
+          @tar = nil
+        end
+      end
+    end
+
+    def add_file(project, role, node, transferable)
+      data = transferable.contents(project, role, node, pwd: role.path)
+      tar.add_file(transferable.remote_path, transferable.file_mode) do |io|
+        io.write(data)
+      end
+    end
+  end
+end

data/lib/cult/cli/common.rb

@@ -1,6 +1,8 @@
 require 'io/console'
 require 'shellwords'
 
+require 'cult/cli/cri_extensions'
+
 module Cult
   module CLI
 

data/lib/cult/cli/console_cmd.rb

@@ -1,20 +1,28 @@
 require 'delegate'
 require 'rainbow'
 
+require 'cult/user_refinements'
+
 module Cult
   module CLI
 
-    class ConsoleContext < SimpleDelegator
-      attr_accessor :original_argv
+    class ConsoleContext < ProjectContext
+      using ::Cult::UserRefinements
 
+      attr_accessor :original_argv
 
+      attr_reader :project
       def initialize(project, argv)
-        super(project)
+        @project = project
+        # super(project)
 
         @original_argv = [$0, *argv]
         ENV['CULT_PROJECT'] = self.path
       end
 
+      def path
+        project.path
+      end
 
       def load_rc
         consolerc = project.location_of(".cultconsolerc")
@@ -29,18 +37,10 @@ module Cult
         super
       end
 
-
-      # Gives us an escape hatch to get the real, non-decorated object
-      def project
-        __getobj__
-      end
-
-
       def cult(*argv)
         system $0, *argv
       end
 
-
       def binding
         super
       end

data/lib/cult/cli/cri_extensions.rb

@@ -4,8 +4,7 @@ module Cult
   module CLI
     class ::String
       def format_description
-        self.gsub(/(\S)\n(\S)/m, '\1 \2')
-            .gsub(/\.[ ]{2}(\S)/m, '. \1')
+        self
       end
     end
 

data/lib/cult/cli/fleet_cmd.rb

@@ -0,0 +1,37 @@
+module Cult
+  module CLI
+    module_function
+    def fleet_cmd
+      fleet = Cri::Command.define do
+        name 'fleet'
+        summary 'Fleet commands'
+
+        run(arguments: 0) do |opts, args, cmd|
+          puts cmd.help
+        end
+      end
+
+      fleet_sync = Cri::Command.define do
+        name    'sync'
+        summary 'Synchronize host information across fleet'
+        description <<~EOD.format_description
+          Processes generates and executes tasks/sync on every node with a
+          current network setup.
+        EOD
+
+        run(arguments: 0..-1) do |opts, args, cmd|
+          nodes = args.empty? ? Cult.project.nodes
+                              : CLI.fetch_items(args, from: Node)
+          nodes.each do |node|
+            c = Commander.new(project: Cult.project, node: node)
+            c.sync!
+            puts "SYNCING #{node}"
+          end
+        end
+      end
+      fleet.add_command(fleet_sync)
+
+      fleet
+    end
+  end
+end

data/lib/cult/cli/init_cmd.rb

@@ -1,5 +1,3 @@
-require 'cult/skel'
-require 'cult/project'
 require 'cult/drivers/load'
 
 module Cult

data/lib/cult/cli/node_cmd.rb

@@ -1,7 +1,5 @@
-require 'cult/skel'
-require 'cult/commander'
-
 require 'SecureRandom'
+require 'fileutils'
 
 module Cult
   module CLI
@@ -32,7 +30,8 @@ module Cult
 
         run(arguments: 1) do |opts, args, cmd|
           node = CLI.fetch_item(args[0], from: Node)
-          exec "ssh", "#{node.user}@#{node.host}"
+          exec "ssh", '-i', node.ssh_private_key_file,
+               "#{node.user}@#{node.host}"
         end
       end
       node.add_command(node_ssh)
@@ -43,8 +42,8 @@ module Cult
         usage       'create [options] NAME...'
         summary     'Create a new node'
         description <<~EOD.format_description
-          This command creates a new node specification.  With --bootstrap,
-          it'll also provision it, so it'll actually exist out there.
+          This command creates a new node specification and then creates it with
+          your provider.
 
           The newly created node will have all the roles listed in --role.  If
           none are specified, it'll have the role "all".  If no name is
@@ -75,8 +74,6 @@ module Cult
         required :I, :image,     'Provider image'
         required :S, :size,      'Provider instance size'
 
-        flag     :b, :bootstrap, 'Bring up node'
-
         run(arguments: 0..-1) do |opts, args, cmd|
           random_suffix = ->(basename) do
             begin
@@ -150,19 +147,18 @@ module Cult
             }
 
             Node.from_data!(Cult.project, data).tap do |node|
-              if opts[:bootstrap]
-                prov_data = provider.provision!(name: node.name,
-                                                image: node_spec[:image],
-                                                size: node_spec[:size],
-                                                zone: node_spec[:zone],
-                                                ssh_key_files: '/Users/mike/.ssh/id_rsa.pub')
-                File.write(Cult.project.dump_name(node.state_path),
-                           Cult.project.dump_object(prov_data))
-
-                c = Commander.new(project: Cult.project, node: node)
-                c.bootstrap!
-                c.install!(node)
-              end
+              prov_data = provider.provision!(name: node.name,
+                                              image: node_spec[:image],
+                                              size: node_spec[:size],
+                                              zone: node_spec[:zone],
+                                              ssh_public_key: node.ssh_public_key_file)
+              prov_data['provider'] = provider.name
+              File.write(Cult.project.dump_name(node.state_path),
+                         Cult.project.dump_object(prov_data))
+
+              c = Commander.new(project: Cult.project, node: node)
+              c.bootstrap!
+              c.install!(node)
             end
           end
 
@@ -170,6 +166,42 @@ module Cult
       end
       node.add_command node_create
 
+      node_destroy = Cri::Command.define do
+        name        'destroy'
+        aliases     'rm'
+        usage       'destroy NODE'
+        summary     'Destroy nodes'
+        description <<~EOD.format_description
+          Destroys all nodes named NODE, or match the pattern described by
+          NODE.
+
+          First, the remote node is destroyed, then the local definition.
+
+          This command respects the global --yes option, otherwise, you will
+          be prompted before each destroy.
+        EOD
+
+        run(arguments: 1..-1) do |opts, args, cmd|
+          nodes = CLI.fetch_items(args, from: Node)
+          nodes.each do |node|
+            if CLI.yes_no?("Destroy node `#{node}`?")
+              puts "destroying #{node}"
+              begin
+                node.provider.destroy!(id: node.definition['id'],
+                                       ssh_key_id: node.definition['ssh_key_id'])
+              rescue Exception => e
+                puts "Exception while remote-destroying node: #{e.to_s}\n" +
+                     "#{e.backtrace}"
+                puts "Continuing, though."
+              end
+              fail unless node.path.match(/#{Regexp.escape(node.name)}/)
+              FileUtils.rm_rf(node.path)
+            end
+          end
+        end
+      end
+      node.add_command(node_destroy)
+
       node_list = Cri::Command.define do
         name        'list'
         aliases     'ls'
@@ -193,7 +225,7 @@ module Cult
           end
 
           nodes.each do |node|
-            puts "Node: #{node.inspect}"
+            puts "#{node.name}\t#{node.provider&.name}\t#{node.roles.map(&:name)}"
           end
         end
       end

data/lib/cult/cli/task_cmd.rb

@@ -1,6 +1,3 @@
-require 'cult/task'
-require 'cult/role'
-
 module Cult
   module CLI
     module_function
@@ -12,21 +9,40 @@ module Cult
         summary     'Task Manipulation'
         usage       'task [command]'
         description <<~EOD.format_description
-
+          Tasks are basically shell scripts.  Or anything with a \#! line, or
+          that can be executed by name.
+
+          Each task belongs to a Role, and the collection of Tasks in a Role,
+          when ran in sequence, define what the Role does.
+
+          For example, you could have a 'database-sever' Role, which would
+          include tasks with filenames like:
+
+            000-add-postgres-apt-repo
+            001-install-postgres
+            002-create-roles
+            003-update-hba
+            004-install-tls-cert
+            005-start-postgres
+
+          All of these Tasks would be run in sequence to define what you
+          consider a `database-server` should look like.  Note that a task's
+          sequence is defined by a leading number, and `task resequence` will
+          neatly line these up for you.
         EOD
 
         run(arguments: 0) do |opts, args, cmd|
           puts cmd.help
-          exit
         end
       end
 
 
-      task_reserial = Cri::Command.define do
+      task_resequence = Cri::Command.define do
         name        'resequence'
+        aliases     'reserial'
         summary     'Resequences task serial numbers'
 
-        flag     :A,  :all,             'Reserial all roles'
+        flag     :A,  :all,             'Re-sequence all roles'
         flag     :G,  :'git-add',       '`git add` each change'
         required :r,  :role,            'Roles to resequence (multiple)',
                       multiple: true
@@ -64,7 +80,7 @@ module Cult
 
           roles.each do |role|
             puts "Resequencing role: `#{role.name}'"
-            tasks = role.tasks.sort_by do |task|
+            tasks = role.build_tasks.sort_by do |task|
               # This makes sure we don't change order for duplicate serials
               [task.serial, task.name]
             end
@@ -98,7 +114,7 @@ module Cult
           end
         end
       end
-      task.add_command(task_reserial)
+      task.add_command(task_resequence)
 
 
       task_sanity = Cri::Command.define do