opswalrus 1.0.0

data/README.md

@@ -0,0 +1,256 @@
+# opswalrus
+
+opswalrus is a tool that runs scripts against hosts. It's kind of like Ansible, but aims to be simpler to use.
+
+# Examples
+
+```bash
+> ops run core/host info
+{
+  success: true,
+  host: {
+    name: "davidlinux",
+    os: "Ubuntu 23.04 (lunar)",
+    kernel: "Linux 6.2.0-1007-lowlatency x86_64 GNU/Linux",
+  }
+}
+```
+
+# Packages and Imports
+
+## Ops Packages
+
+An ops package is a directory containing a package.yml (or package.yaml) file.
+
+- The package.yml (or package.yaml) file is called the package file.
+- The directory containing the package file is called the package directory.
+
+A package file looks like this:
+```yaml
+author: David Ellis
+license: MIT
+version: 1.0.0
+dependencies:
+  core: davidkellis/ops_core
+  apt: davidkellis/ops_apt
+```
+
+## Ops Files
+
+An ops package may also consist of ops files, arranged in an arbitrary directory structure.
+
+- An ops file is a script to do do something, very much like a shell script.
+- An ops file should try to implement an idempotent operation, such that repeated execution of the script results in the same desired state.
+
+An ops file looks like this:
+```
+params:
+  ops_username: string
+  ops_ssh_public_key: string
+  hostname: string
+
+output:
+  success: boolean,
+  error: string?
+
+imports:
+  core: core        # core references the bundled core package referenced in the package.yaml
+  svc: service      # service references the bundled service package referenced in the package.yaml
+...
+
+desc "create the admin group if it doens't exist"
+core.create_group name: "admin"
+
+desc "set up passwordless sudo for admin group users"
+core.replace_line file: "/etc/sudoers",
+                  pattern: "^%admin",
+                  line: "%admin ALL=(ALL) NOPASSWD: ALL",
+                  verify: "/usr/sbin/visudo -cf %s"
+
+desc "create the ops user and make it an admin user"
+core.create_user name: params.ops_username
+
+desc "set up authorized key for id_ansible ssh key (root user)"
+core.ssh.add_authorized_key user: "root", key: ops_ssh_public_key
+
+desc "set up authorized key for id_ansible ssh key (ops user)"
+core.ssh.add_authorized_key user: ops_username, key: ops_ssh_public_key
+
+desc "disable password authentication for root"
+core.replace_line file: '/etc/ssh/sshd_config',
+                  pattern: '^#?PermitRootLogin',
+                  line: 'PermitRootLogin prohibit-password'
+
+desc "restart sshd"
+svc.restart name: "sshd"
+
+{
+  success: true,
+}
+```
+
+An ops file is broken up into two parts. The first part is an optional YAML block that describes the structure of the expected
+input parameters, the structure of the expected JSON output message, and the package dependencies that the script
+needs in order to run.
+
+The YAML block is concluded with an elipsis, `...`, on a line by itself.
+
+The YAML block and its associated trailing elipsis may be omitted.
+
+Following the elipsis that concludes the YAML block is a block of Ruby code. The block of Ruby is executed with a number
+of methods, constants, and libraries that are available as a kind of domain specific language (DSL). This DSL makes
+writing ops scripts feel very much like writing standard bash shell scripts.
+
+Ops file imports are a mapping consisting of a local name and a corresponding package reference.
+
+## Package Bundles
+
+When an ops file is run, the ops runtime will first bundle up the invoked ops file as well as all package dependencies
+and place the bundle of associated ops packages and ops files into an ops bundle directory.
+
+The bundle directory is named ops_bundle, and contains everything needed to run the specified ops file on either the
+local host or a remote host.
+
+The ops command will place the bundle directory in the directory from which the ops command is being run. So, if `pwd`
+returns `/home/david/foo` and the ops command is run from within that directory, then the bundle directory will be placed
+at `/home/david/foo/ops_bundle`.
+
+The one exception to the normal bundle directory placement rule described in the previous paragraph is when the ops
+command is being run from within a directory that is contained within a package directory. In that case, the bundle directory
+will be placed inside the package directory. So, for example, if the directory structure looks like:
+```
+❯ tree pkg
+pkg
+├── apt
+│   ├── install.ops
+│   └── update.ops
+├── core
+│   ├── echo.ops
+│   ├── host
+│   │   ├── info.ops
+│   │   └── info.rb
+│   ├── package.yaml
+│   ├── ssh_copy_id.ops
+│   ├── touch.ops
+│   └── whoami.ops
+├── hardening
+│   └── package.yaml
+├── motd
+│   ├── motd.ops
+│   └── package.yaml
+└── service
+    └── restart.ops
+```
+and the `pwd` command returns `pkg/core/host`, and the ops command is run from within `pkg/core/host`, then the bundle
+directory will be placed at `pkg/core/ops_bundle`.
+
+### Bundle Directory Contents
+
+A bundle directory contains all the dependencies for a given ops file invocation. There are two possible cases:
+1. The invoked ops file is part of a package
+2. The invoked ops file is not part of a package
+
+In case (1), when the ops file being invoked is part of a package, we'll call it P, then the bundle directory will contain
+a copy of the package directory associated with P, as well as all of the package directories associated with all
+transitive package dependencies of P. For example, if the ops file foo.ops is contained within the package directory
+for the Bar package, and if the Bar package depends on the core package and the service package, then the
+directory structure of the bundle directory would be:
+```
+❯ tree ops_bundle
+ops_bundle
+├── Bar
+│   └── foo.ops
+├── core
+│   ├── echo.ops
+│   ├── host
+│   │   ├── info.ops
+│   │   └── info.rb
+│   ├── package.yaml
+│   ├── ssh_copy_id.ops
+│   ├── touch.ops
+│   └── whoami.ops
+└── service
+    └── restart.ops
+```
+
+In case (2), when the ops file being invoked is not part of a package, then the bundle directory will contain a copy
+of the package directories associated with all transitive package dependencies of the ops file being invoked.
+Additionally, the deepest nested directory containing all of the transitive ops file dependencies of the ops file being
+invoked will be copied to the bundle directory.
+
+## Import and Symbol Resolution
+
+When the ops command bundles and runs an ops file, the rules that the runtime uses to resolve references to other ops
+files is as follows; assume the following sample project directory structure:
+
+Project directory structure:
+```
+davidinfra
+├── caddy
+│   ├── install
+│   │   └── debian.ops
+│   └── install.ops
+├── hosts.yml
+├── main.ops
+├── prepare_host
+│   ├── all.ops
+│   ├── hostname.ops
+│   └── ssh.ops
+└── roles
+    └── web.ops
+```
+
+Corresponding bundle directory structure:
+```
+davidinfra
+├── ops_bundle
+│   ├── core
+│   │   └──...
+│   └── davidinfra
+│       ├── caddy
+│       │   ├── install
+│       │   │   └── debian.ops
+│       │   └── install.ops
+│       ├── hosts.yml
+│       ├── main.ops
+│       ├── prepare_host
+│       │   ├── all.ops
+│       │   ├── hostname.ops
+│       │   └── ssh.ops
+│       └── roles
+│           └── web.ops
+├── caddy
+│   ├── install
+│   │   └── debian.ops
+│   └── install.ops
+├── hosts.yml
+├── main.ops
+├── prepare_host
+│   ├── all.ops
+│   ├── hostname.ops
+│   └── ssh.ops
+└── roles
+    └── web.ops
+```
+
+The import and symbol resolution rules are as follows:
+
+1. An ops file implicitly imports all sibling ops files and directories that reside within its same parent directory.
+   Within the lexical scope of an ops file's ruby script, any ops files or subdirectories that are implicitly imported
+   may be referenced by their name.
+   For example:
+   - main.ops may invoke caddy/install.ops with the expression `caddy.install(...)`
+   - all.ops may invoke hostname.ops with the expression `hostname(...)`
+2. If there is an ops file and a directory that share the same name (with the exception of the .ops file extension),
+   then only the install.ops file may be referenced and invoked by other ops files, while the directory of the same name,
+   and its contents, may only be referenced from within the ops file of the matching name. This allows the details of
+   an operation to be encapsulated away and a public API be exposed through the ops file, while the details of the
+   implementation are hidden away in the ops files within the directory.
+   For example:
+   - main.ops may invoke `caddy.install(...)`, but it may not invoke `caddy.install.debian(...)`
+   - install.ops may invoke `install.debian(...)`, and reference other files or subpackages within the caddy/install directory
+3. Ops files may import packages or relative paths:
+   1. a package reference that matches one of the local package names in the dependencies captured in packages.yaml
+   2. a package reference that resolves to a relative path pointing at a package directory
+   3. a relative path that resolves to a directory containing ops files
+   4. a relative path that resolves to an ops file

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/exe/ops

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require 'opswalrus'
+
+exit_status = OpsWalrus::Cli.run(ARGV)
+exit exit_status

data/exe/run_ops_bundle

@@ -0,0 +1,22 @@
+#!/usr/bin/env ruby
+
+require 'opswalrus'
+
+def main
+  zip_file_path = ARGV.shift.to_pathname
+
+  exit_status = 0
+
+  Dir.mktmpdir do |dir|
+    dir = dir.to_pathname
+
+    # unzip the bundle into the temp directory
+    OpsWalrus::DirZipper.unzip(zip_file_path, dir)
+
+    exit_status = OpsWalrus::Cli.run(ARGV)
+  end
+
+  exit exit_status
+end
+
+main

data/lib/opswalrus/app.rb

@@ -0,0 +1,328 @@
+require "citrus"
+require "git"
+require "io/console"
+require "json"
+require "random/formatter"
+require "shellwords"
+require "socket"
+require "stringio"
+require "yaml"
+require "pathname"
+require_relative "host"
+require_relative "hosts_file"
+require_relative "operation_runner"
+require_relative "bundler"
+require_relative "package_file"
+
+class String
+  def escape_single_quotes
+    gsub("'"){"\\'"}
+  end
+
+  def to_pathname
+    Pathname.new(self)
+  end
+end
+
+class Pathname
+  def to_pathname
+    self
+  end
+end
+
+module OpsWalrus
+  class Error < StandardError
+  end
+
+  class App
+    def self.instance(*args)
+      @instance ||= new(*args)
+    end
+
+
+    attr_reader :local_hostname
+
+    def initialize(pwd = Dir.pwd)
+      @verbose = false
+      @sudo_user = nil
+      @sudo_password = nil
+      @inventory_host_references = []
+      @inventory_tag_selections = []
+      @params = nil
+      @pwd = pwd.to_pathname
+      @bundler = Bundler.new(@pwd)
+      @local_hostname = "localhost"
+    end
+
+    def to_s
+      ""  # return empty string because we won't want anyone accidentally printing or inspecting @sudo_password
+    end
+
+    def inspect
+      ""  # return empty string because we won't want anyone accidentally printing or inspecting @sudo_password
+    end
+
+    def set_local_hostname(hostname)
+      hostname = hostname.strip
+      @local_hostname = hostname.empty? ? "localhost" : hostname
+    end
+
+    def set_inventory_hosts(*hosts)
+      hosts.flatten!.compact!
+      @inventory_host_references.concat(hosts).compact!
+    end
+
+    def set_inventory_tags(*tags)
+      tags.flatten!.compact!
+      @inventory_tag_selections.concat(tags).compact!
+    end
+
+    def bundler
+      @bundler
+    end
+
+    def bundle_dir
+      @bundler.bundle_dir
+    end
+
+    def set_verbose(verbose)
+      @verbose = verbose
+    end
+
+    def verbose?
+      @verbose
+    end
+
+    def debug?
+      @verbose == 2
+    end
+
+    def set_pwd(pwd)
+      @pwd = pwd.to_pathname
+      @bundler = Bundler.new(@pwd)
+    end
+
+    def pwd
+      @pwd || raise("No working directory specified")
+    end
+
+    def set_sudo_user(user)
+      @sudo_user = user
+    end
+
+    def sudo_user
+      @sudo_user || "root"
+    end
+
+    def set_sudo_password(password)
+      @sudo_password = password
+    end
+
+    def prompt_sudo_password
+      password = IO::console.getpass("[ops] Enter sudo password to run sudo in local environment: ")
+      set_sudo_password(password)
+      nil
+    end
+
+    def sudo_password
+      @sudo_password
+    end
+
+    # params must be a string representation of a JSON object: '{}' | '{"key1": ... , ...}'
+    def set_params(params)
+      json_hash = JSON.parse(params) rescue nil
+      json_hash = json_hash.is_a?(Hash) ? json_hash : nil
+
+      @params = json_hash   # @params returns a Hash or nil
+    end
+
+    # args is of the form ["github.com/davidkellis/my-package/sub-package1", "operation1", "arg1:val1", "arg2:val2", "arg3:val3"]
+    # if the first argument is the path to a .ops file, then treat it as a local path, and add the containing package
+    #   to the load path
+    # otherwise, copy the
+    # returns the exit status code that the script should terminate with
+    def run(package_operation_and_args)
+      return 0 if package_operation_and_args.empty?
+
+      ops_file_path, operation_kv_args, tmp_dir = get_entry_point_ops_file_and_args(package_operation_and_args)
+      ops_file = OpsFile.new(self, ops_file_path)
+
+      # if the ops file is part of a package, then set the package directory as the app's pwd
+      # puts "run1: #{ops_file.ops_file_path}"
+      if ops_file.package_file && ops_file.package_file.dirname.to_s !~ /#{Bundler::BUNDLE_DIR}/
+        # puts "set pwd: #{ops_file.package_file.dirname}"
+        set_pwd(ops_file.package_file.dirname)
+        rebased_ops_file_relative_path = ops_file.ops_file_path.relative_path_from(ops_file.package_file.dirname)
+        # note: rebased_ops_file_relative_path is a relative path that is relative to ops_file.package_file.dirname
+        # puts "rebased path: #{rebased_ops_file_relative_path}"
+        absolute_ops_file_path = ops_file.package_file.dirname.join(rebased_ops_file_relative_path)
+        # puts "absolute path: #{absolute_ops_file_path}"
+        ops_file = OpsFile.new(self, absolute_ops_file_path)
+      end
+      # puts "run2: #{ops_file.ops_file_path}"
+
+      op = OperationRunner.new(self, ops_file)
+      # if op.requires_sudo?
+      #   prompt_sudo_password unless sudo_password
+      # end
+      # exit_status, out, err, script_output_structure = op.run(operation_kv_args, params_json_hash: @params, verbose: @verbose)
+      result = op.run(operation_kv_args, params_json_hash: @params, verbose: @verbose)
+      exit_status = result.exit_status
+
+      if @verbose
+        puts "Op exit_status"
+        puts exit_status
+
+        # puts "Op stdout"
+        # puts out
+
+        # puts "Op stderr"
+        # puts err
+
+        puts "Op output"
+        # puts script_output_structure ? JSON.pretty_generate(script_output_structure) : nil.inspect
+        puts JSON.pretty_generate(result.value)
+      end
+
+      exit_status
+    ensure
+      FileUtils.remove_entry(tmp_dir) if tmp_dir
+    end
+
+    # package_operation_and_args can take one of the following forms:
+    # - ["github.com/davidkellis/my-package", "operation1", "arg1:val1", "arg2:val2", "arg3:val3"]
+    # - ["foo.zip", "foo/myfile.ops", "arg1:val1", "arg2:val2", "arg3:val3"]
+    # - ["davidkellis/my-package", "operation1", "arg1:val1", "arg2:val2", "arg3:val3"]
+    # - ["davidkellis/my-package", "operation1"]
+    # - ["my-package/operation1", "arg1:val1", "arg2:val2", "arg3:val3"]
+    # - ["./my-package/operation1", "arg1:val1", "arg2:val2", "arg3:val3"]
+    # - ["../../my-package/operation1", "arg1:val1", "arg2:val2", "arg3:val3"]
+    # - ["../../my-package/operation1"]
+    #
+    # returns 3-tuple of the form: [ ops_file_path, operation_kv_args, optional_tmp_dir ]
+    # such that the third item - optional_tmp_dir - if present, should be deleted after the script has completed running
+    def get_entry_point_ops_file_and_args(package_operation_and_args)
+      package_operation_and_args = package_operation_and_args.dup
+      package_or_ops_file_reference = package_operation_and_args.slice!(0, 1).first
+      tmp_dir = nil
+
+      case
+      when File.exist?(package_or_ops_file_reference)
+        first_filepath = package_or_ops_file_reference.to_pathname.realpath
+        ops_file_path = case first_filepath.extname.downcase
+        when ".ops"
+          first_filepath
+        when ".zip"
+          tmp_dir = Dir.mktmpdir.to_pathname
+
+          # unzip the bundle into the temp directory
+          DirZipper.unzip(first_filepath, tmp_dir)
+
+          relative_ops_path = package_operation_and_args.slice!(0, 1).first
+
+          tmp_dir.join(relative_ops_path)
+        else
+          raise Error, "Unknown file type for entrypoint: #{first_filepath}"
+        end
+        operation_kv_args = package_operation_and_args
+        [ops_file_path, operation_kv_args, tmp_dir ]
+      when repo_url = git_repo?(package_or_ops_file_reference)
+        destination_package_path = bundler.download_git_package(repo_url)
+
+        ops_file_path = nil
+        base_path = Pathname.new(destination_package_path)
+        path_parts = 0
+        package_operation_and_args.each do |candidate_path_arg|
+          candidate_base_path = base_path.join(candidate_path_arg)
+          candidate_ops_file = candidate_base_path.sub_ext(".ops")
+          if candidate_ops_file.exist?
+            path_parts += 1
+            ops_file_path = candidate_ops_file
+            break
+          elsif candidate_base_path.exist?
+            path_parts += 1
+          else
+            raise Error, "Operation not found in #{repo_url}: #{candidate_base_path}"
+          end
+          base_path = candidate_base_path
+        end
+        operation_kv_args = package_operation_and_args.drop(path_parts)
+
+        # for an original package_operation_and_args of ["github.com/davidkellis/my-package", "operation1", "arg1:val1", "arg2:val2", "arg3:val3"]
+        # we return: [ "#{pwd}/#{Bundler::BUNDLE_DIR}/github-com-davidkellis-my-package/operation1.ops", ["arg1:val1", "arg2:val2", "arg3:val3"] ]
+        [ops_file_path, operation_kv_args, tmp_dir]
+      else
+        raise Error, "Unknown operation reference: #{package_or_ops_file_reference.inspect}"
+      end
+    end
+
+    # git_repo?("davidkellis/arborist") -> "https://github.com/davidkellis/arborist"
+    # returns the repo URL
+    def git_repo?(repo_reference)
+      candidate_repo_references = [
+        repo_reference,
+        repo_reference =~ /(\.(com|net|org|dev|io|local))\// && "https://#{repo_reference}",
+        repo_reference !~ /github\.com\// && repo_reference =~ /^[a-z\d](?:[a-z\d]|-(?=[a-z\d])){0,38}\/([\w\.@\:\-~]+)$/i && "https://github.com/#{repo_reference}"    # this regex is from https://www.npmjs.com/package/github-username-regex and https://www.debuggex.com/r/H4kRw1G0YPyBFjfm
+      ].compact
+      working_repo_reference = candidate_repo_references.find {|reference| Git.ls_remote(reference) rescue nil }
+      working_repo_reference
+    end
+
+    def bundle_status
+    end
+
+    def bundle_update
+      bundler.update
+    end
+
+    def report_inventory(host_references, tags: nil)
+      selected_hosts = inventory(tags, host_references)
+
+      selected_hosts.each do |host|
+        puts host.summary(verbose?)
+      end
+    end
+
+    # tag_selection is an array of strings
+    def inventory(tag_selection = nil, host_references_override = nil)
+      host_references = host_references_override || @inventory_host_references
+      tags = @inventory_tag_selections + (tag_selection || [])
+      tags.uniq!
+
+      host_references = ["hosts.yml"] if (host_references.nil? || host_references.empty?) && File.exist?("hosts.yml")
+
+      hosts_files, host_strings = host_references.partition {|ref| File.exist?(ref) }
+      hosts_files = hosts_files.map {|file_path| HostsFile.new(file_path) }
+      untagged_hosts = host_strings.map(&:strip).uniq.map {|host| Host.new(host) }
+      inventory_file_hosts = hosts_files.reduce({}) do |host_map, hosts_file|
+        hosts_file.hosts.each do |host|
+          (host_map[host] ||= host).tag!(host.tags)
+        end
+
+        host_map
+      end.keys
+      all_hosts = untagged_hosts + inventory_file_hosts
+
+      selected_hosts = if tags.empty?
+        all_hosts
+      else
+        all_hosts.select do |host|
+          tags.all? {|t| host.tags.include? t }
+        end
+      end
+
+      selected_hosts.sort_by(&:to_s)
+    end
+
+    def unzip(zip_bundle_file = nil, output_dir = nil)
+      bundler.unzip(zip_bundle_file, output_dir)
+    end
+
+    def zip
+      tmpzip = pwd.join("tmpops.zip")
+      FileUtils.rm(tmpzip) if tmpzip.exist?
+      @zip_bundle_path ||= DirZipper.zip(pwd, tmpzip)
+    end
+
+  end
+end

data/lib/opswalrus/bootstrap.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# if brew is already installed, initialize this shell environment with brew
+if [ -x "$(command -v /home/linuxbrew/.linuxbrew/bin/brew)" ]; then
+  eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"
+
+  # exit early if ruby already exists
+  if [ -x "$(command -v ruby)" ]; then
+    echo 'Ruby is already installed.' >&2
+    exit 0
+  fi
+fi
+
+OS=$(cat /etc/os-release | grep "^ID=")
+if echo $OS | grep -q 'ubuntu'; then
+  # update package list
+  sudo apt update -qy
+
+  if [ -f /var/run/reboot-required ]; then
+    echo 'A system reboot is required!'
+    exit 1
+  fi
+
+  # there are probably some services that need restarting because they're using old libraries, so we'll just do the easy thing and reboot
+  sudo DEBIAN_FRONTEND=noninteractive apt install -yq needrestart
+
+  # install homebrew dependencies, per https://docs.brew.sh/Homebrew-on-Linux
+  sudo DEBIAN_FRONTEND=noninteractive apt-get install -yq build-essential procps curl file git
+
+  # restart services that need it
+  sudo needrestart -q -r a
+  sudo needrestart -q -r a
+  sudo needrestart -q -r a
+elif echo $OS | grep -q 'fedora'; then
+  sudo yum groupinstall -y 'Development Tools'
+  sudo yum install -y procps-ng curl file git
+elif echo $OS | grep -q 'arch'; then
+  sudo pacman -Syu --noconfirm base-devel procps-ng curl file git
+else
+  echo "unsupported OS"
+  exit 1
+fi
+
+
+# install homebrew
+NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
+
+# initialize brew in shell session
+eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"
+
+# install gcc, ruby, age
+brew install gcc
+brew install ruby
+brew install age    # https://github.com/FiloSottile/age
+
+# install opswalrus gem
+gem install opswalrus

data/lib/opswalrus/bootstrap_linux_host1.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+# update package list
+sudo apt update -y
+
+# update OS
+# sudo DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confold" -o Dpkg::Options::="--force-confdef" dist-upgrade -q -y --allow-downgrades --allow-remove-essential --allow-change-held-packages
+
+if [ -f /var/run/reboot-required ]; then
+echo 'A system reboot is required!'
+sudo reboot
+fi