mrsk 0.10.1 → 0.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d53c75ef2fad59a884ae2e8a04fff57a68330faf7522ef7bb620affc48a803d6
-  data.tar.gz: b91edccb441e562d24680034df00fbcdf3518aadd570784fbf31d9f2c07a9ebd
+  metadata.gz: 8a09720b016119167213b3c1aab02e56c5e3de0eaa09dc39996d55931048e3e5
+  data.tar.gz: 7b435e7ff711c7267cb3f1771d56791fe76c1fb62950daeeacfe923288c314e8
 SHA512:
-  metadata.gz: d7037c8e7a41acd4cb199b29251245f3e99f2c9cf62ca99c392ba7dfb77defde655f616161e6ef82319ea14438fb27992a8a88861bb49a2a1efe1771f92d60e0
-  data.tar.gz: a0810f91aeb9332ea8401d2a7f8ca847f78a37a85f67b71753734fe2f0ef298435f813fc07fa156bb755e05170afd09f2bdc384504041276ca2502b19ab1d7a6
+  metadata.gz: de31f5f7e47e1f93446603e48a9a1760fca96bc41ee6dfec0660030fb0fbcdcb3fb4145d4b81fda9c3f4eef5de0a205f54e67e7bb27c91d7ebd0c93c3f5d0c57
+  data.tar.gz: 48d36ec263b4ccfd3729059eb85f340717b17406f88e8281955e948fcfe9c30ffe081bad01977e6f9836ecd480361cf94f15cdfe79aa7477c18a6fb6ad9b97bb

data/README.md

@@ -6,6 +6,8 @@ Watch the screencast: https://www.youtube.com/watch?v=LL1cV2FXZ5I
 
 Join us on Discord: https://discord.gg/YgHVT7GCXS
 
+Ask questions: https://github.com/mrsked/mrsk/discussions
+
 ## Installation
 
 If you have a Ruby environment available, you can install MRSK globally with:
@@ -14,13 +16,13 @@ If you have a Ruby environment available, you can install MRSK globally with:
 gem install mrsk
 ```
 
-...otherwise, you can run a dockerized version via an alias (add this to your ${SHELL}rc to simplify re-use):
+...otherwise, you can run a dockerized version via an alias (add this to your .bashrc or similar to simplify re-use):
 
 ```sh
 alias mrsk='docker run --rm -it -v $HOME/.ssh:/root/.ssh -v /var/run/docker.sock:/var/run/docker.sock -v ${PWD}/:/workdir  ghcr.io/mrsked/mrsk'
 ```
 
-Then, inside your app directory, run `mrsk init` (or `mrsk init --bundle` within Rails apps where you want a bin/mrsk binstub). Now edit the new file `config/deploy.yml`. It could look as simple as this:
+Then, inside your app directory, run `mrsk init` (or `mrsk init --bundle` within Rails 7+ apps where you want a bin/mrsk binstub). Now edit the new file `config/deploy.yml`. It could look as simple as this:
 
 ```yaml
 service: hey
@@ -191,6 +193,15 @@ ssh:
   user: app
 ```
 
+If you are using non-root user, you need to bootstrap your servers manually, before using them with MRSK. On Ubuntu, you'd do:
+
+```bash
+sudo apt update
+sudo apt upgrade -y
+sudo apt install -y docker.io curl git
+sudo usermod -a -G docker ubuntu
+```
+
 ### Using a proxy SSH host
 
 If you need to connect to server through a proxy host, you can use `ssh/proxy`:
@@ -207,6 +218,13 @@ ssh:
   proxy: "app@192.168.0.1"
 ```
 
+Also if you need specific proxy command to connect to the server:
+
+```yaml
+ssh:
+  proxy_command: aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --region=us-east-1 ## ssh via aws ssm
+```
+
 ### Using env variables
 
 You can inject env variables into the app containers using `env`:
@@ -288,8 +306,9 @@ You can specialize the default Traefik rules by setting labels on the containers
 
 ```yaml
 labels:
-  traefik.http.routers.hey.rule: Host(`app.hey.com`)
+  traefik.http.routers.hey-web.rule: Host(`app.hey.com`)
 ```
+Traefik rules are in the "service-role-destination" format. The default role will be `web` if no rule is specified. If the destination is not specified, it is not included. To give an example, the above rule would become "traefik.http.routers.hey-web.rule" if it was for the "staging" destination.
 
 Note: The backticks are needed to ensure the rule is passed in correctly and not treated as command substitution by Bash!
 
@@ -312,6 +331,21 @@ servers:
       my-label: "50"
 ```
 
+### Using shell expansion
+
+You can use shell expansion to interpolate values from the host machine into labels and env variables with the `${}` syntax.
+Anything within the curly braces will be executed on the host machine and the result will be interpolated into the label or env variable.
+
+```yaml
+labels:
+  host-machine: "${cat /etc/hostname}"
+
+env:
+  HOST_DEPLOYMENT_DIR: "${PWD}"
+```
+
+Note: Any other occurrence of `$` will be escaped to prevent unwanted shell expansion!
+
 ### Using container options
 
 You can specialize the options used to start containers using the `options` definitions:
@@ -439,9 +473,9 @@ RUN --mount=type=secret,id=GITHUB_TOKEN \
   rm -rf /usr/local/bundle/cache
 ```
 
-### Using command arguments for Traefik
+### Traefik command arguments
 
-You can customize the traefik command line:
+Customize the Traefik command line using `args`:
 
 ```yaml
 traefik:
@@ -450,37 +484,70 @@ traefik:
     accesslog.format: json
 ```
 
-This will start the traefik container with `--accesslog=true accesslog.format=json`.
+This starts the Traefik container with `--accesslog=true --accesslog.format=json` arguments.
 
-### Traefik's host port binding
+### Traefik host port binding
 
-By default Traefik binds to port 80 of the host machine, it can be configured to use an alternative port:
+Traefik binds to port 80 by default. Specify an alternative port using `host_port`:
 
 ```yaml
 traefik:
   host_port: 8080
 ```
 
-### Configure docker options for traefik
+### Traefik version, upgrades, and custom images
 
-We allow users to pass additional docker options to the trafik container like
+MRSK runs the traefik:v2.9 image to track Traefik 2.9.x releases.
+
+To pin Traefik to a specific version or an image published to your registry,
+specify `image`:
+
+```yaml
+traefik:
+  image: traefik:v2.10.0-rc1
+```
+
+This is useful for downgrading Traefik if there's an unexpected breaking
+change in a minor version release, upgrading Traefik to test forthcoming
+releases, or running your own Traefik-derived image.
+
+MRSK has not been tested for compatibility with Traefik 3 betas. Please do!
+
+### Traefik container configuration
+
+Pass additional Docker configuration for the Traefik container using `options`:
 
 ```yaml
 traefik:
   options:
     publish:
-    - 8080:8080
+      - 8080:8080
     volumes:
-    - /tmp/example.json:/tmp/example.json
+      - /tmp/example.json:/tmp/example.json
     memory: 512m
 ```
 
-This will start the traefik container with a command like: `docker run ... --volume /tmp/example.json:/tmp/example.json --publish 8080:8080 `
+This starts the Traefik container with `--volume /tmp/example.json:/tmp/example.json --publish 8080:8080 --memory 512m` arguments to `docker run`.
+
+### Traefik container labels
+
+Add labels to Traefik Docker container.
 
+```yaml
+traefik:
+  labels:
+    traefik.enable: true
+    traefik.http.routers.dashboard.rule: Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
+    traefik.http.routers.dashboard.service: api@internal
+    traefik.http.routers.dashboard.middlewares: auth
+    traefik.http.middlewares.auth.basicauth.users: test:$2y$05$H2o72tMaO.TwY1wNQUV1K.fhjRgLHRDWohFvUZOJHBEtUXNKrqUKi # test:password
+```
+
+This labels Traefik container with `--label traefik.http.routers.dashboard.middlewares=\"auth\"` and so on.
 
-### Configure alternate entrypoints for traefik
+### Traefik alternate entrypoints
 
-You can configure multiple entrypoints for traefik like so:
+You can configure multiple entrypoints for Traefik like so:
 
 ```yaml
 service: myservice
@@ -540,7 +607,7 @@ accessories:
       memory: "2GB"
   redis:
     image: redis:latest
-    role:
+    roles:
       - web
     port: "36379:6379"
     volumes:
@@ -610,18 +677,45 @@ That'll post a line like follows to a preconfigured chatbot in Basecamp:
 [My App] [dhh] Rolled back to version d264c4e92470ad1bd18590f04466787262f605de
 ```
 
-### Using custom healthcheck path or port
+### Healthcheck
+
+MRSK uses Docker healtchecks to check the health of your application during deployment. Traefik uses this same healthcheck status to determine when a container is ready to receive traffic.
 
-MRSK defaults to checking the health of your application again `/up` on port 3000. You can tailor both with the `healthcheck` setting:
+The healthcheck defaults to testing the HTTP response to the path `/up` on port 3000, up to 7 times. You can tailor this behaviour with the `healthcheck` setting:
 
 ```yaml
 healthcheck:
   path: /healthz
   port: 4000
+  max_attempts: 7
 ```
 
 This will ensure your application is configured with a traefik label for the healthcheck against `/healthz` and that the pre-deploy healthcheck that MRSK performs is done against the same path on port 4000.
 
+You can also specify a custom healthcheck command, which is useful for non-HTTP services:
+
+```yaml
+healthcheck:
+  cmd: /bin/check_health
+```
+
+The top-level healthcheck configuration applies to all services that use
+Traefik, by default. You can also specialize the configuration at the role
+level:
+
+```yaml
+servers:
+  job:
+    hosts: ...
+    cmd: bin/jobs
+    healthcheck:
+      cmd: bin/check
+```
+
+The healthcheck allows for an optional `max_attempts` setting, which will attempt the healthcheck up to the specified number of times before failing the deploy. This is useful for applications that take a while to start up. The default is 7.
+
+Note that the HTTP health checks assume that the `curl` command is avilable inside the container. If that's not the case, use the healthcheck's `cmd` option to specify an alternative check that the container supports.
+
 ## Commands
 
 ### Running commands on servers
@@ -761,6 +855,24 @@ mrsk lock acquire -m "Doing maintanence"
 mrsk lock release
 ```
 
+## Rolling deployments
+
+When deploying to large numbers of hosts, you might prefer not to restart your services on every host at the same time.
+
+MRSK's default is to boot new containers on all hosts in parallel. But you can control this by configuring `boot/limit` and `boot/wait` as options:
+
+```yaml
+service: myservice
+
+boot:
+  limit: 10 # Can also specify as a percentage of total hosts, such as "25%"
+  wait: 2
+```
+
+When `limit` is specified, containers will be booted on, at most, `limit` hosts at once. MRSK will pause for `wait` seconds between batches.
+
+These settings only apply when booting containers (using `mrsk deploy`, or `mrsk app boot`). For other commands, MRSK continues to run commands in parallel across all hosts.
+
 ## Stage of development
 
 This is beta software. Commands may still move around. But we're live in production at [37signals](https://37signals.com).

data/lib/mrsk/cli/app.rb

@@ -6,27 +6,33 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
       using_version(version_or_latest) do |version|
         say "Start container with version #{version} using a #{MRSK.config.readiness_delay}s readiness delay (or reboot if already running)...", :magenta
 
-        cli = self
+        on(MRSK.hosts) do
+          execute *MRSK.auditor.record("Tagging #{MRSK.config.absolute_image} as the latest image"), verbosity: :debug
+          execute *MRSK.app.tag_current_as_latest
+        end
 
-        on(MRSK.hosts) do |host|
+        on(MRSK.hosts, **MRSK.boot_strategy) do |host|
           roles = MRSK.roles_on(host)
 
           roles.each do |role|
-            execute *MRSK.auditor(role: role).record("Booted app version #{version}"), verbosity: :debug
-
-            begin
-              if capture_with_info(*MRSK.app(role: role).container_id_for_version(version)).present?
-                tmp_version = "#{version}_#{SecureRandom.hex(8)}"
-                info "Renaming container #{version} to #{tmp_version} as already deployed on #{host}"
-                execute *MRSK.auditor(role: role).record("Renaming container #{version} to #{tmp_version}"), verbosity: :debug
-                execute *MRSK.app(role: role).rename_container(version: version, new_version: tmp_version)
-              end
-
-              old_version = capture_with_info(*MRSK.app(role: role).current_running_version).strip
-              execute *MRSK.app(role: role).run
-              sleep MRSK.config.readiness_delay
-              execute *MRSK.app(role: role).stop(version: old_version), raise_on_non_zero_exit: false if old_version.present?
+            app = MRSK.app(role: role)
+            auditor = MRSK.auditor(role: role)
+
+            execute *auditor.record("Booted app version #{version}"), verbosity: :debug
+
+            if capture_with_info(*app.container_id_for_version(version), raise_on_non_zero_exit: false).present?
+              tmp_version = "#{version}_#{SecureRandom.hex(8)}"
+              info "Renaming container #{version} to #{tmp_version} as already deployed on #{host}"
+              execute *auditor.record("Renaming container #{version} to #{tmp_version}"), verbosity: :debug
+              execute *app.rename_container(version: version, new_version: tmp_version)
             end
+
+            old_version = capture_with_info(*app.current_running_version, raise_on_non_zero_exit: false).strip
+            execute *app.run
+
+            Mrsk::Utils::HealthcheckPoller.wait_for_healthy(pause_after_ready: true) { capture_with_info(*app.status(version: version)) }
+
+            execute *app.stop(version: old_version), raise_on_non_zero_exit: false if old_version.present?
           end
         end
       end
@@ -124,6 +130,31 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
     on(MRSK.hosts) { |host| puts_by_host host, capture_with_info(*MRSK.app.list_containers) }
   end
 
+  desc "stale_containers", "Detect app stale containers"
+  option :stop, aliases: "-s", type: :boolean, default: false, desc: "Stop the stale containers found"
+  def stale_containers
+    with_lock do
+      stop = options[:stop]
+
+      cli = self
+
+      on(MRSK.hosts) do |host|
+        roles = MRSK.roles_on(host)
+
+        roles.each do |role|
+          cli.send(:stale_versions, host: host, role: role).each do |version|
+            if stop
+              puts_by_host host, "Stopping stale container for role #{role} with version #{version}"
+              execute *MRSK.app(role: role).stop(version: version), raise_on_non_zero_exit: false
+            else
+              puts_by_host host,  "Detected stale container for role #{role} with version #{version} (use `mrsk app stale_containers --stop` to stop)"
+            end
+          end
+        end
+      end
+    end
+  end
+
   desc "images", "Show app images on servers"
   def images
     on(MRSK.hosts) { |host| puts_by_host host, capture_with_info(*MRSK.app.list_images) }
@@ -240,6 +271,17 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
       version.presence
     end
 
+    def stale_versions(host:, role:)
+      versions = nil
+      on(host) do
+        versions = \
+          capture_with_info(*MRSK.app(role: role).list_versions, raise_on_non_zero_exit: false)
+          .split("\n")
+          .drop(1)
+      end
+      versions
+    end
+
     def version_or_latest
       options[:version] || "latest"
     end

data/lib/mrsk/cli/base.rb

@@ -6,8 +6,6 @@ module Mrsk::Cli
   class Base < Thor
     include SSHKit::DSL
 
-    class LockError < StandardError; end
-
     def self.exit_on_failure?() true end
 
     class_option :verbose, type: :boolean, aliases: "-v", desc: "Detailed logging"
@@ -79,32 +77,55 @@ module Mrsk::Cli
       end
 
       def with_lock
-        acquire_lock
+        if MRSK.holding_lock?
+          yield
+        else
+          acquire_lock
+
+          begin
+            yield
+          rescue
+            if MRSK.hold_lock_on_error?
+              error "  \e[31mDeploy lock was not released\e[0m"
+            else
+              release_lock
+            end
+
+            raise
+          end
 
-        yield
-      ensure
-        release_lock
+          release_lock
+        end
       end
 
       def acquire_lock
-        if MRSK.lock_count == 0
-          say "Acquiring the deploy lock"
-          on(MRSK.primary_host) { execute *MRSK.lock.acquire("Automatic deploy lock", MRSK.config.version) }
-        end
-        MRSK.lock_count += 1
+        say "Acquiring the deploy lock"
+        on(MRSK.primary_host) { execute *MRSK.lock.acquire("Automatic deploy lock", MRSK.config.version) }
+
+        MRSK.holding_lock = true
       rescue SSHKit::Runner::ExecuteError => e
         if e.message =~ /cannot create directory/
-          invoke "mrsk:cli:lock:status", []
+          on(MRSK.primary_host) { execute *MRSK.lock.status }
+          raise LockError, "Deploy lock found"
+        else
+          raise e
         end
-
-        raise LockError, "Deploy lock found"
       end
 
       def release_lock
-        MRSK.lock_count -= 1
-        if MRSK.lock_count == 0
-          say "Releasing the deploy lock"
-          on(MRSK.primary_host) { execute *MRSK.lock.release }
+        say "Releasing the deploy lock"
+        on(MRSK.primary_host) { execute *MRSK.lock.release }
+
+        MRSK.holding_lock = false
+      end
+
+      def hold_lock_on_error
+        if MRSK.hold_lock_on_error?
+          yield
+        else
+          MRSK.hold_lock_on_error = true
+          yield
+          MRSK.hold_lock_on_error = false
         end
       end
   end

data/lib/mrsk/cli/build.rb

@@ -1,4 +1,6 @@
 class Mrsk::Cli::Build < Mrsk::Cli::Base
+  class BuildError < StandardError; end
+
   desc "deliver", "Build app and push app image to registry then pull image on servers"
   def deliver
     with_lock do
@@ -14,7 +16,9 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
 
       run_locally do
         begin
-          MRSK.with_verbosity(:debug) { execute *MRSK.builder.push }
+          if cli.verify_local_dependencies
+            MRSK.with_verbosity(:debug) { execute *MRSK.builder.push }
+          end
         rescue SSHKit::Command::Failed => e
           if e.message =~ /(no builder)|(no such file or directory)/
             error "Missing compatible builder, so creating a new one first"
@@ -77,4 +81,22 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
       puts capture(*MRSK.builder.info)
     end
   end
+
+
+  desc "", "" # Really a private method, but needed to be invoked from #push
+  def verify_local_dependencies
+    run_locally do
+      begin
+        execute *MRSK.builder.ensure_local_dependencies_installed
+      rescue SSHKit::Command::Failed => e
+        build_error = e.message =~ /command not found/ ?
+          "Docker is not installed locally" :
+          "Docker buildx plugin is not installed locally"
+
+        raise BuildError, build_error
+      end
+    end
+
+    true
+  end
 end

data/lib/mrsk/cli/healthcheck.rb

@@ -1,8 +1,4 @@
 class Mrsk::Cli::Healthcheck < Mrsk::Cli::Base
-  MAX_ATTEMPTS = 7
-
-  class HealthcheckError < StandardError; end
-
   default_command :perform
 
   desc "perform", "Health check current app version"
@@ -10,37 +6,10 @@ class Mrsk::Cli::Healthcheck < Mrsk::Cli::Base
     on(MRSK.primary_host) do
       begin
         execute *MRSK.healthcheck.run
-
-        target = "Health check against #{MRSK.config.healthcheck["path"]}"
-        attempt = 1
-
-        begin
-          status = capture_with_info(*MRSK.healthcheck.curl)
-
-          if status == "200"
-            info "#{target} succeeded with 200 OK!"
-          else
-            raise HealthcheckError, "#{target} failed with status #{status}"
-          end
-        rescue SSHKit::Command::Failed
-          if attempt <= MAX_ATTEMPTS
-            info "#{target} failed to respond, retrying in #{attempt}s..."
-            sleep attempt
-            attempt += 1
-
-            retry
-          else
-            raise
-          end
-        end
-      rescue SSHKit::Command::Failed, HealthcheckError => e
+        Mrsk::Utils::HealthcheckPoller.wait_for_healthy { capture_with_info(*MRSK.healthcheck.status) }
+      rescue Mrsk::Utils::HealthcheckPoller::HealthcheckError => e
         error capture_with_info(*MRSK.healthcheck.logs)
-
-        if e.message =~ /curl/
-          raise SSHKit::Command::Failed, "#{target} failed to return 200 OK!"
-        else
-          raise
-        end
+        raise
       ensure
         execute *MRSK.healthcheck.stop, raise_on_non_zero_exit: false
         execute *MRSK.healthcheck.remove, raise_on_non_zero_exit: false

data/lib/mrsk/cli/lock.rb

@@ -12,7 +12,7 @@ class Mrsk::Cli::Lock < Mrsk::Cli::Base
     message = options[:message]
     handle_missing_lock do
       on(MRSK.primary_host) { execute *MRSK.lock.acquire(message, MRSK.config.version) }
-      say "Set the deploy lock"
+      say "Acquired the deploy lock"
     end
   end
 
@@ -20,7 +20,7 @@ class Mrsk::Cli::Lock < Mrsk::Cli::Base
   def release
     handle_missing_lock do
       on(MRSK.primary_host) { execute *MRSK.lock.release }
-      say "Removed the deploy lock"
+      say "Released the deploy lock"
     end
   end