mrsk 0.11.0 → 0.12.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3a8c9aac5626518667d963bd8c0b8e757be3dfe2cfed585f1e7d0783e2e017a4
4
- data.tar.gz: 9e7c336b776e518d98a3a25a7a3c47858cf40f2d8398c881d8c3ec3f224d1222
3
+ metadata.gz: 8a09720b016119167213b3c1aab02e56c5e3de0eaa09dc39996d55931048e3e5
4
+ data.tar.gz: 7b435e7ff711c7267cb3f1771d56791fe76c1fb62950daeeacfe923288c314e8
5
5
  SHA512:
6
- metadata.gz: 3889961797501b12da9ca958021b98a5a2b1167943813156241e5da44cc35b3d6ace84b56f4850b50e351cd995be099821d791e2136001f37ba700f8310e5128
7
- data.tar.gz: a15e1ad08dfc1c8471f0fa341d865ffc237d88aae229cd250b49feb1d4ca10e750ddc2dec85c392974017c13c6d7b2f583fb3af85e840d5dddcbed19c8f90712
6
+ metadata.gz: de31f5f7e47e1f93446603e48a9a1760fca96bc41ee6dfec0660030fb0fbcdcb3fb4145d4b81fda9c3f4eef5de0a205f54e67e7bb27c91d7ebd0c93c3f5d0c57
7
+ data.tar.gz: 48d36ec263b4ccfd3729059eb85f340717b17406f88e8281955e948fcfe9c30ffe081bad01977e6f9836ecd480361cf94f15cdfe79aa7477c18a6fb6ad9b97bb
data/README.md CHANGED
@@ -331,6 +331,21 @@ servers:
331
331
  my-label: "50"
332
332
  ```
333
333
 
334
+ ### Using shell expansion
335
+
336
+ You can use shell expansion to interpolate values from the host machine into labels and env variables with the `${}` syntax.
337
+ Anything within the curly braces will be executed on the host machine and the result will be interpolated into the label or env variable.
338
+
339
+ ```yaml
340
+ labels:
341
+ host-machine: "${cat /etc/hostname}"
342
+
343
+ env:
344
+ HOST_DEPLOYMENT_DIR: "${PWD}"
345
+ ```
346
+
347
+ Note: Any other occurrence of `$` will be escaped to prevent unwanted shell expansion!
348
+
334
349
  ### Using container options
335
350
 
336
351
  You can specialize the options used to start containers using the `options` definitions:
@@ -514,18 +529,18 @@ traefik:
514
529
 
515
530
  This starts the Traefik container with `--volume /tmp/example.json:/tmp/example.json --publish 8080:8080 --memory 512m` arguments to `docker run`.
516
531
 
517
- ### Traefik container lables
532
+ ### Traefik container labels
518
533
 
519
534
  Add labels to Traefik Docker container.
520
535
 
521
536
  ```yaml
522
537
  traefik:
523
- lables:
524
- - traefik.enable: true
525
- - traefik.http.routers.dashboard.rule: Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
526
- - traefik.http.routers.dashboard.service: api@internal
527
- - traefik.http.routers.dashboard.middlewares: auth
528
- - traefik.http.middlewares.auth.basicauth.users: test:$2y$05$H2o72tMaO.TwY1wNQUV1K.fhjRgLHRDWohFvUZOJHBEtUXNKrqUKi # test:password
538
+ labels:
539
+ traefik.enable: true
540
+ traefik.http.routers.dashboard.rule: Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
541
+ traefik.http.routers.dashboard.service: api@internal
542
+ traefik.http.routers.dashboard.middlewares: auth
543
+ traefik.http.middlewares.auth.basicauth.users: test:$2y$05$H2o72tMaO.TwY1wNQUV1K.fhjRgLHRDWohFvUZOJHBEtUXNKrqUKi # test:password
529
544
  ```
530
545
 
531
546
  This labels Traefik container with `--label traefik.http.routers.dashboard.middlewares=\"auth\"` and so on.
@@ -662,9 +677,11 @@ That'll post a line like follows to a preconfigured chatbot in Basecamp:
662
677
  [My App] [dhh] Rolled back to version d264c4e92470ad1bd18590f04466787262f605de
663
678
  ```
664
679
 
665
- ### Custom healthcheck
680
+ ### Healthcheck
666
681
 
667
- MRSK defaults to checking the health of your application again `/up` on port 3000 up to 7 times. You can tailor the behaviour with the `healthcheck` setting:
682
+ MRSK uses Docker healtchecks to check the health of your application during deployment. Traefik uses this same healthcheck status to determine when a container is ready to receive traffic.
683
+
684
+ The healthcheck defaults to testing the HTTP response to the path `/up` on port 3000, up to 7 times. You can tailor this behaviour with the `healthcheck` setting:
668
685
 
669
686
  ```yaml
670
687
  healthcheck:
@@ -675,7 +692,29 @@ healthcheck:
675
692
 
676
693
  This will ensure your application is configured with a traefik label for the healthcheck against `/healthz` and that the pre-deploy healthcheck that MRSK performs is done against the same path on port 4000.
677
694
 
678
- The healthcheck also allows for an optional `max_attempts` setting, which will attempt the healthcheck up to the specified number of times before failing the deploy. This is useful for applications that take a while to start up. The default is 7.
695
+ You can also specify a custom healthcheck command, which is useful for non-HTTP services:
696
+
697
+ ```yaml
698
+ healthcheck:
699
+ cmd: /bin/check_health
700
+ ```
701
+
702
+ The top-level healthcheck configuration applies to all services that use
703
+ Traefik, by default. You can also specialize the configuration at the role
704
+ level:
705
+
706
+ ```yaml
707
+ servers:
708
+ job:
709
+ hosts: ...
710
+ cmd: bin/jobs
711
+ healthcheck:
712
+ cmd: bin/check
713
+ ```
714
+
715
+ The healthcheck allows for an optional `max_attempts` setting, which will attempt the healthcheck up to the specified number of times before failing the deploy. This is useful for applications that take a while to start up. The default is 7.
716
+
717
+ Note that the HTTP health checks assume that the `curl` command is avilable inside the container. If that's not the case, use the healthcheck's `cmd` option to specify an alternative check that the container supports.
679
718
 
680
719
  ## Commands
681
720
 
@@ -816,6 +855,24 @@ mrsk lock acquire -m "Doing maintanence"
816
855
  mrsk lock release
817
856
  ```
818
857
 
858
+ ## Rolling deployments
859
+
860
+ When deploying to large numbers of hosts, you might prefer not to restart your services on every host at the same time.
861
+
862
+ MRSK's default is to boot new containers on all hosts in parallel. But you can control this by configuring `boot/limit` and `boot/wait` as options:
863
+
864
+ ```yaml
865
+ service: myservice
866
+
867
+ boot:
868
+ limit: 10 # Can also specify as a percentage of total hosts, such as "25%"
869
+ wait: 2
870
+ ```
871
+
872
+ When `limit` is specified, containers will be booted on, at most, `limit` hosts at once. MRSK will pause for `wait` seconds between batches.
873
+
874
+ These settings only apply when booting containers (using `mrsk deploy`, or `mrsk app boot`). For other commands, MRSK continues to run commands in parallel across all hosts.
875
+
819
876
  ## Stage of development
820
877
 
821
878
  This is beta software. Commands may still move around. But we're live in production at [37signals](https://37signals.com).
data/lib/mrsk/cli/app.rb CHANGED
@@ -6,27 +6,33 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
6
6
  using_version(version_or_latest) do |version|
7
7
  say "Start container with version #{version} using a #{MRSK.config.readiness_delay}s readiness delay (or reboot if already running)...", :magenta
8
8
 
9
- cli = self
9
+ on(MRSK.hosts) do
10
+ execute *MRSK.auditor.record("Tagging #{MRSK.config.absolute_image} as the latest image"), verbosity: :debug
11
+ execute *MRSK.app.tag_current_as_latest
12
+ end
10
13
 
11
- on(MRSK.hosts) do |host|
14
+ on(MRSK.hosts, **MRSK.boot_strategy) do |host|
12
15
  roles = MRSK.roles_on(host)
13
16
 
14
17
  roles.each do |role|
15
- execute *MRSK.auditor(role: role).record("Booted app version #{version}"), verbosity: :debug
16
-
17
- begin
18
- if capture_with_info(*MRSK.app(role: role).container_id_for_version(version)).present?
19
- tmp_version = "#{version}_#{SecureRandom.hex(8)}"
20
- info "Renaming container #{version} to #{tmp_version} as already deployed on #{host}"
21
- execute *MRSK.auditor(role: role).record("Renaming container #{version} to #{tmp_version}"), verbosity: :debug
22
- execute *MRSK.app(role: role).rename_container(version: version, new_version: tmp_version)
23
- end
24
-
25
- old_version = capture_with_info(*MRSK.app(role: role).current_running_version).strip
26
- execute *MRSK.app(role: role).run
27
- sleep MRSK.config.readiness_delay
28
- execute *MRSK.app(role: role).stop(version: old_version), raise_on_non_zero_exit: false if old_version.present?
18
+ app = MRSK.app(role: role)
19
+ auditor = MRSK.auditor(role: role)
20
+
21
+ execute *auditor.record("Booted app version #{version}"), verbosity: :debug
22
+
23
+ if capture_with_info(*app.container_id_for_version(version), raise_on_non_zero_exit: false).present?
24
+ tmp_version = "#{version}_#{SecureRandom.hex(8)}"
25
+ info "Renaming container #{version} to #{tmp_version} as already deployed on #{host}"
26
+ execute *auditor.record("Renaming container #{version} to #{tmp_version}"), verbosity: :debug
27
+ execute *app.rename_container(version: version, new_version: tmp_version)
29
28
  end
29
+
30
+ old_version = capture_with_info(*app.current_running_version, raise_on_non_zero_exit: false).strip
31
+ execute *app.run
32
+
33
+ Mrsk::Utils::HealthcheckPoller.wait_for_healthy(pause_after_ready: true) { capture_with_info(*app.status(version: version)) }
34
+
35
+ execute *app.stop(version: old_version), raise_on_non_zero_exit: false if old_version.present?
30
36
  end
31
37
  end
32
38
  end
@@ -124,6 +130,31 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
124
130
  on(MRSK.hosts) { |host| puts_by_host host, capture_with_info(*MRSK.app.list_containers) }
125
131
  end
126
132
 
133
+ desc "stale_containers", "Detect app stale containers"
134
+ option :stop, aliases: "-s", type: :boolean, default: false, desc: "Stop the stale containers found"
135
+ def stale_containers
136
+ with_lock do
137
+ stop = options[:stop]
138
+
139
+ cli = self
140
+
141
+ on(MRSK.hosts) do |host|
142
+ roles = MRSK.roles_on(host)
143
+
144
+ roles.each do |role|
145
+ cli.send(:stale_versions, host: host, role: role).each do |version|
146
+ if stop
147
+ puts_by_host host, "Stopping stale container for role #{role} with version #{version}"
148
+ execute *MRSK.app(role: role).stop(version: version), raise_on_non_zero_exit: false
149
+ else
150
+ puts_by_host host, "Detected stale container for role #{role} with version #{version} (use `mrsk app stale_containers --stop` to stop)"
151
+ end
152
+ end
153
+ end
154
+ end
155
+ end
156
+ end
157
+
127
158
  desc "images", "Show app images on servers"
128
159
  def images
129
160
  on(MRSK.hosts) { |host| puts_by_host host, capture_with_info(*MRSK.app.list_images) }
@@ -240,6 +271,17 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
240
271
  version.presence
241
272
  end
242
273
 
274
+ def stale_versions(host:, role:)
275
+ versions = nil
276
+ on(host) do
277
+ versions = \
278
+ capture_with_info(*MRSK.app(role: role).list_versions, raise_on_non_zero_exit: false)
279
+ .split("\n")
280
+ .drop(1)
281
+ end
282
+ versions
283
+ end
284
+
243
285
  def version_or_latest
244
286
  options[:version] || "latest"
245
287
  end
data/lib/mrsk/cli/base.rb CHANGED
@@ -77,25 +77,35 @@ module Mrsk::Cli
77
77
  end
78
78
 
79
79
  def with_lock
80
- acquire_lock
81
-
82
- yield
80
+ if MRSK.holding_lock?
81
+ yield
82
+ else
83
+ acquire_lock
84
+
85
+ begin
86
+ yield
87
+ rescue
88
+ if MRSK.hold_lock_on_error?
89
+ error " \e[31mDeploy lock was not released\e[0m"
90
+ else
91
+ release_lock
92
+ end
93
+
94
+ raise
95
+ end
83
96
 
84
- release_lock
85
- rescue
86
- error " \e[31mDeploy lock was not released\e[0m" if MRSK.lock_count > 0
87
- raise
97
+ release_lock
98
+ end
88
99
  end
89
100
 
90
101
  def acquire_lock
91
- if MRSK.lock_count == 0
92
- say "Acquiring the deploy lock"
93
- on(MRSK.primary_host) { execute *MRSK.lock.acquire("Automatic deploy lock", MRSK.config.version) }
94
- end
95
- MRSK.lock_count += 1
102
+ say "Acquiring the deploy lock"
103
+ on(MRSK.primary_host) { execute *MRSK.lock.acquire("Automatic deploy lock", MRSK.config.version) }
104
+
105
+ MRSK.holding_lock = true
96
106
  rescue SSHKit::Runner::ExecuteError => e
97
107
  if e.message =~ /cannot create directory/
98
- invoke "mrsk:cli:lock:status", []
108
+ on(MRSK.primary_host) { execute *MRSK.lock.status }
99
109
  raise LockError, "Deploy lock found"
100
110
  else
101
111
  raise e
@@ -103,10 +113,19 @@ module Mrsk::Cli
103
113
  end
104
114
 
105
115
  def release_lock
106
- MRSK.lock_count -= 1
107
- if MRSK.lock_count == 0
108
- say "Releasing the deploy lock"
109
- on(MRSK.primary_host) { execute *MRSK.lock.release }
116
+ say "Releasing the deploy lock"
117
+ on(MRSK.primary_host) { execute *MRSK.lock.release }
118
+
119
+ MRSK.holding_lock = false
120
+ end
121
+
122
+ def hold_lock_on_error
123
+ if MRSK.hold_lock_on_error?
124
+ yield
125
+ else
126
+ MRSK.hold_lock_on_error = true
127
+ yield
128
+ MRSK.hold_lock_on_error = false
110
129
  end
111
130
  end
112
131
  end
@@ -1,4 +1,6 @@
1
1
  class Mrsk::Cli::Build < Mrsk::Cli::Base
2
+ class BuildError < StandardError; end
3
+
2
4
  desc "deliver", "Build app and push app image to registry then pull image on servers"
3
5
  def deliver
4
6
  with_lock do
@@ -14,7 +16,9 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
14
16
 
15
17
  run_locally do
16
18
  begin
17
- MRSK.with_verbosity(:debug) { execute *MRSK.builder.push }
19
+ if cli.verify_local_dependencies
20
+ MRSK.with_verbosity(:debug) { execute *MRSK.builder.push }
21
+ end
18
22
  rescue SSHKit::Command::Failed => e
19
23
  if e.message =~ /(no builder)|(no such file or directory)/
20
24
  error "Missing compatible builder, so creating a new one first"
@@ -77,4 +81,22 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
77
81
  puts capture(*MRSK.builder.info)
78
82
  end
79
83
  end
84
+
85
+
86
+ desc "", "" # Really a private method, but needed to be invoked from #push
87
+ def verify_local_dependencies
88
+ run_locally do
89
+ begin
90
+ execute *MRSK.builder.ensure_local_dependencies_installed
91
+ rescue SSHKit::Command::Failed => e
92
+ build_error = e.message =~ /command not found/ ?
93
+ "Docker is not installed locally" :
94
+ "Docker buildx plugin is not installed locally"
95
+
96
+ raise BuildError, build_error
97
+ end
98
+ end
99
+
100
+ true
101
+ end
80
102
  end
@@ -1,7 +1,4 @@
1
1
  class Mrsk::Cli::Healthcheck < Mrsk::Cli::Base
2
-
3
- class HealthcheckError < StandardError; end
4
-
5
2
  default_command :perform
6
3
 
7
4
  desc "perform", "Health check current app version"
@@ -9,38 +6,10 @@ class Mrsk::Cli::Healthcheck < Mrsk::Cli::Base
9
6
  on(MRSK.primary_host) do
10
7
  begin
11
8
  execute *MRSK.healthcheck.run
12
-
13
- target = "Health check against #{MRSK.config.healthcheck["path"]}"
14
- attempt = 1
15
- max_attempts = MRSK.config.healthcheck["max_attempts"]
16
-
17
- begin
18
- status = capture_with_info(*MRSK.healthcheck.curl)
19
-
20
- if status == "200"
21
- info "#{target} succeeded with 200 OK!"
22
- else
23
- raise HealthcheckError, "#{target} failed with status #{status}"
24
- end
25
- rescue SSHKit::Command::Failed
26
- if attempt <= max_attempts
27
- info "#{target} failed to respond, retrying in #{attempt}s (attempt #{attempt}/#{max_attempts})..."
28
- sleep attempt
29
- attempt += 1
30
-
31
- retry
32
- else
33
- raise
34
- end
35
- end
36
- rescue SSHKit::Command::Failed, HealthcheckError => e
9
+ Mrsk::Utils::HealthcheckPoller.wait_for_healthy { capture_with_info(*MRSK.healthcheck.status) }
10
+ rescue Mrsk::Utils::HealthcheckPoller::HealthcheckError => e
37
11
  error capture_with_info(*MRSK.healthcheck.logs)
38
-
39
- if e.message =~ /curl/
40
- raise SSHKit::Command::Failed, "#{target} failed to return 200 OK!"
41
- else
42
- raise
43
- end
12
+ raise
44
13
  ensure
45
14
  execute *MRSK.healthcheck.stop, raise_on_non_zero_exit: false
46
15
  execute *MRSK.healthcheck.remove, raise_on_non_zero_exit: false
data/lib/mrsk/cli/main.rb CHANGED
@@ -17,9 +17,6 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
17
17
  invoke_options = deploy_options
18
18
 
19
19
  runtime = print_runtime do
20
- say "Ensure curl and Docker are installed...", :magenta
21
- invoke "mrsk:cli:server:bootstrap", [], invoke_options
22
-
23
20
  say "Log into image registry...", :magenta
24
21
  invoke "mrsk:cli:registry:login", [], invoke_options
25
22
 
@@ -37,7 +34,12 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
37
34
  say "Ensure app can pass healthcheck...", :magenta
38
35
  invoke "mrsk:cli:healthcheck:perform", [], invoke_options
39
36
 
40
- invoke "mrsk:cli:app:boot", [], invoke_options
37
+ say "Detect stale containers...", :magenta
38
+ invoke "mrsk:cli:app:stale_containers", [], invoke_options
39
+
40
+ hold_lock_on_error do
41
+ invoke "mrsk:cli:app:boot", [], invoke_options
42
+ end
41
43
 
42
44
  say "Prune old containers and images...", :magenta
43
45
  invoke "mrsk:cli:prune:all", [], invoke_options
@@ -65,7 +67,12 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
65
67
  say "Ensure app can pass healthcheck...", :magenta
66
68
  invoke "mrsk:cli:healthcheck:perform", [], invoke_options
67
69
 
68
- invoke "mrsk:cli:app:boot", [], invoke_options
70
+ say "Detect stale containers...", :magenta
71
+ invoke "mrsk:cli:app:stale_containers", [], invoke_options
72
+
73
+ hold_lock_on_error do
74
+ invoke "mrsk:cli:app:boot", [], invoke_options
75
+ end
69
76
  end
70
77
 
71
78
  audit_broadcast "Redeployed #{service_version} in #{runtime.round} seconds" unless options[:skip_broadcast]
@@ -75,34 +82,41 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
75
82
  desc "rollback [VERSION]", "Rollback app to VERSION"
76
83
  def rollback(version)
77
84
  with_lock do
78
- MRSK.config.version = version
79
-
80
- if container_available?(version)
81
- say "Start version #{version}, then wait #{MRSK.config.readiness_delay}s for app to boot before stopping the old version...", :magenta
85
+ invoke_options = deploy_options
82
86
 
83
- cli = self
87
+ hold_lock_on_error do
88
+ MRSK.config.version = version
84
89
  old_version = nil
85
90
 
86
- on(MRSK.hosts) do |host|
87
- roles = MRSK.roles_on(host)
91
+ if container_available?(version)
92
+ say "Start version #{version}, then wait #{MRSK.config.readiness_delay}s for app to boot before stopping the old version...", :magenta
93
+
94
+ on(MRSK.hosts) do
95
+ execute *MRSK.auditor.record("Tagging #{MRSK.config.absolute_image} as the latest image"), verbosity: :debug
96
+ execute *MRSK.app.tag_current_as_latest
97
+ end
98
+
99
+ on(MRSK.hosts) do |host|
100
+ roles = MRSK.roles_on(host)
88
101
 
89
- roles.each do |role|
90
- app = MRSK.app(role: role)
91
- old_version = capture_with_info(*app.current_running_version).strip.presence
102
+ roles.each do |role|
103
+ app = MRSK.app(role: role)
104
+ old_version = capture_with_info(*app.current_running_version).strip.presence
92
105
 
93
- execute *app.start
106
+ execute *app.start
94
107
 
95
- if old_version
96
- sleep MRSK.config.readiness_delay
108
+ if old_version
109
+ sleep MRSK.config.readiness_delay
97
110
 
98
- execute *app.stop(version: old_version), raise_on_non_zero_exit: false
111
+ execute *app.stop(version: old_version), raise_on_non_zero_exit: false
112
+ end
99
113
  end
100
114
  end
101
- end
102
115
 
103
- audit_broadcast "Rolled back #{service_version(Mrsk::Utils.abbreviate_version(old_version))} to #{service_version}" unless options[:skip_broadcast]
104
- else
105
- say "The app version '#{version}' is not available as a container (use 'mrsk app containers' for available versions)", :red
116
+ audit_broadcast "Rolled back #{service_version(Mrsk::Utils.abbreviate_version(old_version))} to #{service_version}" unless options[:skip_broadcast]
117
+ else
118
+ say "The app version '#{version}' is not available as a container (use 'mrsk app containers' for available versions)", :red
119
+ end
106
120
  end
107
121
  end
108
122
  end
@@ -219,15 +233,24 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
219
233
  subcommand "lock", Mrsk::Cli::Lock
220
234
 
221
235
  private
222
- def container_available?(version, host: MRSK.primary_host)
223
- available = nil
224
-
225
- on(host) do
226
- first_role = MRSK.roles_on(host).first
227
- available = capture_with_info(*MRSK.app(role: first_role).container_id_for_version(version)).present?
236
+ def container_available?(version)
237
+ begin
238
+ on(MRSK.hosts) do
239
+ MRSK.roles_on(host).each do |role|
240
+ container_id = capture_with_info(*MRSK.app(role: role).container_id_for_version(version))
241
+ raise "Container not found" unless container_id.present?
242
+ end
243
+ end
244
+ rescue SSHKit::Runner::ExecuteError => e
245
+ if e.message =~ /Container not found/
246
+ say "Error looking for container version #{version}: #{e.message}"
247
+ return false
248
+ else
249
+ raise
250
+ end
228
251
  end
229
252
 
230
- available
253
+ true
231
254
  end
232
255
 
233
256
  def deploy_options
@@ -7,7 +7,7 @@ class Mrsk::Cli::Prune < Mrsk::Cli::Base
7
7
  end
8
8
  end
9
9
 
10
- desc "images", "Prune unused images older than 7 days"
10
+ desc "images", "Prune dangling images"
11
11
  def images
12
12
  with_lock do
13
13
  on(MRSK.hosts) do
@@ -17,7 +17,7 @@ class Mrsk::Cli::Prune < Mrsk::Cli::Base
17
17
  end
18
18
  end
19
19
 
20
- desc "containers", "Prune stopped containers older than 3 days"
20
+ desc "containers", "Prune all stopped containers, except the last 5"
21
21
  def containers
22
22
  with_lock do
23
23
  on(MRSK.hosts) do
@@ -1,17 +1,21 @@
1
1
  class Mrsk::Cli::Server < Mrsk::Cli::Base
2
- desc "bootstrap", "Ensure curl and Docker are installed on servers"
2
+ desc "bootstrap", "Set up Docker to run MRSK apps"
3
3
  def bootstrap
4
- with_lock do
5
- on(MRSK.hosts + MRSK.accessory_hosts) do
6
- dependencies_to_install = Array.new.tap do |dependencies|
7
- dependencies << "curl" unless execute "which curl", raise_on_non_zero_exit: false
8
- dependencies << "docker.io" unless execute "which docker", raise_on_non_zero_exit: false
9
- end
4
+ missing = []
10
5
 
11
- if dependencies_to_install.any?
12
- execute "apt-get update -y && apt-get install #{dependencies_to_install.join(" ")} -y"
6
+ on(MRSK.hosts | MRSK.accessory_hosts) do |host|
7
+ unless execute(*MRSK.docker.installed?, raise_on_non_zero_exit: false)
8
+ if execute(*MRSK.docker.superuser?, raise_on_non_zero_exit: false)
9
+ info "Missing Docker on #{host}. Installing…"
10
+ execute *MRSK.docker.install
11
+ else
12
+ missing << host
13
13
  end
14
14
  end
15
15
  end
16
+
17
+ if missing.any?
18
+ raise "Docker is not installed on #{missing.join(", ")} and can't be automatically installed without having root access and the `curl` command available. Install Docker manually: https://docs.docker.com/engine/install/"
19
+ end
16
20
  end
17
21
  end
@@ -94,7 +94,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
94
94
  end
95
95
  end
96
96
 
97
- desc "remove_container", "Remove Traefik image from servers", hide: true
97
+ desc "remove_image", "Remove Traefik image from servers", hide: true
98
98
  def remove_image
99
99
  with_lock do
100
100
  on(MRSK.traefik_hosts) do
@@ -2,11 +2,12 @@ require "active_support/core_ext/enumerable"
2
2
  require "active_support/core_ext/module/delegation"
3
3
 
4
4
  class Mrsk::Commander
5
- attr_accessor :verbosity, :lock_count
5
+ attr_accessor :verbosity, :holding_lock, :hold_lock_on_error
6
6
 
7
7
  def initialize
8
8
  self.verbosity = :info
9
- self.lock_count = 0
9
+ self.holding_lock = false
10
+ self.hold_lock_on_error = false
10
11
  end
11
12
 
12
13
  def config
@@ -35,7 +36,7 @@ class Mrsk::Commander
35
36
  end
36
37
 
37
38
  def primary_host
38
- specific_hosts&.first || config.primary_web_host
39
+ specific_hosts&.first || specific_roles&.first&.primary_host || config.primary_web_host
39
40
  end
40
41
 
41
42
  def roles
@@ -50,6 +51,14 @@ class Mrsk::Commander
50
51
  end
51
52
  end
52
53
 
54
+ def boot_strategy
55
+ if config.boot.limit.present?
56
+ { in: :groups, limit: config.boot.limit, wait: config.boot.wait }
57
+ else
58
+ {}
59
+ end
60
+ end
61
+
53
62
  def roles_on(host)
54
63
  roles.select { |role| role.hosts.include?(host.to_s) }.map(&:name)
55
64
  end
@@ -83,6 +92,10 @@ class Mrsk::Commander
83
92
  @builder ||= Mrsk::Commands::Builder.new(config)
84
93
  end
85
94
 
95
+ def docker
96
+ @docker ||= Mrsk::Commands::Docker.new(config)
97
+ end
98
+
86
99
  def healthcheck
87
100
  @healthcheck ||= Mrsk::Commands::Healthcheck.new(config)
88
101
  end
@@ -115,6 +128,14 @@ class Mrsk::Commander
115
128
  SSHKit.config.output_verbosity = old_level
116
129
  end
117
130
 
131
+ def holding_lock?
132
+ self.holding_lock
133
+ end
134
+
135
+ def hold_lock_on_error?
136
+ self.hold_lock_on_error
137
+ end
138
+
118
139
  private
119
140
  # Lazy setup of SSHKit
120
141
  def configure_sshkit_with(config)
@@ -15,6 +15,7 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
15
15
  "--name", container_name,
16
16
  "-e", "MRSK_CONTAINER_NAME=\"#{container_name}\"",
17
17
  *role.env_args,
18
+ *role.health_check_args,
18
19
  *config.logging_args,
19
20
  *config.volume_args,
20
21
  *role.label_args,
@@ -27,9 +28,13 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
27
28
  docker :start, container_name
28
29
  end
29
30
 
31
+ def status(version:)
32
+ pipe container_id_for_version(version), xargs(docker(:inspect, "--format", DOCKER_HEALTH_STATUS_FORMAT))
33
+ end
34
+
30
35
  def stop(version: nil)
31
36
  pipe \
32
- version ? container_id_for_version(version) : current_container_id,
37
+ version ? container_id_for_version(version) : current_running_container_id,
33
38
  xargs(config.stop_wait_time ? docker(:stop, "-t", config.stop_wait_time) : docker(:stop))
34
39
  end
35
40
 
@@ -40,7 +45,7 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
40
45
 
41
46
  def logs(since: nil, lines: nil, grep: nil)
42
47
  pipe \
43
- current_container_id,
48
+ current_running_container_id,
44
49
  "xargs docker logs#{" --since #{since}" if since}#{" --tail #{lines}" if lines} 2>&1",
45
50
  ("grep '#{grep}'" if grep)
46
51
  end
@@ -48,7 +53,7 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
48
53
  def follow_logs(host:, grep: nil)
49
54
  run_over_ssh \
50
55
  pipe(
51
- current_container_id,
56
+ current_running_container_id,
52
57
  "xargs docker logs --timestamps --tail 10 --follow 2>&1",
53
58
  (%(grep "#{grep}") if grep)
54
59
  ),
@@ -82,8 +87,8 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
82
87
  end
83
88
 
84
89
 
85
- def current_container_id
86
- docker :ps, "--quiet", *filter_args
90
+ def current_running_container_id
91
+ docker :ps, "--quiet", *filter_args(status: :running), "--latest"
87
92
  end
88
93
 
89
94
  def container_id_for_version(version)
@@ -91,11 +96,14 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
91
96
  end
92
97
 
93
98
  def current_running_version
94
- # FIXME: Find more graceful way to extract the version from "app-version" than using sed and tail!
99
+ list_versions("--latest", status: :running)
100
+ end
101
+
102
+ def list_versions(*docker_args, status: nil)
95
103
  pipe \
96
- docker(:ps, *filter_args, "--format", '"{{.Names}}"'),
97
- %(sed 's/-/\\n/g'),
98
- "tail -n 1"
104
+ docker(:ps, *filter_args(status: status), *docker_args, "--format", '"{{.Names}}"'),
105
+ %(grep -oE "\\-[^-]+$"), # Extract SHA from "service-role-dest-SHA"
106
+ %(cut -c 2-)
99
107
  end
100
108
 
101
109
  def list_containers
@@ -128,20 +136,25 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
128
136
  docker :image, :prune, "--all", "--force", *filter_args
129
137
  end
130
138
 
139
+ def tag_current_as_latest
140
+ docker :tag, config.absolute_image, config.latest_image
141
+ end
142
+
131
143
 
132
144
  private
133
145
  def container_name(version = nil)
134
146
  [ config.service, role, config.destination, version || config.version ].compact.join("-")
135
147
  end
136
148
 
137
- def filter_args
138
- argumentize "--filter", filters
149
+ def filter_args(status: nil)
150
+ argumentize "--filter", filters(status: status)
139
151
  end
140
152
 
141
- def filters
153
+ def filters(status: nil)
142
154
  [ "label=service=#{config.service}" ].tap do |filters|
143
155
  filters << "label=destination=#{config.destination}" if config.destination
144
156
  filters << "label=role=#{role}" if role
157
+ filters << "status=#{status}" if status
145
158
  end
146
159
  end
147
160
  end
@@ -2,6 +2,8 @@ module Mrsk::Commands
2
2
  class Base
3
3
  delegate :sensitive, :argumentize, to: Mrsk::Utils
4
4
 
5
+ DOCKER_HEALTH_STATUS_FORMAT = "'{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'"
6
+
5
7
  attr_accessor :config
6
8
 
7
9
  def initialize(config)
@@ -1,4 +1,7 @@
1
+
1
2
  class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
3
+ class BuilderError < StandardError; end
4
+
2
5
  delegate :argumentize, to: Mrsk::Utils
3
6
 
4
7
  def clean
@@ -7,7 +10,6 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
7
10
 
8
11
  def pull
9
12
  docker :pull, config.absolute_image
10
- docker :pull, config.latest_image
11
13
  end
12
14
 
13
15
  def build_options
@@ -18,6 +20,7 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
18
20
  context
19
21
  end
20
22
 
23
+
21
24
  private
22
25
  def build_tags
23
26
  [ "-t", config.absolute_image, "-t", config.latest_image ]
@@ -36,7 +39,11 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
36
39
  end
37
40
 
38
41
  def build_dockerfile
39
- argumentize "--file", dockerfile
42
+ if Pathname.new(File.expand_path(dockerfile)).exist?
43
+ argumentize "--file", dockerfile
44
+ else
45
+ raise BuilderError, "Missing #{dockerfile}"
46
+ end
40
47
  end
41
48
 
42
49
  def args
@@ -2,7 +2,7 @@ class Mrsk::Commands::Builder < Mrsk::Commands::Base
2
2
  delegate :create, :remove, :push, :clean, :pull, :info, to: :target
3
3
 
4
4
  def name
5
- target.class.to_s.remove("Mrsk::Commands::Builder::").underscore
5
+ target.class.to_s.remove("Mrsk::Commands::Builder::").underscore.inquiry
6
6
  end
7
7
 
8
8
  def target
@@ -33,4 +33,24 @@ class Mrsk::Commands::Builder < Mrsk::Commands::Base
33
33
  def multiarch_remote
34
34
  @multiarch_remote ||= Mrsk::Commands::Builder::Multiarch::Remote.new(config)
35
35
  end
36
+
37
+
38
+ def ensure_local_dependencies_installed
39
+ if name.native?
40
+ ensure_local_docker_installed
41
+ else
42
+ combine \
43
+ ensure_local_docker_installed,
44
+ ensure_local_buildx_installed
45
+ end
46
+ end
47
+
48
+ private
49
+ def ensure_local_docker_installed
50
+ docker "--version"
51
+ end
52
+
53
+ def ensure_local_buildx_installed
54
+ docker :buildx, "version"
55
+ end
36
56
  end
@@ -0,0 +1,21 @@
1
+ class Mrsk::Commands::Docker < Mrsk::Commands::Base
2
+ # Install Docker using the https://github.com/docker/docker-install convenience script.
3
+ def install
4
+ pipe [ :curl, "-fsSL", "https://get.docker.com" ], :sh
5
+ end
6
+
7
+ # Checks the Docker client version. Fails if Docker is not installed.
8
+ def installed?
9
+ docker "-v"
10
+ end
11
+
12
+ # Checks the Docker server version. Fails if Docker is not running.
13
+ def running?
14
+ docker :version
15
+ end
16
+
17
+ # Do we have superuser access to install Docker and start system services?
18
+ def superuser?
19
+ [ '[ "${EUID:-$(id -u)}" -eq 0 ]' ]
20
+ end
21
+ end
@@ -11,14 +11,15 @@ class Mrsk::Commands::Healthcheck < Mrsk::Commands::Base
11
11
  "--label", "service=#{container_name}",
12
12
  "-e", "MRSK_CONTAINER_NAME=\"#{container_name}\"",
13
13
  *web.env_args,
14
+ *web.health_check_args,
14
15
  *config.volume_args,
15
16
  *web.option_args,
16
17
  config.absolute_image,
17
18
  web.cmd
18
19
  end
19
20
 
20
- def curl
21
- [ :curl, "--silent", "--output", "/dev/null", "--write-out", "'%{http_code}'", "--max-time", "2", health_url ]
21
+ def status
22
+ pipe container_id, xargs(docker(:inspect, "--format", DOCKER_HEALTH_STATUS_FORMAT))
22
23
  end
23
24
 
24
25
  def logs
@@ -2,11 +2,19 @@ require "active_support/duration"
2
2
  require "active_support/core_ext/numeric/time"
3
3
 
4
4
  class Mrsk::Commands::Prune < Mrsk::Commands::Base
5
- def images(until_hours: 7.days.in_hours.to_i)
6
- docker :image, :prune, "--all", "--force", "--filter", "label=service=#{config.service}", "--filter", "until=#{until_hours}h"
5
+ def images
6
+ docker :image, :prune, "--all", "--force", "--filter", "label=service=#{config.service}", "--filter", "dangling=true"
7
7
  end
8
8
 
9
- def containers(until_hours: 3.days.in_hours.to_i)
10
- docker :container, :prune, "--force", "--filter", "label=service=#{config.service}", "--filter", "until=#{until_hours}h"
9
+ def containers(keep_last: 5)
10
+ pipe \
11
+ docker(:ps, "-q", "-a", "--filter", "label=service=#{config.service}", *stopped_containers_filters),
12
+ "tail -n +#{keep_last + 1}",
13
+ "while read container_id; do docker rm $container_id; done"
11
14
  end
15
+
16
+ private
17
+ def stopped_containers_filters
18
+ [ "created", "exited", "dead" ].flat_map { |status| ["--filter", "status=#{status}"] }
19
+ end
12
20
  end
@@ -0,0 +1,20 @@
1
+ class Mrsk::Configuration::Boot
2
+ def initialize(config:)
3
+ @options = config.raw_config.boot || {}
4
+ @host_count = config.all_hosts.count
5
+ end
6
+
7
+ def limit
8
+ limit = @options["limit"]
9
+
10
+ if limit.to_s.end_with?("%")
11
+ @host_count * limit.to_i / 100
12
+ else
13
+ limit
14
+ end
15
+ end
16
+
17
+ def wait
18
+ @options["wait"]
19
+ end
20
+ end
@@ -35,6 +35,21 @@ class Mrsk::Configuration::Role
35
35
  argumentize_env_with_secrets env
36
36
  end
37
37
 
38
+ def health_check_args
39
+ if health_check_cmd.present?
40
+ optionize({ "health-cmd" => health_check_cmd, "health-interval" => "1s" })
41
+ else
42
+ []
43
+ end
44
+ end
45
+
46
+ def health_check_cmd
47
+ options = specializations["healthcheck"] || {}
48
+ options = config.healthcheck.merge(options) if running_traefik?
49
+
50
+ options["cmd"] || http_health_check(port: options["port"], path: options["path"])
51
+ end
52
+
38
53
  def cmd
39
54
  specializations["cmd"]
40
55
  end
@@ -74,9 +89,10 @@ class Mrsk::Configuration::Role
74
89
  def traefik_labels
75
90
  if running_traefik?
76
91
  {
92
+ # Setting a service property ensures that the generated service name will be consistent between versions
93
+ "traefik.http.services.#{traefik_service}.loadbalancer.server.scheme" => "http",
94
+
77
95
  "traefik.http.routers.#{traefik_service}.rule" => "PathPrefix(`/`)",
78
- "traefik.http.services.#{traefik_service}.loadbalancer.healthcheck.path" => config.healthcheck["path"],
79
- "traefik.http.services.#{traefik_service}.loadbalancer.healthcheck.interval" => "1s",
80
96
  "traefik.http.middlewares.#{traefik_service}-retry.retry.attempts" => "5",
81
97
  "traefik.http.middlewares.#{traefik_service}-retry.retry.initialinterval" => "500ms",
82
98
  "traefik.http.routers.#{traefik_service}.middlewares" => "#{traefik_service}-retry@docker"
@@ -125,4 +141,8 @@ class Mrsk::Configuration::Role
125
141
  new_env["clear"] = (clear_app_env + clear_role_env).uniq
126
142
  end
127
143
  end
144
+
145
+ def http_health_check(port:, path:)
146
+ "curl -f #{URI.join("http://localhost:#{port}", path)} || exit 1" if path.present? || port.present?
147
+ end
128
148
  end
@@ -87,6 +87,10 @@ class Mrsk::Configuration
87
87
  roles.select(&:running_traefik?).flat_map(&:hosts).uniq
88
88
  end
89
89
 
90
+ def boot
91
+ Mrsk::Configuration::Boot.new(config: self)
92
+ end
93
+
90
94
 
91
95
  def repository
92
96
  [ raw_config.registry["server"], image ].compact.join("/")
@@ -2,8 +2,8 @@ require "sshkit"
2
2
  require "sshkit/dsl"
3
3
 
4
4
  class SSHKit::Backend::Abstract
5
- def capture_with_info(*args)
6
- capture(*args, verbosity: Logger::INFO)
5
+ def capture_with_info(*args, **kwargs)
6
+ capture(*args, **kwargs, verbosity: Logger::INFO)
7
7
  end
8
8
 
9
9
  def puts_by_host(host, output, type: "App")
@@ -0,0 +1,39 @@
1
+ class Mrsk::Utils::HealthcheckPoller
2
+ TRAEFIK_HEALTHY_DELAY = 2
3
+
4
+ class HealthcheckError < StandardError; end
5
+
6
+ class << self
7
+ def wait_for_healthy(pause_after_ready: false, &block)
8
+ attempt = 1
9
+ max_attempts = MRSK.config.healthcheck["max_attempts"]
10
+
11
+ begin
12
+ case status = block.call
13
+ when "healthy"
14
+ sleep TRAEFIK_HEALTHY_DELAY if pause_after_ready
15
+ when "running" # No health check configured
16
+ sleep MRSK.config.readiness_delay if pause_after_ready
17
+ else
18
+ raise HealthcheckError, "container not ready (#{status})"
19
+ end
20
+ rescue HealthcheckError => e
21
+ if attempt <= max_attempts
22
+ info "#{e.message}, retrying in #{attempt}s (attempt #{attempt}/#{max_attempts})..."
23
+ sleep attempt
24
+ attempt += 1
25
+ retry
26
+ else
27
+ raise
28
+ end
29
+ end
30
+
31
+ info "Container is healthy!"
32
+ end
33
+
34
+ private
35
+ def info(message)
36
+ SSHKit.config.output.info(message)
37
+ end
38
+ end
39
+ end
data/lib/mrsk/utils.rb CHANGED
@@ -1,6 +1,8 @@
1
1
  module Mrsk::Utils
2
2
  extend self
3
3
 
4
+ DOLLAR_SIGN_WITHOUT_SHELL_EXPANSION_REGEX = /\$(?!{[^\}]*\})/
5
+
4
6
  # Return a list of escaped shell arguments using the same named argument against the passed attributes (hash or array).
5
7
  def argumentize(argument, attributes, sensitive: false)
6
8
  Array(attributes).flat_map do |key, value|
@@ -75,7 +77,9 @@ module Mrsk::Utils
75
77
 
76
78
  # Escape a value to make it safe for shell use.
77
79
  def escape_shell_value(value)
78
- value.to_s.dump.gsub(/`/, '\\\\`')
80
+ value.to_s.dump
81
+ .gsub(/`/, '\\\\`')
82
+ .gsub(DOLLAR_SIGN_WITHOUT_SHELL_EXPANSION_REGEX, '\$')
79
83
  end
80
84
 
81
85
  # Abbreviate a git revhash for concise display
data/lib/mrsk/version.rb CHANGED
@@ -1,3 +1,3 @@
1
1
  module Mrsk
2
- VERSION = "0.11.0"
2
+ VERSION = "0.12.0"
3
3
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: mrsk
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.11.0
4
+ version: 0.12.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - David Heinemeier Hansson
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2023-04-17 00:00:00.000000000 Z
11
+ date: 2023-05-02 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activesupport
@@ -201,6 +201,7 @@ files:
201
201
  - lib/mrsk/commands/builder/multiarch/remote.rb
202
202
  - lib/mrsk/commands/builder/native.rb
203
203
  - lib/mrsk/commands/builder/native/remote.rb
204
+ - lib/mrsk/commands/docker.rb
204
205
  - lib/mrsk/commands/healthcheck.rb
205
206
  - lib/mrsk/commands/lock.rb
206
207
  - lib/mrsk/commands/prune.rb
@@ -208,9 +209,11 @@ files:
208
209
  - lib/mrsk/commands/traefik.rb
209
210
  - lib/mrsk/configuration.rb
210
211
  - lib/mrsk/configuration/accessory.rb
212
+ - lib/mrsk/configuration/boot.rb
211
213
  - lib/mrsk/configuration/role.rb
212
214
  - lib/mrsk/sshkit_with_ext.rb
213
215
  - lib/mrsk/utils.rb
216
+ - lib/mrsk/utils/healthcheck_poller.rb
214
217
  - lib/mrsk/utils/sensitive.rb
215
218
  - lib/mrsk/version.rb
216
219
  homepage: https://github.com/rails/mrsk