Bump version for 0.15.0

Updated README with info for Rails <7 usage

CONTRIBUTING.md

@@ -27,7 +27,7 @@ Please keep the following guidelines in mind when opening a pull request:
 - Add tests for your changes, if possible.
 - Ensure that your changes don't break existing functionality.
 
-#### Commit message guidline
+#### Commit message guidelines
 
 A good commit message should describe what changed and why.
 

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    mrsk (0.13.0)
+    mrsk (0.15.0)
       activesupport (>= 7.0)
       bcrypt_pbkdf (~> 1.0)
       dotenv (~> 2.8)

README.md

@@ -63,11 +63,29 @@ This will:
 
 Voila! All the servers are now serving the app on port 80. If you're just running a single server, you're ready to go. If you're running multiple servers, you need to put a load balancer in front of them. For subsequent deploys, or if your servers already have Docker and curl installed, you can just run `mrsk deploy`.
 
+### Rails <7 usage
+
+MRSK is not needed to be in your application Gemfile to be used. However, if you want to guarantee specific MRSK version in your CI/CD workflows, you can create a separate Gemfile for MRSK, for example, `gemfile/mrsk.gemfile`:
+
+```ruby
+source 'https://rubygems.org'
+
+gem 'mrsk', '~> 0.14'
+```
+
+Bundle with `BUNDLE_GEMFILE=gemfiles/mrsk.gemfile bundle`.
+
+After this MRSK can be used for deployment:
+
+```sh
+BUNDLE_GEMFILE=gemfiles/mrsk.gemfile bundle exec mrsk deploy
+```
+
 ## Vision
 
 In the past decade+, there's been an explosion in commercial offerings that make deploying web apps easier. Heroku kicked it off with an incredible offering that stayed ahead of the competition seemingly forever. These days we have excellent alternatives like Fly.io and Render. And hosted Kubernetes is making things easier too on AWS, GCP, Digital Ocean, and elsewhere. But these are all offerings that have you renting computers in the cloud at a premium. If you want to run on your own hardware, or even just have a clear migration path to do so in the future, you need to carefully consider how locked in you get to these commercial platforms. Preferably before the bills swallow your business whole!
 
-MRSK seeks to bring the advance in ergonomics pioneered by these commercial offerings to deploying web apps anywhere. Whether that's low-cost cloud options without the managed-service markup from the likes of Digital Ocean, Hetzner, OVH, etc, or it's your own colocated bare metal. To MRSK, it's all the same. Feed the config file a list of IP addresses with vanilla Ubuntu servers that have seen no prep beyond an added SSH key, and you'll be running in literally minutes.
+MRSK seeks to bring the advance in ergonomics pioneered by these commercial offerings to deploying web apps anywhere. Whether that's low-cost cloud options without the managed-service markup from the likes of Digital Ocean, Hetzner, OVH, etc., or it's your own colocated bare metal. To MRSK, it's all the same. Feed the config file a list of IP addresses with vanilla Ubuntu servers that have seen no prep beyond an added SSH key, and you'll be running in literally minutes.
 
 This approach gives you enormous portability. You can have your web app deployed on several clouds at ease like this. Or you can buy the baseline with your own hardware, then deploy to a cloud before a big seasonal spike to get more capacity. When you're not locked into a single provider from a tooling perspective, there are a lot of compelling options available.
 
@@ -123,6 +141,8 @@ This template can safely be checked into git. Then everyone deploying the app ca
 
 If you need separate env variables for different destinations, you can set them with `.env.destination.erb` for the template, which will generate `.env.staging` when run with `mrsk envify -d staging`.
 
+Note: If you utilize biometrics with 1Password you can remove the `session_token` related parts in the example and just call `op read op://Vault/Docker Hub/password -n`.
+
 #### Bitwarden as a secret store
 
 If you are using open source secret store like bitwarden, you can create `.env.erb` as a template which looks up the secrets.
@@ -206,13 +226,13 @@ ssh:
   user: app
 ```
 
-If you are using non-root user, you need to bootstrap your servers manually, before using them with MRSK. On Ubuntu, you'd do:
+If you are using non-root user (`app` as above example), you need to bootstrap your servers manually, before using them with MRSK. On Ubuntu, you'd do:
 
 ```bash
 sudo apt update
 sudo apt upgrade -y
 sudo apt install -y docker.io curl git
-sudo usermod -a -G docker ubuntu
+sudo usermod -a -G docker app
 ```
 
 ### Using a proxy SSH host
@@ -238,6 +258,15 @@ ssh:
   proxy_command: aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --region=us-east-1 ## ssh via aws ssm
 ```
 
+### Configuring the SSH log level
+
+```yaml
+ssh:
+  log_level: debug
+```
+
+Valid levels are `debug`, `info`, `warn`, `error` and `fatal` (default).
+
 ### Using env variables
 
 You can inject env variables into the app containers using `env`:
@@ -380,6 +409,16 @@ servers:
 
 That'll start the job containers with `docker run ... --cap-add --cpu-count 4 ...`.
 
+### Setting a minimum version
+
+You can set the minimum MRSK version with:
+
+```yaml
+minimum_version: 0.13.3
+```
+
+Note: versions <= 0.13.2 will ignore this setting.
+
 ### Configuring logging
 
 You can configure the logging driver and options passed to Docker using `logging`:
@@ -463,6 +502,37 @@ builder:
   context: ".."
 ```
 
+### Using multistage builder cache
+
+Docker multistage build cache can singlehandedly speed up your builds by a lot. Currently MRSK only supports using the GHA cache or the Registry cache:
+
+```yaml
+# Using GHA cache
+builder:
+  cache:
+    type: gha
+
+# Using Registry cache
+builder:
+  cache:
+    type: registry
+
+# Using Registry cache with different cache image
+builder:
+  cache:
+    type: registry
+    # default image name is <image>-build-cache
+    image: application-cache-image
+
+# Using Registry cache with additinonal cache-to options
+builder:
+  cache:
+    type: registry
+    options: mode=max,image-manifest=true,oci-mediatypes=true
+```
+
+For further insights into build cache optimization, check out documentation on Docker's official website: https://docs.docker.com/build/cache/.
+
 ### Using build secrets for new images
 
 Some images need a secret passed in during build time, like a GITHUB_TOKEN, to give access to private gem repositories. This can be done by having the secret in ENV, then referencing it in the builder configuration:
@@ -581,6 +651,16 @@ traefik:
     entrypoints.otherentrypoint.address: ':9000'
 ```
 
+### Rebooting Traefik
+
+If you make changes to Traefik args or labels, you'll need to reboot with:
+
+`mrsk traefik reboot`
+
+In production, reboot the Traefik containers one by one with a slower but safer approach, using a rolling reboot:
+
+`mrsk traefik reboot --rolling`
+
 ### Configuring build args for new images
 
 Build arguments that aren't secret can also be configured:
@@ -670,7 +750,7 @@ This assumes the Cron settings are stored in `config/crontab`.
 
 ### Healthcheck
 
-MRSK uses Docker healtchecks to check the health of your application during deployment. Traefik uses this same healthcheck status to determine when a container is ready to receive traffic.
+MRSK uses Docker healthchecks to check the health of your application during deployment. Traefik uses this same healthcheck status to determine when a container is ready to receive traffic.
 
 The healthcheck defaults to testing the HTTP response to the path `/up` on port 3000, up to 7 times. You can tailor this behaviour with the `healthcheck` setting:
 
@@ -825,7 +905,7 @@ If you wish to remove the entire application, including Traefik, containers, ima
 
 ## Locking
 
-Commands that are unsafe to run concurrently will take a deploy lock while they run. The lock is the `mrsk_lock` directory on the primary server.
+Commands that are unsafe to run concurrently will take a deploy lock while they run. The lock is the `mrsk_lock-<service>` directory on the primary server.
 
 You can check the lock status with:
 
@@ -840,7 +920,7 @@ Message: Automatic deploy lock
 You can also manually acquire and release the lock
 
 ```
-mrsk lock acquire -m "Doing maintanence"
+mrsk lock acquire -m "Doing maintenance"
 ```
 
 ```
@@ -881,12 +961,14 @@ firing a JSON webhook. These variables include:
 - `MRSK_RECORDED_AT` - UTC timestamp in ISO 8601 format, e.g. `2023-04-14T17:07:31Z`
 - `MRSK_PERFORMER` - the local user performing the command (from `whoami`)
 - `MRSK_SERVICE_VERSION` - an abbreviated service and version for use in messages, e.g. app@150b24f
-- `MRSK_VERSION` - an full version being deployed
+- `MRSK_VERSION` - the full version being deployed
+- `MRSK_HOSTS` - a comma-separated list of the hosts targeted by the command
+- `MRSK_COMMAND` - The command we are running
+- `MRSK_SUBCOMMAND` - optional: The subcommand we are running
 - `MRSK_DESTINATION` - optional: destination, e.g. "staging"
-- `MRSK_HOSTS` - a comma separated list of the hosts targeted by the command
 - `MRSK_ROLE` - optional: role targeted, e.g. "web"
 
-There are three hooks:
+There are four hooks:
 
 1. pre-connect
 Called before taking the deploy lock. For checks that need to run before connecting to remote hosts - e.g. DNS warming.
@@ -894,9 +976,11 @@ Called before taking the deploy lock. For checks that need to run before connect
 2. pre-build
 Used for pre-build checks - e.g. there are no uncommitted changes or that CI has passed.
 
-3. post-deploy - run after a deploy, redeploy or rollback
+3. pre-deploy
+For final checks before deploying, e.g. checking CI completed
 
-This hook is also passed a `MRSK_RUNTIME` env variable.
+3. post-deploy - run after a deploy, redeploy or rollback.
+This hook is also passed a `MRSK_RUNTIME` env variable set to the total seconds the deploy took.
 
 This could be used to broadcast a deployment message, or register the new version with an APM.
 
@@ -907,7 +991,7 @@ The command could look something like:
 curl -q -d content="[My App] ${MRSK_PERFORMER} Rolled back to version ${MRSK_VERSION}" https://3.basecamp.com/XXXXX/integrations/XXXXX/buckets/XXXXX/chats/XXXXX/lines
 ```
 
-That'll post a line like follows to a preconfigured chatbot in Basecamp:
+That'll post a line like the following to a preconfigured chatbot in Basecamp:
 
 ```
 [My App] [dhh] Rolled back to version d264c4e92470ad1bd18590f04466787262f605de

lib/mrsk/cli/accessory.rb

@@ -1,7 +1,7 @@
 class Mrsk::Cli::Accessory < Mrsk::Cli::Base
   desc "boot [NAME]", "Boot new accessory service on host (use NAME=all to boot all accessories)"
-  def boot(name)
+  def boot(name, login: true)
-    with_lock do
+    mutating do
       if name == "all"
         MRSK.accessory_names.each { |accessory_name| boot(accessory_name) }
       else
@@ -10,7 +10,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
           upload(name)
 
           on(accessory.hosts) do
-            execute *MRSK.registry.login
+            execute *MRSK.registry.login if login
             execute *MRSK.auditor.record("Booted #{name} accessory"), verbosity: :debug
             execute *accessory.run
           end
@@ -21,7 +21,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
 
   desc "upload [NAME]", "Upload accessory files to host", hide: true
   def upload(name)
-    with_lock do
+    mutating do
       with_accessory(name) do |accessory|
         on(accessory.hosts) do
           accessory.files.each do |(local, remote)|
@@ -38,7 +38,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
 
   desc "directories [NAME]", "Create accessory directories on host", hide: true
   def directories(name)
-    with_lock do
+    mutating do
       with_accessory(name) do |accessory|
         on(accessory.hosts) do
           accessory.directories.keys.each do |host_path|
@@ -51,18 +51,22 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
 
   desc "reboot [NAME]", "Reboot existing accessory on host (stop container, remove container, start new container)"
   def reboot(name)
-    with_lock do
+    mutating do
       with_accessory(name) do |accessory|
+        on(accessory.hosts) do
+          execute *MRSK.registry.login
+        end
+
         stop(name)
         remove_container(name)
-        boot(name)
+        boot(name, login: false)
       end
     end
   end
 
   desc "start [NAME]", "Start existing accessory container on host"
   def start(name)
-    with_lock do
+    mutating do
       with_accessory(name) do |accessory|
         on(accessory.hosts) do
           execute *MRSK.auditor.record("Started #{name} accessory"), verbosity: :debug
@@ -74,7 +78,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
 
   desc "stop [NAME]", "Stop existing accessory container on host"
   def stop(name)
-    with_lock do
+    mutating do
       with_accessory(name) do |accessory|
         on(accessory.hosts) do
           execute *MRSK.auditor.record("Stopped #{name} accessory"), verbosity: :debug
@@ -86,7 +90,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
 
   desc "restart [NAME]", "Restart existing accessory container on host"
   def restart(name)
-    with_lock do
+    mutating do
       with_accessory(name) do
         stop(name)
         start(name)
@@ -165,7 +169,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
   desc "remove [NAME]", "Remove accessory container, image and data directory from host (use NAME=all to remove all accessories)"
   option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
   def remove(name)
-    with_lock do
+    mutating do
       if name == "all"
         MRSK.accessory_names.each { |accessory_name| remove(accessory_name) }
       else
@@ -183,7 +187,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
 
   desc "remove_container [NAME]", "Remove accessory container from host", hide: true
   def remove_container(name)
-    with_lock do
+    mutating do
       with_accessory(name) do |accessory|
         on(accessory.hosts) do
           execute *MRSK.auditor.record("Remove #{name} accessory container"), verbosity: :debug
@@ -195,7 +199,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
 
   desc "remove_image [NAME]", "Remove accessory image from host", hide: true
   def remove_image(name)
-    with_lock do
+    mutating do
       with_accessory(name) do |accessory|
         on(accessory.hosts) do
           execute *MRSK.auditor.record("Removed #{name} accessory image"), verbosity: :debug
@@ -207,7 +211,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
 
   desc "remove_service_directory [NAME]", "Remove accessory directory used for uploaded files and data directories from host", hide: true
   def remove_service_directory(name)
-    with_lock do
+    mutating do
       with_accessory(name) do |accessory|
         on(accessory.hosts) do
           execute *accessory.remove_service_directory

lib/mrsk/cli/app.rb

@@ -1,7 +1,7 @@
 class Mrsk::Cli::App < Mrsk::Cli::Base
   desc "boot", "Boot app on servers (or reboot app if already running)"
   def boot
-    with_lock do
+    mutating do
       hold_lock_on_error do
         say "Get most recent version available as an image...", :magenta unless options[:version]
         using_version(version_or_latest) do |version|
@@ -29,7 +29,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
               execute *auditor.record("Booted app version #{version}"), verbosity: :debug
 
               old_version = capture_with_info(*app.current_running_version, raise_on_non_zero_exit: false).strip
-              execute *app.start_or_run
+              execute *app.start_or_run(hostname: "#{host}-#{SecureRandom.hex(6)}")
 
               Mrsk::Utils::HealthcheckPoller.wait_for_healthy(pause_after_ready: true) { capture_with_info(*app.status(version: version)) }
 
@@ -43,7 +43,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
 
   desc "start", "Start existing app container on servers"
   def start
-    with_lock do
+    mutating do
       on(MRSK.hosts) do |host|
         roles = MRSK.roles_on(host)
 
@@ -57,7 +57,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
 
   desc "stop", "Stop app container on servers"
   def stop
-    with_lock do
+    mutating do
       on(MRSK.hosts) do |host|
         roles = MRSK.roles_on(host)
 
@@ -135,7 +135,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
   desc "stale_containers", "Detect app stale containers"
   option :stop, aliases: "-s", type: :boolean, default: false, desc: "Stop the stale containers found"
   def stale_containers
-    with_lock do
+    mutating do
       stop = options[:stop]
 
       cli = self
@@ -202,7 +202,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
 
   desc "remove", "Remove app containers and images from servers"
   def remove
-    with_lock do
+    mutating do
       stop
       remove_containers
       remove_images
@@ -211,7 +211,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
 
   desc "remove_container [VERSION]", "Remove app container with given version from servers", hide: true
   def remove_container(version)
-    with_lock do
+    mutating do
       on(MRSK.hosts) do |host|
         roles = MRSK.roles_on(host)
 
@@ -225,7 +225,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
 
   desc "remove_containers", "Remove all app containers from servers", hide: true
   def remove_containers
-    with_lock do
+    mutating do
       on(MRSK.hosts) do |host|
         roles = MRSK.roles_on(host)
 
@@ -239,7 +239,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
 
   desc "remove_images", "Remove all app images from servers", hide: true
   def remove_images
-    with_lock do
+    mutating do
       on(MRSK.hosts) do
         execute *MRSK.auditor.record("Removed all app images"), verbosity: :debug
         execute *MRSK.app.remove_images

lib/mrsk/cli/base.rb

@@ -72,10 +72,11 @@ module Mrsk::Cli
         puts "  Finished all in #{sprintf("%.1f seconds", runtime)}"
       end
 
-      def with_lock
+      def mutating
-        if MRSK.holding_lock?
+        return yield if MRSK.holding_lock?
-          yield
+
-        else
+        MRSK.config.ensure_env_available
+
         run_hook "pre-connect"
 
         acquire_lock
@@ -94,7 +95,6 @@ module Mrsk::Cli
 
         release_lock
       end
-      end
 
       def acquire_lock
         raise_if_locked do
@@ -133,15 +133,39 @@ module Mrsk::Cli
         end
       end
 
-      def run_hook(hook, **details)
+      def run_hook(hook, **extra_details)
         if !options[:skip_hooks] && MRSK.hook.hook_exists?(hook)
+          details = { hosts: MRSK.hosts.join(","), command: command, subcommand: subcommand }
+
           say "Running the #{hook} hook...", :magenta
           run_locally do
-            MRSK.with_verbosity(:debug) { execute *MRSK.hook.run(hook, **details, hosts: MRSK.hosts.join(",")) }
+            MRSK.with_verbosity(:debug) { execute *MRSK.hook.run(hook, **details, **extra_details) }
           rescue SSHKit::Command::Failed
             raise HookError.new("Hook `#{hook}` failed")
           end
         end
       end
+
+      def command
+        @mrsk_command ||= begin
+          invocation_class, invocation_commands = *first_invocation
+          if invocation_class == Mrsk::Cli::Main
+            invocation_commands[0]
+          else
+            Mrsk::Cli::Main.subcommand_classes.find { |command, clazz| clazz == invocation_class }[0]
+          end
+        end
+      end
+
+      def subcommand
+        @mrsk_subcommand ||= begin
+          invocation_class, invocation_commands = *first_invocation
+          invocation_commands[0] if invocation_class != Mrsk::Cli::Main
+        end
+      end
+
+      def first_invocation
+        instance_variable_get("@_invocations").first
+      end
     end
 end

lib/mrsk/cli/build.rb

@@ -3,7 +3,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
 
   desc "deliver", "Build app and push app image to registry then pull image on servers"
   def deliver
-    with_lock do
+    mutating do
       push
       pull
     end
@@ -11,7 +11,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
 
   desc "push", "Build and push app image to registry"
   def push
-    with_lock do
+    mutating do
       cli = self
 
       verify_local_dependencies
@@ -37,7 +37,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
 
   desc "pull", "Pull app image from registry onto servers"
   def pull
-    with_lock do
+    mutating do
       on(MRSK.hosts) do
         execute *MRSK.auditor.record("Pulled image with version #{MRSK.config.version}"), verbosity: :debug
         execute *MRSK.builder.clean, raise_on_non_zero_exit: false
@@ -48,7 +48,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
 
   desc "create", "Create a build setup"
   def create
-    with_lock do
+    mutating do
       run_locally do
         begin
           debug "Using builder: #{MRSK.builder.name}"
@@ -67,7 +67,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
 
   desc "remove", "Remove build setup"
   def remove
-    with_lock do
+    mutating do
       run_locally do
         debug "Using builder: #{MRSK.builder.name}"
         execute *MRSK.builder.remove

lib/mrsk/cli/lock.rb

@@ -7,7 +7,7 @@ class Mrsk::Cli::Lock < Mrsk::Cli::Base
   end
 
   desc "acquire", "Acquire the deploy lock"
-  option :message, aliases: "-m", type: :string, desc: "A lock mesasge", required: true
+  option :message, aliases: "-m", type: :string, desc: "A lock message", required: true
   def acquire
     message = options[:message]
     raise_if_locked do

lib/mrsk/cli/main.rb

@@ -2,7 +2,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
   desc "setup", "Setup all accessories and deploy app to servers"
   def setup
     print_runtime do
-      with_lock do
+      mutating do
         invoke "mrsk:cli:server:bootstrap"
         invoke "mrsk:cli:accessory:boot", [ "all" ]
         deploy
@@ -14,7 +14,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
   option :skip_push, aliases: "-P", type: :boolean, default: false, desc: "Skip image build and push"
   def deploy
     runtime = print_runtime do
-      with_lock do
+      mutating do
         invoke_options = deploy_options
 
         say "Log into image registry...", :magenta
@@ -28,6 +28,8 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
           invoke "mrsk:cli:build:deliver", [], invoke_options
         end
 
+        run_hook "pre-deploy"
+
         say "Ensure Traefik is running...", :magenta
         invoke "mrsk:cli:traefik:boot", [], invoke_options
 
@@ -51,7 +53,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
   option :skip_push, aliases: "-P", type: :boolean, default: false, desc: "Skip image build and push"
   def redeploy
     runtime = print_runtime do
-      with_lock do
+      mutating do
         invoke_options = deploy_options
 
         if options[:skip_push]
@@ -62,6 +64,8 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
           invoke "mrsk:cli:build:deliver", [], invoke_options
         end
 
+        run_hook "pre-deploy"
+
         say "Ensure app can pass healthcheck...", :magenta
         invoke "mrsk:cli:healthcheck:perform", [], invoke_options
 
@@ -79,13 +83,15 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
   def rollback(version)
     rolled_back = false
     runtime = print_runtime do
-      with_lock do
+      mutating do
         invoke_options = deploy_options
 
         MRSK.config.version = version
         old_version = nil
 
         if container_available?(version)
+          run_hook "pre-deploy"
+
           invoke "mrsk:cli:app:boot", [], invoke_options.merge(version: version)
           rolled_back = true
         else
@@ -174,7 +180,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
   desc "remove", "Remove Traefik, app, accessories, and registry session from servers"
   option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
   def remove
-    with_lock do
+    mutating do
       if options[:confirmed] || ask("This will remove all containers and images. Are you sure?", limited_to: %w( y N ), default: "N") == "y"
         invoke "mrsk:cli:traefik:remove", [], options.without(:confirmed)
         invoke "mrsk:cli:app:remove", [], options.without(:confirmed)

lib/mrsk/cli/prune.rb

@@ -1,7 +1,7 @@
 class Mrsk::Cli::Prune < Mrsk::Cli::Base
   desc "all", "Prune unused images and stopped containers"
   def all
-    with_lock do
+    mutating do
       containers
       images
     end
@@ -9,17 +9,18 @@ class Mrsk::Cli::Prune < Mrsk::Cli::Base
 
   desc "images", "Prune dangling images"
   def images
-    with_lock do
+    mutating do
       on(MRSK.hosts) do
         execute *MRSK.auditor.record("Pruned images"), verbosity: :debug
-        execute *MRSK.prune.images
+        execute *MRSK.prune.dangling_images
+        execute *MRSK.prune.tagged_images
       end
     end
   end
 
   desc "containers", "Prune all stopped containers, except the last 5"
   def containers
-    with_lock do
+    mutating do
       on(MRSK.hosts) do
         execute *MRSK.auditor.record("Pruned containers"), verbosity: :debug
         execute *MRSK.prune.containers

lib/mrsk/cli/templates/sample_hooks/pre-deploy.sample

@@ -0,0 +1,109 @@
+#!/usr/bin/env ruby
+
+# A sample pre-deploy hook
+#
+# Checks the Github status of the build, waiting for a pending build to complete for up to 720 seconds.
+#
+# Fails unless the combined status is "success"
+#
+# These environment variables are available:
+# MRSK_RECORDED_AT
+# MRSK_PERFORMER
+# MRSK_VERSION
+# MRSK_HOSTS
+# MRSK_COMMAND
+# MRSK_SUBCOMMAND
+# MRSK_ROLE (if set)
+# MRSK_DESTINATION (if set)
+
+# Only check the build status for production deployments
+if ENV["MRSK_COMMAND"] == "rollback" || ENV["MRSK_DESTINATION"] != "production"
+  exit 0
+end
+
+require "bundler/inline"
+
+# true = install gems so this is fast on repeat invocations
+gemfile(true, quiet: true) do
+  source "https://rubygems.org"
+
+  gem "octokit"
+  gem "faraday-retry"
+end
+
+MAX_ATTEMPTS = 72
+ATTEMPTS_GAP = 10
+
+def exit_with_error(message)
+  $stderr.puts message
+  exit 1
+end
+
+class GithubStatusChecks
+  attr_reader :remote_url, :git_sha, :github_client, :combined_status
+
+  def initialize
+    @remote_url = `git config --get remote.origin.url`.strip.delete_prefix("https://github.com/")
+    @git_sha = `git rev-parse HEAD`.strip
+    @github_client = Octokit::Client.new(access_token: ENV["GITHUB_TOKEN"])
+    refresh!
+  end
+
+  def refresh!
+    @combined_status = github_client.combined_status(remote_url, git_sha)
+  end
+
+  def state
+    combined_status[:state]
+  end
+
+  def first_status_url
+    first_status = combined_status[:statuses].find { |status| status[:state] == state }
+    first_status && first_status[:target_url]
+  end
+
+  def complete_count
+    combined_status[:statuses].count { |status| status[:state] != "pending"}
+  end
+
+  def total_count
+    combined_status[:statuses].count
+  end
+
+  def current_status
+    if total_count > 0
+      "Completed #{complete_count}/#{total_count} checks, see #{first_status_url} ..."
+    else
+      "Build not started..."
+    end
+  end
+end
+
+
+$stdout.sync = true
+
+puts "Checking build status..."
+attempts = 0
+checks = GithubStatusChecks.new
+
+begin
+  loop do
+    case checks.state
+    when "success"
+      puts "Checks passed, see #{checks.first_status_url}"
+      exit 0
+    when "failure"
+      exit_with_error "Checks failed, see #{checks.first_status_url}"
+    when "pending"
+      attempts += 1
+    end
+
+    exit_with_error "Checks are still pending, gave up after #{MAX_ATTEMPTS * ATTEMPTS_GAP} seconds" if attempts == MAX_ATTEMPTS
+
+    puts checks.current_status
+    sleep(ATTEMPTS_GAP)
+    checks.refresh!
+  end
+rescue Octokit::NotFound
+  exit_with_error "Build status could not be found"
+end

lib/mrsk/cli/traefik.rb

@@ -1,7 +1,7 @@
 class Mrsk::Cli::Traefik < Mrsk::Cli::Base
   desc "boot", "Boot Traefik on servers"
   def boot
-    with_lock do
+    mutating do
       on(MRSK.traefik_hosts) do
         execute *MRSK.registry.login
         execute *MRSK.traefik.run, raise_on_non_zero_exit: false
@@ -10,17 +10,22 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
   end
 
   desc "reboot", "Reboot Traefik on servers (stop container, remove container, start new container)"
+  option :rolling, type: :boolean, default: false, desc: "Reboot traefik on hosts in sequence, rather than in parallel"
   def reboot
-    with_lock do
+    mutating do
-      stop
+      on(MRSK.traefik_hosts, in: options[:rolling] ? :sequence : :parallel) do
-      remove_container
+        execute *MRSK.auditor.record("Rebooted traefik"), verbosity: :debug
-      boot
+        execute *MRSK.registry.login
+        execute *MRSK.traefik.stop, raise_on_non_zero_exit: false
+        execute *MRSK.traefik.remove_container
+        execute *MRSK.traefik.run, raise_on_non_zero_exit: false
+      end
     end
   end
 
   desc "start", "Start existing Traefik container on servers"
   def start
-    with_lock do
+    mutating do
       on(MRSK.traefik_hosts) do
         execute *MRSK.auditor.record("Started traefik"), verbosity: :debug
         execute *MRSK.traefik.start, raise_on_non_zero_exit: false
@@ -30,7 +35,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
 
   desc "stop", "Stop existing Traefik container on servers"
   def stop
-    with_lock do
+    mutating do
       on(MRSK.traefik_hosts) do
         execute *MRSK.auditor.record("Stopped traefik"), verbosity: :debug
         execute *MRSK.traefik.stop, raise_on_non_zero_exit: false
@@ -40,7 +45,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
 
   desc "restart", "Restart existing Traefik container on servers"
   def restart
-    with_lock do
+    mutating do
       stop
       start
     end
@@ -77,7 +82,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
 
   desc "remove", "Remove Traefik container and image from servers"
   def remove
-    with_lock do
+    mutating do
       stop
       remove_container
       remove_image
@@ -86,7 +91,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
 
   desc "remove_container", "Remove Traefik container from servers", hide: true
   def remove_container
-    with_lock do
+    mutating do
       on(MRSK.traefik_hosts) do
         execute *MRSK.auditor.record("Removed traefik container"), verbosity: :debug
         execute *MRSK.traefik.remove_container
@@ -96,7 +101,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
 
   desc "remove_image", "Remove Traefik image from servers", hide: true
   def remove_image
-    with_lock do
+    mutating do
       on(MRSK.traefik_hosts) do
         execute *MRSK.auditor.record("Removed traefik image"), verbosity: :debug
         execute *MRSK.traefik.remove_image

lib/mrsk/commands/app.rb

@@ -1,4 +1,6 @@
 class Mrsk::Commands::App < Mrsk::Commands::Base
+  ACTIVE_DOCKER_STATUSES = [ :running, :restarting ]
+
   attr_reader :role
 
   def initialize(config, role: nil)
@@ -6,17 +8,18 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
     @role = role
   end
 
-  def start_or_run
+  def start_or_run(hostname: nil)
-    combine start, run, by: "||"
+    combine start, run(hostname: hostname), by: "||"
   end
 
-  def run
+  def run(hostname: nil)
     role = config.role(self.role)
 
     docker :run,
       "--detach",
       "--restart unless-stopped",
       "--name", container_name,
+      *(["--hostname", hostname] if hostname),
       "-e", "MRSK_CONTAINER_NAME=\"#{container_name}\"",
       *role.env_args,
       *role.health_check_args,
@@ -92,7 +95,7 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
 
 
   def current_running_container_id
-    docker :ps, "--quiet", *filter_args(status: :running), "--latest"
+    docker :ps, "--quiet", *filter_args(statuses: ACTIVE_DOCKER_STATUSES), "--latest"
   end
 
   def container_id_for_version(version, only_running: false)
@@ -100,12 +103,12 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
   end
 
   def current_running_version
-    list_versions("--latest", status: :running)
+    list_versions("--latest", statuses: ACTIVE_DOCKER_STATUSES)
   end
 
-  def list_versions(*docker_args, status: nil)
+  def list_versions(*docker_args, statuses: nil)
     pipe \
-      docker(:ps, *filter_args(status: status), *docker_args, "--format", '"{{.Names}}"'),
+      docker(:ps, *filter_args(statuses: statuses), *docker_args, "--format", '"{{.Names}}"'),
       %(grep -oE "\\-[^-]+$"), # Extract SHA from "service-role-dest-SHA"
       %(cut -c 2-)
   end
@@ -150,15 +153,17 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
       [ config.service, role, config.destination, version || config.version ].compact.join("-")
     end
 
-    def filter_args(status: nil)
+    def filter_args(statuses: nil)
-      argumentize "--filter", filters(status: status)
+      argumentize "--filter", filters(statuses: statuses)
     end
 
-    def filters(status: nil)
+    def filters(statuses: nil)
       [ "label=service=#{config.service}" ].tap do |filters|
         filters << "label=destination=#{config.destination}" if config.destination
         filters << "label=role=#{role}" if role
-        filters << "status=#{status}" if status
+        statuses&.each do |status|
+          filters << "status=#{status}"
+        end
       end
     end
 end

lib/mrsk/commands/base.rb

@@ -13,7 +13,11 @@ module Mrsk::Commands
 
     def run_over_ssh(*command, host:)
       "ssh".tap do |cmd|
-        cmd << " -J #{config.ssh_proxy.jump_proxies}" if config.ssh_proxy
+        if config.ssh_proxy && config.ssh_proxy.is_a?(Net::SSH::Proxy::Jump)
+          cmd << " -J #{config.ssh_proxy.jump_proxies}"
+        elsif config.ssh_proxy && config.ssh_proxy.is_a?(Net::SSH::Proxy::Command)
+          cmd << " -o ProxyCommand='#{config.ssh_proxy.command_line_template}'"
+        end
         cmd << " -t #{config.ssh_user}@#{host} '#{command.join(" ")}'"
       end
     end

lib/mrsk/commands/builder.rb

@@ -7,11 +7,13 @@ class Mrsk::Commands::Builder < Mrsk::Commands::Base
 
   def target
     case
-    when config.builder && config.builder["multiarch"] == false
+    when !config.builder.multiarch? && !config.builder.cached?
       native
-    when config.builder && config.builder["local"] && config.builder["remote"]
+    when !config.builder.multiarch? && config.builder.cached?
+      native_cached
+    when config.builder.local? && config.builder.remote?
       multiarch_remote
-    when config.builder && config.builder["remote"]
+    when config.builder.remote?
       native_remote
     else
       multiarch
@@ -22,6 +24,10 @@ class Mrsk::Commands::Builder < Mrsk::Commands::Base
     @native ||= Mrsk::Commands::Builder::Native.new(config)
   end
 
+  def native_cached
+    @native ||= Mrsk::Commands::Builder::Native::Cached.new(config)
+  end
+
   def native_remote
     @native ||= Mrsk::Commands::Builder::Native::Remote.new(config)
   end

lib/mrsk/commands/builder/base.rb

@@ -3,6 +3,7 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
   class BuilderError < StandardError; end
 
   delegate :argumentize, to: Mrsk::Utils
+  delegate :args, :secrets, :dockerfile, :local_arch, :local_host, :remote_arch, :remote_host, :cache_from, :cache_to, to: :builder_config
 
   def clean
     docker :image, :rm, "--force", config.absolute_image
@@ -13,11 +14,11 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
   end
 
   def build_options
-    [ *build_tags, *build_labels, *build_args, *build_secrets, *build_dockerfile ]
+    [ *build_tags, *build_cache, *build_labels, *build_args, *build_secrets, *build_dockerfile ]
   end
 
   def build_context
-    context
+    config.builder.context
   end
 
 
@@ -26,6 +27,13 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
       [ "-t", config.absolute_image, "-t", config.latest_image ]
     end
 
+    def build_cache
+      if cache_to && cache_from
+        ["--cache-to", cache_to,
+          "--cache-from", cache_from]
+      end
+    end
+
     def build_labels
       argumentize "--label", { service: config.service }
     end
@@ -46,19 +54,7 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
       end
     end
 
-    def args
+    def builder_config
-      (config.builder && config.builder["args"]) || {}
+      config.builder
-    end
-
-    def secrets
-      (config.builder && config.builder["secrets"]) || []
-    end
-
-    def dockerfile
-      (config.builder && config.builder["dockerfile"]) || "Dockerfile"
-    end
-
-    def context
-      (config.builder && config.builder["context"]) || "."
     end
 end

lib/mrsk/commands/builder/multiarch/remote.rb

@@ -22,17 +22,17 @@ class Mrsk::Commands::Builder::Multiarch::Remote < Mrsk::Commands::Builder::Mult
     end
 
     def create_local_buildx
-      docker :buildx, :create, "--name", builder_name, builder_name_with_arch(local["arch"]), "--platform", "linux/#{local["arch"]}"
+      docker :buildx, :create, "--name", builder_name, builder_name_with_arch(local_arch), "--platform", "linux/#{local_arch}"
     end
 
     def append_remote_buildx
-      docker :buildx, :create, "--append", "--name", builder_name, builder_name_with_arch(remote["arch"]), "--platform", "linux/#{remote["arch"]}"
+      docker :buildx, :create, "--append", "--name", builder_name, builder_name_with_arch(remote_arch), "--platform", "linux/#{remote_arch}"
     end
 
     def create_contexts
       combine \
-        create_context(local["arch"], local["host"]),
+        create_context(local_arch, local_host),
-        create_context(remote["arch"], remote["host"])
+        create_context(remote_arch, remote_host)
     end
 
     def create_context(arch, host)
@@ -41,19 +41,11 @@ class Mrsk::Commands::Builder::Multiarch::Remote < Mrsk::Commands::Builder::Mult
 
     def remove_contexts
       combine \
-        remove_context(local["arch"]),
+        remove_context(local_arch),
-        remove_context(remote["arch"])
+        remove_context(remote_arch)
     end
 
     def remove_context(arch)
       docker :context, :rm, builder_name_with_arch(arch)
     end
-
-    def local
-      config.builder["local"]
-    end
-
-    def remote
-      config.builder["remote"]
-    end
 end

lib/mrsk/commands/builder/native.rb

@@ -1,10 +1,10 @@
 class Mrsk::Commands::Builder::Native < Mrsk::Commands::Builder::Base
   def create
-    # No-op on native
+    # No-op on native without cache
   end
 
   def remove
-    # No-op on native
+    # No-op on native without cache
   end
 
   def push

lib/mrsk/commands/builder/native/cached.rb

@@ -0,0 +1,16 @@
+class Mrsk::Commands::Builder::Native::Cached < Mrsk::Commands::Builder::Native
+  def create
+    docker :buildx, :create, "--use", "--driver=docker-container"
+  end
+
+  def remove
+    docker :buildx, :rm, builder_name
+  end
+
+  def push
+    docker :buildx, :build,
+      "--push",
+      *build_options,
+      build_context
+  end
+end

lib/mrsk/commands/builder/native/remote.rb

@@ -28,29 +28,21 @@ class Mrsk::Commands::Builder::Native::Remote < Mrsk::Commands::Builder::Native
 
 
   private
-    def arch
-      config.builder["remote"]["arch"]
-    end
-
-    def host
-      config.builder["remote"]["host"]
-    end
-
     def builder_name
       "mrsk-#{config.service}-native-remote"
     end
 
     def builder_name_with_arch
-      "#{builder_name}-#{arch}"
+      "#{builder_name}-#{remote_arch}"
     end
 
     def platform
-      "linux/#{arch}"
+      "linux/#{remote_arch}"
     end
 
     def create_context
       docker :context, :create,
-        builder_name_with_arch, "--description", "'#{builder_name} #{arch} native host'", "--docker", "'host=#{host}'"
+        builder_name_with_arch, "--description", "'#{builder_name} #{remote_arch} native host'", "--docker", "'host=#{remote_host}'"
     end
 
     def remove_context

lib/mrsk/commands/lock.rb

@@ -40,7 +40,7 @@ class Mrsk::Commands::Lock < Mrsk::Commands::Base
     end
 
     def lock_dir
-      :mrsk_lock
+      "mrsk_lock-#{config.service}"
     end
 
     def lock_details_file

lib/mrsk/commands/prune.rb

@@ -2,13 +2,20 @@ require "active_support/duration"
 require "active_support/core_ext/numeric/time"
 
 class Mrsk::Commands::Prune < Mrsk::Commands::Base
-  def images
+  def dangling_images
     docker :image, :prune, "--force", "--filter", "label=service=#{config.service}", "--filter", "dangling=true"
   end
 
+  def tagged_images
+    pipe \
+      docker(:image, :ls, *service_filter, "--format", "'{{.ID}} {{.Repository}}:{{.Tag}}'"),
+      "grep -v -w \"#{active_image_list}\"",
+      "while read image tag; do docker rmi $tag; done"
+  end
+
   def containers(keep_last: 5)
     pipe \
-      docker(:ps, "-q", "-a", "--filter", "label=service=#{config.service}", *stopped_containers_filters),
+      docker(:ps, "-q", "-a", *service_filter, *stopped_containers_filters),
       "tail -n +#{keep_last + 1}",
       "while read container_id; do docker rm $container_id; done"
   end
@@ -17,4 +24,15 @@ class Mrsk::Commands::Prune < Mrsk::Commands::Base
     def stopped_containers_filters
       [ "created", "exited", "dead" ].flat_map { |status| ["--filter", "status=#{status}"] }
     end
+
+    def active_image_list
+      # Pull the images that are used by any containers
+      # Append repo:latest - to avoid deleting the latest tag
+      # Append repo:<none> - to avoid deleting dangling images that are in use. Unused dangling images are deleted separately
+      "$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=#{config.service} | tr -d '\\n')#{config.latest_image}\\|#{config.repository}:<none>"
+    end
+
+    def service_filter
+      [ "--filter", "label=service=#{config.service}" ]
+    end
 end

lib/mrsk/commands/traefik.rb

@@ -3,6 +3,9 @@ class Mrsk::Commands::Traefik < Mrsk::Commands::Base
 
   DEFAULT_IMAGE = "traefik:v2.9"
   CONTAINER_PORT = 80
+  DEFAULT_ARGS = {
+    'log.level' => 'DEBUG'
+  }
 
   def run
     docker :run, "--name traefik",
@@ -16,7 +19,6 @@ class Mrsk::Commands::Traefik < Mrsk::Commands::Base
       *docker_options_args,
       image,
       "--providers.docker",
-      "--log.level=DEBUG",
       *cmd_option_args
   end
 
@@ -86,9 +88,9 @@ class Mrsk::Commands::Traefik < Mrsk::Commands::Base
 
     def cmd_option_args
       if args = config.traefik["args"]
-        optionize args, with: "="
+        optionize DEFAULT_ARGS.merge(args), with: "="
       else
-        []
+        optionize DEFAULT_ARGS, with: "="
       end
     end
 

lib/mrsk/configuration.rb

@@ -6,7 +6,7 @@ require "erb"
 require "net/ssh/proxy/jump"
 
 class Mrsk::Configuration
-  delegate :service, :image, :servers, :env, :labels, :registry, :builder, :stop_wait_time, :hooks_path, to: :raw_config, allow_nil: true
+  delegate :service, :image, :servers, :env, :labels, :registry, :stop_wait_time, :hooks_path, to: :raw_config, allow_nil: true
   delegate :argumentize, :argumentize_env_with_secrets, :optionize, to: Mrsk::Utils
 
   attr_accessor :destination
@@ -153,7 +153,15 @@ class Mrsk::Configuration
   end
 
   def ssh_options
-    { user: ssh_user, proxy: ssh_proxy, auth_methods: [ "publickey" ] }.compact
+    { user: ssh_user, proxy: ssh_proxy, auth_methods: [ "publickey" ], logger: ssh_logger }.compact
+  end
+
+  def ssh_logger
+    @ssh_logger ||= ::Logger.new(STDERR).tap { |logger| logger.level = ssh_log_level }
+  end
+
+  def ssh_log_level
+    (raw_config.ssh && raw_config.ssh["log_level"]) || ::Logger::FATAL
   end
 
 
@@ -165,8 +173,12 @@ class Mrsk::Configuration
     raw_config.readiness_delay || 7
   end
 
+  def minimum_version
+    raw_config.minimum_version
+  end
+
   def valid?
-    ensure_required_keys_present && ensure_env_available
+    ensure_required_keys_present && ensure_valid_mrsk_version
   end
 
 
@@ -181,8 +193,9 @@ class Mrsk::Configuration
       service_with_version: service_with_version,
       env_args: env_args,
       volume_args: volume_args,
-      ssh_options: ssh_options,
+      ssh_options: ssh_options.except(:logger),
-      builder: raw_config.builder,
+      ssh_log_level: ssh_log_level,
+      builder: builder.to_h,
       accessories: raw_config.accessories,
       logging: logging_args,
       healthcheck: healthcheck
@@ -197,6 +210,18 @@ class Mrsk::Configuration
     raw_config.hooks_path || ".mrsk/hooks"
   end
 
+  def builder
+    Mrsk::Configuration::Builder.new(config: self)
+  end
+
+  # Will raise KeyError if any secret ENVs are missing
+  def ensure_env_available
+    env_args
+    roles.each(&:env_args)
+
+    true
+  end
+
   private
     # Will raise ArgumentError if any required config keys are missing
     def ensure_required_keys_present
@@ -221,14 +246,15 @@ class Mrsk::Configuration
       true
     end
 
-    # Will raise KeyError if any secret ENVs are missing
+    def ensure_valid_mrsk_version
-    def ensure_env_available
+      if minimum_version && Gem::Version.new(minimum_version) > Gem::Version.new(Mrsk::VERSION)
-      env_args
+        raise ArgumentError, "Current version is #{Mrsk::VERSION}, minimum required is #{minimum_version}"
-      roles.each(&:env_args)
+      end
 
       true
     end
 
+
     def role_names
       raw_config.servers.is_a?(Array) ? [ "web" ] : raw_config.servers.keys.sort
     end

lib/mrsk/configuration/builder.rb

@@ -0,0 +1,114 @@
+class Mrsk::Configuration::Builder
+  def initialize(config:)
+    @options = config.raw_config.builder || {}
+    @image = config.image
+    @server = config.registry["server"]
+
+    valid?
+  end
+
+  def to_h
+    @options
+  end
+
+  def multiarch?
+    @options["multiarch"] != false
+  end
+
+  def local?
+    !!@options["local"]
+  end
+
+  def remote?
+    !!@options["remote"]
+  end
+
+  def cached?
+    !!@options["cache"]
+  end
+
+  def args
+    @options["args"] || {}
+  end
+
+  def secrets
+    @options["secrets"] || []
+  end
+
+  def dockerfile
+    @options["dockerfile"] || "Dockerfile"
+  end
+
+  def context
+    @options["context"] || "."
+  end
+
+  def local_arch
+    @options["local"]["arch"] if local?
+  end
+
+  def local_host
+    @options["local"]["host"] if local?
+  end
+
+  def remote_arch
+    @options["remote"]["arch"] if remote?
+  end
+
+  def remote_host
+    @options["remote"]["host"] if remote?
+  end
+
+  def cache_from
+    if cached?
+      case @options["cache"]["type"]
+      when "gha"
+        cache_from_config_for_gha
+      when "registry"
+        cache_from_config_for_registry
+      end
+    end
+  end
+
+  def cache_to
+    if cached?
+      case @options["cache"]["type"]
+      when "gha"
+        cache_to_config_for_gha
+      when "registry"
+        cache_to_config_for_registry
+      end
+    end
+  end
+
+  private
+    def valid?
+      if @options["local"] && !@options["remote"]
+        raise ArgumentError, "You must specify both local and remote builder config for remote multiarch builds"
+      end
+
+      if @options["cache"] && @options["cache"]["type"]
+        raise ArgumentError, "Invalid cache type: #{@options["cache"]["type"]}" unless ["gha", "registry"].include?(@options["cache"]["type"])
+      end
+    end
+
+    def cache_image
+      @options["cache"]&.fetch("image", nil) || "#{@image}-build-cache"
+    end
+
+    def cache_from_config_for_gha
+      "type=gha"
+    end
+
+    def cache_from_config_for_registry
+      [ "type=registry", "ref=#{@server}/#{cache_image}" ].compact.join(",")
+    end
+
+    def cache_to_config_for_gha
+      [ "type=gha", @options["cache"]&.fetch("options", nil)].compact.join(",")
+    end
+
+    def cache_to_config_for_registry
+      [ "type=registry", @options["cache"]&.fetch("options", nil), "ref=#{@server}/#{cache_image}" ].compact.join(",")
+    end
+end

lib/mrsk/version.rb

@@ -1,3 +1,3 @@
 module Mrsk
-  VERSION = "0.13.0"
+  VERSION = "0.15.0"
 end

test/cli/accessory_test.rb

@@ -40,9 +40,10 @@ class CliAccessoryTest < CliTestCase
   end
 
   test "reboot" do
+    Mrsk::Commands::Registry.any_instance.expects(:login)
     Mrsk::Cli::Accessory.any_instance.expects(:stop).with("mysql")
     Mrsk::Cli::Accessory.any_instance.expects(:remove_container).with("mysql")
-    Mrsk::Cli::Accessory.any_instance.expects(:boot).with("mysql")
+    Mrsk::Cli::Accessory.any_instance.expects(:boot).with("mysql", login: false)
 
     run_command("reboot", "mysql")
   end

test/cli/app_test.rb

@@ -5,7 +5,7 @@ class CliAppTest < CliTestCase
     stub_running
     run_command("boot").tap do |output|
       assert_match "docker tag dhh/app:latest dhh/app:latest", output
-      assert_match "docker run --detach --restart unless-stopped", output
+      assert_match /docker run --detach --restart unless-stopped --name app-web-latest --hostname 1.1.1.1-[0-9a-f]{12} /, output
       assert_match "docker container ls --all --filter name=^app-web-123$ --quiet | xargs docker stop", output
     end
   end
@@ -22,13 +22,13 @@ class CliAppTest < CliTestCase
       .returns("running") # health check
 
     SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
-      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
+      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--filter", "status=restarting", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
       .returns("123") # old version
 
     run_command("boot").tap do |output|
       assert_match /Renaming container .* to .* as already deployed on 1.1.1.1/, output # Rename
       assert_match /docker rename app-web-latest app-web-latest_replaced_[0-9a-f]{16}/, output
-      assert_match "docker run --detach --restart unless-stopped", output
+      assert_match /docker run --detach --restart unless-stopped --name app-web-latest --hostname 1.1.1.1-[0-9a-f]{12} /, output
       assert_match "docker container ls --all --filter name=^app-web-123$ --quiet | xargs docker stop", output
     end
   ensure
@@ -65,7 +65,7 @@ class CliAppTest < CliTestCase
 
   test "stop" do
     run_command("stop").tap do |output|
-      assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop", output
+      assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop", output
     end
   end
 
@@ -98,7 +98,7 @@ class CliAppTest < CliTestCase
 
   test "remove" do
     run_command("remove").tap do |output|
-      assert_match /#{Regexp.escape("docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop")}/, output
+      assert_match /#{Regexp.escape("docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop")}/, output
       assert_match /#{Regexp.escape("docker container prune --force --filter label=service=app")}/, output
       assert_match /#{Regexp.escape("docker image prune --all --force --filter label=service=app")}/, output
     end
@@ -130,7 +130,7 @@ class CliAppTest < CliTestCase
 
   test "exec with reuse" do
     run_command("exec", "--reuse", "ruby -v").tap do |output|
-      assert_match "docker ps --filter label=service=app --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output # Get current version
+      assert_match "docker ps --filter label=service=app --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output # Get current version
       assert_match "docker exec app-web-999 ruby -v", output
     end
   end
@@ -149,28 +149,28 @@ class CliAppTest < CliTestCase
 
   test "logs" do
     SSHKit::Backend::Abstract.any_instance.stubs(:exec)
-      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest| xargs docker logs --timestamps --tail 10 2>&1'")
+      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest| xargs docker logs --timestamps --tail 10 2>&1'")
 
-    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --tail 100 2>&1", run_command("logs")
+    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --tail 100 2>&1", run_command("logs")
   end
 
   test "logs with follow" do
     SSHKit::Backend::Abstract.any_instance.stubs(:exec)
-      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1'")
+      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1'")
 
-    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1", run_command("logs", "--follow")
+    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1", run_command("logs", "--follow")
   end
 
   test "version" do
     run_command("version").tap do |output|
-      assert_match "docker ps --filter label=service=app --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
+      assert_match "docker ps --filter label=service=app --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
     end
   end
 
 
   test "version through main" do
     stdouted { Mrsk::Cli::Main.start(["app", "version", "-c", "test/fixtures/deploy_with_accessories.yml", "--hosts", "1.1.1.1"]) }.tap do |output|
-      assert_match "docker ps --filter label=service=app --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
+      assert_match "docker ps --filter label=service=app --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
     end
   end
 

test/cli/build_test.rb

@@ -9,7 +9,11 @@ class CliBuildTest < CliTestCase
   end
 
   test "push" do
+    Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
+    hook_variables = { version: 999, service_version: "app@999", hosts: "1.1.1.1,1.1.1.2,1.1.1.3,1.1.1.4", command: "build", subcommand: "push" }
+
     run_command("push").tap do |output|
+      assert_hook_ran "pre-build", output, **hook_variables
       assert_match /docker --version && docker buildx version/, output
       assert_match /docker buildx build --push --platform linux\/amd64,linux\/arm64 --builder mrsk-app-multiarch -t dhh\/app:999 -t dhh\/app:latest --label service="app" --file Dockerfile \. as .*@localhost/, output
     end

test/cli/cli_test_case.rb

@@ -27,19 +27,30 @@ class CliTestCase < ActiveSupport::TestCase
         .raises(SSHKit::Command::Failed.new("failed"))
     end
 
-    def ensure_hook_runs(hook)
-      Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
-      SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-        .with { |*args| args != [".mrsk/hooks/#{hook}"] }
-      SSHKit::Backend::Abstract.any_instance.expects(:execute)
-        .with { |*args| args.first == ".mrsk/hooks/#{hook}" }
-        .once
-    end
-
     def stub_locking
       SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-        .with { |arg1, arg2| arg1 == :mkdir && arg2 == :mrsk_lock }
+        .with { |arg1, arg2| arg1 == :mkdir && arg2 == "mrsk_lock-app" }
       SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-        .with { |arg1, arg2| arg1 == :rm && arg2 == "mrsk_lock/details" }
+        .with { |arg1, arg2| arg1 == :rm && arg2 == "mrsk_lock-app/details" }
+    end
+
+    def assert_hook_ran(hook, output, version:, service_version:, hosts:, command:, subcommand: nil, runtime: nil)
+      performer = `whoami`.strip
+
+      assert_match "Running the #{hook} hook...\n", output
+
+      expected = %r{Running\s/usr/bin/env\s\.mrsk/hooks/#{hook}\sas\s#{performer}@localhost\n\s
+        DEBUG\s\[[0-9a-f]*\]\sCommand:\s\(\sexport\s
+        MRSK_RECORDED_AT=\"\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\dZ\"\s
+        MRSK_PERFORMER=\"#{performer}\"\s
+        MRSK_VERSION=\"#{version}\"\s
+        MRSK_SERVICE_VERSION=\"#{service_version}\"\s
+        MRSK_HOSTS=\"#{hosts}\"\s
+        MRSK_COMMAND=\"#{command}\"\s
+        #{"MRSK_SUBCOMMAND=\\\"#{subcommand}\\\"\\s" if subcommand}
+        #{"MRSK_RUNTIME=\\\"#{runtime}\\\"\\s" if runtime}
+        ;\s/usr/bin/env\s\.mrsk/hooks/#{hook} }x
+
+      assert_match expected, output
     end
 end

test/cli/main_test.rb

@@ -21,16 +21,18 @@ class CliMainTest < CliTestCase
     Mrsk::Cli::Main.any_instance.expects(:invoke).with("mrsk:cli:prune:all", [], invoke_options)
 
     Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
+    hook_variables = { version: 999, service_version: "app@999", hosts: "1.1.1.1,1.1.1.2", command: "deploy" }
 
     run_command("deploy").tap do |output|
-      assert_match /Running the pre-connect hook.../, output
+      assert_hook_ran "pre-connect", output, **hook_variables
       assert_match /Log into image registry/, output
       assert_match /Build and push app image/, output
+      assert_hook_ran "pre-deploy", output, **hook_variables
       assert_match /Ensure Traefik is running/, output
       assert_match /Ensure app can pass healthcheck/, output
       assert_match /Detect stale containers/, output
       assert_match /Prune old containers and images/, output
-      assert_match /Running the post-deploy hook.../, output
+      assert_hook_ran "post-deploy", output, **hook_variables, runtime: 0
     end
   end
 
@@ -61,11 +63,11 @@ class CliMainTest < CliTestCase
     Thread.report_on_exception = false
 
     SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-      .with { |*arg| arg[0..1] == [:mkdir, :mrsk_lock] }
+      .with { |*arg| arg[0..1] == [:mkdir, 'mrsk_lock-app'] }
-      .raises(RuntimeError, "mkdir: cannot create directory ‘mrsk_lock’: File exists")
+      .raises(RuntimeError, "mkdir: cannot create directory ‘mrsk_lock-app’: File exists")
 
     SSHKit::Backend::Abstract.any_instance.expects(:capture_with_debug)
-      .with(:stat, :mrsk_lock, ">", "/dev/null", "&&", :cat, "mrsk_lock/details", "|", :base64, "-d")
+      .with(:stat, 'mrsk_lock-app', ">", "/dev/null", "&&", :cat, "mrsk_lock-app/details", "|", :base64, "-d")
 
     assert_raises(Mrsk::Cli::LockError) do
       run_command("deploy")
@@ -76,7 +78,7 @@ class CliMainTest < CliTestCase
     Thread.report_on_exception = false
 
     SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-      .with { |*arg| arg[0..1] == [:mkdir, :mrsk_lock] }
+      .with { |*arg| arg[0..1] == [:mkdir, 'mrsk_lock-app'] }
       .raises(SocketError, "getaddrinfo: nodename nor servname provided, or not known")
 
     assert_raises(SSHKit::Runner::ExecuteError) do
@@ -114,6 +116,12 @@ class CliMainTest < CliTestCase
     end
   end
 
+  test "deploy with missing secrets" do
+    assert_raises(KeyError) do
+      run_command("deploy", config_file: "deploy_with_secrets")
+    end
+  end
+
   test "redeploy" do
     invoke_options = { "config_file" => "test/fixtures/deploy_simple.yml", "version" => "999", "skip_hooks" => false }
 
@@ -124,10 +132,15 @@ class CliMainTest < CliTestCase
 
     Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
 
+    hook_variables = { version: 999, service_version: "app@999", hosts: "1.1.1.1,1.1.1.2", command: "redeploy" }
+
     run_command("redeploy").tap do |output|
+      assert_hook_ran "pre-connect", output, **hook_variables
       assert_match /Build and push app image/, output
+      assert_hook_ran "pre-deploy", output, **hook_variables
+      assert_match /Running the pre-deploy hook.../, output
       assert_match /Ensure app can pass healthcheck/, output
-      assert_match /Running the post-deploy hook.../, output
+      assert_hook_ran "post-deploy", output, **hook_variables, runtime: "0"
     end
   end
 
@@ -165,7 +178,7 @@ class CliMainTest < CliTestCase
         .with(:docker, :container, :ls, "--all", "--filter", "name=^app-#{role}-123$", "--quiet")
         .returns("version-to-rollback\n").at_least_once
       SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
-        .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=#{role}", "--filter", "status=running", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
+        .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=#{role}", "--filter", "status=running", "--filter", "status=restarting", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
         .returns("version-to-rollback\n").at_least_once
       SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
         .with(:docker, :container, :ls, "--all", "--filter", "name=^app-#{role}-123$", "--quiet", "|", :xargs, :docker, :inspect, "--format", "'{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'")
@@ -173,13 +186,15 @@ class CliMainTest < CliTestCase
     end
 
     Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
+    hook_variables = { version: 123, service_version: "app@123", hosts: "1.1.1.1,1.1.1.2,1.1.1.3,1.1.1.4", command: "rollback" }
 
     run_command("rollback", "123", config_file: "deploy_with_accessories").tap do |output|
       assert_match "Start container with version 123", output
+      assert_hook_ran "pre-deploy", output, **hook_variables
       assert_match "docker tag dhh/app:123 dhh/app:latest", output
       assert_match "docker start app-web-123", output
       assert_match "docker container ls --all --filter name=^app-web-version-to-rollback$ --quiet | xargs docker stop", output, "Should stop the container that was previously running"
-      assert_match "Running the post-deploy hook...", output
+      assert_hook_ran "post-deploy", output, **hook_variables, runtime: "0"
     end
   end
 
@@ -192,7 +207,7 @@ class CliMainTest < CliTestCase
       .with(:docker, :container, :ls, "--filter", "name=^app-web-123$", "--quiet", raise_on_non_zero_exit: false)
       .returns("").at_least_once
     SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
-      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
+      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--filter", "status=restarting", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
       .returns("").at_least_once
     SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
       .with(:docker, :container, :ls, "--all", "--filter", "name=^app-web-123$", "--quiet", "|", :xargs, :docker, :inspect, "--format", "'{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'")

test/cli/prune_test.rb

@@ -10,7 +10,8 @@ class CliPruneTest < CliTestCase
 
   test "images" do
     run_command("images").tap do |output|
-      assert_match /docker image prune --force --filter label=service=app --filter dangling=true on 1.1.1.\d/, output
+      assert_match "docker image prune --force --filter label=service=app --filter dangling=true on 1.1.1.", output
+      assert_match "docker image ls --filter label=service=app --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w \"$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=app | tr -d '\\n')dhh/app:latest\\|dhh/app:<none>\" | while read image tag; do docker rmi $tag; done on 1.1.1.", output
     end
   end
 

test/cli/server_test.rb

@@ -11,7 +11,7 @@ class CliServerTest < CliTestCase
     SSHKit::Backend::Abstract.any_instance.expects(:execute).with(:docker, "-v", raise_on_non_zero_exit: false).returns(false).at_least_once
     SSHKit::Backend::Abstract.any_instance.expects(:execute).with('[ "${EUID:-$(id -u)}" -eq 0 ]', raise_on_non_zero_exit: false).returns(false).at_least_once
 
-    assert_raise RuntimeError, "Docker is not installed on 1.1.1.1, 1.1.1.3, 1.1.1.4, 1.1.1.2 and can't be automatically intalled without having root access and the `curl` command available. Install Docker manually: https://docs.docker.com/engine/install/" do
+    assert_raise RuntimeError, "Docker is not installed on 1.1.1.1, 1.1.1.3, 1.1.1.4, 1.1.1.2 and can't be automatically installed without having root access and the `curl` command available. Install Docker manually: https://docs.docker.com/engine/install/" do
       run_command("bootstrap")
     end
   end

test/cli/traefik_test.rb

@@ -4,16 +4,24 @@ class CliTraefikTest < CliTestCase
   test "boot" do
     run_command("boot").tap do |output|
       assert_match "docker login", output
-      assert_match "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=DEBUG", output
+      assert_match "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=\"DEBUG\"", output
     end
   end
 
   test "reboot" do
-    Mrsk::Cli::Traefik.any_instance.expects(:stop)
+    Mrsk::Commands::Registry.any_instance.expects(:login).twice
-    Mrsk::Cli::Traefik.any_instance.expects(:remove_container)
-    Mrsk::Cli::Traefik.any_instance.expects(:boot)
 
-    run_command("reboot")
+    run_command("reboot").tap do |output|
+      assert_match "docker container stop traefik", output
+      assert_match "docker container prune --force --filter label=org.opencontainers.image.title=Traefik", output
+      assert_match "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=\"DEBUG\"", output
+    end
+  end
+
+  test "reboot --rolling" do
+    run_command("reboot", "--rolling").tap do |output|
+      assert_match "Running docker container prune --force --filter label=org.opencontainers.image.title=Traefik on 1.1.1.1", output.lines[3]
+    end
   end
 
   test "start" do

test/commander_test.rb

@@ -65,7 +65,7 @@ class CommanderTest < ActiveSupport::TestCase
   end
 
   test "percentage-based group strategy" do
-    configure_with(:deploy_with_precentage_boot_strategy)
+    configure_with(:deploy_with_percentage_boot_strategy)
 
     assert_equal({ in: :groups, limit: 1, wait: 2 }, @mrsk.boot_strategy)
   end

test/commands/app_test.rb

@@ -17,6 +17,12 @@ class CommandsAppTest < ActiveSupport::TestCase
       new_command.run.join(" ")
   end
 
+  test "run with hostname" do
+    assert_equal \
+      "docker run --detach --restart unless-stopped --name app-web-999 --hostname myhost -e MRSK_CONTAINER_NAME=\"app-web-999\" -e RAILS_MASTER_KEY=\"456\" --health-cmd \"curl -f http://localhost:3000/up || exit 1\" --health-interval \"1s\" --log-opt max-size=\"10m\" --label service=\"app\" --label role=\"web\" --label traefik.http.services.app-web.loadbalancer.server.scheme=\"http\" --label traefik.http.routers.app-web.rule=\"PathPrefix(\\`/\\`)\" --label traefik.http.middlewares.app-web-retry.retry.attempts=\"5\" --label traefik.http.middlewares.app-web-retry.retry.initialinterval=\"500ms\" --label traefik.http.routers.app-web.middlewares=\"app-web-retry@docker\" dhh/app:999",
+      new_command.run(hostname: "myhost").join(" ")
+  end
+
   test "run with volumes" do
     @config[:volumes] = ["/local/path:/container/path" ]
 
@@ -77,16 +83,28 @@ class CommandsAppTest < ActiveSupport::TestCase
       new_command.start.join(" ")
   end
 
+  test "start_or_run" do
+    assert_equal \
+      "docker start app-web-999 || docker run --detach --restart unless-stopped --name app-web-999 -e MRSK_CONTAINER_NAME=\"app-web-999\" -e RAILS_MASTER_KEY=\"456\" --health-cmd \"curl -f http://localhost:3000/up || exit 1\" --health-interval \"1s\" --log-opt max-size=\"10m\" --label service=\"app\" --label role=\"web\" --label traefik.http.services.app-web.loadbalancer.server.scheme=\"http\" --label traefik.http.routers.app-web.rule=\"PathPrefix(\\`/\\`)\" --label traefik.http.middlewares.app-web-retry.retry.attempts=\"5\" --label traefik.http.middlewares.app-web-retry.retry.initialinterval=\"500ms\" --label traefik.http.routers.app-web.middlewares=\"app-web-retry@docker\" dhh/app:999",
+      new_command.start_or_run.join(" ")
+  end
+
+  test "start_or_run with hostname" do
+    assert_equal \
+      "docker start app-web-999 || docker run --detach --restart unless-stopped --name app-web-999 --hostname myhost -e MRSK_CONTAINER_NAME=\"app-web-999\" -e RAILS_MASTER_KEY=\"456\" --health-cmd \"curl -f http://localhost:3000/up || exit 1\" --health-interval \"1s\" --log-opt max-size=\"10m\" --label service=\"app\" --label role=\"web\" --label traefik.http.services.app-web.loadbalancer.server.scheme=\"http\" --label traefik.http.routers.app-web.rule=\"PathPrefix(\\`/\\`)\" --label traefik.http.middlewares.app-web-retry.retry.attempts=\"5\" --label traefik.http.middlewares.app-web-retry.retry.initialinterval=\"500ms\" --label traefik.http.routers.app-web.middlewares=\"app-web-retry@docker\" dhh/app:999",
+      new_command.start_or_run(hostname: "myhost").join(" ")
+  end
+
   test "stop" do
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop",
       new_command.stop.join(" ")
   end
 
   test "stop with custom stop wait time" do
     @config[:stop_wait_time] = 30
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop -t 30",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop -t 30",
       new_command.stop.join(" ")
   end
 
@@ -112,37 +130,37 @@ class CommandsAppTest < ActiveSupport::TestCase
 
   test "logs" do
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs 2>&1",
       new_command.logs.join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --since 5m 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --since 5m 2>&1",
       new_command.logs(since: "5m").join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --tail 100 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --tail 100 2>&1",
       new_command.logs(lines: "100").join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --since 5m --tail 100 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --since 5m --tail 100 2>&1",
       new_command.logs(since: "5m", lines: "100").join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs 2>&1 | grep 'my-id'",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs 2>&1 | grep 'my-id'",
       new_command.logs(grep: "my-id").join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --since 5m 2>&1 | grep 'my-id'",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --since 5m 2>&1 | grep 'my-id'",
       new_command.logs(since: "5m", grep: "my-id").join(" ")
   end
 
   test "follow logs" do
     assert_match \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1",
       new_command.follow_logs(host: "app-1")
 
     assert_match \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1 | grep \"Completed\"",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1 | grep \"Completed\"",
       new_command.follow_logs(host: "app-1", grep: "Completed")
   end
 
@@ -193,17 +211,21 @@ class CommandsAppTest < ActiveSupport::TestCase
     assert_equal "ssh -J root@2.2.2.2 -t app@1.1.1.1 'ls'", new_command.run_over_ssh("ls", host: "1.1.1.1")
   end
 
+  test "run over ssh with proxy_command" do
+    @config[:ssh] = { "proxy_command" => "ssh -W %h:%p user@proxy-server" }
+    assert_equal "ssh -o ProxyCommand='ssh -W %h:%p user@proxy-server' -t root@1.1.1.1 'ls'", new_command.run_over_ssh("ls", host: "1.1.1.1")
+  end
 
   test "current_running_container_id" do
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest",
       new_command.current_running_container_id.join(" ")
   end
 
   test "current_running_container_id with destination" do
     @destination = "staging"
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=destination=staging --filter label=role=web --filter status=running --latest",
+      "docker ps --quiet --filter label=service=app --filter label=destination=staging --filter label=role=web --filter status=running --filter status=restarting --latest",
       new_command.current_running_container_id.join(" ")
   end
 
@@ -215,7 +237,7 @@ class CommandsAppTest < ActiveSupport::TestCase
 
   test "current_running_version" do
     assert_equal \
-      "docker ps --filter label=service=app --filter label=role=web --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
+      "docker ps --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
       new_command.current_running_version.join(" ")
   end
 
@@ -225,8 +247,8 @@ class CommandsAppTest < ActiveSupport::TestCase
       new_command.list_versions.join(" ")
 
     assert_equal \
-      "docker ps --filter label=service=app --filter label=role=web --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
+      "docker ps --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
-      new_command.list_versions("--latest", status: :running).join(" ")
+      new_command.list_versions("--latest", statuses: [ :running, :restarting ]).join(" ")
   end
 
   test "list_containers" do

test/commands/builder_test.rb

@@ -6,10 +6,10 @@ class CommandsBuilderTest < ActiveSupport::TestCase
   end
 
   test "target multiarch by default" do
-    builder = new_builder_command
+    builder = new_builder_command(builder: { "cache" => { "type" => "gha" }})
     assert_equal "multiarch", builder.name
     assert_equal \
-      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch -t dhh/app:123 -t dhh/app:latest --label service=\"app\" --file Dockerfile .",
+      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch -t dhh/app:123 -t dhh/app:latest --cache-to type=gha --cache-from type=gha --label service=\"app\" --file Dockerfile .",
       builder.push.join(" ")
   end
 
@@ -21,19 +21,27 @@ class CommandsBuilderTest < ActiveSupport::TestCase
       builder.push.join(" ")
   end
 
+  test "target native cached when multiarch is off and cache is set" do
+    builder = new_builder_command(builder: { "multiarch" => false, "cache" => { "type" => "gha" }})
+    assert_equal "native/cached", builder.name
+    assert_equal \
+      "docker buildx build --push -t dhh/app:123 -t dhh/app:latest --cache-to type=gha --cache-from type=gha --label service=\"app\" --file Dockerfile .",
+      builder.push.join(" ")
+  end
+
   test "target multiarch remote when local and remote is set" do
-    builder = new_builder_command(builder: { "local" => { }, "remote" => { } })
+    builder = new_builder_command(builder: { "local" => { }, "remote" => { }, "cache" => { "type" => "gha" } })
     assert_equal "multiarch/remote", builder.name
     assert_equal \
-      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch-remote -t dhh/app:123 -t dhh/app:latest --label service=\"app\" --file Dockerfile .",
+      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch-remote -t dhh/app:123 -t dhh/app:latest --cache-to type=gha --cache-from type=gha --label service=\"app\" --file Dockerfile .",
       builder.push.join(" ")
   end
 
   test "target native remote when only remote is set" do
-    builder = new_builder_command(builder: { "remote" => { "arch" => "amd64" } })
+    builder = new_builder_command(builder: { "remote" => { "arch" => "amd64" }, "cache" => { "type" => "gha" } })
     assert_equal "native/remote", builder.name
     assert_equal \
-      "docker buildx build --push --platform linux/amd64 --builder mrsk-app-native-remote -t dhh/app:123 -t dhh/app:latest --label service=\"app\" --file Dockerfile .",
+      "docker buildx build --push --platform linux/amd64 --builder mrsk-app-native-remote -t dhh/app:123 -t dhh/app:latest --cache-to type=gha --cache-from type=gha --label service=\"app\" --file Dockerfile .",
       builder.push.join(" ")
   end
 
@@ -88,7 +96,7 @@ class CommandsBuilderTest < ActiveSupport::TestCase
       builder.push.join(" ")
   end
 
-  test "native push with with build secrets" do
+  test "native push with build secrets" do
     builder = new_builder_command(builder: { "multiarch" => false, "secrets" => [ "a", "b" ] })
     assert_equal \
       "docker build -t dhh/app:123 -t dhh/app:latest --label service=\"app\" --secret id=\"a\" --secret id=\"b\" --file Dockerfile . && docker push dhh/app:123 && docker push dhh/app:latest",

test/commands/lock_test.rb

@@ -10,19 +10,19 @@ class CommandsLockTest < ActiveSupport::TestCase
 
   test "status" do
     assert_equal \
-      "stat mrsk_lock > /dev/null && cat mrsk_lock/details | base64 -d",
+      "stat mrsk_lock-app > /dev/null && cat mrsk_lock-app/details | base64 -d",
       new_command.status.join(" ")
   end
 
   test "acquire" do
     assert_match \
-      /mkdir mrsk_lock && echo ".*" > mrsk_lock\/details/m,
+      /mkdir mrsk_lock-app && echo ".*" > mrsk_lock-app\/details/m,
       new_command.acquire("Hello", "123").join(" ")
   end
 
   test "release" do
     assert_match \
-      "rm mrsk_lock/details && rm -r mrsk_lock",
+      "rm mrsk_lock-app/details && rm -r mrsk_lock-app",
       new_command.release.join(" ")
   end
 

test/commands/prune_test.rb

@@ -8,10 +8,16 @@ class CommandsPruneTest < ActiveSupport::TestCase
     }
   end
 
-  test "images" do
+  test "dangling images" do
     assert_equal \
       "docker image prune --force --filter label=service=app --filter dangling=true",
-      new_command.images.join(" ")
+      new_command.dangling_images.join(" ")
+  end
+
+  test "tagged images" do
+    assert_equal \
+      "docker image ls --filter label=service=app --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w \"$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=app | tr -d '\\n')dhh/app:latest\\|dhh/app:<none>\" | while read image tag; do docker rmi $tag; done",
+      new_command.tagged_images.join(" ")
   end
 
   test "containers" do

test/commands/traefik_test.rb

@@ -18,67 +18,67 @@ class CommandsTraefikTest < ActiveSupport::TestCase
 
   test "run" do
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
 
     @config[:traefik]["host_port"] = "8080"
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 8080:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 8080:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
   end
 
   test "run with ports configured" do
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
 
     @config[:traefik]["options"] = {"publish" => %w[9000:9000 9001:9001]}
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --publish \"9000:9000\" --publish \"9001:9001\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --publish \"9000:9000\" --publish \"9001:9001\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
   end
 
   test "run with volumes configured" do
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
 
     @config[:traefik]["options"] = {"volume" => %w[./letsencrypt/acme.json:/letsencrypt/acme.json] }
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
   end
 
   test "run with several options configured" do
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
 
     @config[:traefik]["options"] = {"volume" => %w[./letsencrypt/acme.json:/letsencrypt/acme.json], "publish" => %w[8080:8080], "memory" => "512m"}
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" --publish \"8080:8080\" --memory \"512m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" --publish \"8080:8080\" --memory \"512m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
   end
 
   test "run with labels configured" do
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
 
     @config[:traefik]["labels"] = { "traefik.http.routers.dashboard.service" => "api@internal", "traefik.http.routers.dashboard.middlewares" => "auth" }
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --label traefik.http.routers.dashboard.service=\"api@internal\" --label traefik.http.routers.dashboard.middlewares=\"auth\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --label traefik.http.routers.dashboard.service=\"api@internal\" --label traefik.http.routers.dashboard.middlewares=\"auth\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
   end
 
   test "run with env configured" do
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
 
     @config[:traefik]["env"] = { "secret" => %w[EXAMPLE_API_KEY] }
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock -e EXAMPLE_API_KEY=\"456\" --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock -e EXAMPLE_API_KEY=\"456\" --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
   end
 
@@ -86,7 +86,7 @@ class CommandsTraefikTest < ActiveSupport::TestCase
     @config.delete(:traefik)
 
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=DEBUG",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=\"DEBUG\"",
       new_command.run.join(" ")
   end
 
@@ -94,7 +94,15 @@ class CommandsTraefikTest < ActiveSupport::TestCase
     @config[:logging] = { "driver" => "local", "options" => { "max-size" => "100m", "max-file" => "3" } }
 
     assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-driver \"local\" --log-opt max-size=\"100m\" --log-opt max-file=\"3\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-driver \"local\" --log-opt max-size=\"100m\" --log-opt max-file=\"3\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      new_command.run.join(" ")
+  end
+
+  test "run with default args overriden" do
+    @config[:traefik]["args"]["log.level"] = "ERROR"
+
+    assert_equal \
+    "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"ERROR\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
       new_command.run.join(" ")
   end
 

test/configuration/builder_test.rb

@@ -0,0 +1,151 @@
+require "test_helper"
+
+class ConfigurationBuilderTest < ActiveSupport::TestCase
+  setup do
+    @deploy = {
+      service: "app", image: "dhh/app", registry: { "username" => "dhh", "password" => "secret" },
+      servers: [ "1.1.1.1" ]
+    }
+
+    @config = Mrsk::Configuration.new(@deploy)
+
+    @deploy_with_builder_option = {
+      service: "app", image: "dhh/app", registry: { "username" => "dhh", "password" => "secret" },
+      servers: [ "1.1.1.1" ],
+      builder: {}
+    }
+
+    @config_with_builder_option = Mrsk::Configuration.new(@deploy_with_builder_option)
+  end
+
+  test "multiarch?" do
+    assert_equal true, @config.builder.multiarch?
+  end
+
+  test "setting multiarch to false" do
+    @deploy_with_builder_option[:builder] = { "multiarch" => false }
+
+    assert_equal false, @config_with_builder_option.builder.multiarch?
+  end
+
+  test "local?" do
+    assert_equal false, @config.builder.local?
+  end
+
+  test "remote?" do
+    assert_equal false, @config.builder.remote?
+  end
+
+  test "remote_arch" do
+    assert_nil @config.builder.remote_arch
+  end
+
+  test "remote_host" do
+    assert_nil @config.builder.remote_host
+  end
+
+  test "remote config is missing when local is specified" do
+    @deploy_with_builder_option[:builder] = { "local" => { "arch" => "arm64", "host" => "unix:///Users/<%= `whoami`.strip %>/.docker/run/docker.sock" } }
+
+    assert_raises(ArgumentError) do
+      @config_with_builder_option.builder
+    end
+  end
+
+  test "setting both local and remote configs" do
+    @deploy_with_builder_option[:builder] = {
+      "local" => { "arch" => "arm64", "host" => "unix:///Users/<%= `whoami`.strip %>/.docker/run/docker.sock" },
+      "remote" => { "arch" => "amd64", "host" => "ssh://root@192.168.0.1" }
+    }
+
+    assert_equal true, @config_with_builder_option.builder.local?
+    assert_equal true, @config_with_builder_option.builder.remote?
+
+    assert_equal "amd64", @config_with_builder_option.builder.remote_arch
+    assert_equal "ssh://root@192.168.0.1", @config_with_builder_option.builder.remote_host
+
+    assert_equal "arm64", @config_with_builder_option.builder.local_arch
+    assert_equal "unix:///Users/<%= `whoami`.strip %>/.docker/run/docker.sock", @config_with_builder_option.builder.local_host
+  end
+
+  test "cached?" do
+    assert_equal false, @config.builder.cached?
+  end
+
+  test "invalid cache type specified" do
+    @deploy_with_builder_option[:builder] = { "cache" => { "type" => "invalid" } }
+
+    assert_raises(ArgumentError) do
+      @config_with_builder_option.builder
+    end
+  end
+
+  test "cache_from" do
+    assert_nil @config.builder.cache_from
+  end
+
+  test "cache_to" do
+    assert_nil @config.builder.cache_to
+  end
+
+  test "setting gha cache" do
+    @deploy_with_builder_option[:builder] = { "cache" => { "type" => "gha", "options" => "mode=max" } }
+
+    assert_equal "type=gha", @config_with_builder_option.builder.cache_from
+    assert_equal "type=gha,mode=max", @config_with_builder_option.builder.cache_to
+  end
+
+  test "setting registry cache" do
+    @deploy_with_builder_option[:builder] = { "cache" => { "type" => "registry", "options" => "mode=max,image-manifest=true,oci-mediatypes=true" } }
+
+    assert_equal "type=registry,ref=/dhh/app-build-cache", @config_with_builder_option.builder.cache_from
+    assert_equal "type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref=/dhh/app-build-cache", @config_with_builder_option.builder.cache_to
+  end
+
+  test "setting registry cache with image" do
+    @deploy_with_builder_option[:builder] = { "cache" => { "type" => "registry", "image" => "mrsk", "options" => "mode=max" } }
+
+    assert_equal "type=registry,ref=/mrsk", @config_with_builder_option.builder.cache_from
+    assert_equal "type=registry,mode=max,ref=/mrsk", @config_with_builder_option.builder.cache_to
+  end
+
+  test "args" do
+    assert_equal({}, @config.builder.args)
+  end
+
+  test "setting args" do
+    @deploy_with_builder_option[:builder] = { "args" => { "key" => "value" } }
+
+    assert_equal({ "key" => "value" }, @config_with_builder_option.builder.args)
+  end
+
+  test "secrets" do
+    assert_equal [], @config.builder.secrets
+  end
+
+  test "setting secrets" do
+    @deploy_with_builder_option[:builder] = { "secrets" => ["GITHUB_TOKEN"] }
+
+    assert_equal ["GITHUB_TOKEN"], @config_with_builder_option.builder.secrets
+  end
+
+  test "dockerfile" do
+    assert_equal "Dockerfile", @config.builder.dockerfile
+  end
+
+  test "setting dockerfile" do
+    @deploy_with_builder_option[:builder] = { "dockerfile" => "Dockerfile.dev" }
+
+    assert_equal "Dockerfile.dev", @config_with_builder_option.builder.dockerfile
+  end
+
+  test "context" do
+    assert_equal ".", @config.builder.context
+  end
+
+  test "setting context" do
+    @deploy_with_builder_option[:builder] = { "context" => ".." }
+
+    assert_equal "..", @config_with_builder_option.builder.context
+  end
+end

test/configuration_test.rb

@@ -166,7 +166,7 @@ class ConfigurationTest < ActiveSupport::TestCase
     assert_raises(KeyError) do
       config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!({
         env: { "secret" => [ "PASSWORD" ] }
-      }) })
+      }) }).ensure_env_available
     end
   end
 
@@ -211,17 +211,21 @@ class ConfigurationTest < ActiveSupport::TestCase
     assert_equal "root", @config.ssh_options[:user]
 
     config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(ssh: { "user" => "app" }) })
-    assert_equal "app", @config.ssh_options[:user]
+    assert_equal "app", config.ssh_options[:user]
+    assert_equal 4, config.ssh_options[:logger].level
+
+    config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(ssh: { "log_level" => "debug" }) })
+    assert_equal 0, config.ssh_options[:logger].level
   end
 
   test "ssh options with proxy host" do
     config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(ssh: { "proxy" => "1.2.3.4" }) })
-    assert_equal "root@1.2.3.4", @config.ssh_options[:proxy].jump_proxies
+    assert_equal "root@1.2.3.4", config.ssh_options[:proxy].jump_proxies
   end
 
   test "ssh options with proxy host and user" do
     config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(ssh: { "proxy" => "app@1.2.3.4" }) })
-    assert_equal "app@1.2.3.4", @config.ssh_options[:proxy].jump_proxies
+    assert_equal "app@1.2.3.4", config.ssh_options[:proxy].jump_proxies
   end
 
   test "volume_args" do
@@ -266,6 +270,37 @@ class ConfigurationTest < ActiveSupport::TestCase
   end
 
   test "to_h" do
-    assert_equal({ :roles=>["web"], :hosts=>["1.1.1.1", "1.1.1.2"], :primary_host=>"1.1.1.1", :version=>"missing", :repository=>"dhh/app", :absolute_image=>"dhh/app:missing", :service_with_version=>"app-missing", :env_args=>["-e", "REDIS_URL=\"redis://x/y\""], :ssh_options=>{:user=>"root", :auth_methods=>["publickey"]}, :volume_args=>["--volume", "/local/path:/container/path"], :logging=>["--log-opt", "max-size=\"10m\""], :healthcheck=>{"path"=>"/up", "port"=>3000, "max_attempts" => 7 }}, @config.to_h)
+    assert_equal({
+      :roles=>["web"],
+      :hosts=>["1.1.1.1", "1.1.1.2"],
+      :primary_host=>"1.1.1.1",
+      :version=>"missing",
+      :repository=>"dhh/app",
+      :absolute_image=>"dhh/app:missing",
+      :service_with_version=>"app-missing",
+      :env_args=>["-e", "REDIS_URL=\"redis://x/y\""],
+      :ssh_options=>{:user=>"root", :auth_methods=>["publickey"]},
+      :ssh_log_level=>4,
+      :volume_args=>["--volume", "/local/path:/container/path"],
+      :builder=>{},
+      :logging=>["--log-opt", "max-size=\"10m\""],
+      :healthcheck=>{"path"=>"/up", "port"=>3000, "max_attempts" => 7 }
+    }, @config.to_h)
+  end
+
+  test "min version is lower" do
+    config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(minimum_version: "0.0.1") })
+    assert_equal "0.0.1", config.minimum_version
+  end
+
+  test "min version is equal" do
+    config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(minimum_version: Mrsk::VERSION) })
+    assert_equal Mrsk::VERSION, config.minimum_version
+  end
+
+  test "min version is higher" do
+    assert_raises(ArgumentError) do
+      Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(minimum_version: "10000.0.0") })
+    end
   end
 end

test/fixtures/deploy_with_secrets.yml

@@ -0,0 +1,11 @@
+service: app
+image: dhh/app
+servers:
+  - "1.1.1.1"
+  - "1.1.1.2"
+registry:
+  username: user
+  password: pw
+env:
+  secret:
+    - PASSWORD

test/integration/docker/deployer/Dockerfile

@@ -17,6 +17,7 @@ RUN apt-get update --fix-missing && apt-get install -y docker-ce docker-ce-cli c
 COPY *.sh .
 COPY app/ .
 
+RUN rm -rf /root/.ssh
 RUN ln -s /shared/ssh /root/.ssh
 RUN mkdir -p /etc/docker/certs.d/registry:4443 && ln -s /shared/certs/domain.crt /etc/docker/certs.d/registry:4443/ca.crt
 

test/integration/docker/deployer/app/.mrsk/hooks/pre-deploy

@@ -0,0 +1,3 @@
+#!/bin/sh
+echo "Deployed!"
+mkdir -p /tmp/${TEST_ID} && touch /tmp/${TEST_ID}/pre-deploy

test/integration/docker/load_balancer/default.conf

@@ -8,9 +8,9 @@ server {
 
   location / {
     proxy_pass http://loadbalancer;
-    proxy_connect_timeout       5;
+    proxy_connect_timeout       10;
-    proxy_send_timeout          5;
+    proxy_send_timeout          10;
-    proxy_read_timeout          5;
+    proxy_read_timeout          10;
-    send_timeout                5;
+    send_timeout                10;
   }
 }

test/integration/integration_test.rb

@@ -10,6 +10,13 @@ class IntegrationTest < ActiveSupport::TestCase
   end
 
   teardown do
+    unless passed?
+      [:deployer, :vm1, :vm2, :shared, :load_balancer, :registry].each do |container|
+        puts
+        puts "Logs for #{container}:"
+        docker_compose :logs, container
+      end
+    end
     docker_compose "down -t 1"
   end
 
@@ -54,7 +61,7 @@ class IntegrationTest < ActiveSupport::TestCase
       assert_equal "404", response.code
     end
 
-    def wait_for_app_to_be_up(timeout: 10, up_count: 3)
+    def wait_for_app_to_be_up(timeout: 20, up_count: 3)
       timeout_at = Time.now + timeout
       up_times = 0
       response = app_response

test/integration/main_test.rb

@@ -8,16 +8,16 @@ class MainTest < IntegrationTest
 
     mrsk :deploy
     assert_app_is_up version: first_version
-    assert_hooks_ran "pre-connect", "pre-build", "post-deploy"
+    assert_hooks_ran "pre-connect", "pre-build", "pre-deploy", "post-deploy"
 
     second_version = update_app_rev
 
     mrsk :redeploy
     assert_app_is_up version: second_version
-    assert_hooks_ran "pre-connect", "pre-build", "post-deploy"
+    assert_hooks_ran "pre-connect", "pre-build", "pre-deploy", "post-deploy"
 
     mrsk :rollback, first_version
-    assert_hooks_ran "pre-connect", "post-deploy"
+    assert_hooks_ran "pre-connect", "pre-deploy", "post-deploy"
     assert_app_is_up version: first_version
 
     details = mrsk :details, capture: true