Bump version for 0.15.0

Merge pull request #357 from igor-alexandrov/documentation-update
Updated README with info for Rails <7 usage
2023-07-24 13:43:50 +01:00 · 2023-07-24 14:39:48 +02:00 · 2023-07-19 16:43:59 +01:00 · 2023-07-19 15:03:15 +01:00 · 2023-07-19 14:58:46 +01:00 · 2023-07-19 14:46:16 +01:00
50 changed files with 951 additions and 255 deletions
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -27,7 +27,7 @@ Please keep the following guidelines in mind when opening a pull request:
 - Add tests for your changes, if possible.
 - Ensure that your changes don't break existing functionality.

-#### Commit message guidline
+#### Commit message guidelines

 A good commit message should describe what changed and why.

--- a/2
+++ b/2
@@ -21,7 +21,7 @@ RUN apk add --no-cache --update build-base git docker openrc openssh-client-defa

 # Copy the rest of our application code into the container.
 # We do this after bundle install, to avoid having to run bundle
-# everytime we do small fixes in the source code.
+# every time we do small fixes in the source code.
 COPY . .

 # Install the gem locally from the project folder
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -1,7 +1,7 @@
 PATH
  remote: .
  specs:
-    mrsk (0.13.0)
+    mrsk (0.15.0)
      activesupport (>= 7.0)
      bcrypt_pbkdf (~> 1.0)
      dotenv (~> 2.8)
--- a/README.md
+++ b/README.md
@@ -63,11 +63,29 @@ This will:

 Voila! All the servers are now serving the app on port 80. If you're just running a single server, you're ready to go. If you're running multiple servers, you need to put a load balancer in front of them. For subsequent deploys, or if your servers already have Docker and curl installed, you can just run `mrsk deploy`.

+### Rails <7 usage
+
+MRSK is not needed to be in your application Gemfile to be used. However, if you want to guarantee specific MRSK version in your CI/CD workflows, you can create a separate Gemfile for MRSK, for example, `gemfile/mrsk.gemfile`:
+
+```ruby
+source 'https://rubygems.org'
+
+gem 'mrsk', '~> 0.14'
+```
+
+Bundle with `BUNDLE_GEMFILE=gemfiles/mrsk.gemfile bundle`.
+
+After this MRSK can be used for deployment:
+
+```sh
+BUNDLE_GEMFILE=gemfiles/mrsk.gemfile bundle exec mrsk deploy
+```
+
 ## Vision

 In the past decade+, there's been an explosion in commercial offerings that make deploying web apps easier. Heroku kicked it off with an incredible offering that stayed ahead of the competition seemingly forever. These days we have excellent alternatives like Fly.io and Render. And hosted Kubernetes is making things easier too on AWS, GCP, Digital Ocean, and elsewhere. But these are all offerings that have you renting computers in the cloud at a premium. If you want to run on your own hardware, or even just have a clear migration path to do so in the future, you need to carefully consider how locked in you get to these commercial platforms. Preferably before the bills swallow your business whole!

-MRSK seeks to bring the advance in ergonomics pioneered by these commercial offerings to deploying web apps anywhere. Whether that's low-cost cloud options without the managed-service markup from the likes of Digital Ocean, Hetzner, OVH, etc, or it's your own colocated bare metal. To MRSK, it's all the same. Feed the config file a list of IP addresses with vanilla Ubuntu servers that have seen no prep beyond an added SSH key, and you'll be running in literally minutes.
+MRSK seeks to bring the advance in ergonomics pioneered by these commercial offerings to deploying web apps anywhere. Whether that's low-cost cloud options without the managed-service markup from the likes of Digital Ocean, Hetzner, OVH, etc., or it's your own colocated bare metal. To MRSK, it's all the same. Feed the config file a list of IP addresses with vanilla Ubuntu servers that have seen no prep beyond an added SSH key, and you'll be running in literally minutes.

 This approach gives you enormous portability. You can have your web app deployed on several clouds at ease like this. Or you can buy the baseline with your own hardware, then deploy to a cloud before a big seasonal spike to get more capacity. When you're not locked into a single provider from a tooling perspective, there are a lot of compelling options available.

@@ -123,6 +141,8 @@ This template can safely be checked into git. Then everyone deploying the app ca

 If you need separate env variables for different destinations, you can set them with `.env.destination.erb` for the template, which will generate `.env.staging` when run with `mrsk envify -d staging`.

+Note: If you utilize biometrics with 1Password you can remove the `session_token` related parts in the example and just call `op read op://Vault/Docker Hub/password -n`.
+
 #### Bitwarden as a secret store

 If you are using open source secret store like bitwarden, you can create `.env.erb` as a template which looks up the secrets.
@@ -206,13 +226,13 @@ ssh:
  user: app
 ```

-If you are using non-root user, you need to bootstrap your servers manually, before using them with MRSK. On Ubuntu, you'd do:
+If you are using non-root user (`app` as above example), you need to bootstrap your servers manually, before using them with MRSK. On Ubuntu, you'd do:

 ```bash
 sudo apt update
 sudo apt upgrade -y
 sudo apt install -y docker.io curl git
-sudo usermod -a -G docker ubuntu
+sudo usermod -a -G docker app
 ```

 ### Using a proxy SSH host
@@ -238,6 +258,15 @@ ssh:
  proxy_command: aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --region=us-east-1 ## ssh via aws ssm
 ```

+### Configuring the SSH log level
+
+```yaml
+ssh:
+  log_level: debug
+```
+
+Valid levels are `debug`, `info`, `warn`, `error` and `fatal` (default).
+
 ### Using env variables

 You can inject env variables into the app containers using `env`:
@@ -380,6 +409,16 @@ servers:

 That'll start the job containers with `docker run ... --cap-add --cpu-count 4 ...`.

+### Setting a minimum version
+
+You can set the minimum MRSK version with:
+
+```yaml
+minimum_version: 0.13.3
+```
+
+Note: versions <= 0.13.2 will ignore this setting.
+
 ### Configuring logging

 You can configure the logging driver and options passed to Docker using `logging`:
@@ -463,6 +502,37 @@ builder:
  context: ".."
 ```

+### Using multistage builder cache
+
+Docker multistage build cache can singlehandedly speed up your builds by a lot. Currently MRSK only supports using the GHA cache or the Registry cache:
+
+```yaml
+# Using GHA cache
+builder:
+  cache:
+    type: gha
+
+# Using Registry cache
+builder:
+  cache:
+    type: registry
+
+# Using Registry cache with different cache image
+builder:
+  cache:
+    type: registry
+    # default image name is <image>-build-cache
+    image: application-cache-image
+
+# Using Registry cache with additinonal cache-to options
+builder:
+  cache:
+    type: registry
+    options: mode=max,image-manifest=true,oci-mediatypes=true
+```
+
+For further insights into build cache optimization, check out documentation on Docker's official website: https://docs.docker.com/build/cache/.
+
 ### Using build secrets for new images

 Some images need a secret passed in during build time, like a GITHUB_TOKEN, to give access to private gem repositories. This can be done by having the secret in ENV, then referencing it in the builder configuration:
@@ -581,6 +651,16 @@ traefik:
    entrypoints.otherentrypoint.address: ':9000'
 ```

+### Rebooting Traefik
+
+If you make changes to Traefik args or labels, you'll need to reboot with:
+
+`mrsk traefik reboot`
+
+In production, reboot the Traefik containers one by one with a slower but safer approach, using a rolling reboot:
+
+`mrsk traefik reboot --rolling`
+
 ### Configuring build args for new images

 Build arguments that aren't secret can also be configured:
@@ -670,7 +750,7 @@ This assumes the Cron settings are stored in `config/crontab`.

 ### Healthcheck

-MRSK uses Docker healtchecks to check the health of your application during deployment. Traefik uses this same healthcheck status to determine when a container is ready to receive traffic.
+MRSK uses Docker healthchecks to check the health of your application during deployment. Traefik uses this same healthcheck status to determine when a container is ready to receive traffic.

 The healthcheck defaults to testing the HTTP response to the path `/up` on port 3000, up to 7 times. You can tailor this behaviour with the `healthcheck` setting:

@@ -825,7 +905,7 @@ If you wish to remove the entire application, including Traefik, containers, ima

 ## Locking

-Commands that are unsafe to run concurrently will take a deploy lock while they run. The lock is the `mrsk_lock` directory on the primary server.
+Commands that are unsafe to run concurrently will take a deploy lock while they run. The lock is the `mrsk_lock-<service>` directory on the primary server.

 You can check the lock status with:

@@ -840,7 +920,7 @@ Message: Automatic deploy lock
 You can also manually acquire and release the lock

 ```
-mrsk lock acquire -m "Doing maintanence"
+mrsk lock acquire -m "Doing maintenance"
 ```

 ```
@@ -881,12 +961,14 @@ firing a JSON webhook. These variables include:
 - `MRSK_RECORDED_AT` - UTC timestamp in ISO 8601 format, e.g. `2023-04-14T17:07:31Z`
 - `MRSK_PERFORMER` - the local user performing the command (from `whoami`)
 - `MRSK_SERVICE_VERSION` - an abbreviated service and version for use in messages, e.g. app@150b24f
- `MRSK_VERSION` - an full version being deployed
+- `MRSK_VERSION` - the full version being deployed
+- `MRSK_HOSTS` - a comma-separated list of the hosts targeted by the command
+- `MRSK_COMMAND` - The command we are running
+- `MRSK_SUBCOMMAND` - optional: The subcommand we are running
 - `MRSK_DESTINATION` - optional: destination, e.g. "staging"
- `MRSK_HOSTS` - a comma separated list of the hosts targeted by the command
 - `MRSK_ROLE` - optional: role targeted, e.g. "web"

-There are three hooks:
+There are four hooks:

 1. pre-connect
 Called before taking the deploy lock. For checks that need to run before connecting to remote hosts - e.g. DNS warming.
@@ -894,9 +976,11 @@ Called before taking the deploy lock. For checks that need to run before connect
 2. pre-build
 Used for pre-build checks - e.g. there are no uncommitted changes or that CI has passed.

-3. post-deploy - run after a deploy, redeploy or rollback
+3. pre-deploy
+For final checks before deploying, e.g. checking CI completed

-This hook is also passed a `MRSK_RUNTIME` env variable.
+3. post-deploy - run after a deploy, redeploy or rollback.
+This hook is also passed a `MRSK_RUNTIME` env variable set to the total seconds the deploy took.

 This could be used to broadcast a deployment message, or register the new version with an APM.

@@ -907,7 +991,7 @@ The command could look something like:
 curl -q -d content="[My App] ${MRSK_PERFORMER} Rolled back to version ${MRSK_VERSION}" https://3.basecamp.com/XXXXX/integrations/XXXXX/buckets/XXXXX/chats/XXXXX/lines
 ```

-That'll post a line like follows to a preconfigured chatbot in Basecamp:
+That'll post a line like the following to a preconfigured chatbot in Basecamp:

 ```
 [My App] [dhh] Rolled back to version d264c4e92470ad1bd18590f04466787262f605de
--- a/lib/mrsk/cli/accessory.rb
+++ b/lib/mrsk/cli/accessory.rb
@@ -1,7 +1,7 @@
 class Mrsk::Cli::Accessory < Mrsk::Cli::Base
  desc "boot [NAME]", "Boot new accessory service on host (use NAME=all to boot all accessories)"
-  def boot(name)
-    with_lock do
+  def boot(name, login: true)
+    mutating do
      if name == "all"
        MRSK.accessory_names.each { |accessory_name| boot(accessory_name) }
      else
@@ -10,7 +10,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
          upload(name)

          on(accessory.hosts) do
-            execute *MRSK.registry.login
+            execute *MRSK.registry.login if login
            execute *MRSK.auditor.record("Booted #{name} accessory"), verbosity: :debug
            execute *accessory.run
          end
@@ -21,7 +21,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base

  desc "upload [NAME]", "Upload accessory files to host", hide: true
  def upload(name)
-    with_lock do
+    mutating do
      with_accessory(name) do |accessory|
        on(accessory.hosts) do
          accessory.files.each do |(local, remote)|
@@ -38,7 +38,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base

  desc "directories [NAME]", "Create accessory directories on host", hide: true
  def directories(name)
-    with_lock do
+    mutating do
      with_accessory(name) do |accessory|
        on(accessory.hosts) do
          accessory.directories.keys.each do |host_path|
@@ -51,18 +51,22 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base

  desc "reboot [NAME]", "Reboot existing accessory on host (stop container, remove container, start new container)"
  def reboot(name)
-    with_lock do
+    mutating do
      with_accessory(name) do |accessory|
+        on(accessory.hosts) do
+          execute *MRSK.registry.login
+        end
+
        stop(name)
        remove_container(name)
-        boot(name)
+        boot(name, login: false)
      end
    end
  end

  desc "start [NAME]", "Start existing accessory container on host"
  def start(name)
-    with_lock do
+    mutating do
      with_accessory(name) do |accessory|
        on(accessory.hosts) do
          execute *MRSK.auditor.record("Started #{name} accessory"), verbosity: :debug
@@ -74,7 +78,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base

  desc "stop [NAME]", "Stop existing accessory container on host"
  def stop(name)
-    with_lock do
+    mutating do
      with_accessory(name) do |accessory|
        on(accessory.hosts) do
          execute *MRSK.auditor.record("Stopped #{name} accessory"), verbosity: :debug
@@ -86,7 +90,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base

  desc "restart [NAME]", "Restart existing accessory container on host"
  def restart(name)
-    with_lock do
+    mutating do
      with_accessory(name) do
        stop(name)
        start(name)
@@ -165,7 +169,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
  desc "remove [NAME]", "Remove accessory container, image and data directory from host (use NAME=all to remove all accessories)"
  option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
  def remove(name)
-    with_lock do
+    mutating do
      if name == "all"
        MRSK.accessory_names.each { |accessory_name| remove(accessory_name) }
      else
@@ -183,7 +187,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base

  desc "remove_container [NAME]", "Remove accessory container from host", hide: true
  def remove_container(name)
-    with_lock do
+    mutating do
      with_accessory(name) do |accessory|
        on(accessory.hosts) do
          execute *MRSK.auditor.record("Remove #{name} accessory container"), verbosity: :debug
@@ -195,7 +199,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base

  desc "remove_image [NAME]", "Remove accessory image from host", hide: true
  def remove_image(name)
-    with_lock do
+    mutating do
      with_accessory(name) do |accessory|
        on(accessory.hosts) do
          execute *MRSK.auditor.record("Removed #{name} accessory image"), verbosity: :debug
@@ -207,7 +211,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base

  desc "remove_service_directory [NAME]", "Remove accessory directory used for uploaded files and data directories from host", hide: true
  def remove_service_directory(name)
-    with_lock do
+    mutating do
      with_accessory(name) do |accessory|
        on(accessory.hosts) do
          execute *accessory.remove_service_directory
--- a/lib/mrsk/cli/app.rb
+++ b/lib/mrsk/cli/app.rb
@@ -1,7 +1,7 @@
 class Mrsk::Cli::App < Mrsk::Cli::Base
  desc "boot", "Boot app on servers (or reboot app if already running)"
  def boot
-    with_lock do
+    mutating do
      hold_lock_on_error do
        say "Get most recent version available as an image...", :magenta unless options[:version]
        using_version(version_or_latest) do |version|
@@ -29,7 +29,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
              execute *auditor.record("Booted app version #{version}"), verbosity: :debug

              old_version = capture_with_info(*app.current_running_version, raise_on_non_zero_exit: false).strip
-              execute *app.start_or_run
+              execute *app.start_or_run(hostname: "#{host}-#{SecureRandom.hex(6)}")

              Mrsk::Utils::HealthcheckPoller.wait_for_healthy(pause_after_ready: true) { capture_with_info(*app.status(version: version)) }

@@ -43,7 +43,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base

  desc "start", "Start existing app container on servers"
  def start
-    with_lock do
+    mutating do
      on(MRSK.hosts) do |host|
        roles = MRSK.roles_on(host)

@@ -57,7 +57,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base

  desc "stop", "Stop app container on servers"
  def stop
-    with_lock do
+    mutating do
      on(MRSK.hosts) do |host|
        roles = MRSK.roles_on(host)

@@ -135,7 +135,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
  desc "stale_containers", "Detect app stale containers"
  option :stop, aliases: "-s", type: :boolean, default: false, desc: "Stop the stale containers found"
  def stale_containers
-    with_lock do
+    mutating do
      stop = options[:stop]

      cli = self
@@ -202,7 +202,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base

  desc "remove", "Remove app containers and images from servers"
  def remove
-    with_lock do
+    mutating do
      stop
      remove_containers
      remove_images
@@ -211,7 +211,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base

  desc "remove_container [VERSION]", "Remove app container with given version from servers", hide: true
  def remove_container(version)
-    with_lock do
+    mutating do
      on(MRSK.hosts) do |host|
        roles = MRSK.roles_on(host)

@@ -225,7 +225,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base

  desc "remove_containers", "Remove all app containers from servers", hide: true
  def remove_containers
-    with_lock do
+    mutating do
      on(MRSK.hosts) do |host|
        roles = MRSK.roles_on(host)

@@ -239,7 +239,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base

  desc "remove_images", "Remove all app images from servers", hide: true
  def remove_images
-    with_lock do
+    mutating do
      on(MRSK.hosts) do
        execute *MRSK.auditor.record("Removed all app images"), verbosity: :debug
        execute *MRSK.app.remove_images
--- a/lib/mrsk/cli/base.rb
+++ b/lib/mrsk/cli/base.rb
@@ -72,28 +72,28 @@ module Mrsk::Cli
        puts "  Finished all in #{sprintf("%.1f seconds", runtime)}"
      end

-      def with_lock
-        if MRSK.holding_lock?
+      def mutating
+        return yield if MRSK.holding_lock?
+
+        MRSK.config.ensure_env_available
+
+        run_hook "pre-connect"
+
+        acquire_lock
+
+        begin
          yield
-        else
-          run_hook "pre-connect"
-
-          acquire_lock
-
-          begin
-            yield
-          rescue
-            if MRSK.hold_lock_on_error?
-              error "  \e[31mDeploy lock was not released\e[0m"
-            else
-              release_lock
-            end
-
-            raise
+        rescue
+          if MRSK.hold_lock_on_error?
+            error "  \e[31mDeploy lock was not released\e[0m"
+          else
+            release_lock
          end

-          release_lock
+          raise
        end
+
+        release_lock
      end

      def acquire_lock
@@ -133,15 +133,39 @@ module Mrsk::Cli
        end
      end

-      def run_hook(hook, **details)
+      def run_hook(hook, **extra_details)
        if !options[:skip_hooks] && MRSK.hook.hook_exists?(hook)
+          details = { hosts: MRSK.hosts.join(","), command: command, subcommand: subcommand }
+
          say "Running the #{hook} hook...", :magenta
          run_locally do
-            MRSK.with_verbosity(:debug) { execute *MRSK.hook.run(hook, **details, hosts: MRSK.hosts.join(",")) }
+            MRSK.with_verbosity(:debug) { execute *MRSK.hook.run(hook, **details, **extra_details) }
          rescue SSHKit::Command::Failed
            raise HookError.new("Hook `#{hook}` failed")
          end
        end
      end
-  end
+
+      def command
+        @mrsk_command ||= begin
+          invocation_class, invocation_commands = *first_invocation
+          if invocation_class == Mrsk::Cli::Main
+            invocation_commands[0]
+          else
+            Mrsk::Cli::Main.subcommand_classes.find { |command, clazz| clazz == invocation_class }[0]
+          end
+        end
+      end
+
+      def subcommand
+        @mrsk_subcommand ||= begin
+          invocation_class, invocation_commands = *first_invocation
+          invocation_commands[0] if invocation_class != Mrsk::Cli::Main
+        end
+      end
+
+      def first_invocation
+        instance_variable_get("@_invocations").first
+      end
+    end
 end
--- a/lib/mrsk/cli/build.rb
+++ b/lib/mrsk/cli/build.rb
@@ -3,7 +3,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base

  desc "deliver", "Build app and push app image to registry then pull image on servers"
  def deliver
-    with_lock do
+    mutating do
      push
      pull
    end
@@ -11,7 +11,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base

  desc "push", "Build and push app image to registry"
  def push
-    with_lock do
+    mutating do
      cli = self

      verify_local_dependencies
@@ -37,7 +37,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base

  desc "pull", "Pull app image from registry onto servers"
  def pull
-    with_lock do
+    mutating do
      on(MRSK.hosts) do
        execute *MRSK.auditor.record("Pulled image with version #{MRSK.config.version}"), verbosity: :debug
        execute *MRSK.builder.clean, raise_on_non_zero_exit: false
@@ -48,7 +48,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base

  desc "create", "Create a build setup"
  def create
-    with_lock do
+    mutating do
      run_locally do
        begin
          debug "Using builder: #{MRSK.builder.name}"
@@ -67,7 +67,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base

  desc "remove", "Remove build setup"
  def remove
-    with_lock do
+    mutating do
      run_locally do
        debug "Using builder: #{MRSK.builder.name}"
        execute *MRSK.builder.remove
--- a/lib/mrsk/cli/lock.rb
+++ b/lib/mrsk/cli/lock.rb
@@ -7,7 +7,7 @@ class Mrsk::Cli::Lock < Mrsk::Cli::Base
  end

  desc "acquire", "Acquire the deploy lock"
-  option :message, aliases: "-m", type: :string, desc: "A lock mesasge", required: true
+  option :message, aliases: "-m", type: :string, desc: "A lock message", required: true
  def acquire
    message = options[:message]
    raise_if_locked do
--- a/lib/mrsk/cli/main.rb
+++ b/lib/mrsk/cli/main.rb
@@ -2,7 +2,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
  desc "setup", "Setup all accessories and deploy app to servers"
  def setup
    print_runtime do
-      with_lock do
+      mutating do
        invoke "mrsk:cli:server:bootstrap"
        invoke "mrsk:cli:accessory:boot", [ "all" ]
        deploy
@@ -14,7 +14,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
  option :skip_push, aliases: "-P", type: :boolean, default: false, desc: "Skip image build and push"
  def deploy
    runtime = print_runtime do
-      with_lock do
+      mutating do
        invoke_options = deploy_options

        say "Log into image registry...", :magenta
@@ -28,6 +28,8 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
          invoke "mrsk:cli:build:deliver", [], invoke_options
        end

+        run_hook "pre-deploy"
+
        say "Ensure Traefik is running...", :magenta
        invoke "mrsk:cli:traefik:boot", [], invoke_options

@@ -51,7 +53,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
  option :skip_push, aliases: "-P", type: :boolean, default: false, desc: "Skip image build and push"
  def redeploy
    runtime = print_runtime do
-      with_lock do
+      mutating do
        invoke_options = deploy_options

        if options[:skip_push]
@@ -62,6 +64,8 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
          invoke "mrsk:cli:build:deliver", [], invoke_options
        end

+        run_hook "pre-deploy"
+
        say "Ensure app can pass healthcheck...", :magenta
        invoke "mrsk:cli:healthcheck:perform", [], invoke_options

@@ -79,13 +83,15 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
  def rollback(version)
    rolled_back = false
    runtime = print_runtime do
-      with_lock do
+      mutating do
        invoke_options = deploy_options

        MRSK.config.version = version
        old_version = nil

        if container_available?(version)
+          run_hook "pre-deploy"
+
          invoke "mrsk:cli:app:boot", [], invoke_options.merge(version: version)
          rolled_back = true
        else
@@ -174,7 +180,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
  desc "remove", "Remove Traefik, app, accessories, and registry session from servers"
  option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
  def remove
-    with_lock do
+    mutating do
      if options[:confirmed] || ask("This will remove all containers and images. Are you sure?", limited_to: %w( y N ), default: "N") == "y"
        invoke "mrsk:cli:traefik:remove", [], options.without(:confirmed)
        invoke "mrsk:cli:app:remove", [], options.without(:confirmed)
--- a/lib/mrsk/cli/prune.rb
+++ b/lib/mrsk/cli/prune.rb
@@ -1,7 +1,7 @@
 class Mrsk::Cli::Prune < Mrsk::Cli::Base
  desc "all", "Prune unused images and stopped containers"
  def all
-    with_lock do
+    mutating do
      containers
      images
    end
@@ -9,17 +9,18 @@ class Mrsk::Cli::Prune < Mrsk::Cli::Base

  desc "images", "Prune dangling images"
  def images
-    with_lock do
+    mutating do
      on(MRSK.hosts) do
        execute *MRSK.auditor.record("Pruned images"), verbosity: :debug
-        execute *MRSK.prune.images
+        execute *MRSK.prune.dangling_images
+        execute *MRSK.prune.tagged_images
      end
    end
  end

  desc "containers", "Prune all stopped containers, except the last 5"
  def containers
-    with_lock do
+    mutating do
      on(MRSK.hosts) do
        execute *MRSK.auditor.record("Pruned containers"), verbosity: :debug
        execute *MRSK.prune.containers
--- a/lib/mrsk/cli/templates/sample_hooks/pre-deploy.sample
+++ b/lib/mrsk/cli/templates/sample_hooks/pre-deploy.sample
@@ -0,0 +1,109 @@
+#!/usr/bin/env ruby
+
+# A sample pre-deploy hook
+#
+# Checks the Github status of the build, waiting for a pending build to complete for up to 720 seconds.
+#
+# Fails unless the combined status is "success"
+#
+# These environment variables are available:
+# MRSK_RECORDED_AT
+# MRSK_PERFORMER
+# MRSK_VERSION
+# MRSK_HOSTS
+# MRSK_COMMAND
+# MRSK_SUBCOMMAND
+# MRSK_ROLE (if set)
+# MRSK_DESTINATION (if set)
+
+# Only check the build status for production deployments
+if ENV["MRSK_COMMAND"] == "rollback" || ENV["MRSK_DESTINATION"] != "production"
+  exit 0
+end
+
+require "bundler/inline"
+
+# true = install gems so this is fast on repeat invocations
+gemfile(true, quiet: true) do
+  source "https://rubygems.org"
+
+  gem "octokit"
+  gem "faraday-retry"
+end
+
+MAX_ATTEMPTS = 72
+ATTEMPTS_GAP = 10
+
+def exit_with_error(message)
+  $stderr.puts message
+  exit 1
+end
+
+class GithubStatusChecks
+  attr_reader :remote_url, :git_sha, :github_client, :combined_status
+
+  def initialize
+    @remote_url = `git config --get remote.origin.url`.strip.delete_prefix("https://github.com/")
+    @git_sha = `git rev-parse HEAD`.strip
+    @github_client = Octokit::Client.new(access_token: ENV["GITHUB_TOKEN"])
+    refresh!
+  end
+
+  def refresh!
+    @combined_status = github_client.combined_status(remote_url, git_sha)
+  end
+
+  def state
+    combined_status[:state]
+  end
+
+  def first_status_url
+    first_status = combined_status[:statuses].find { |status| status[:state] == state }
+    first_status && first_status[:target_url]
+  end
+
+  def complete_count
+    combined_status[:statuses].count { |status| status[:state] != "pending"}
+  end
+
+  def total_count
+    combined_status[:statuses].count
+  end
+
+  def current_status
+    if total_count > 0
+      "Completed #{complete_count}/#{total_count} checks, see #{first_status_url} ..."
+    else
+      "Build not started..."
+    end
+  end
+end
+
+
+$stdout.sync = true
+
+puts "Checking build status..."
+attempts = 0
+checks = GithubStatusChecks.new
+
+begin
+  loop do
+    case checks.state
+    when "success"
+      puts "Checks passed, see #{checks.first_status_url}"
+      exit 0
+    when "failure"
+      exit_with_error "Checks failed, see #{checks.first_status_url}"
+    when "pending"
+      attempts += 1
+    end
+
+    exit_with_error "Checks are still pending, gave up after #{MAX_ATTEMPTS * ATTEMPTS_GAP} seconds" if attempts == MAX_ATTEMPTS
+
+    puts checks.current_status
+    sleep(ATTEMPTS_GAP)
+    checks.refresh!
+  end
+rescue Octokit::NotFound
+  exit_with_error "Build status could not be found"
+end
--- a/lib/mrsk/cli/traefik.rb
+++ b/lib/mrsk/cli/traefik.rb
@@ -1,7 +1,7 @@
 class Mrsk::Cli::Traefik < Mrsk::Cli::Base
  desc "boot", "Boot Traefik on servers"
  def boot
-    with_lock do
+    mutating do
      on(MRSK.traefik_hosts) do
        execute *MRSK.registry.login
        execute *MRSK.traefik.run, raise_on_non_zero_exit: false
@@ -10,17 +10,22 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
  end

  desc "reboot", "Reboot Traefik on servers (stop container, remove container, start new container)"
+  option :rolling, type: :boolean, default: false, desc: "Reboot traefik on hosts in sequence, rather than in parallel"
  def reboot
-    with_lock do
-      stop
-      remove_container
-      boot
+    mutating do
+      on(MRSK.traefik_hosts, in: options[:rolling] ? :sequence : :parallel) do
+        execute *MRSK.auditor.record("Rebooted traefik"), verbosity: :debug
+        execute *MRSK.registry.login
+        execute *MRSK.traefik.stop, raise_on_non_zero_exit: false
+        execute *MRSK.traefik.remove_container
+        execute *MRSK.traefik.run, raise_on_non_zero_exit: false
+      end
    end
  end

  desc "start", "Start existing Traefik container on servers"
  def start
-    with_lock do
+    mutating do
      on(MRSK.traefik_hosts) do
        execute *MRSK.auditor.record("Started traefik"), verbosity: :debug
        execute *MRSK.traefik.start, raise_on_non_zero_exit: false
@@ -30,7 +35,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base

  desc "stop", "Stop existing Traefik container on servers"
  def stop
-    with_lock do
+    mutating do
      on(MRSK.traefik_hosts) do
        execute *MRSK.auditor.record("Stopped traefik"), verbosity: :debug
        execute *MRSK.traefik.stop, raise_on_non_zero_exit: false
@@ -40,7 +45,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base

  desc "restart", "Restart existing Traefik container on servers"
  def restart
-    with_lock do
+    mutating do
      stop
      start
    end
@@ -77,7 +82,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base

  desc "remove", "Remove Traefik container and image from servers"
  def remove
-    with_lock do
+    mutating do
      stop
      remove_container
      remove_image
@@ -86,7 +91,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base

  desc "remove_container", "Remove Traefik container from servers", hide: true
  def remove_container
-    with_lock do
+    mutating do
      on(MRSK.traefik_hosts) do
        execute *MRSK.auditor.record("Removed traefik container"), verbosity: :debug
        execute *MRSK.traefik.remove_container
@@ -96,7 +101,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base

  desc "remove_image", "Remove Traefik image from servers", hide: true
  def remove_image
-    with_lock do
+    mutating do
      on(MRSK.traefik_hosts) do
        execute *MRSK.auditor.record("Removed traefik image"), verbosity: :debug
        execute *MRSK.traefik.remove_image
--- a/lib/mrsk/commands/app.rb
+++ b/lib/mrsk/commands/app.rb
@@ -1,4 +1,6 @@
 class Mrsk::Commands::App < Mrsk::Commands::Base
+  ACTIVE_DOCKER_STATUSES = [ :running, :restarting ]
+
  attr_reader :role

  def initialize(config, role: nil)
@@ -6,17 +8,18 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
    @role = role
  end

-  def start_or_run
-    combine start, run, by: "||"
+  def start_or_run(hostname: nil)
+    combine start, run(hostname: hostname), by: "||"
  end

-  def run
+  def run(hostname: nil)
    role = config.role(self.role)

    docker :run,
      "--detach",
      "--restart unless-stopped",
      "--name", container_name,
+      *(["--hostname", hostname] if hostname),
      "-e", "MRSK_CONTAINER_NAME=\"#{container_name}\"",
      *role.env_args,
      *role.health_check_args,
@@ -92,7 +95,7 @@ class Mrsk::Commands::App < Mrsk::Commands::Base


  def current_running_container_id
-    docker :ps, "--quiet", *filter_args(status: :running), "--latest"
+    docker :ps, "--quiet", *filter_args(statuses: ACTIVE_DOCKER_STATUSES), "--latest"
  end

  def container_id_for_version(version, only_running: false)
@@ -100,12 +103,12 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
  end

  def current_running_version
-    list_versions("--latest", status: :running)
+    list_versions("--latest", statuses: ACTIVE_DOCKER_STATUSES)
  end

-  def list_versions(*docker_args, status: nil)
+  def list_versions(*docker_args, statuses: nil)
    pipe \
-      docker(:ps, *filter_args(status: status), *docker_args, "--format", '"{{.Names}}"'),
+      docker(:ps, *filter_args(statuses: statuses), *docker_args, "--format", '"{{.Names}}"'),
      %(grep -oE "\\-[^-]+$"), # Extract SHA from "service-role-dest-SHA"
      %(cut -c 2-)
  end
@@ -150,15 +153,17 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
      [ config.service, role, config.destination, version || config.version ].compact.join("-")
    end

-    def filter_args(status: nil)
-      argumentize "--filter", filters(status: status)
+    def filter_args(statuses: nil)
+      argumentize "--filter", filters(statuses: statuses)
    end

-    def filters(status: nil)
+    def filters(statuses: nil)
      [ "label=service=#{config.service}" ].tap do |filters|
        filters << "label=destination=#{config.destination}" if config.destination
        filters << "label=role=#{role}" if role
-        filters << "status=#{status}" if status
+        statuses&.each do |status|
+          filters << "status=#{status}"
+        end
      end
    end
 end
--- a/lib/mrsk/commands/base.rb
+++ b/lib/mrsk/commands/base.rb
@@ -13,7 +13,11 @@ module Mrsk::Commands

    def run_over_ssh(*command, host:)
      "ssh".tap do |cmd|
-        cmd << " -J #{config.ssh_proxy.jump_proxies}" if config.ssh_proxy
+        if config.ssh_proxy && config.ssh_proxy.is_a?(Net::SSH::Proxy::Jump)
+          cmd << " -J #{config.ssh_proxy.jump_proxies}"
+        elsif config.ssh_proxy && config.ssh_proxy.is_a?(Net::SSH::Proxy::Command)
+          cmd << " -o ProxyCommand='#{config.ssh_proxy.command_line_template}'"
+        end
        cmd << " -t #{config.ssh_user}@#{host} '#{command.join(" ")}'"
      end
    end
--- a/lib/mrsk/commands/builder.rb
+++ b/lib/mrsk/commands/builder.rb
@@ -7,11 +7,13 @@ class Mrsk::Commands::Builder < Mrsk::Commands::Base

  def target
    case
-    when config.builder && config.builder["multiarch"] == false
+    when !config.builder.multiarch? && !config.builder.cached?
      native
-    when config.builder && config.builder["local"] && config.builder["remote"]
+    when !config.builder.multiarch? && config.builder.cached?
+      native_cached
+    when config.builder.local? && config.builder.remote?
      multiarch_remote
-    when config.builder && config.builder["remote"]
+    when config.builder.remote?
      native_remote
    else
      multiarch
@@ -22,6 +24,10 @@ class Mrsk::Commands::Builder < Mrsk::Commands::Base
    @native ||= Mrsk::Commands::Builder::Native.new(config)
  end

+  def native_cached
+    @native ||= Mrsk::Commands::Builder::Native::Cached.new(config)
+  end
+
  def native_remote
    @native ||= Mrsk::Commands::Builder::Native::Remote.new(config)
  end
--- a/lib/mrsk/commands/builder/base.rb
+++ b/lib/mrsk/commands/builder/base.rb
@@ -3,6 +3,7 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
  class BuilderError < StandardError; end

  delegate :argumentize, to: Mrsk::Utils
+  delegate :args, :secrets, :dockerfile, :local_arch, :local_host, :remote_arch, :remote_host, :cache_from, :cache_to, to: :builder_config

  def clean
    docker :image, :rm, "--force", config.absolute_image
@@ -13,11 +14,11 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
  end

  def build_options
-    [ *build_tags, *build_labels, *build_args, *build_secrets, *build_dockerfile ]
+    [ *build_tags, *build_cache, *build_labels, *build_args, *build_secrets, *build_dockerfile ]
  end

  def build_context
-    context
+    config.builder.context
  end


@@ -26,6 +27,13 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
      [ "-t", config.absolute_image, "-t", config.latest_image ]
    end

+    def build_cache
+      if cache_to && cache_from
+        ["--cache-to", cache_to,
+          "--cache-from", cache_from]
+      end
+    end
+
    def build_labels
      argumentize "--label", { service: config.service }
    end
@@ -46,19 +54,7 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
      end
    end

-    def args
-      (config.builder && config.builder["args"]) || {}
-    end
-
-    def secrets
-      (config.builder && config.builder["secrets"]) || []
-    end
-
-    def dockerfile
-      (config.builder && config.builder["dockerfile"]) || "Dockerfile"
-    end
-
-    def context
-      (config.builder && config.builder["context"]) || "."
+    def builder_config
+      config.builder
    end
 end
--- a/lib/mrsk/commands/builder/multiarch/remote.rb
+++ b/lib/mrsk/commands/builder/multiarch/remote.rb
@@ -22,17 +22,17 @@ class Mrsk::Commands::Builder::Multiarch::Remote < Mrsk::Commands::Builder::Mult
    end

    def create_local_buildx
-      docker :buildx, :create, "--name", builder_name, builder_name_with_arch(local["arch"]), "--platform", "linux/#{local["arch"]}"
+      docker :buildx, :create, "--name", builder_name, builder_name_with_arch(local_arch), "--platform", "linux/#{local_arch}"
    end

    def append_remote_buildx
-      docker :buildx, :create, "--append", "--name", builder_name, builder_name_with_arch(remote["arch"]), "--platform", "linux/#{remote["arch"]}"
+      docker :buildx, :create, "--append", "--name", builder_name, builder_name_with_arch(remote_arch), "--platform", "linux/#{remote_arch}"
    end

    def create_contexts
      combine \
-        create_context(local["arch"], local["host"]),
-        create_context(remote["arch"], remote["host"])
+        create_context(local_arch, local_host),
+        create_context(remote_arch, remote_host)
    end

    def create_context(arch, host)
@@ -41,19 +41,11 @@ class Mrsk::Commands::Builder::Multiarch::Remote < Mrsk::Commands::Builder::Mult

    def remove_contexts
      combine \
-        remove_context(local["arch"]),
-        remove_context(remote["arch"])
+        remove_context(local_arch),
+        remove_context(remote_arch)
    end

    def remove_context(arch)
      docker :context, :rm, builder_name_with_arch(arch)
    end
-
-    def local
-      config.builder["local"]
-    end
-
-    def remote
-      config.builder["remote"]
-    end
 end
--- a/lib/mrsk/commands/builder/native.rb
+++ b/lib/mrsk/commands/builder/native.rb
@@ -1,10 +1,10 @@
 class Mrsk::Commands::Builder::Native < Mrsk::Commands::Builder::Base
  def create
-    # No-op on native
+    # No-op on native without cache
  end

  def remove
-    # No-op on native
+    # No-op on native without cache
  end

  def push
--- a/lib/mrsk/commands/builder/native/cached.rb
+++ b/lib/mrsk/commands/builder/native/cached.rb
@@ -0,0 +1,16 @@
+class Mrsk::Commands::Builder::Native::Cached < Mrsk::Commands::Builder::Native
+  def create
+    docker :buildx, :create, "--use", "--driver=docker-container"
+  end
+
+  def remove
+    docker :buildx, :rm, builder_name
+  end
+
+  def push
+    docker :buildx, :build,
+      "--push",
+      *build_options,
+      build_context
+  end
+end
--- a/lib/mrsk/commands/builder/native/remote.rb
+++ b/lib/mrsk/commands/builder/native/remote.rb
@@ -28,29 +28,21 @@ class Mrsk::Commands::Builder::Native::Remote < Mrsk::Commands::Builder::Native


  private
-    def arch
-      config.builder["remote"]["arch"]
-    end
-
-    def host
-      config.builder["remote"]["host"]
-    end
-
    def builder_name
      "mrsk-#{config.service}-native-remote"
    end

    def builder_name_with_arch
-      "#{builder_name}-#{arch}"
+      "#{builder_name}-#{remote_arch}"
    end

    def platform
-      "linux/#{arch}"
+      "linux/#{remote_arch}"
    end

    def create_context
      docker :context, :create,
-        builder_name_with_arch, "--description", "'#{builder_name} #{arch} native host'", "--docker", "'host=#{host}'"
+        builder_name_with_arch, "--description", "'#{builder_name} #{remote_arch} native host'", "--docker", "'host=#{remote_host}'"
    end

    def remove_context
--- a/lib/mrsk/commands/lock.rb
+++ b/lib/mrsk/commands/lock.rb
@@ -40,7 +40,7 @@ class Mrsk::Commands::Lock < Mrsk::Commands::Base
    end

    def lock_dir
-      :mrsk_lock
+      "mrsk_lock-#{config.service}"
    end

    def lock_details_file
--- a/lib/mrsk/commands/prune.rb
+++ b/lib/mrsk/commands/prune.rb
@@ -2,13 +2,20 @@ require "active_support/duration"
 require "active_support/core_ext/numeric/time"

 class Mrsk::Commands::Prune < Mrsk::Commands::Base
-  def images
+  def dangling_images
    docker :image, :prune, "--force", "--filter", "label=service=#{config.service}", "--filter", "dangling=true"
  end

+  def tagged_images
+    pipe \
+      docker(:image, :ls, *service_filter, "--format", "'{{.ID}} {{.Repository}}:{{.Tag}}'"),
+      "grep -v -w \"#{active_image_list}\"",
+      "while read image tag; do docker rmi $tag; done"
+  end
+
  def containers(keep_last: 5)
    pipe \
-      docker(:ps, "-q", "-a", "--filter", "label=service=#{config.service}", *stopped_containers_filters),
+      docker(:ps, "-q", "-a", *service_filter, *stopped_containers_filters),
      "tail -n +#{keep_last + 1}",
      "while read container_id; do docker rm $container_id; done"
  end
@@ -17,4 +24,15 @@ class Mrsk::Commands::Prune < Mrsk::Commands::Base
    def stopped_containers_filters
      [ "created", "exited", "dead" ].flat_map { |status| ["--filter", "status=#{status}"] }
    end
+
+    def active_image_list
+      # Pull the images that are used by any containers
+      # Append repo:latest - to avoid deleting the latest tag
+      # Append repo:<none> - to avoid deleting dangling images that are in use. Unused dangling images are deleted separately
+      "$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=#{config.service} | tr -d '\\n')#{config.latest_image}\\|#{config.repository}:<none>"
+    end
+
+    def service_filter
+      [ "--filter", "label=service=#{config.service}" ]
+    end
 end
--- a/lib/mrsk/commands/traefik.rb
+++ b/lib/mrsk/commands/traefik.rb
@@ -3,6 +3,9 @@ class Mrsk::Commands::Traefik < Mrsk::Commands::Base

  DEFAULT_IMAGE = "traefik:v2.9"
  CONTAINER_PORT = 80
+  DEFAULT_ARGS = {
+    'log.level' => 'DEBUG'
+  }

  def run
    docker :run, "--name traefik",
@@ -16,7 +19,6 @@ class Mrsk::Commands::Traefik < Mrsk::Commands::Base
      *docker_options_args,
      image,
      "--providers.docker",
-      "--log.level=DEBUG",
      *cmd_option_args
  end

@@ -86,9 +88,9 @@ class Mrsk::Commands::Traefik < Mrsk::Commands::Base

    def cmd_option_args
      if args = config.traefik["args"]
-        optionize args, with: "="
+        optionize DEFAULT_ARGS.merge(args), with: "="
      else
-        []
+        optionize DEFAULT_ARGS, with: "="
      end
    end

--- a/lib/mrsk/configuration.rb
+++ b/lib/mrsk/configuration.rb
@@ -6,7 +6,7 @@ require "erb"
 require "net/ssh/proxy/jump"

 class Mrsk::Configuration
-  delegate :service, :image, :servers, :env, :labels, :registry, :builder, :stop_wait_time, :hooks_path, to: :raw_config, allow_nil: true
+  delegate :service, :image, :servers, :env, :labels, :registry, :stop_wait_time, :hooks_path, to: :raw_config, allow_nil: true
  delegate :argumentize, :argumentize_env_with_secrets, :optionize, to: Mrsk::Utils

  attr_accessor :destination
@@ -153,7 +153,15 @@ class Mrsk::Configuration
  end

  def ssh_options
-    { user: ssh_user, proxy: ssh_proxy, auth_methods: [ "publickey" ] }.compact
+    { user: ssh_user, proxy: ssh_proxy, auth_methods: [ "publickey" ], logger: ssh_logger }.compact
+  end
+
+  def ssh_logger
+    @ssh_logger ||= ::Logger.new(STDERR).tap { |logger| logger.level = ssh_log_level }
+  end
+
+  def ssh_log_level
+    (raw_config.ssh && raw_config.ssh["log_level"]) || ::Logger::FATAL
  end


@@ -165,8 +173,12 @@ class Mrsk::Configuration
    raw_config.readiness_delay || 7
  end

+  def minimum_version
+    raw_config.minimum_version
+  end
+
  def valid?
-    ensure_required_keys_present && ensure_env_available
+    ensure_required_keys_present && ensure_valid_mrsk_version
  end


@@ -181,8 +193,9 @@ class Mrsk::Configuration
      service_with_version: service_with_version,
      env_args: env_args,
      volume_args: volume_args,
-      ssh_options: ssh_options,
-      builder: raw_config.builder,
+      ssh_options: ssh_options.except(:logger),
+      ssh_log_level: ssh_log_level,
+      builder: builder.to_h,
      accessories: raw_config.accessories,
      logging: logging_args,
      healthcheck: healthcheck
@@ -197,6 +210,18 @@ class Mrsk::Configuration
    raw_config.hooks_path || ".mrsk/hooks"
  end

+  def builder
+    Mrsk::Configuration::Builder.new(config: self)
+  end
+
+  # Will raise KeyError if any secret ENVs are missing
+  def ensure_env_available
+    env_args
+    roles.each(&:env_args)
+
+    true
+  end
+
  private
    # Will raise ArgumentError if any required config keys are missing
    def ensure_required_keys_present
@@ -221,14 +246,15 @@ class Mrsk::Configuration
      true
    end

-    # Will raise KeyError if any secret ENVs are missing
-    def ensure_env_available
-      env_args
-      roles.each(&:env_args)
+    def ensure_valid_mrsk_version
+      if minimum_version && Gem::Version.new(minimum_version) > Gem::Version.new(Mrsk::VERSION)
+        raise ArgumentError, "Current version is #{Mrsk::VERSION}, minimum required is #{minimum_version}"
+      end

      true
    end

+
    def role_names
      raw_config.servers.is_a?(Array) ? [ "web" ] : raw_config.servers.keys.sort
    end
--- a/lib/mrsk/configuration/builder.rb
+++ b/lib/mrsk/configuration/builder.rb
@@ -0,0 +1,114 @@
+class Mrsk::Configuration::Builder
+  def initialize(config:)
+    @options = config.raw_config.builder || {}
+    @image = config.image
+    @server = config.registry["server"]
+
+    valid?
+  end
+
+  def to_h
+    @options
+  end
+
+  def multiarch?
+    @options["multiarch"] != false
+  end
+
+  def local?
+    !!@options["local"]
+  end
+
+  def remote?
+    !!@options["remote"]
+  end
+
+  def cached?
+    !!@options["cache"]
+  end
+
+  def args
+    @options["args"] || {}
+  end
+
+  def secrets
+    @options["secrets"] || []
+  end
+
+  def dockerfile
+    @options["dockerfile"] || "Dockerfile"
+  end
+
+  def context
+    @options["context"] || "."
+  end
+
+  def local_arch
+    @options["local"]["arch"] if local?
+  end
+
+  def local_host
+    @options["local"]["host"] if local?
+  end
+
+  def remote_arch
+    @options["remote"]["arch"] if remote?
+  end
+
+  def remote_host
+    @options["remote"]["host"] if remote?
+  end
+
+  def cache_from
+    if cached?
+      case @options["cache"]["type"]
+      when "gha"
+        cache_from_config_for_gha
+      when "registry"
+        cache_from_config_for_registry
+      end
+    end
+  end
+
+  def cache_to
+    if cached?
+      case @options["cache"]["type"]
+      when "gha"
+        cache_to_config_for_gha
+      when "registry"
+        cache_to_config_for_registry
+      end
+    end
+  end
+
+  private
+    def valid?
+      if @options["local"] && !@options["remote"]
+        raise ArgumentError, "You must specify both local and remote builder config for remote multiarch builds"
+      end
+
+      if @options["cache"] && @options["cache"]["type"]
+        raise ArgumentError, "Invalid cache type: #{@options["cache"]["type"]}" unless ["gha", "registry"].include?(@options["cache"]["type"])
+      end
+    end
+
+    def cache_image
+      @options["cache"]&.fetch("image", nil) || "#{@image}-build-cache"
+    end
+
+    def cache_from_config_for_gha
+      "type=gha"
+    end
+
+    def cache_from_config_for_registry
+      [ "type=registry", "ref=#{@server}/#{cache_image}" ].compact.join(",")
+    end
+
+    def cache_to_config_for_gha
+      [ "type=gha", @options["cache"]&.fetch("options", nil)].compact.join(",")
+    end
+
+    def cache_to_config_for_registry
+      [ "type=registry", @options["cache"]&.fetch("options", nil), "ref=#{@server}/#{cache_image}" ].compact.join(",")
+    end
+end
--- a/lib/mrsk/version.rb
+++ b/lib/mrsk/version.rb
@@ -1,3 +1,3 @@
 module Mrsk
-  VERSION = "0.13.0"
+  VERSION = "0.15.0"
 end
--- a/test/cli/accessory_test.rb
+++ b/test/cli/accessory_test.rb
@@ -40,9 +40,10 @@ class CliAccessoryTest < CliTestCase
  end

  test "reboot" do
+    Mrsk::Commands::Registry.any_instance.expects(:login)
    Mrsk::Cli::Accessory.any_instance.expects(:stop).with("mysql")
    Mrsk::Cli::Accessory.any_instance.expects(:remove_container).with("mysql")
-    Mrsk::Cli::Accessory.any_instance.expects(:boot).with("mysql")
+    Mrsk::Cli::Accessory.any_instance.expects(:boot).with("mysql", login: false)

    run_command("reboot", "mysql")
  end
--- a/test/cli/app_test.rb
+++ b/test/cli/app_test.rb
@@ -5,7 +5,7 @@ class CliAppTest < CliTestCase
    stub_running
    run_command("boot").tap do |output|
      assert_match "docker tag dhh/app:latest dhh/app:latest", output
-      assert_match "docker run --detach --restart unless-stopped", output
+      assert_match /docker run --detach --restart unless-stopped --name app-web-latest --hostname 1.1.1.1-[0-9a-f]{12} /, output
      assert_match "docker container ls --all --filter name=^app-web-123$ --quiet | xargs docker stop", output
    end
  end
@@ -22,13 +22,13 @@ class CliAppTest < CliTestCase
      .returns("running") # health check

    SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
-      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
+      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--filter", "status=restarting", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
      .returns("123") # old version

    run_command("boot").tap do |output|
      assert_match /Renaming container .* to .* as already deployed on 1.1.1.1/, output # Rename
      assert_match /docker rename app-web-latest app-web-latest_replaced_[0-9a-f]{16}/, output
-      assert_match "docker run --detach --restart unless-stopped", output
+      assert_match /docker run --detach --restart unless-stopped --name app-web-latest --hostname 1.1.1.1-[0-9a-f]{12} /, output
      assert_match "docker container ls --all --filter name=^app-web-123$ --quiet | xargs docker stop", output
    end
  ensure
@@ -65,7 +65,7 @@ class CliAppTest < CliTestCase

  test "stop" do
    run_command("stop").tap do |output|
-      assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop", output
+      assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop", output
    end
  end

@@ -98,7 +98,7 @@ class CliAppTest < CliTestCase

  test "remove" do
    run_command("remove").tap do |output|
-      assert_match /#{Regexp.escape("docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop")}/, output
+      assert_match /#{Regexp.escape("docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop")}/, output
      assert_match /#{Regexp.escape("docker container prune --force --filter label=service=app")}/, output
      assert_match /#{Regexp.escape("docker image prune --all --force --filter label=service=app")}/, output
    end
@@ -130,7 +130,7 @@ class CliAppTest < CliTestCase

  test "exec with reuse" do
    run_command("exec", "--reuse", "ruby -v").tap do |output|
-      assert_match "docker ps --filter label=service=app --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output # Get current version
+      assert_match "docker ps --filter label=service=app --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output # Get current version
      assert_match "docker exec app-web-999 ruby -v", output
    end
  end
@@ -149,28 +149,28 @@ class CliAppTest < CliTestCase

  test "logs" do
    SSHKit::Backend::Abstract.any_instance.stubs(:exec)
-      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest| xargs docker logs --timestamps --tail 10 2>&1'")
+      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest| xargs docker logs --timestamps --tail 10 2>&1'")

-    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --tail 100 2>&1", run_command("logs")
+    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --tail 100 2>&1", run_command("logs")
  end

  test "logs with follow" do
    SSHKit::Backend::Abstract.any_instance.stubs(:exec)
-      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1'")
+      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1'")

-    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1", run_command("logs", "--follow")
+    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1", run_command("logs", "--follow")
  end

  test "version" do
    run_command("version").tap do |output|
-      assert_match "docker ps --filter label=service=app --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
+      assert_match "docker ps --filter label=service=app --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
    end
  end


  test "version through main" do
    stdouted { Mrsk::Cli::Main.start(["app", "version", "-c", "test/fixtures/deploy_with_accessories.yml", "--hosts", "1.1.1.1"]) }.tap do |output|
-      assert_match "docker ps --filter label=service=app --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
+      assert_match "docker ps --filter label=service=app --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
    end
  end

--- a/test/cli/build_test.rb
+++ b/test/cli/build_test.rb
@@ -9,7 +9,11 @@ class CliBuildTest < CliTestCase
  end

  test "push" do
+    Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
+    hook_variables = { version: 999, service_version: "app@999", hosts: "1.1.1.1,1.1.1.2,1.1.1.3,1.1.1.4", command: "build", subcommand: "push" }
+
    run_command("push").tap do |output|
+      assert_hook_ran "pre-build", output, **hook_variables
      assert_match /docker --version && docker buildx version/, output
      assert_match /docker buildx build --push --platform linux\/amd64,linux\/arm64 --builder mrsk-app-multiarch -t dhh\/app:999 -t dhh\/app:latest --label service="app" --file Dockerfile \. as .*@localhost/, output
    end
--- a/test/cli/cli_test_case.rb
+++ b/test/cli/cli_test_case.rb
@@ -27,19 +27,30 @@ class CliTestCase < ActiveSupport::TestCase
        .raises(SSHKit::Command::Failed.new("failed"))
    end

-    def ensure_hook_runs(hook)
-      Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
-      SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-        .with { |*args| args != [".mrsk/hooks/#{hook}"] }
-      SSHKit::Backend::Abstract.any_instance.expects(:execute)
-        .with { |*args| args.first == ".mrsk/hooks/#{hook}" }
-        .once
-    end
-
    def stub_locking
      SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-        .with { |arg1, arg2| arg1 == :mkdir && arg2 == :mrsk_lock }
+        .with { |arg1, arg2| arg1 == :mkdir && arg2 == "mrsk_lock-app" }
      SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-        .with { |arg1, arg2| arg1 == :rm && arg2 == "mrsk_lock/details" }
+        .with { |arg1, arg2| arg1 == :rm && arg2 == "mrsk_lock-app/details" }
+    end
+
+    def assert_hook_ran(hook, output, version:, service_version:, hosts:, command:, subcommand: nil, runtime: nil)
+      performer = `whoami`.strip
+
+      assert_match "Running the #{hook} hook...\n", output
+
+      expected = %r{Running\s/usr/bin/env\s\.mrsk/hooks/#{hook}\sas\s#{performer}@localhost\n\s
+        DEBUG\s\[[0-9a-f]*\]\sCommand:\s\(\sexport\s
+        MRSK_RECORDED_AT=\"\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\dZ\"\s
+        MRSK_PERFORMER=\"#{performer}\"\s
+        MRSK_VERSION=\"#{version}\"\s
+        MRSK_SERVICE_VERSION=\"#{service_version}\"\s
+        MRSK_HOSTS=\"#{hosts}\"\s
+        MRSK_COMMAND=\"#{command}\"\s
+        #{"MRSK_SUBCOMMAND=\\\"#{subcommand}\\\"\\s" if subcommand}
+        #{"MRSK_RUNTIME=\\\"#{runtime}\\\"\\s" if runtime}
+        ;\s/usr/bin/env\s\.mrsk/hooks/#{hook} }x
+
+      assert_match expected, output
    end
 end
--- a/test/cli/main_test.rb
+++ b/test/cli/main_test.rb
@@ -21,16 +21,18 @@ class CliMainTest < CliTestCase
    Mrsk::Cli::Main.any_instance.expects(:invoke).with("mrsk:cli:prune:all", [], invoke_options)

    Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
+    hook_variables = { version: 999, service_version: "app@999", hosts: "1.1.1.1,1.1.1.2", command: "deploy" }

    run_command("deploy").tap do |output|
-      assert_match /Running the pre-connect hook.../, output
+      assert_hook_ran "pre-connect", output, **hook_variables
      assert_match /Log into image registry/, output
      assert_match /Build and push app image/, output
+      assert_hook_ran "pre-deploy", output, **hook_variables
      assert_match /Ensure Traefik is running/, output
      assert_match /Ensure app can pass healthcheck/, output
      assert_match /Detect stale containers/, output
      assert_match /Prune old containers and images/, output
-      assert_match /Running the post-deploy hook.../, output
+      assert_hook_ran "post-deploy", output, **hook_variables, runtime: 0
    end
  end

@@ -61,11 +63,11 @@ class CliMainTest < CliTestCase
    Thread.report_on_exception = false

    SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-      .with { |*arg| arg[0..1] == [:mkdir, :mrsk_lock] }
-      .raises(RuntimeError, "mkdir: cannot create directory ‘mrsk_lock’: File exists")
+      .with { |*arg| arg[0..1] == [:mkdir, 'mrsk_lock-app'] }
+      .raises(RuntimeError, "mkdir: cannot create directory ‘mrsk_lock-app’: File exists")

    SSHKit::Backend::Abstract.any_instance.expects(:capture_with_debug)
-      .with(:stat, :mrsk_lock, ">", "/dev/null", "&&", :cat, "mrsk_lock/details", "|", :base64, "-d")
+      .with(:stat, 'mrsk_lock-app', ">", "/dev/null", "&&", :cat, "mrsk_lock-app/details", "|", :base64, "-d")

    assert_raises(Mrsk::Cli::LockError) do
      run_command("deploy")
@@ -76,7 +78,7 @@ class CliMainTest < CliTestCase
    Thread.report_on_exception = false

    SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-      .with { |*arg| arg[0..1] == [:mkdir, :mrsk_lock] }
+      .with { |*arg| arg[0..1] == [:mkdir, 'mrsk_lock-app'] }
      .raises(SocketError, "getaddrinfo: nodename nor servname provided, or not known")

    assert_raises(SSHKit::Runner::ExecuteError) do
@@ -114,6 +116,12 @@ class CliMainTest < CliTestCase
    end
  end

+  test "deploy with missing secrets" do
+    assert_raises(KeyError) do
+      run_command("deploy", config_file: "deploy_with_secrets")
+    end
+  end
+
  test "redeploy" do
    invoke_options = { "config_file" => "test/fixtures/deploy_simple.yml", "version" => "999", "skip_hooks" => false }

@@ -124,10 +132,15 @@ class CliMainTest < CliTestCase

    Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)

+    hook_variables = { version: 999, service_version: "app@999", hosts: "1.1.1.1,1.1.1.2", command: "redeploy" }
+
    run_command("redeploy").tap do |output|
+      assert_hook_ran "pre-connect", output, **hook_variables
      assert_match /Build and push app image/, output
+      assert_hook_ran "pre-deploy", output, **hook_variables
+      assert_match /Running the pre-deploy hook.../, output
      assert_match /Ensure app can pass healthcheck/, output
-      assert_match /Running the post-deploy hook.../, output
+      assert_hook_ran "post-deploy", output, **hook_variables, runtime: "0"
    end
  end

@@ -165,7 +178,7 @@ class CliMainTest < CliTestCase
        .with(:docker, :container, :ls, "--all", "--filter", "name=^app-#{role}-123$", "--quiet")
        .returns("version-to-rollback\n").at_least_once
      SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
-        .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=#{role}", "--filter", "status=running", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
+        .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=#{role}", "--filter", "status=running", "--filter", "status=restarting", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
        .returns("version-to-rollback\n").at_least_once
      SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
        .with(:docker, :container, :ls, "--all", "--filter", "name=^app-#{role}-123$", "--quiet", "|", :xargs, :docker, :inspect, "--format", "'{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'")
@@ -173,13 +186,15 @@ class CliMainTest < CliTestCase
    end

    Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
+    hook_variables = { version: 123, service_version: "app@123", hosts: "1.1.1.1,1.1.1.2,1.1.1.3,1.1.1.4", command: "rollback" }

    run_command("rollback", "123", config_file: "deploy_with_accessories").tap do |output|
      assert_match "Start container with version 123", output
+      assert_hook_ran "pre-deploy", output, **hook_variables
      assert_match "docker tag dhh/app:123 dhh/app:latest", output
      assert_match "docker start app-web-123", output
      assert_match "docker container ls --all --filter name=^app-web-version-to-rollback$ --quiet | xargs docker stop", output, "Should stop the container that was previously running"
-      assert_match "Running the post-deploy hook...", output
+      assert_hook_ran "post-deploy", output, **hook_variables, runtime: "0"
    end
  end

@@ -192,7 +207,7 @@ class CliMainTest < CliTestCase
      .with(:docker, :container, :ls, "--filter", "name=^app-web-123$", "--quiet", raise_on_non_zero_exit: false)
      .returns("").at_least_once
    SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
-      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
+      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--filter", "status=restarting", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
      .returns("").at_least_once
    SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
      .with(:docker, :container, :ls, "--all", "--filter", "name=^app-web-123$", "--quiet", "|", :xargs, :docker, :inspect, "--format", "'{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'")
--- a/test/cli/prune_test.rb
+++ b/test/cli/prune_test.rb
@@ -10,7 +10,8 @@ class CliPruneTest < CliTestCase

  test "images" do
    run_command("images").tap do |output|
-      assert_match /docker image prune --force --filter label=service=app --filter dangling=true on 1.1.1.\d/, output
+      assert_match "docker image prune --force --filter label=service=app --filter dangling=true on 1.1.1.", output
+      assert_match "docker image ls --filter label=service=app --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w \"$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=app | tr -d '\\n')dhh/app:latest\\|dhh/app:<none>\" | while read image tag; do docker rmi $tag; done on 1.1.1.", output
    end
  end

--- a/test/cli/server_test.rb
+++ b/test/cli/server_test.rb
@@ -11,7 +11,7 @@ class CliServerTest < CliTestCase
    SSHKit::Backend::Abstract.any_instance.expects(:execute).with(:docker, "-v", raise_on_non_zero_exit: false).returns(false).at_least_once
    SSHKit::Backend::Abstract.any_instance.expects(:execute).with('[ "${EUID:-$(id -u)}" -eq 0 ]', raise_on_non_zero_exit: false).returns(false).at_least_once

-    assert_raise RuntimeError, "Docker is not installed on 1.1.1.1, 1.1.1.3, 1.1.1.4, 1.1.1.2 and can't be automatically intalled without having root access and the `curl` command available. Install Docker manually: https://docs.docker.com/engine/install/" do
+    assert_raise RuntimeError, "Docker is not installed on 1.1.1.1, 1.1.1.3, 1.1.1.4, 1.1.1.2 and can't be automatically installed without having root access and the `curl` command available. Install Docker manually: https://docs.docker.com/engine/install/" do
      run_command("bootstrap")
    end
  end
--- a/test/cli/traefik_test.rb
+++ b/test/cli/traefik_test.rb
@@ -4,16 +4,24 @@ class CliTraefikTest < CliTestCase
  test "boot" do
    run_command("boot").tap do |output|
      assert_match "docker login", output
-      assert_match "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=DEBUG", output
+      assert_match "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=\"DEBUG\"", output
    end
  end

  test "reboot" do
-    Mrsk::Cli::Traefik.any_instance.expects(:stop)
-    Mrsk::Cli::Traefik.any_instance.expects(:remove_container)
-    Mrsk::Cli::Traefik.any_instance.expects(:boot)
+    Mrsk::Commands::Registry.any_instance.expects(:login).twice

-    run_command("reboot")
+    run_command("reboot").tap do |output|
+      assert_match "docker container stop traefik", output
+      assert_match "docker container prune --force --filter label=org.opencontainers.image.title=Traefik", output
+      assert_match "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=\"DEBUG\"", output
+    end
+  end
+
+  test "reboot --rolling" do
+    run_command("reboot", "--rolling").tap do |output|
+      assert_match "Running docker container prune --force --filter label=org.opencontainers.image.title=Traefik on 1.1.1.1", output.lines[3]
+    end
  end

  test "start" do
--- a/test/commander_test.rb
+++ b/test/commander_test.rb
@@ -65,7 +65,7 @@ class CommanderTest < ActiveSupport::TestCase
  end

  test "percentage-based group strategy" do
-    configure_with(:deploy_with_precentage_boot_strategy)
+    configure_with(:deploy_with_percentage_boot_strategy)

    assert_equal({ in: :groups, limit: 1, wait: 2 }, @mrsk.boot_strategy)
  end
--- a/test/commands/app_test.rb
+++ b/test/commands/app_test.rb
@@ -17,6 +17,12 @@ class CommandsAppTest < ActiveSupport::TestCase
      new_command.run.join(" ")
  end

+  test "run with hostname" do
+    assert_equal \
+      "docker run --detach --restart unless-stopped --name app-web-999 --hostname myhost -e MRSK_CONTAINER_NAME=\"app-web-999\" -e RAILS_MASTER_KEY=\"456\" --health-cmd \"curl -f http://localhost:3000/up || exit 1\" --health-interval \"1s\" --log-opt max-size=\"10m\" --label service=\"app\" --label role=\"web\" --label traefik.http.services.app-web.loadbalancer.server.scheme=\"http\" --label traefik.http.routers.app-web.rule=\"PathPrefix(\\`/\\`)\" --label traefik.http.middlewares.app-web-retry.retry.attempts=\"5\" --label traefik.http.middlewares.app-web-retry.retry.initialinterval=\"500ms\" --label traefik.http.routers.app-web.middlewares=\"app-web-retry@docker\" dhh/app:999",
+      new_command.run(hostname: "myhost").join(" ")
+  end
+
  test "run with volumes" do
    @config[:volumes] = ["/local/path:/container/path" ]

@@ -77,16 +83,28 @@ class CommandsAppTest < ActiveSupport::TestCase
      new_command.start.join(" ")
  end

+  test "start_or_run" do
+    assert_equal \
+      "docker start app-web-999 || docker run --detach --restart unless-stopped --name app-web-999 -e MRSK_CONTAINER_NAME=\"app-web-999\" -e RAILS_MASTER_KEY=\"456\" --health-cmd \"curl -f http://localhost:3000/up || exit 1\" --health-interval \"1s\" --log-opt max-size=\"10m\" --label service=\"app\" --label role=\"web\" --label traefik.http.services.app-web.loadbalancer.server.scheme=\"http\" --label traefik.http.routers.app-web.rule=\"PathPrefix(\\`/\\`)\" --label traefik.http.middlewares.app-web-retry.retry.attempts=\"5\" --label traefik.http.middlewares.app-web-retry.retry.initialinterval=\"500ms\" --label traefik.http.routers.app-web.middlewares=\"app-web-retry@docker\" dhh/app:999",
+      new_command.start_or_run.join(" ")
+  end
+
+  test "start_or_run with hostname" do
+    assert_equal \
+      "docker start app-web-999 || docker run --detach --restart unless-stopped --name app-web-999 --hostname myhost -e MRSK_CONTAINER_NAME=\"app-web-999\" -e RAILS_MASTER_KEY=\"456\" --health-cmd \"curl -f http://localhost:3000/up || exit 1\" --health-interval \"1s\" --log-opt max-size=\"10m\" --label service=\"app\" --label role=\"web\" --label traefik.http.services.app-web.loadbalancer.server.scheme=\"http\" --label traefik.http.routers.app-web.rule=\"PathPrefix(\\`/\\`)\" --label traefik.http.middlewares.app-web-retry.retry.attempts=\"5\" --label traefik.http.middlewares.app-web-retry.retry.initialinterval=\"500ms\" --label traefik.http.routers.app-web.middlewares=\"app-web-retry@docker\" dhh/app:999",
+      new_command.start_or_run(hostname: "myhost").join(" ")
+  end
+
  test "stop" do
    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop",
      new_command.stop.join(" ")
  end

  test "stop with custom stop wait time" do
    @config[:stop_wait_time] = 30
    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop -t 30",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop -t 30",
      new_command.stop.join(" ")
  end

@@ -112,37 +130,37 @@ class CommandsAppTest < ActiveSupport::TestCase

  test "logs" do
    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs 2>&1",
      new_command.logs.join(" ")

    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --since 5m 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --since 5m 2>&1",
      new_command.logs(since: "5m").join(" ")

    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --tail 100 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --tail 100 2>&1",
      new_command.logs(lines: "100").join(" ")

    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --since 5m --tail 100 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --since 5m --tail 100 2>&1",
      new_command.logs(since: "5m", lines: "100").join(" ")

    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs 2>&1 | grep 'my-id'",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs 2>&1 | grep 'my-id'",
      new_command.logs(grep: "my-id").join(" ")

    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --since 5m 2>&1 | grep 'my-id'",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --since 5m 2>&1 | grep 'my-id'",
      new_command.logs(since: "5m", grep: "my-id").join(" ")
  end

  test "follow logs" do
    assert_match \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1",
      new_command.follow_logs(host: "app-1")

    assert_match \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1 | grep \"Completed\"",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1 | grep \"Completed\"",
      new_command.follow_logs(host: "app-1", grep: "Completed")
  end

@@ -193,17 +211,21 @@ class CommandsAppTest < ActiveSupport::TestCase
    assert_equal "ssh -J root@2.2.2.2 -t app@1.1.1.1 'ls'", new_command.run_over_ssh("ls", host: "1.1.1.1")
  end

+  test "run over ssh with proxy_command" do
+    @config[:ssh] = { "proxy_command" => "ssh -W %h:%p user@proxy-server" }
+    assert_equal "ssh -o ProxyCommand='ssh -W %h:%p user@proxy-server' -t root@1.1.1.1 'ls'", new_command.run_over_ssh("ls", host: "1.1.1.1")
+  end

  test "current_running_container_id" do
    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest",
      new_command.current_running_container_id.join(" ")
  end

  test "current_running_container_id with destination" do
    @destination = "staging"
    assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=destination=staging --filter label=role=web --filter status=running --latest",
+      "docker ps --quiet --filter label=service=app --filter label=destination=staging --filter label=role=web --filter status=running --filter status=restarting --latest",
      new_command.current_running_container_id.join(" ")
  end

@@ -215,7 +237,7 @@ class CommandsAppTest < ActiveSupport::TestCase

  test "current_running_version" do
    assert_equal \
-      "docker ps --filter label=service=app --filter label=role=web --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
+      "docker ps --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
      new_command.current_running_version.join(" ")
  end

@@ -225,8 +247,8 @@ class CommandsAppTest < ActiveSupport::TestCase
      new_command.list_versions.join(" ")

    assert_equal \
-      "docker ps --filter label=service=app --filter label=role=web --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
-      new_command.list_versions("--latest", status: :running).join(" ")
+      "docker ps --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
+      new_command.list_versions("--latest", statuses: [ :running, :restarting ]).join(" ")
  end

  test "list_containers" do
--- a/test/commands/builder_test.rb
+++ b/test/commands/builder_test.rb
@@ -6,10 +6,10 @@ class CommandsBuilderTest < ActiveSupport::TestCase
  end

  test "target multiarch by default" do
-    builder = new_builder_command
+    builder = new_builder_command(builder: { "cache" => { "type" => "gha" }})
    assert_equal "multiarch", builder.name
    assert_equal \
-      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch -t dhh/app:123 -t dhh/app:latest --label service=\"app\" --file Dockerfile .",
+      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch -t dhh/app:123 -t dhh/app:latest --cache-to type=gha --cache-from type=gha --label service=\"app\" --file Dockerfile .",
      builder.push.join(" ")
  end

@@ -21,19 +21,27 @@ class CommandsBuilderTest < ActiveSupport::TestCase
      builder.push.join(" ")
  end

+  test "target native cached when multiarch is off and cache is set" do
+    builder = new_builder_command(builder: { "multiarch" => false, "cache" => { "type" => "gha" }})
+    assert_equal "native/cached", builder.name
+    assert_equal \
+      "docker buildx build --push -t dhh/app:123 -t dhh/app:latest --cache-to type=gha --cache-from type=gha --label service=\"app\" --file Dockerfile .",
+      builder.push.join(" ")
+  end
+
  test "target multiarch remote when local and remote is set" do
-    builder = new_builder_command(builder: { "local" => { }, "remote" => { } })
+    builder = new_builder_command(builder: { "local" => { }, "remote" => { }, "cache" => { "type" => "gha" } })
    assert_equal "multiarch/remote", builder.name
    assert_equal \
-      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch-remote -t dhh/app:123 -t dhh/app:latest --label service=\"app\" --file Dockerfile .",
+      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch-remote -t dhh/app:123 -t dhh/app:latest --cache-to type=gha --cache-from type=gha --label service=\"app\" --file Dockerfile .",
      builder.push.join(" ")
  end

  test "target native remote when only remote is set" do
-    builder = new_builder_command(builder: { "remote" => { "arch" => "amd64" } })
+    builder = new_builder_command(builder: { "remote" => { "arch" => "amd64" }, "cache" => { "type" => "gha" } })
    assert_equal "native/remote", builder.name
    assert_equal \
-      "docker buildx build --push --platform linux/amd64 --builder mrsk-app-native-remote -t dhh/app:123 -t dhh/app:latest --label service=\"app\" --file Dockerfile .",
+      "docker buildx build --push --platform linux/amd64 --builder mrsk-app-native-remote -t dhh/app:123 -t dhh/app:latest --cache-to type=gha --cache-from type=gha --label service=\"app\" --file Dockerfile .",
      builder.push.join(" ")
  end

@@ -88,7 +96,7 @@ class CommandsBuilderTest < ActiveSupport::TestCase
      builder.push.join(" ")
  end

-  test "native push with with build secrets" do
+  test "native push with build secrets" do
    builder = new_builder_command(builder: { "multiarch" => false, "secrets" => [ "a", "b" ] })
    assert_equal \
      "docker build -t dhh/app:123 -t dhh/app:latest --label service=\"app\" --secret id=\"a\" --secret id=\"b\" --file Dockerfile . && docker push dhh/app:123 && docker push dhh/app:latest",
--- a/test/commands/lock_test.rb
+++ b/test/commands/lock_test.rb
@@ -10,19 +10,19 @@ class CommandsLockTest < ActiveSupport::TestCase

  test "status" do
    assert_equal \
-      "stat mrsk_lock > /dev/null && cat mrsk_lock/details | base64 -d",
+      "stat mrsk_lock-app > /dev/null && cat mrsk_lock-app/details | base64 -d",
      new_command.status.join(" ")
  end

  test "acquire" do
    assert_match \
-      /mkdir mrsk_lock && echo ".*" > mrsk_lock\/details/m,
+      /mkdir mrsk_lock-app && echo ".*" > mrsk_lock-app\/details/m,
      new_command.acquire("Hello", "123").join(" ")
  end

  test "release" do
    assert_match \
-      "rm mrsk_lock/details && rm -r mrsk_lock",
+      "rm mrsk_lock-app/details && rm -r mrsk_lock-app",
      new_command.release.join(" ")
  end

--- a/test/commands/prune_test.rb
+++ b/test/commands/prune_test.rb
@@ -8,10 +8,16 @@ class CommandsPruneTest < ActiveSupport::TestCase
    }
  end

-  test "images" do
+  test "dangling images" do
    assert_equal \
      "docker image prune --force --filter label=service=app --filter dangling=true",
-      new_command.images.join(" ")
+      new_command.dangling_images.join(" ")
+  end
+
+  test "tagged images" do
+    assert_equal \
+      "docker image ls --filter label=service=app --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w \"$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=app | tr -d '\\n')dhh/app:latest\\|dhh/app:<none>\" | while read image tag; do docker rmi $tag; done",
+      new_command.tagged_images.join(" ")
  end

  test "containers" do
--- a/test/commands/traefik_test.rb
+++ b/test/commands/traefik_test.rb
@@ -18,67 +18,67 @@ class CommandsTraefikTest < ActiveSupport::TestCase

  test "run" do
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")

    @config[:traefik]["host_port"] = "8080"
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 8080:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 8080:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")
  end

  test "run with ports configured" do
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")

    @config[:traefik]["options"] = {"publish" => %w[9000:9000 9001:9001]}
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --publish \"9000:9000\" --publish \"9001:9001\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --publish \"9000:9000\" --publish \"9001:9001\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")
  end

  test "run with volumes configured" do
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")

    @config[:traefik]["options"] = {"volume" => %w[./letsencrypt/acme.json:/letsencrypt/acme.json] }
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")
  end

  test "run with several options configured" do
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")

    @config[:traefik]["options"] = {"volume" => %w[./letsencrypt/acme.json:/letsencrypt/acme.json], "publish" => %w[8080:8080], "memory" => "512m"}
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" --publish \"8080:8080\" --memory \"512m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" --publish \"8080:8080\" --memory \"512m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")
  end

  test "run with labels configured" do
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")

    @config[:traefik]["labels"] = { "traefik.http.routers.dashboard.service" => "api@internal", "traefik.http.routers.dashboard.middlewares" => "auth" }
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --label traefik.http.routers.dashboard.service=\"api@internal\" --label traefik.http.routers.dashboard.middlewares=\"auth\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" --label traefik.http.routers.dashboard.service=\"api@internal\" --label traefik.http.routers.dashboard.middlewares=\"auth\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")
  end

  test "run with env configured" do
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")

    @config[:traefik]["env"] = { "secret" => %w[EXAMPLE_API_KEY] }
    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock -e EXAMPLE_API_KEY=\"456\" --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock -e EXAMPLE_API_KEY=\"456\" --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")
  end

@@ -86,7 +86,7 @@ class CommandsTraefikTest < ActiveSupport::TestCase
    @config.delete(:traefik)

    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=DEBUG",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{Mrsk::Commands::Traefik::DEFAULT_IMAGE} --providers.docker --log.level=\"DEBUG\"",
      new_command.run.join(" ")
  end

@@ -94,7 +94,15 @@ class CommandsTraefikTest < ActiveSupport::TestCase
    @config[:logging] = { "driver" => "local", "options" => { "max-size" => "100m", "max-file" => "3" } }

    assert_equal \
-      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-driver \"local\" --log-opt max-size=\"100m\" --log-opt max-file=\"3\" #{@image} --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-driver \"local\" --log-opt max-size=\"100m\" --log-opt max-file=\"3\" #{@image} --providers.docker --log.level=\"DEBUG\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
+      new_command.run.join(" ")
+  end
+
+  test "run with default args overriden" do
+    @config[:traefik]["args"]["log.level"] = "ERROR"
+
+    assert_equal \
+    "docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --log-opt max-size=\"10m\" #{@image} --providers.docker --log.level=\"ERROR\" --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
      new_command.run.join(" ")
  end

--- a/test/configuration/builder_test.rb
+++ b/test/configuration/builder_test.rb
@@ -0,0 +1,151 @@
+require "test_helper"
+
+class ConfigurationBuilderTest < ActiveSupport::TestCase
+  setup do
+    @deploy = {
+      service: "app", image: "dhh/app", registry: { "username" => "dhh", "password" => "secret" },
+      servers: [ "1.1.1.1" ]
+    }
+
+    @config = Mrsk::Configuration.new(@deploy)
+
+    @deploy_with_builder_option = {
+      service: "app", image: "dhh/app", registry: { "username" => "dhh", "password" => "secret" },
+      servers: [ "1.1.1.1" ],
+      builder: {}
+    }
+
+    @config_with_builder_option = Mrsk::Configuration.new(@deploy_with_builder_option)
+  end
+
+  test "multiarch?" do
+    assert_equal true, @config.builder.multiarch?
+  end
+
+  test "setting multiarch to false" do
+    @deploy_with_builder_option[:builder] = { "multiarch" => false }
+
+    assert_equal false, @config_with_builder_option.builder.multiarch?
+  end
+
+  test "local?" do
+    assert_equal false, @config.builder.local?
+  end
+
+  test "remote?" do
+    assert_equal false, @config.builder.remote?
+  end
+
+  test "remote_arch" do
+    assert_nil @config.builder.remote_arch
+  end
+
+  test "remote_host" do
+    assert_nil @config.builder.remote_host
+  end
+
+  test "remote config is missing when local is specified" do
+    @deploy_with_builder_option[:builder] = { "local" => { "arch" => "arm64", "host" => "unix:///Users/<%= `whoami`.strip %>/.docker/run/docker.sock" } }
+
+    assert_raises(ArgumentError) do
+      @config_with_builder_option.builder
+    end
+  end
+
+  test "setting both local and remote configs" do
+    @deploy_with_builder_option[:builder] = {
+      "local" => { "arch" => "arm64", "host" => "unix:///Users/<%= `whoami`.strip %>/.docker/run/docker.sock" },
+      "remote" => { "arch" => "amd64", "host" => "ssh://root@192.168.0.1" }
+    }
+
+    assert_equal true, @config_with_builder_option.builder.local?
+    assert_equal true, @config_with_builder_option.builder.remote?
+
+    assert_equal "amd64", @config_with_builder_option.builder.remote_arch
+    assert_equal "ssh://root@192.168.0.1", @config_with_builder_option.builder.remote_host
+
+    assert_equal "arm64", @config_with_builder_option.builder.local_arch
+    assert_equal "unix:///Users/<%= `whoami`.strip %>/.docker/run/docker.sock", @config_with_builder_option.builder.local_host
+  end
+
+  test "cached?" do
+    assert_equal false, @config.builder.cached?
+  end
+
+  test "invalid cache type specified" do
+    @deploy_with_builder_option[:builder] = { "cache" => { "type" => "invalid" } }
+
+    assert_raises(ArgumentError) do
+      @config_with_builder_option.builder
+    end
+  end
+
+  test "cache_from" do
+    assert_nil @config.builder.cache_from
+  end
+
+  test "cache_to" do
+    assert_nil @config.builder.cache_to
+  end
+
+  test "setting gha cache" do
+    @deploy_with_builder_option[:builder] = { "cache" => { "type" => "gha", "options" => "mode=max" } }
+
+    assert_equal "type=gha", @config_with_builder_option.builder.cache_from
+    assert_equal "type=gha,mode=max", @config_with_builder_option.builder.cache_to
+  end
+
+  test "setting registry cache" do
+    @deploy_with_builder_option[:builder] = { "cache" => { "type" => "registry", "options" => "mode=max,image-manifest=true,oci-mediatypes=true" } }
+
+    assert_equal "type=registry,ref=/dhh/app-build-cache", @config_with_builder_option.builder.cache_from
+    assert_equal "type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref=/dhh/app-build-cache", @config_with_builder_option.builder.cache_to
+  end
+
+  test "setting registry cache with image" do
+    @deploy_with_builder_option[:builder] = { "cache" => { "type" => "registry", "image" => "mrsk", "options" => "mode=max" } }
+
+    assert_equal "type=registry,ref=/mrsk", @config_with_builder_option.builder.cache_from
+    assert_equal "type=registry,mode=max,ref=/mrsk", @config_with_builder_option.builder.cache_to
+  end
+
+  test "args" do
+    assert_equal({}, @config.builder.args)
+  end
+
+  test "setting args" do
+    @deploy_with_builder_option[:builder] = { "args" => { "key" => "value" } }
+
+    assert_equal({ "key" => "value" }, @config_with_builder_option.builder.args)
+  end
+
+  test "secrets" do
+    assert_equal [], @config.builder.secrets
+  end
+
+  test "setting secrets" do
+    @deploy_with_builder_option[:builder] = { "secrets" => ["GITHUB_TOKEN"] }
+
+    assert_equal ["GITHUB_TOKEN"], @config_with_builder_option.builder.secrets
+  end
+
+  test "dockerfile" do
+    assert_equal "Dockerfile", @config.builder.dockerfile
+  end
+
+  test "setting dockerfile" do
+    @deploy_with_builder_option[:builder] = { "dockerfile" => "Dockerfile.dev" }
+
+    assert_equal "Dockerfile.dev", @config_with_builder_option.builder.dockerfile
+  end
+
+  test "context" do
+    assert_equal ".", @config.builder.context
+  end
+
+  test "setting context" do
+    @deploy_with_builder_option[:builder] = { "context" => ".." }
+
+    assert_equal "..", @config_with_builder_option.builder.context
+  end
+end
--- a/test/configuration_test.rb
+++ b/test/configuration_test.rb
@@ -166,7 +166,7 @@ class ConfigurationTest < ActiveSupport::TestCase
    assert_raises(KeyError) do
      config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!({
        env: { "secret" => [ "PASSWORD" ] }
-      }) })
+      }) }).ensure_env_available
    end
  end

@@ -211,17 +211,21 @@ class ConfigurationTest < ActiveSupport::TestCase
    assert_equal "root", @config.ssh_options[:user]

    config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(ssh: { "user" => "app" }) })
-    assert_equal "app", @config.ssh_options[:user]
+    assert_equal "app", config.ssh_options[:user]
+    assert_equal 4, config.ssh_options[:logger].level
+
+    config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(ssh: { "log_level" => "debug" }) })
+    assert_equal 0, config.ssh_options[:logger].level
  end

  test "ssh options with proxy host" do
    config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(ssh: { "proxy" => "1.2.3.4" }) })
-    assert_equal "root@1.2.3.4", @config.ssh_options[:proxy].jump_proxies
+    assert_equal "root@1.2.3.4", config.ssh_options[:proxy].jump_proxies
  end

  test "ssh options with proxy host and user" do
    config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(ssh: { "proxy" => "app@1.2.3.4" }) })
-    assert_equal "app@1.2.3.4", @config.ssh_options[:proxy].jump_proxies
+    assert_equal "app@1.2.3.4", config.ssh_options[:proxy].jump_proxies
  end

  test "volume_args" do
@@ -266,6 +270,37 @@ class ConfigurationTest < ActiveSupport::TestCase
  end

  test "to_h" do
-    assert_equal({ :roles=>["web"], :hosts=>["1.1.1.1", "1.1.1.2"], :primary_host=>"1.1.1.1", :version=>"missing", :repository=>"dhh/app", :absolute_image=>"dhh/app:missing", :service_with_version=>"app-missing", :env_args=>["-e", "REDIS_URL=\"redis://x/y\""], :ssh_options=>{:user=>"root", :auth_methods=>["publickey"]}, :volume_args=>["--volume", "/local/path:/container/path"], :logging=>["--log-opt", "max-size=\"10m\""], :healthcheck=>{"path"=>"/up", "port"=>3000, "max_attempts" => 7 }}, @config.to_h)
+    assert_equal({
+      :roles=>["web"],
+      :hosts=>["1.1.1.1", "1.1.1.2"],
+      :primary_host=>"1.1.1.1",
+      :version=>"missing",
+      :repository=>"dhh/app",
+      :absolute_image=>"dhh/app:missing",
+      :service_with_version=>"app-missing",
+      :env_args=>["-e", "REDIS_URL=\"redis://x/y\""],
+      :ssh_options=>{:user=>"root", :auth_methods=>["publickey"]},
+      :ssh_log_level=>4,
+      :volume_args=>["--volume", "/local/path:/container/path"],
+      :builder=>{},
+      :logging=>["--log-opt", "max-size=\"10m\""],
+      :healthcheck=>{"path"=>"/up", "port"=>3000, "max_attempts" => 7 }
+    }, @config.to_h)
+  end
+
+  test "min version is lower" do
+    config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(minimum_version: "0.0.1") })
+    assert_equal "0.0.1", config.minimum_version
+  end
+
+  test "min version is equal" do
+    config = Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(minimum_version: Mrsk::VERSION) })
+    assert_equal Mrsk::VERSION, config.minimum_version
+  end
+
+  test "min version is higher" do
+    assert_raises(ArgumentError) do
+      Mrsk::Configuration.new(@deploy.tap { |c| c.merge!(minimum_version: "10000.0.0") })
+    end
  end
 end
--- a/test/fixtures/deploy_with_percentage_boot_strategy.yml
+++ b/test/fixtures/deploy_with_percentage_boot_strategy.yml
--- a/test/fixtures/deploy_with_secrets.yml
+++ b/test/fixtures/deploy_with_secrets.yml
@@ -0,0 +1,11 @@
+service: app
+image: dhh/app
+servers:
+  - "1.1.1.1"
+  - "1.1.1.2"
+registry:
+  username: user
+  password: pw
+env:
+  secret:
+    - PASSWORD
--- a/test/integration/docker/deployer/Dockerfile
+++ b/test/integration/docker/deployer/Dockerfile
@@ -17,6 +17,7 @@ RUN apt-get update --fix-missing && apt-get install -y docker-ce docker-ce-cli c
 COPY *.sh .
 COPY app/ .

+RUN rm -rf /root/.ssh
 RUN ln -s /shared/ssh /root/.ssh
 RUN mkdir -p /etc/docker/certs.d/registry:4443 && ln -s /shared/certs/domain.crt /etc/docker/certs.d/registry:4443/ca.crt

--- a/test/integration/docker/deployer/app/.mrsk/hooks/pre-deploy
+++ b/test/integration/docker/deployer/app/.mrsk/hooks/pre-deploy
@@ -0,0 +1,3 @@
+#!/bin/sh
+echo "Deployed!"
+mkdir -p /tmp/${TEST_ID} && touch /tmp/${TEST_ID}/pre-deploy
--- a/test/integration/docker/load_balancer/default.conf
+++ b/test/integration/docker/load_balancer/default.conf
@@ -8,9 +8,9 @@ server {

  location / {
    proxy_pass http://loadbalancer;
-    proxy_connect_timeout       5;
-    proxy_send_timeout          5;
-    proxy_read_timeout          5;
-    send_timeout                5;
+    proxy_connect_timeout       10;
+    proxy_send_timeout          10;
+    proxy_read_timeout          10;
+    send_timeout                10;
  }
 }
--- a/test/integration/integration_test.rb
+++ b/test/integration/integration_test.rb
@@ -10,6 +10,13 @@ class IntegrationTest < ActiveSupport::TestCase
  end

  teardown do
+    unless passed?
+      [:deployer, :vm1, :vm2, :shared, :load_balancer, :registry].each do |container|
+        puts
+        puts "Logs for #{container}:"
+        docker_compose :logs, container
+      end
+    end
    docker_compose "down -t 1"
  end

@@ -54,7 +61,7 @@ class IntegrationTest < ActiveSupport::TestCase
      assert_equal "404", response.code
    end

-    def wait_for_app_to_be_up(timeout: 10, up_count: 3)
+    def wait_for_app_to_be_up(timeout: 20, up_count: 3)
      timeout_at = Time.now + timeout
      up_times = 0
      response = app_response
--- a/test/integration/main_test.rb
+++ b/test/integration/main_test.rb
@@ -8,16 +8,16 @@ class MainTest < IntegrationTest

    mrsk :deploy
    assert_app_is_up version: first_version
-    assert_hooks_ran "pre-connect", "pre-build", "post-deploy"
+    assert_hooks_ran "pre-connect", "pre-build", "pre-deploy", "post-deploy"

    second_version = update_app_rev

    mrsk :redeploy
    assert_app_is_up version: second_version
-    assert_hooks_ran "pre-connect", "pre-build", "post-deploy"
+    assert_hooks_ran "pre-connect", "pre-build", "pre-deploy", "post-deploy"

    mrsk :rollback, first_version
-    assert_hooks_ran "pre-connect", "post-deploy"
+    assert_hooks_ran "pre-connect", "pre-deploy", "post-deploy"
    assert_app_is_up version: first_version

    details = mrsk :details, capture: true
Author	SHA1	Message	Date
Donal McBreen	61b7dc90f2	Bump version for 0.15.0	2023-07-24 13:43:50 +01:00
David Heinemeier Hansson	f6442513ae	Merge pull request #357 from igor-alexandrov/documentation-update Updated README with info for Rails <7 usage	2023-07-24 14:39:48 +02:00
Donal McBreen	5e8df58e6b	Merge pull request #393 from basecamp/rolling-traefik-restarts Support a --rolling option for traefik reboots	2023-07-19 16:43:59 +01:00
Lewis Buckley	9d5a6d1321	Document the rolling option for traefik reboots	2023-07-19 15:03:15 +01:00
Lewis Buckley	ecfd258093	Document the rolling reboot option	2023-07-19 14:58:46 +01:00
Lewis Buckley	313f89a108	Merge branch 'main' into rolling-traefik-restarts * main: Removed not needed MRSK.traefik.run command in Traefil reboot Updated README with locking directory name Include service name to lock details Configurable SSH log levels Add registry container output to debug Minor tweaks to hooks section in readme Update README.md Updated README.md to make setup examples consistent Login to the registry proactively before stoping Accessory and Traefik	2023-07-19 14:46:16 +01:00
Lewis Buckley	9ab448e186	Support a --rolling option for traefik reboots	2023-07-19 14:39:27 +01:00
Donal McBreen	e1433f3895	Merge pull request #349 from igor-alexandrov/login-to-registry-proactively Login to the registry proactively before stoping Accessory and Traefik	2023-07-19 13:36:00 +01:00
Igor Alexandrov	a29e188c90	Removed not needed MRSK.traefik.run command in Traefil reboot	2023-07-19 15:08:42 +04:00
Donal McBreen	95e3915991	Merge pull request #386 from mrsked/ssh-log-levels Configurable SSH log levels	2023-07-17 14:09:21 +01:00
Donal McBreen	30d342183d	Merge pull request #387 from igor-alexandrov/lock-with-service-name Include service name to lock details	2023-07-17 14:07:22 +01:00
Igor Alexandrov	83f5f3f053	Updated README with locking directory name	2023-07-17 10:39:50 +04:00
Igor Alexandrov	e6ca270537	Include service name to lock details	2023-07-15 21:50:39 +04:00
Donal McBreen	cd88c49c42	Configurable SSH log levels Allow ssh log_level to be set to debug connection issues.	2023-07-14 16:08:47 +01:00
Donal McBreen	d03195ce1c	Merge pull request #385 from mrsked/integration-test-registry-debug Add registry container output to debug	2023-07-14 13:50:31 +01:00
Donal McBreen	da1c049829	Add registry container output to debug	2023-07-14 13:41:30 +01:00
Donal McBreen	4095e1853d	Merge pull request #356 from helgeblod/fix-readme-username-typo Update README.md to make setup examples consistent	2023-07-12 13:24:51 +01:00
Donal McBreen	dbc9989730	Merge pull request #364 from nickhammond/patch-2 Update README.md	2023-07-12 13:22:53 +01:00
Donal McBreen	e493369453	Merge pull request #379 from nickhammond/patch-3 Minor tweaks to hooks section in readme	2023-07-12 11:29:36 +01:00
Nick Hammond	e760cfa457	Minor tweaks to hooks section in readme	2023-07-08 10:59:55 -06:00
Nick Hammond	f8d651af0d	Update README.md Add a note about utilizing biometrics with the envify example.	2023-06-28 10:03:27 -06:00
Igor Alexandrov	08172be375	Updated README with info for Rails <7 usage	2023-06-26 16:05:24 +04:00
Jonas Helgemo	a3cc2317e2	Updated README.md to make setup examples consistent - SSH and apt examples should use same username	2023-06-26 11:57:23 +02:00
Igor Alexandrov	2746a48e88	Login to the registry proactively before stoping Accessory and Traefik	2023-06-22 15:13:47 +04:00
David Heinemeier Hansson	9a501867b4	Bump version for 0.14.0	2023-06-20 17:02:42 +02:00
David Heinemeier Hansson	c5397ff51e	Merge pull request #342 from mrsked/only-require-secrets-when-mutating Only require secrets when mutating	2023-06-20 16:50:14 +02:00
Donal McBreen	4950f61a87	Only require secrets when mutating Rename `with_lock` to more generic `mutating` and move the env_args check to that point. This allows read-only actions to be run without requiring secrets.	2023-06-20 15:39:51 +01:00
David Heinemeier Hansson	08d8790851	Merge pull request #337 from igor-alexandrov/feature/cache Support for Docker multistage build cache	2023-06-20 11:38:46 +02:00
Igor Alexandrov	02256ac8fe	More code style improvements	2023-06-19 18:22:07 +04:00
Igor Alexandrov	dadd8225da	Various code style improvements	2023-06-18 23:39:44 +04:00
Igor Alexandrov	aa28ee0f3e	Inroduce Native::Cached builder	2023-06-18 22:45:04 +04:00
David Heinemeier Hansson	2007ab475e	Merge pull request #327 from bestfriendfinance/fix-run-over-ssh-with-proxy-command Add support for proxy_command to run_over_ssh	2023-06-18 17:18:50 +02:00
Igor Alexandrov	4df3389d09	Added support for multistage build cache	2023-06-18 19:02:10 +04:00
Matt Robinson	21b13bf8d3	Add support for proxy_command to run_over_ssh	2023-06-16 08:22:10 -03:00
David Heinemeier Hansson	6e6f696717	Merge pull request #335 from mrsked/specify-min-version Add a minimum version setting	2023-06-16 10:02:40 +02:00
Donal McBreen	98c12a254e	Add a minimum version setting Allow a minimum MRSK version to be specified in the config.	2023-06-15 14:53:03 +01:00
Donal McBreen	f0301d2007	Merge pull request #328 from igor-alexandrov/319-override-default-traefik-args Added ability to override default Traefik command line arguments	2023-06-15 14:49:56 +01:00
Igor Alexandrov	d3f5e9efe8	Updated Traefik CLI test	2023-06-15 17:11:20 +04:00
Igor Alexandrov	d9b3fac17a	Added ability to override default Traefik command line arguments	2023-06-15 15:41:20 +04:00
Donal McBreen	cd5c41ddbe	Merge pull request #334 from basecamp/fix-ssh-symlink Fix ssh symlink	2023-06-15 12:14:30 +01:00
Donal McBreen	a14c6141e5	Dump container logs on failure	2023-06-15 12:05:50 +01:00
Donal McBreen	95d6ee5031	Remove /root/.ssh before symlinking Ensure the symlinks are created correctly whether or not /root/.ssh already exists.	2023-06-15 12:02:56 +01:00
David Heinemeier Hansson	80a4ca4f8a	Merge pull request #331 from basecamp/fix-sample-pre-deploy-hook Fix up the sample pre-deploy hook	2023-06-12 14:09:29 +02:00
Donal McBreen	12ca865e71	Fix up the sample pre-deploy hook - Fix the shebang - Extract a GithubStatusChecks class	2023-06-12 12:47:51 +01:00
David Heinemeier Hansson	66b4a0ea40	Bump version for 0.13.2	2023-06-01 12:54:06 +02:00
David Heinemeier Hansson	04b39ea798	Fix spelling	2023-05-31 18:15:21 +02:00
David Heinemeier Hansson	ae55a7b5d8	Merge pull request #325 from mrsked/revert-324-fix-filename-typo Revert "Fix typo in fixture filename"	2023-05-31 18:14:51 +02:00
David Heinemeier Hansson	601cfbd95e	Revert "Fix typo in fixture filename"	2023-05-31 18:14:43 +02:00
David Heinemeier Hansson	9fdc85c2e6	Merge pull request #324 from basecamp/fix-filename-typo Fix typo in fixture filename	2023-05-31 18:13:11 +02:00
Donal McBreen	222eda6085	Fix typo in fixture filename Following on from https://github.com/mrsked/mrsk/pull/320, fix the fixture filename as well	2023-05-31 17:11:13 +01:00
David Heinemeier Hansson	504a09ef1d	Merge pull request #318 from basecamp/pre-deploy-hook Add a pre-deploy hook	2023-05-31 17:59:46 +02:00
David Heinemeier Hansson	5a25f073f7	Merge pull request #320 from jsoref/spelling Spelling	2023-05-31 17:59:18 +02:00
David Heinemeier Hansson	c8f521c0e8	Merge pull request #323 from basecamp/prefix-docker-host-with-real-host Prefix container hostname with the underlying one	2023-05-31 17:58:55 +02:00
Donal McBreen	28d6a131a9	Prefix container hostname with the underlying one To make it easier to identity where a docker container is running, prefix its hostname with the underlying one from the host. Docker chooses a 12 character random hex string by default, so we'll keep that as the suffix.	2023-05-31 16:22:25 +01:00
David Heinemeier Hansson	3a9075b8ba	Merge pull request #321 from basecamp/more-robust-image-pruning	2023-05-31 13:02:48 +02:00
Donal McBreen	079d9538bb	Improve image pruning robustness If you different images with the same git SHA, on the second deploy the tag is moved and the first image becomes untagged. It may however still be attached to an existing container. To handle this: 1. Initially prune dangling images - this will remove any untagged images that are not attached to an existing image 2. Then filter out the untagged images when deleting tagged images - any that remain will be attached to a container. The second issue is that `docker container ls -a --format '{{.Image}}` will sometimes return the image id rather than a tag. This means that the image doesn't get filtered out when we grep to remove the active images. To fix that we'll grep against both the image id and repo:tag.	2023-05-31 10:17:52 +01:00
Josh Soref	8e94c21729	spelling: with Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>	2023-05-29 20:46:34 -04:00
Josh Soref	b536fcfa43	spelling: percentage Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>	2023-05-29 20:46:34 -04:00
Josh Soref	85005be07f	spelling: message Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>	2023-05-29 20:46:34 -04:00
Josh Soref	fc00392d68	spelling: installed Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>	2023-05-29 20:46:34 -04:00
Josh Soref	fe9affa349	spelling: healthchecks Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>	2023-05-29 20:46:34 -04:00
Josh Soref	3ecb3a4bfc	spelling: guidelines Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>	2023-05-29 20:46:34 -04:00
Josh Soref	787812cdc2	spelling: every time Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>	2023-05-29 20:46:34 -04:00
Josh Soref	91fb85d6b5	spelling: etc. Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>	2023-05-29 20:46:34 -04:00
Donal McBreen	db0bf6bb16	Add a pre-deploy hook Useful for checking the status of CI before deploying. Doing this at this point in the deployment maximises the parallelisation of building and running CI.	2023-05-29 16:06:41 +01:00
David Heinemeier Hansson	de2de19434	Merge pull request #315 from basecamp/prune-unused-images Prune unused images correctly	2023-05-29 11:42:49 +02:00
David Heinemeier Hansson	f9fbebaa72	Merge pull request #316 from f440/fix-typo Fix typo	2023-05-29 11:42:26 +02:00
Donal McBreen	1e300f3798	Wait longer for app to come up	2023-05-29 08:31:19 +01:00
f440	0373f6c4de	Fix typo	2023-05-27 16:27:19 +09:00
Donal McBreen	9037088f99	Increase nginx timeouts in load balancer	2023-05-25 17:31:20 +01:00
Donal McBreen	ff7a1e6726	Prune unused images correctly dangling=true doesn't prune any images, as we are not creating dangling images. Using --all should remove unused images, but it considers the Git SHA tag on the latest image to be unused (presumably because there are two tags, the SHA and latest and the running container is only considered to be using "latest"). As a result it deletes the tag, which means that we can't rollback to that SHA later. Its a bit more complicated to only remove images that are not referenced by any containers. First we find the tags we want to keep from the containers (running and stopped). Then we append the latest tag to that list. Then we get a full list of image tags and remove those tags from that list (using `grep -v -w`). Finally we pass the tags to `docker rmi`. That either deletes the tag if there are other references to the image or both the tag and the image if it is the only one.	2023-05-25 17:16:46 +01:00
David Heinemeier Hansson	602aa43496	Bump version for 0.13.1	2023-05-25 14:04:29 +02:00
David Heinemeier Hansson	e35334e5fe	Merge pull request #313 from basecamp/stop-restarting-containers Stop containers with restarting status	2023-05-25 14:04:09 +02:00
Donal McBreen	cedb8d900f	Stop containers with restarting status When stopping the old container we need to also look for ones with a restarting status.	2023-05-25 12:10:26 +01:00