Bump version for 0.13.2

Revert "Fix typo in fixture filename"

CONTRIBUTING.md

@@ -27,7 +27,7 @@ Please keep the following guidelines in mind when opening a pull request:
 - Add tests for your changes, if possible.
 - Ensure that your changes don't break existing functionality.
 
-#### Commit message guidline
+#### Commit message guidelines
 
 A good commit message should describe what changed and why.
 

Dockerfile

@@ -21,7 +21,7 @@ RUN apk add --no-cache --update build-base git docker openrc openssh-client-defa
 
 # Copy the rest of our application code into the container.
 # We do this after bundle install, to avoid having to run bundle
-# everytime we do small fixes in the source code.
+# every time we do small fixes in the source code.
 COPY . .
 
 # Install the gem locally from the project folder

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    mrsk (0.13.0)
+    mrsk (0.13.2)
       activesupport (>= 7.0)
       bcrypt_pbkdf (~> 1.0)
       dotenv (~> 2.8)

README.md

@@ -67,7 +67,7 @@ Voila! All the servers are now serving the app on port 80. If you're just runnin
 
 In the past decade+, there's been an explosion in commercial offerings that make deploying web apps easier. Heroku kicked it off with an incredible offering that stayed ahead of the competition seemingly forever. These days we have excellent alternatives like Fly.io and Render. And hosted Kubernetes is making things easier too on AWS, GCP, Digital Ocean, and elsewhere. But these are all offerings that have you renting computers in the cloud at a premium. If you want to run on your own hardware, or even just have a clear migration path to do so in the future, you need to carefully consider how locked in you get to these commercial platforms. Preferably before the bills swallow your business whole!
 
-MRSK seeks to bring the advance in ergonomics pioneered by these commercial offerings to deploying web apps anywhere. Whether that's low-cost cloud options without the managed-service markup from the likes of Digital Ocean, Hetzner, OVH, etc, or it's your own colocated bare metal. To MRSK, it's all the same. Feed the config file a list of IP addresses with vanilla Ubuntu servers that have seen no prep beyond an added SSH key, and you'll be running in literally minutes.
+MRSK seeks to bring the advance in ergonomics pioneered by these commercial offerings to deploying web apps anywhere. Whether that's low-cost cloud options without the managed-service markup from the likes of Digital Ocean, Hetzner, OVH, etc., or it's your own colocated bare metal. To MRSK, it's all the same. Feed the config file a list of IP addresses with vanilla Ubuntu servers that have seen no prep beyond an added SSH key, and you'll be running in literally minutes.
 
 This approach gives you enormous portability. You can have your web app deployed on several clouds at ease like this. Or you can buy the baseline with your own hardware, then deploy to a cloud before a big seasonal spike to get more capacity. When you're not locked into a single provider from a tooling perspective, there are a lot of compelling options available.
 
@@ -670,7 +670,7 @@ This assumes the Cron settings are stored in `config/crontab`.
 
 ### Healthcheck
 
-MRSK uses Docker healtchecks to check the health of your application during deployment. Traefik uses this same healthcheck status to determine when a container is ready to receive traffic.
+MRSK uses Docker healthchecks to check the health of your application during deployment. Traefik uses this same healthcheck status to determine when a container is ready to receive traffic.
 
 The healthcheck defaults to testing the HTTP response to the path `/up` on port 3000, up to 7 times. You can tailor this behaviour with the `healthcheck` setting:
 
@@ -840,7 +840,7 @@ Message: Automatic deploy lock
 You can also manually acquire and release the lock
 
 ```
-mrsk lock acquire -m "Doing maintanence"
+mrsk lock acquire -m "Doing maintenance"
 ```
 
 ```
@@ -882,11 +882,13 @@ firing a JSON webhook. These variables include:
 - `MRSK_PERFORMER` - the local user performing the command (from `whoami`)
 - `MRSK_SERVICE_VERSION` - an abbreviated service and version for use in messages, e.g. app@150b24f
 - `MRSK_VERSION` - an full version being deployed
-- `MRSK_DESTINATION` - optional: destination, e.g. "staging"
 - `MRSK_HOSTS` - a comma separated list of the hosts targeted by the command
+- `MRSK_COMMAND` - The command we are running
+- `MRSK_SUBCOMMAND` - optional: The subcommand we are running
+- `MRSK_DESTINATION` - optional: destination, e.g. "staging"
 - `MRSK_ROLE` - optional: role targeted, e.g. "web"
 
-There are three hooks:
+There are four hooks:
 
 1. pre-connect
 Called before taking the deploy lock. For checks that need to run before connecting to remote hosts - e.g. DNS warming.
@@ -894,6 +896,9 @@ Called before taking the deploy lock. For checks that need to run before connect
 2. pre-build
 Used for pre-build checks - e.g. there are no uncommitted changes or that CI has passed.
 
+3. pre-deploy
+For final checks before deploying, e.g. checking CI completed
+
 3. post-deploy - run after a deploy, redeploy or rollback
 
 This hook is also passed a `MRSK_RUNTIME` env variable.

lib/mrsk/cli/app.rb

@@ -29,7 +29,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
               execute *auditor.record("Booted app version #{version}"), verbosity: :debug
 
               old_version = capture_with_info(*app.current_running_version, raise_on_non_zero_exit: false).strip
-              execute *app.start_or_run
+              execute *app.start_or_run(hostname: "#{host}-#{SecureRandom.hex(6)}")
 
               Mrsk::Utils::HealthcheckPoller.wait_for_healthy(pause_after_ready: true) { capture_with_info(*app.status(version: version)) }
 

lib/mrsk/cli/base.rb

@@ -133,15 +133,39 @@ module Mrsk::Cli
         end
       end
 
-      def run_hook(hook, **details)
+      def run_hook(hook, **extra_details)
         if !options[:skip_hooks] && MRSK.hook.hook_exists?(hook)
+          details = { hosts: MRSK.hosts.join(","), command: command, subcommand: subcommand }
+
           say "Running the #{hook} hook...", :magenta
           run_locally do
-            MRSK.with_verbosity(:debug) { execute *MRSK.hook.run(hook, **details, hosts: MRSK.hosts.join(",")) }
+            MRSK.with_verbosity(:debug) { execute *MRSK.hook.run(hook, **details, **extra_details) }
           rescue SSHKit::Command::Failed
             raise HookError.new("Hook `#{hook}` failed")
           end
         end
       end
-  end
+
+      def command
+        @mrsk_command ||= begin
+          invocation_class, invocation_commands = *first_invocation
+          if invocation_class == Mrsk::Cli::Main
+            invocation_commands[0]
+          else
+            Mrsk::Cli::Main.subcommand_classes.find { |command, clazz| clazz == invocation_class }[0]
+          end
+        end
+      end
+
+      def subcommand
+        @mrsk_subcommand ||= begin
+          invocation_class, invocation_commands = *first_invocation
+          invocation_commands[0] if invocation_class != Mrsk::Cli::Main
+        end
+      end
+
+      def first_invocation
+        instance_variable_get("@_invocations").first
+      end
+    end
 end

lib/mrsk/cli/lock.rb

@@ -7,7 +7,7 @@ class Mrsk::Cli::Lock < Mrsk::Cli::Base
   end
 
   desc "acquire", "Acquire the deploy lock"
-  option :message, aliases: "-m", type: :string, desc: "A lock mesasge", required: true
+  option :message, aliases: "-m", type: :string, desc: "A lock message", required: true
   def acquire
     message = options[:message]
     raise_if_locked do

lib/mrsk/cli/main.rb

@@ -28,6 +28,8 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
           invoke "mrsk:cli:build:deliver", [], invoke_options
         end
 
+        run_hook "pre-deploy"
+
         say "Ensure Traefik is running...", :magenta
         invoke "mrsk:cli:traefik:boot", [], invoke_options
 
@@ -62,6 +64,8 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
           invoke "mrsk:cli:build:deliver", [], invoke_options
         end
 
+        run_hook "pre-deploy"
+
         say "Ensure app can pass healthcheck...", :magenta
         invoke "mrsk:cli:healthcheck:perform", [], invoke_options
 
@@ -86,6 +90,8 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
         old_version = nil
 
         if container_available?(version)
+          run_hook "pre-deploy"
+
           invoke "mrsk:cli:app:boot", [], invoke_options.merge(version: version)
           rolled_back = true
         else

lib/mrsk/cli/prune.rb

@@ -12,7 +12,8 @@ class Mrsk::Cli::Prune < Mrsk::Cli::Base
     with_lock do
       on(MRSK.hosts) do
         execute *MRSK.auditor.record("Pruned images"), verbosity: :debug
-        execute *MRSK.prune.images
+        execute *MRSK.prune.dangling_images
+        execute *MRSK.prune.tagged_images
       end
     end
   end

lib/mrsk/cli/templates/sample_hooks/pre-deploy.sample

@@ -0,0 +1,82 @@
+#!/bin/sh
+
+# A sample pre-deploy hook
+#
+# Checks the Github status of the build, waiting for a pending build to complete for up to 720 seconds.
+#
+# Fails unless the combined status is "success"
+#
+# These environment variables are available:
+# MRSK_RECORDED_AT
+# MRSK_PERFORMER
+# MRSK_VERSION
+# MRSK_HOSTS
+# MRSK_COMMAND
+# MRSK_SUBCOMMAND
+# MRSK_ROLE (if set)
+# MRSK_DESTINATION (if set)
+
+#!/usr/bin/env ruby
+
+# Only check the build status for production deployments
+if ENV["MRSK_COMMAND"] == "rollback" || ENV["MRSK_DESTINATION"] != "production"
+  exit 0
+end
+
+require "bundler/inline"
+
+# true = install gems so this is fast on repeat invocations
+gemfile(true, quiet: true) do
+  source "https://rubygems.org"
+
+  gem "octokit"
+  gem "faraday-retry"
+end
+
+MAX_ATTEMPTS = 72
+ATTEMPTS_GAP = 10
+
+def exit_with_error(message)
+  $stderr.puts message
+  exit 1
+end
+
+def first_status_url(combined_status, state)
+  first_status = combined_status[:statuses].find { |status| status[:state] == state }
+  first_status && first_status[:target_url]
+end
+
+remote_url = `git config --get remote.origin.url`.strip.delete_prefix("https://github.com/")
+git_sha = `git rev-parse HEAD`.strip
+
+repository = Octokit::Repository.from_url(remote_url)
+github_client = Octokit::Client.new(access_token: ENV["GITHUB_TOKEN"])
+attempts = 0
+
+begin
+  loop do
+    combined_status = github_client.combined_status(remote_url, git_sha)
+    state = combined_status[:state]
+    first_status_url = first_status_url(combined_status, state)
+
+    case state
+    when "success"
+      puts "Build passed, see #{first_status_url}"
+      exit 0
+    when "failure"
+      exit_with_error "Build failed, see #{first_status_url}"
+    when "pending"
+      attempts += 1
+    end
+
+    puts "Waiting #{ATTEMPTS_GAP} more seconds for build to complete#{", see #{first_status_url}" if first_status_url}..."
+
+    if attempts == MAX_ATTEMPTS
+      exit_with_error "Build status is still pending, gave up after #{MAX_ATTEMPTS * ATTEMPTS_GAP} seconds"
+    end
+
+    sleep(ATTEMPTS_GAP)
+  end
+rescue Octokit::NotFound
+  exit_with_error "Build status could not be found"
+end

lib/mrsk/commands/app.rb

@@ -1,4 +1,6 @@
 class Mrsk::Commands::App < Mrsk::Commands::Base
+  ACTIVE_DOCKER_STATUSES = [ :running, :restarting ]
+
   attr_reader :role
 
   def initialize(config, role: nil)
@@ -6,17 +8,18 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
     @role = role
   end
 
-  def start_or_run
+  def start_or_run(hostname: nil)
-    combine start, run, by: "||"
+    combine start, run(hostname: hostname), by: "||"
   end
 
-  def run
+  def run(hostname: nil)
     role = config.role(self.role)
 
     docker :run,
       "--detach",
       "--restart unless-stopped",
       "--name", container_name,
+      *(["--hostname", hostname] if hostname),
       "-e", "MRSK_CONTAINER_NAME=\"#{container_name}\"",
       *role.env_args,
       *role.health_check_args,
@@ -92,7 +95,7 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
 
 
   def current_running_container_id
-    docker :ps, "--quiet", *filter_args(status: :running), "--latest"
+    docker :ps, "--quiet", *filter_args(statuses: ACTIVE_DOCKER_STATUSES), "--latest"
   end
 
   def container_id_for_version(version, only_running: false)
@@ -100,12 +103,12 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
   end
 
   def current_running_version
-    list_versions("--latest", status: :running)
+    list_versions("--latest", statuses: ACTIVE_DOCKER_STATUSES)
   end
 
-  def list_versions(*docker_args, status: nil)
+  def list_versions(*docker_args, statuses: nil)
     pipe \
-      docker(:ps, *filter_args(status: status), *docker_args, "--format", '"{{.Names}}"'),
+      docker(:ps, *filter_args(statuses: statuses), *docker_args, "--format", '"{{.Names}}"'),
       %(grep -oE "\\-[^-]+$"), # Extract SHA from "service-role-dest-SHA"
       %(cut -c 2-)
   end
@@ -150,15 +153,17 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
       [ config.service, role, config.destination, version || config.version ].compact.join("-")
     end
 
-    def filter_args(status: nil)
+    def filter_args(statuses: nil)
-      argumentize "--filter", filters(status: status)
+      argumentize "--filter", filters(statuses: statuses)
     end
 
-    def filters(status: nil)
+    def filters(statuses: nil)
       [ "label=service=#{config.service}" ].tap do |filters|
         filters << "label=destination=#{config.destination}" if config.destination
         filters << "label=role=#{role}" if role
-        filters << "status=#{status}" if status
+        statuses&.each do |status|
+          filters << "status=#{status}"
+        end
       end
     end
 end

lib/mrsk/commands/prune.rb

@@ -2,13 +2,20 @@ require "active_support/duration"
 require "active_support/core_ext/numeric/time"
 
 class Mrsk::Commands::Prune < Mrsk::Commands::Base
-  def images
+  def dangling_images
     docker :image, :prune, "--force", "--filter", "label=service=#{config.service}", "--filter", "dangling=true"
   end
 
+  def tagged_images
+    pipe \
+      docker(:image, :ls, *service_filter, "--format", "'{{.ID}} {{.Repository}}:{{.Tag}}'"),
+      "grep -v -w \"#{active_image_list}\"",
+      "while read image tag; do docker rmi $tag; done"
+  end
+
   def containers(keep_last: 5)
     pipe \
-      docker(:ps, "-q", "-a", "--filter", "label=service=#{config.service}", *stopped_containers_filters),
+      docker(:ps, "-q", "-a", *service_filter, *stopped_containers_filters),
       "tail -n +#{keep_last + 1}",
       "while read container_id; do docker rm $container_id; done"
   end
@@ -17,4 +24,15 @@ class Mrsk::Commands::Prune < Mrsk::Commands::Base
     def stopped_containers_filters
       [ "created", "exited", "dead" ].flat_map { |status| ["--filter", "status=#{status}"] }
     end
+
+    def active_image_list
+      # Pull the images that are used by any containers
+      # Append repo:latest - to avoid deleting the latest tag
+      # Append repo:<none> - to avoid deleting dangling images that are in use. Unused dangling images are deleted separately
+      "$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=#{config.service} | tr -d '\\n')#{config.latest_image}\\|#{config.repository}:<none>"
+    end
+
+    def service_filter
+      [ "--filter", "label=service=#{config.service}" ]
+    end
 end

lib/mrsk/version.rb

@@ -1,3 +1,3 @@
 module Mrsk
-  VERSION = "0.13.0"
+  VERSION = "0.13.2"
 end

test/cli/app_test.rb

@@ -5,7 +5,7 @@ class CliAppTest < CliTestCase
     stub_running
     run_command("boot").tap do |output|
       assert_match "docker tag dhh/app:latest dhh/app:latest", output
-      assert_match "docker run --detach --restart unless-stopped", output
+      assert_match /docker run --detach --restart unless-stopped --name app-web-latest --hostname 1.1.1.1-[0-9a-f]{12} /, output
       assert_match "docker container ls --all --filter name=^app-web-123$ --quiet | xargs docker stop", output
     end
   end
@@ -22,13 +22,13 @@ class CliAppTest < CliTestCase
       .returns("running") # health check
 
     SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
-      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
+      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--filter", "status=restarting", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
       .returns("123") # old version
 
     run_command("boot").tap do |output|
       assert_match /Renaming container .* to .* as already deployed on 1.1.1.1/, output # Rename
       assert_match /docker rename app-web-latest app-web-latest_replaced_[0-9a-f]{16}/, output
-      assert_match "docker run --detach --restart unless-stopped", output
+      assert_match /docker run --detach --restart unless-stopped --name app-web-latest --hostname 1.1.1.1-[0-9a-f]{12} /, output
       assert_match "docker container ls --all --filter name=^app-web-123$ --quiet | xargs docker stop", output
     end
   ensure
@@ -65,7 +65,7 @@ class CliAppTest < CliTestCase
 
   test "stop" do
     run_command("stop").tap do |output|
-      assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop", output
+      assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop", output
     end
   end
 
@@ -98,7 +98,7 @@ class CliAppTest < CliTestCase
 
   test "remove" do
     run_command("remove").tap do |output|
-      assert_match /#{Regexp.escape("docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop")}/, output
+      assert_match /#{Regexp.escape("docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop")}/, output
       assert_match /#{Regexp.escape("docker container prune --force --filter label=service=app")}/, output
       assert_match /#{Regexp.escape("docker image prune --all --force --filter label=service=app")}/, output
     end
@@ -130,7 +130,7 @@ class CliAppTest < CliTestCase
 
   test "exec with reuse" do
     run_command("exec", "--reuse", "ruby -v").tap do |output|
-      assert_match "docker ps --filter label=service=app --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output # Get current version
+      assert_match "docker ps --filter label=service=app --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output # Get current version
       assert_match "docker exec app-web-999 ruby -v", output
     end
   end
@@ -149,28 +149,28 @@ class CliAppTest < CliTestCase
 
   test "logs" do
     SSHKit::Backend::Abstract.any_instance.stubs(:exec)
-      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest| xargs docker logs --timestamps --tail 10 2>&1'")
+      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest| xargs docker logs --timestamps --tail 10 2>&1'")
 
-    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --tail 100 2>&1", run_command("logs")
+    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --tail 100 2>&1", run_command("logs")
   end
 
   test "logs with follow" do
     SSHKit::Backend::Abstract.any_instance.stubs(:exec)
-      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1'")
+      .with("ssh -t root@1.1.1.1 'docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1'")
 
-    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1", run_command("logs", "--follow")
+    assert_match "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1", run_command("logs", "--follow")
   end
 
   test "version" do
     run_command("version").tap do |output|
-      assert_match "docker ps --filter label=service=app --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
+      assert_match "docker ps --filter label=service=app --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
     end
   end
 
 
   test "version through main" do
     stdouted { Mrsk::Cli::Main.start(["app", "version", "-c", "test/fixtures/deploy_with_accessories.yml", "--hosts", "1.1.1.1"]) }.tap do |output|
-      assert_match "docker ps --filter label=service=app --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
+      assert_match "docker ps --filter label=service=app --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-", output
     end
   end
 

test/cli/build_test.rb

@@ -9,7 +9,11 @@ class CliBuildTest < CliTestCase
   end
 
   test "push" do
+    Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
+    hook_variables = { version: 999, service_version: "app@999", hosts: "1.1.1.1,1.1.1.2,1.1.1.3,1.1.1.4", command: "build", subcommand: "push" }
+
     run_command("push").tap do |output|
+      assert_hook_ran "pre-build", output, **hook_variables
       assert_match /docker --version && docker buildx version/, output
       assert_match /docker buildx build --push --platform linux\/amd64,linux\/arm64 --builder mrsk-app-multiarch -t dhh\/app:999 -t dhh\/app:latest --label service="app" --file Dockerfile \. as .*@localhost/, output
     end

test/cli/cli_test_case.rb

@@ -27,19 +27,30 @@ class CliTestCase < ActiveSupport::TestCase
         .raises(SSHKit::Command::Failed.new("failed"))
     end
 
-    def ensure_hook_runs(hook)
-      Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
-      SSHKit::Backend::Abstract.any_instance.stubs(:execute)
-        .with { |*args| args != [".mrsk/hooks/#{hook}"] }
-      SSHKit::Backend::Abstract.any_instance.expects(:execute)
-        .with { |*args| args.first == ".mrsk/hooks/#{hook}" }
-        .once
-    end
-
     def stub_locking
       SSHKit::Backend::Abstract.any_instance.stubs(:execute)
         .with { |arg1, arg2| arg1 == :mkdir && arg2 == :mrsk_lock }
       SSHKit::Backend::Abstract.any_instance.stubs(:execute)
         .with { |arg1, arg2| arg1 == :rm && arg2 == "mrsk_lock/details" }
     end
+
+    def assert_hook_ran(hook, output, version:, service_version:, hosts:, command:, subcommand: nil, runtime: nil)
+      performer = `whoami`.strip
+
+      assert_match "Running the #{hook} hook...\n", output
+
+      expected = %r{Running\s/usr/bin/env\s\.mrsk/hooks/#{hook}\sas\s#{performer}@localhost\n\s
+        DEBUG\s\[[0-9a-f]*\]\sCommand:\s\(\sexport\s
+        MRSK_RECORDED_AT=\"\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\dZ\"\s
+        MRSK_PERFORMER=\"#{performer}\"\s
+        MRSK_VERSION=\"#{version}\"\s
+        MRSK_SERVICE_VERSION=\"#{service_version}\"\s
+        MRSK_HOSTS=\"#{hosts}\"\s
+        MRSK_COMMAND=\"#{command}\"\s
+        #{"MRSK_SUBCOMMAND=\\\"#{subcommand}\\\"\\s" if subcommand}
+        #{"MRSK_RUNTIME=\\\"#{runtime}\\\"\\s" if runtime}
+        ;\s/usr/bin/env\s\.mrsk/hooks/#{hook} }x
+
+      assert_match expected, output
+    end
 end

test/cli/main_test.rb

@@ -21,16 +21,18 @@ class CliMainTest < CliTestCase
     Mrsk::Cli::Main.any_instance.expects(:invoke).with("mrsk:cli:prune:all", [], invoke_options)
 
     Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
+    hook_variables = { version: 999, service_version: "app@999", hosts: "1.1.1.1,1.1.1.2", command: "deploy" }
 
     run_command("deploy").tap do |output|
-      assert_match /Running the pre-connect hook.../, output
+      assert_hook_ran "pre-connect", output, **hook_variables
       assert_match /Log into image registry/, output
       assert_match /Build and push app image/, output
+      assert_hook_ran "pre-deploy", output, **hook_variables
       assert_match /Ensure Traefik is running/, output
       assert_match /Ensure app can pass healthcheck/, output
       assert_match /Detect stale containers/, output
       assert_match /Prune old containers and images/, output
-      assert_match /Running the post-deploy hook.../, output
+      assert_hook_ran "post-deploy", output, **hook_variables, runtime: 0
     end
   end
 
@@ -124,10 +126,15 @@ class CliMainTest < CliTestCase
 
     Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
 
+    hook_variables = { version: 999, service_version: "app@999", hosts: "1.1.1.1,1.1.1.2", command: "redeploy" }
+
     run_command("redeploy").tap do |output|
+      assert_hook_ran "pre-connect", output, **hook_variables
       assert_match /Build and push app image/, output
+      assert_hook_ran "pre-deploy", output, **hook_variables
+      assert_match /Running the pre-deploy hook.../, output
       assert_match /Ensure app can pass healthcheck/, output
-      assert_match /Running the post-deploy hook.../, output
+      assert_hook_ran "post-deploy", output, **hook_variables, runtime: "0"
     end
   end
 
@@ -165,7 +172,7 @@ class CliMainTest < CliTestCase
         .with(:docker, :container, :ls, "--all", "--filter", "name=^app-#{role}-123$", "--quiet")
         .returns("version-to-rollback\n").at_least_once
       SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
-        .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=#{role}", "--filter", "status=running", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
+        .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=#{role}", "--filter", "status=running", "--filter", "status=restarting", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
         .returns("version-to-rollback\n").at_least_once
       SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
         .with(:docker, :container, :ls, "--all", "--filter", "name=^app-#{role}-123$", "--quiet", "|", :xargs, :docker, :inspect, "--format", "'{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'")
@@ -173,13 +180,15 @@ class CliMainTest < CliTestCase
     end
 
     Mrsk::Commands::Hook.any_instance.stubs(:hook_exists?).returns(true)
+    hook_variables = { version: 123, service_version: "app@123", hosts: "1.1.1.1,1.1.1.2,1.1.1.3,1.1.1.4", command: "rollback" }
 
     run_command("rollback", "123", config_file: "deploy_with_accessories").tap do |output|
       assert_match "Start container with version 123", output
+      assert_hook_ran "pre-deploy", output, **hook_variables
       assert_match "docker tag dhh/app:123 dhh/app:latest", output
       assert_match "docker start app-web-123", output
       assert_match "docker container ls --all --filter name=^app-web-version-to-rollback$ --quiet | xargs docker stop", output, "Should stop the container that was previously running"
-      assert_match "Running the post-deploy hook...", output
+      assert_hook_ran "post-deploy", output, **hook_variables, runtime: "0"
     end
   end
 
@@ -192,7 +201,7 @@ class CliMainTest < CliTestCase
       .with(:docker, :container, :ls, "--filter", "name=^app-web-123$", "--quiet", raise_on_non_zero_exit: false)
       .returns("").at_least_once
     SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
-      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
+      .with(:docker, :ps, "--filter", "label=service=app", "--filter", "label=role=web", "--filter", "status=running", "--filter", "status=restarting", "--latest", "--format", "\"{{.Names}}\"", "|", "grep -oE \"\\-[^-]+$\"", "|", "cut -c 2-", raise_on_non_zero_exit: false)
       .returns("").at_least_once
     SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info)
       .with(:docker, :container, :ls, "--all", "--filter", "name=^app-web-123$", "--quiet", "|", :xargs, :docker, :inspect, "--format", "'{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'")

test/cli/prune_test.rb

@@ -10,7 +10,8 @@ class CliPruneTest < CliTestCase
 
   test "images" do
     run_command("images").tap do |output|
-      assert_match /docker image prune --force --filter label=service=app --filter dangling=true on 1.1.1.\d/, output
+      assert_match "docker image prune --force --filter label=service=app --filter dangling=true on 1.1.1.", output
+      assert_match "docker image ls --filter label=service=app --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w \"$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=app | tr -d '\\n')dhh/app:latest\\|dhh/app:<none>\" | while read image tag; do docker rmi $tag; done on 1.1.1.", output
     end
   end
 

test/cli/server_test.rb

@@ -11,7 +11,7 @@ class CliServerTest < CliTestCase
     SSHKit::Backend::Abstract.any_instance.expects(:execute).with(:docker, "-v", raise_on_non_zero_exit: false).returns(false).at_least_once
     SSHKit::Backend::Abstract.any_instance.expects(:execute).with('[ "${EUID:-$(id -u)}" -eq 0 ]', raise_on_non_zero_exit: false).returns(false).at_least_once
 
-    assert_raise RuntimeError, "Docker is not installed on 1.1.1.1, 1.1.1.3, 1.1.1.4, 1.1.1.2 and can't be automatically intalled without having root access and the `curl` command available. Install Docker manually: https://docs.docker.com/engine/install/" do
+    assert_raise RuntimeError, "Docker is not installed on 1.1.1.1, 1.1.1.3, 1.1.1.4, 1.1.1.2 and can't be automatically installed without having root access and the `curl` command available. Install Docker manually: https://docs.docker.com/engine/install/" do
       run_command("bootstrap")
     end
   end

test/commander_test.rb

@@ -65,7 +65,7 @@ class CommanderTest < ActiveSupport::TestCase
   end
 
   test "percentage-based group strategy" do
-    configure_with(:deploy_with_precentage_boot_strategy)
+    configure_with(:deploy_with_percentage_boot_strategy)
 
     assert_equal({ in: :groups, limit: 1, wait: 2 }, @mrsk.boot_strategy)
   end

test/commands/app_test.rb

@@ -17,6 +17,12 @@ class CommandsAppTest < ActiveSupport::TestCase
       new_command.run.join(" ")
   end
 
+  test "run with hostname" do
+    assert_equal \
+      "docker run --detach --restart unless-stopped --name app-web-999 --hostname myhost -e MRSK_CONTAINER_NAME=\"app-web-999\" -e RAILS_MASTER_KEY=\"456\" --health-cmd \"curl -f http://localhost:3000/up || exit 1\" --health-interval \"1s\" --log-opt max-size=\"10m\" --label service=\"app\" --label role=\"web\" --label traefik.http.services.app-web.loadbalancer.server.scheme=\"http\" --label traefik.http.routers.app-web.rule=\"PathPrefix(\\`/\\`)\" --label traefik.http.middlewares.app-web-retry.retry.attempts=\"5\" --label traefik.http.middlewares.app-web-retry.retry.initialinterval=\"500ms\" --label traefik.http.routers.app-web.middlewares=\"app-web-retry@docker\" dhh/app:999",
+      new_command.run(hostname: "myhost").join(" ")
+  end
+
   test "run with volumes" do
     @config[:volumes] = ["/local/path:/container/path" ]
 
@@ -77,16 +83,28 @@ class CommandsAppTest < ActiveSupport::TestCase
       new_command.start.join(" ")
   end
 
+  test "start_or_run" do
+    assert_equal \
+      "docker start app-web-999 || docker run --detach --restart unless-stopped --name app-web-999 -e MRSK_CONTAINER_NAME=\"app-web-999\" -e RAILS_MASTER_KEY=\"456\" --health-cmd \"curl -f http://localhost:3000/up || exit 1\" --health-interval \"1s\" --log-opt max-size=\"10m\" --label service=\"app\" --label role=\"web\" --label traefik.http.services.app-web.loadbalancer.server.scheme=\"http\" --label traefik.http.routers.app-web.rule=\"PathPrefix(\\`/\\`)\" --label traefik.http.middlewares.app-web-retry.retry.attempts=\"5\" --label traefik.http.middlewares.app-web-retry.retry.initialinterval=\"500ms\" --label traefik.http.routers.app-web.middlewares=\"app-web-retry@docker\" dhh/app:999",
+      new_command.start_or_run.join(" ")
+  end
+
+  test "start_or_run with hostname" do
+    assert_equal \
+      "docker start app-web-999 || docker run --detach --restart unless-stopped --name app-web-999 --hostname myhost -e MRSK_CONTAINER_NAME=\"app-web-999\" -e RAILS_MASTER_KEY=\"456\" --health-cmd \"curl -f http://localhost:3000/up || exit 1\" --health-interval \"1s\" --log-opt max-size=\"10m\" --label service=\"app\" --label role=\"web\" --label traefik.http.services.app-web.loadbalancer.server.scheme=\"http\" --label traefik.http.routers.app-web.rule=\"PathPrefix(\\`/\\`)\" --label traefik.http.middlewares.app-web-retry.retry.attempts=\"5\" --label traefik.http.middlewares.app-web-retry.retry.initialinterval=\"500ms\" --label traefik.http.routers.app-web.middlewares=\"app-web-retry@docker\" dhh/app:999",
+      new_command.start_or_run(hostname: "myhost").join(" ")
+  end
+
   test "stop" do
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop",
       new_command.stop.join(" ")
   end
 
   test "stop with custom stop wait time" do
     @config[:stop_wait_time] = 30
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker stop -t 30",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker stop -t 30",
       new_command.stop.join(" ")
   end
 
@@ -112,37 +130,37 @@ class CommandsAppTest < ActiveSupport::TestCase
 
   test "logs" do
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs 2>&1",
       new_command.logs.join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --since 5m 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --since 5m 2>&1",
       new_command.logs(since: "5m").join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --tail 100 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --tail 100 2>&1",
       new_command.logs(lines: "100").join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --since 5m --tail 100 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --since 5m --tail 100 2>&1",
       new_command.logs(since: "5m", lines: "100").join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs 2>&1 | grep 'my-id'",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs 2>&1 | grep 'my-id'",
       new_command.logs(grep: "my-id").join(" ")
 
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --since 5m 2>&1 | grep 'my-id'",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --since 5m 2>&1 | grep 'my-id'",
       new_command.logs(since: "5m", grep: "my-id").join(" ")
   end
 
   test "follow logs" do
     assert_match \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1",
       new_command.follow_logs(host: "app-1")
 
     assert_match \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1 | grep \"Completed\"",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest | xargs docker logs --timestamps --tail 10 --follow 2>&1 | grep \"Completed\"",
       new_command.follow_logs(host: "app-1", grep: "Completed")
   end
 
@@ -196,14 +214,14 @@ class CommandsAppTest < ActiveSupport::TestCase
 
   test "current_running_container_id" do
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --latest",
+      "docker ps --quiet --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest",
       new_command.current_running_container_id.join(" ")
   end
 
   test "current_running_container_id with destination" do
     @destination = "staging"
     assert_equal \
-      "docker ps --quiet --filter label=service=app --filter label=destination=staging --filter label=role=web --filter status=running --latest",
+      "docker ps --quiet --filter label=service=app --filter label=destination=staging --filter label=role=web --filter status=running --filter status=restarting --latest",
       new_command.current_running_container_id.join(" ")
   end
 
@@ -215,7 +233,7 @@ class CommandsAppTest < ActiveSupport::TestCase
 
   test "current_running_version" do
     assert_equal \
-      "docker ps --filter label=service=app --filter label=role=web --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
+      "docker ps --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
       new_command.current_running_version.join(" ")
   end
 
@@ -225,8 +243,8 @@ class CommandsAppTest < ActiveSupport::TestCase
       new_command.list_versions.join(" ")
 
     assert_equal \
-      "docker ps --filter label=service=app --filter label=role=web --filter status=running --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
+      "docker ps --filter label=service=app --filter label=role=web --filter status=running --filter status=restarting --latest --format \"{{.Names}}\" | grep -oE \"\\-[^-]+$\" | cut -c 2-",
-      new_command.list_versions("--latest", status: :running).join(" ")
+      new_command.list_versions("--latest", statuses: [ :running, :restarting ]).join(" ")
   end
 
   test "list_containers" do

test/commands/builder_test.rb

@@ -88,7 +88,7 @@ class CommandsBuilderTest < ActiveSupport::TestCase
       builder.push.join(" ")
   end
 
-  test "native push with with build secrets" do
+  test "native push with build secrets" do
     builder = new_builder_command(builder: { "multiarch" => false, "secrets" => [ "a", "b" ] })
     assert_equal \
       "docker build -t dhh/app:123 -t dhh/app:latest --label service=\"app\" --secret id=\"a\" --secret id=\"b\" --file Dockerfile . && docker push dhh/app:123 && docker push dhh/app:latest",

test/commands/prune_test.rb

@@ -8,10 +8,16 @@ class CommandsPruneTest < ActiveSupport::TestCase
     }
   end
 
-  test "images" do
+  test "dangling images" do
     assert_equal \
       "docker image prune --force --filter label=service=app --filter dangling=true",
-      new_command.images.join(" ")
+      new_command.dangling_images.join(" ")
+  end
+
+  test "tagged images" do
+    assert_equal \
+      "docker image ls --filter label=service=app --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w \"$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=app | tr -d '\\n')dhh/app:latest\\|dhh/app:<none>\" | while read image tag; do docker rmi $tag; done",
+      new_command.tagged_images.join(" ")
   end
 
   test "containers" do

test/integration/docker/deployer/app/.mrsk/hooks/pre-deploy

@@ -0,0 +1,3 @@
+#!/bin/sh
+echo "Deployed!"
+mkdir -p /tmp/${TEST_ID} && touch /tmp/${TEST_ID}/pre-deploy

test/integration/docker/load_balancer/default.conf

@@ -8,9 +8,9 @@ server {
 
   location / {
     proxy_pass http://loadbalancer;
-    proxy_connect_timeout       5;
+    proxy_connect_timeout       10;
-    proxy_send_timeout          5;
+    proxy_send_timeout          10;
-    proxy_read_timeout          5;
+    proxy_read_timeout          10;
-    send_timeout                5;
+    send_timeout                10;
   }
 }

test/integration/integration_test.rb

@@ -54,7 +54,7 @@ class IntegrationTest < ActiveSupport::TestCase
       assert_equal "404", response.code
     end
 
-    def wait_for_app_to_be_up(timeout: 10, up_count: 3)
+    def wait_for_app_to_be_up(timeout: 20, up_count: 3)
       timeout_at = Time.now + timeout
       up_times = 0
       response = app_response

test/integration/main_test.rb

@@ -8,16 +8,16 @@ class MainTest < IntegrationTest
 
     mrsk :deploy
     assert_app_is_up version: first_version
-    assert_hooks_ran "pre-connect", "pre-build", "post-deploy"
+    assert_hooks_ran "pre-connect", "pre-build", "pre-deploy", "post-deploy"
 
     second_version = update_app_rev
 
     mrsk :redeploy
     assert_app_is_up version: second_version
-    assert_hooks_ran "pre-connect", "pre-build", "post-deploy"
+    assert_hooks_ran "pre-connect", "pre-build", "pre-deploy", "post-deploy"
 
     mrsk :rollback, first_version
-    assert_hooks_ran "pre-connect", "post-deploy"
+    assert_hooks_ran "pre-connect", "pre-deploy", "post-deploy"
     assert_app_is_up version: first_version
 
     details = mrsk :details, capture: true