Bump version for 0.7.0

Add healthcheck before deploy

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    mrsk (0.6.2)
+    mrsk (0.7.0)
       activesupport (>= 7.0)
       dotenv (~> 2.8)
       sshkit (~> 1.21)

README.md

@@ -1,6 +1,6 @@
 # MRSK
 
-MRSK deploys web apps in containers to servers running Docker with zero downtime. It uses the dynamic reverse-proxy Traefik to hold requests while the new application container is started and the old one is stopped. It works seamlessly across multiple hosts, using SSHKit to execute commands.
+MRSK deploys web apps in containers to servers running Docker with zero downtime. It uses the dynamic reverse-proxy Traefik to hold requests while the new application container is started and the old one is stopped. It works seamlessly across multiple hosts, using SSHKit to execute commands. It was built for Rails applications, but works with any type of web app that can be bundled with Docker.
 
 ## Installation
 
@@ -20,7 +20,7 @@ env:
     - RAILS_MASTER_KEY
 ```
 
-Then edit your `.env` file to add your registry password as `MRSK_REGISTRY_PASSWORD` (and your `RAILS_MASTER_KEY` for production with a Rails app).
+Then edit your `.env` file to add your registry password as `MRSK_REGISTRY_PASSWORD` (and your `RAILS_MASTER_KEY` for production with a Rails app). 
 
 Now you're ready to deploy to the servers:
 
@@ -37,12 +37,23 @@ This will:
 5. Push the image to the registry.
 6. Pull the image from the registry on the servers.
 7. Ensure Traefik is running and accepting traffic on port 80.
-8. Stop any containers running a previous versions of the app.
-9. Start a new container with the version of the app that matches the current git version hash.
-10. Prune unused images and stopped containers to ensure servers don't fill up.
+8. Ensure your app responds with `200 OK` to `GET /up`.
+9. Stop any containers running a previous versions of the app.
+10. Start a new container with the version of the app that matches the current git version hash.
+11. Prune unused images and stopped containers to ensure servers don't fill up.
 
 Voila! All the servers are now serving the app on port 80. If you're just running a single server, you're ready to go. If you're running multiple servers, you need to put a load balancer in front of them.
 
+## Vision
+
+In the past decade+, there's been an explosion in commercial offerings that make deploying web apps easier. Heroku kicked it off with an incredible offering that stayed ahead of the competition seemingly forever. These days we have excellent alternatives like Fly.io and Render. And hosted Kubernetes is making things easier too on AWS, GCP, Digital Ocean, and elsewhere. But these are all offerings that have you renting computers in the cloud at a premium. If you want to run on our own hardware, or even just have a clear migration path to do so, you need to carefully consider how locked in you get to these commercial platforms. Preferably before the bills swallow your business whole!
+
+MRSK seeks to bring the advance in ergonomics pioneered by these commercial offerings to deploying web apps anywhere. Whether that's low-cost cloud options without the managed-service markup from the likes of Digital Ocean, Hetzner, OVH, etc, or it's your own colocated metal. To MRSK, it's all the same. Feed the config file a list of IP addresses with vanilla Ubuntu servers that have seen no prep beyond an added SSH key, and you'll be running in literally minutes.
+
+This structure also gives you enormous portability. You can have your web app deployed on several clouds at ease like this. Or you can buy the baseline with your own hardware, then deploy to a cloud before a big seasonal spike to get more capacity. When you're not locked into a single provider from a tooling perspective, there's a lot of compelling options available.
+
+Ultimately, MRSK is meant to compress the complexity of going to production using open source tooling that isn't tied to any commercial offering. Not to zero, though. You're probably still better off with a fully managed service if basic Linux or Docker is still difficult, but from an early stage when those concepts are familiar.
+
 ## Why not just run Capistrano, Kubernetes or Docker Swarm?
 
 MRSK basically is Capistrano for Containers, which allow us to use vanilla servers as the hosts. No need to ensure that the servers have just the right version of Ruby or other dependencies you need. That all lives in the Docker image now. You can boot a brand new Ubuntu (or whatever) server, add it to the deploy servers of MRSK, and it'll be auto-provisioned with Docker, and run right away. Docker's layer caching also allows for quicker deployments with less mucking about on the server. And the images built for MRSK can be used for CI or later introspection.
@@ -173,10 +184,10 @@ You can specialize the default Traefik rules by setting labels on the containers
 
 ```
 labels:
-  traefik.http.routers.hey.rule: '''Host(`app.hey.com`)'''
+  traefik.http.routers.hey.rule: Host(\`app.hey.com\`)
 ```
 
-Note: The extra quotes are needed to ensure the rule is passed in correctly!
+Note: The escaped backticks are needed to ensure the rule is passed in correctly and not treated as command substitution by Bash!
 
 This allows you to run multiple applications on the same server sharing the same Traefik instance and port.
 See https://doc.traefik.io/traefik/routing/routers/#rule for a full list of available routing rules.
@@ -253,11 +264,11 @@ This build secret can then be referenced in the Dockerfile:
 # Copy Gemfiles
 COPY Gemfile Gemfile.lock ./
 
-# Install dependencies, including private repositories via access token (then remove git configs with exposed GITHUB_TOKEN)
+# Install dependencies, including private repositories via access token (then remove bundle cache with exposed GITHUB_TOKEN)
 RUN --mount=type=secret,id=GITHUB_TOKEN \
   BUNDLE_GITHUB__COM=x-access-token:$(cat /run/secrets/GITHUB_TOKEN) \
   bundle install && \
-  find /usr/local/bundle/cache/bundler/git -name "config" -delete
+  rm -rf /usr/local/bundle/cache
 ```
 
 ### Using command arguments for Traefik
@@ -266,12 +277,13 @@ You can customize the traefik command line:
 
 ```yaml
 traefik:
-  accesslog: true
-  accesslog.format: json
-  metrics.prometheus: true
-  metrics.prometheus.buckets: 0.1,0.3,1.2,5.0
+  args:
+    accesslog: true
+    accesslog.format: json
 ```
 
+This will start the traefik container with `--accesslog=true accesslog.format=json`.
+
 ### Configuring build args for new images
 
 Build arguments that aren't secret can also be configured:
@@ -334,6 +346,41 @@ This template can safely be checked into git. Then everyone deploying the app ca
 
 If you need separate env variables for different destinations, you can set them with `.env.destination.erb` for the template, which will generate `.env.staging` when run with `mrsk envify -d staging`.
 
+### Using audit broadcasts
+
+If you'd like to broadcast audits of deploys, rollbacks, etc to a chatroom or elsewhere, you can configure the `audit_broadcast_cmd` setting with the path to a bin file that reads the audit line from STDIN, and then does whatever with it:
+
+```yaml
+audit_broadcast_cmd:
+  bin/audit_broadcast
+```
+
+The broadcast command could look something like:
+
+```bash
+#!/usr/bin/env bash
+read
+curl -q -d content="[My app] ${REPLY}" https://3.basecamp.com/XXXXX/integrations/XXXXX/buckets/XXXXX/chats/XXXXX/lines
+```
+
+That'll post a line like follows to a preconfigured chatbot in Basecamp:
+
+```
+[My App] [2023-02-18 11:29:52] [dhh] Rolled back to version d264c4e92470ad1bd18590f04466787262f605de
+```
+
+### Using custom healthcheck path or port
+
+MRSK defaults to checking the health of your application again `/up` on port 3000. You can tailor both with the `healthcheck` setting:
+
+```yaml
+healthcheck:
+  path: /healthz
+  port: 4000
+```
+
+This will ensure your application is configured with a traefik label for the healthcheck against `/healthz` and that the pre-deploy healthcheck that MRSK performs is done against the same path on port 4000.
+
 ## Commands
 
 ### Running commands on servers

lib/mrsk/cli/accessory.rb

@@ -9,9 +9,11 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
         upload(name)
 
         on(accessory.host) do
-          execute *MRSK.auditor.record("accessory #{name} boot"), verbosity: :debug
+          execute *MRSK.auditor.record("Booted #{name} accessory"), verbosity: :debug
           execute *accessory.run
         end
+
+        audit_broadcast "Booted accessory #{name}"
       end
     end
   end
@@ -20,8 +22,6 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
   def upload(name)
     with_accessory(name) do |accessory|
       on(accessory.host) do
-        execute *MRSK.auditor.record("accessory #{name} upload files"), verbosity: :debug
-
         accessory.files.each do |(local, remote)|
           accessory.ensure_local_file_present(local)
 
@@ -37,8 +37,6 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
   def directories(name)
     with_accessory(name) do |accessory|
       on(accessory.host) do
-        execute *MRSK.auditor.record("accessory #{name} create directories"), verbosity: :debug
-
         accessory.directories.keys.each do |host_path|
           execute *accessory.make_directory(host_path)
         end
@@ -59,7 +57,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
   def start(name)
     with_accessory(name) do |accessory|
       on(accessory.host) do
-        execute *MRSK.auditor.record("accessory #{name} start"), verbosity: :debug
+        execute *MRSK.auditor.record("Started #{name} accessory"), verbosity: :debug
         execute *accessory.start
       end
     end
@@ -69,7 +67,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
   def stop(name)
     with_accessory(name) do |accessory|
       on(accessory.host) do
-        execute *MRSK.auditor.record("accessory #{name} stop"), verbosity: :debug
+        execute *MRSK.auditor.record("Stopped #{name} accessory"), verbosity: :debug
         execute *accessory.stop, raise_on_non_zero_exit: false
       end
     end
@@ -111,14 +109,14 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
       when options[:reuse]
         say "Launching command from existing container...", :magenta
         on(accessory.host) do
-          execute *MRSK.auditor.record("accessory #{name} cmd '#{cmd}'"), verbosity: :debug
+          execute *MRSK.auditor.record("Executed cmd '#{cmd}' on #{name} accessory"), verbosity: :debug
           capture_with_info(*accessory.execute_in_existing_container(cmd))
         end
 
       else
         say "Launching command from new container...", :magenta
         on(accessory.host) do
-          execute *MRSK.auditor.record("accessory #{name} cmd '#{cmd}'"), verbosity: :debug
+          execute *MRSK.auditor.record("Executed cmd '#{cmd}' on #{name} accessory"), verbosity: :debug
           capture_with_info(*accessory.execute_in_new_container(cmd))
         end
       end
@@ -169,7 +167,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
   def remove_container(name)
     with_accessory(name) do |accessory|
       on(accessory.host) do
-        execute *MRSK.auditor.record("accessory #{name} remove container"), verbosity: :debug
+        execute *MRSK.auditor.record("Remove #{name} accessory container"), verbosity: :debug
         execute *accessory.remove_container
       end
     end
@@ -179,7 +177,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
   def remove_image(name)
     with_accessory(name) do |accessory|
       on(accessory.host) do
-        execute *MRSK.auditor.record("accessory #{name} remove image"), verbosity: :debug
+        execute *MRSK.auditor.record("Removed #{name} accessory image"), verbosity: :debug
         execute *accessory.remove_image
       end
     end
@@ -189,7 +187,6 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
   def remove_service_directory(name)
     with_accessory(name) do |accessory|
       on(accessory.host) do
-        execute *MRSK.auditor.record("accessory #{name} remove service directory"), verbosity: :debug
         execute *accessory.remove_service_directory
       end
     end

lib/mrsk/cli/app.rb

@@ -7,7 +7,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
 
       MRSK.config.roles.each do |role|
         on(role.hosts) do |host|
-          execute *MRSK.auditor.record("app boot version #{version}"), verbosity: :debug
+          execute *MRSK.auditor.record("Booted app version #{version}"), verbosity: :debug
 
           begin
             execute *MRSK.app.stop, raise_on_non_zero_exit: false
@@ -15,7 +15,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
           rescue SSHKit::Command::Failed => e
             if e.message =~ /already in use/
               error "Rebooting container with same version already deployed on #{host}"
-              execute *MRSK.auditor.record("app rebooted with version #{version}"), verbosity: :debug
+              execute *MRSK.auditor.record("Rebooted app version #{version}"), verbosity: :debug
 
               execute *MRSK.app.remove_container(version: version)
               execute *MRSK.app.run(role: role.name)
@@ -31,7 +31,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
   desc "start", "Start existing app on servers (use --version=<git-hash> to designate specific version)"
   def start
     on(MRSK.hosts) do
-      execute *MRSK.auditor.record("app start version #{MRSK.version}"), verbosity: :debug
+      execute *MRSK.auditor.record("Started app version #{MRSK.version}"), verbosity: :debug
       execute *MRSK.app.start, raise_on_non_zero_exit: false
     end
   end
@@ -39,7 +39,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
   desc "stop", "Stop app on servers"
   def stop
     on(MRSK.hosts) do
-      execute *MRSK.auditor.record("app stop"), verbosity: :debug
+      execute *MRSK.auditor.record("Stopped app"), verbosity: :debug
       execute *MRSK.app.stop, raise_on_non_zero_exit: false
     end
   end
@@ -74,7 +74,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
         say "Launching command with version #{version} from existing container...", :magenta
 
         on(MRSK.hosts) do |host|
-          execute *MRSK.auditor.record("app cmd '#{cmd}' with version #{version}"), verbosity: :debug
+          execute *MRSK.auditor.record("Executed cmd '#{cmd}' on app version #{version}"), verbosity: :debug
           puts_by_host host, capture_with_info(*MRSK.app.execute_in_existing_container(cmd))
         end
       end
@@ -84,7 +84,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
       using_version(options[:version] || most_recent_version_available) do |version|
         say "Launching command with version #{version} from new container...", :magenta
         on(MRSK.hosts) do |host|
-          execute *MRSK.auditor.record("app cmd '#{cmd}' with version #{version}"), verbosity: :debug
+          execute *MRSK.auditor.record("Executed cmd '#{cmd}' on app version #{version}"), verbosity: :debug
           puts_by_host host, capture_with_info(*MRSK.app.execute_in_new_container(cmd))
         end
       end
@@ -140,7 +140,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
   desc "remove_container [VERSION]", "Remove app container with given version from servers"
   def remove_container(version)
     on(MRSK.hosts) do
-      execute *MRSK.auditor.record("app remove container #{version}"), verbosity: :debug
+      execute *MRSK.auditor.record("Removed app container with version #{version}"), verbosity: :debug
       execute *MRSK.app.remove_container(version: version)
     end
   end
@@ -148,7 +148,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
   desc "remove_containers", "Remove all app containers from servers"
   def remove_containers
     on(MRSK.hosts) do
-      execute *MRSK.auditor.record("app remove containers"), verbosity: :debug
+      execute *MRSK.auditor.record("Removed all app containers"), verbosity: :debug
       execute *MRSK.app.remove_containers
     end
   end
@@ -156,7 +156,7 @@ class Mrsk::Cli::App < Mrsk::Cli::Base
   desc "remove_images", "Remove all app images from servers"
   def remove_images
     on(MRSK.hosts) do
-      execute *MRSK.auditor.record("app remove images"), verbosity: :debug
+      execute *MRSK.auditor.record("Removed all app images"), verbosity: :debug
       execute *MRSK.app.remove_images
     end
   end

lib/mrsk/cli/base.rb

@@ -59,9 +59,14 @@ module Mrsk::Cli
       def print_runtime
         started_at = Time.now
         yield
+        return Time.now - started_at
       ensure
         runtime = Time.now - started_at
         puts "  Finished all in #{sprintf("%.1f seconds", runtime)}"
       end
+
+      def audit_broadcast(line)
+        run_locally { execute *MRSK.auditor.broadcast(line), verbosity: :debug }
+      end
   end
 end

lib/mrsk/cli/build.rb

@@ -29,7 +29,7 @@ class Mrsk::Cli::Build < Mrsk::Cli::Base
   desc "pull", "Pull app image from the registry onto servers"
   def pull
     on(MRSK.hosts) do
-      execute *MRSK.auditor.record("build pull image #{MRSK.version}"), verbosity: :debug
+      execute *MRSK.auditor.record("Pulled image with version #{MRSK.version}"), verbosity: :debug
       execute *MRSK.builder.pull
     end
   end

lib/mrsk/cli/healthcheck.rb

@@ -0,0 +1,29 @@
+class Mrsk::Cli::Healthcheck < Mrsk::Cli::Base
+  desc "perform", "Health check the current version of the app"
+  def perform
+    on(MRSK.primary_host) do
+      begin
+        execute *MRSK.healthcheck.run
+
+        target = "Health check against #{MRSK.config.healthcheck["path"]}"
+
+        if capture_with_info(*MRSK.healthcheck.curl) == "200"
+          info "#{target} succeeded with 200 OK!"
+        else
+          # Catches 1xx, 2xx, 3xx
+          raise SSHKit::Command::Failed, "#{target} failed to return 200 OK!"
+        end
+      rescue SSHKit::Command::Failed => e
+        if e.message =~ /curl/
+          # Catches 4xx, 5xx
+          raise SSHKit::Command::Failed, "#{target} failed to return 200 OK!"
+        else
+          raise
+        end
+      ensure
+        execute *MRSK.healthcheck.stop, raise_on_non_zero_exit: false
+        execute *MRSK.healthcheck.remove, raise_on_non_zero_exit: false
+      end
+    end
+  end
+end

lib/mrsk/cli/main.rb

@@ -10,7 +10,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
 
   desc "deploy", "Deploy the app to servers"
   def deploy
-    print_runtime do
+    runtime = print_runtime do
       say "Ensure Docker is installed...", :magenta
       invoke "mrsk:cli:server:bootstrap"
 
@@ -23,33 +23,48 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
       say "Ensure Traefik is running...", :magenta
       invoke "mrsk:cli:traefik:boot"
 
+      say "Ensure app can pass healthcheck...", :magenta
+      invoke "mrsk:cli:healthcheck:perform"
+
       invoke "mrsk:cli:app:boot"
 
       say "Prune old containers and images...", :magenta
       invoke "mrsk:cli:prune:all"
     end
+
+    audit_broadcast "Deployed app in #{runtime.to_i} seconds"
   end
 
   desc "redeploy", "Deploy new version of the app to servers (without bootstrapping servers, starting Traefik, pruning, and registry login)"
   def redeploy
-    print_runtime do
+    runtime = print_runtime do
       say "Build and push app image...", :magenta
       invoke "mrsk:cli:build:deliver"
 
+      say "Ensure app can pass healthcheck...", :magenta
+      invoke "mrsk:cli:healthcheck:perform"
+
       invoke "mrsk:cli:app:boot"
     end
+
+    audit_broadcast "Redeployed app in #{runtime.to_i} seconds"
   end
 
   desc "rollback [VERSION]", "Rollback the app to VERSION"
   def rollback(version)
     MRSK.version = version
 
-    cli = self
+    if container_name_available?(MRSK.config.service_with_version)
+      say "Stop current version, then start version #{version}...", :magenta
 
-    cli.say "Stop current version, then start version #{version}...", :magenta
-    on(MRSK.hosts) do
-      execute *MRSK.app.stop, raise_on_non_zero_exit: false
-      execute *MRSK.app.start
+      on(MRSK.hosts) do |host|
+        execute *MRSK.app.stop, raise_on_non_zero_exit: false
+        execute *MRSK.app.start
+      end
+
+      audit_broadcast "Rolled back app to version #{version}"
+    else
+      say "The app version '#{version}' is not available as a container (use 'mrsk app containers' for available versions)", :red
     end
   end
 
@@ -107,10 +122,14 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
   desc "envify", "Create .env by evaluating .env.erb (or .env.staging.erb -> .env.staging when using -d staging)"
   def envify
     if destination = options[:destination]
-      File.write(".env.#{destination}", ERB.new(IO.read(Pathname.new(File.expand_path(".env.#{destination}.erb")))).result)
+      env_template_path = ".env.#{destination}.erb"
+      env_path          = ".env.#{destination}"
     else
-      File.write(".env", ERB.new(IO.read(Pathname.new(File.expand_path(".env.erb")))).result)
+      env_template_path = ".env.erb"
+      env_path          = ".env"
     end
+
+    File.write(env_path, ERB.new(File.read(env_template_path)).result, perm: 0600)
   end
 
   desc "remove", "Remove Traefik, app, and registry session from servers"
@@ -134,6 +153,9 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
   desc "build", "Build the application image"
   subcommand "build", Mrsk::Cli::Build
 
+  desc "healthcheck", "Healthcheck the application"
+  subcommand "healthcheck", Mrsk::Cli::Healthcheck
+
   desc "prune", "Prune old application images and containers"
   subcommand "prune", Mrsk::Cli::Prune
 
@@ -145,4 +167,11 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
 
   desc "traefik", "Manage the Traefik load balancer"
   subcommand "traefik", Mrsk::Cli::Traefik
+
+  private
+    def container_name_available?(container_name, host: MRSK.primary_host)
+      container_names = nil
+      on(host) { container_names = capture_with_info(*MRSK.app.list_container_names).split("\n") }
+      Array(container_names).include?(container_name)
+    end
 end

lib/mrsk/cli/prune.rb

@@ -5,10 +5,10 @@ class Mrsk::Cli::Prune < Mrsk::Cli::Base
     invoke :images
   end
 
-  desc "images", "Prune unused images older than 30 days"
+  desc "images", "Prune unused images older than 7 days"
   def images
     on(MRSK.hosts) do
-      execute *MRSK.auditor.record("prune images"), verbosity: :debug
+      execute *MRSK.auditor.record("Pruned images"), verbosity: :debug
       execute *MRSK.prune.images
     end
   end
@@ -16,7 +16,7 @@ class Mrsk::Cli::Prune < Mrsk::Cli::Base
   desc "containers", "Prune stopped containers for the service older than 3 days"
   def containers
     on(MRSK.hosts) do
-      execute *MRSK.auditor.record("prune containers"), verbosity: :debug
+      execute *MRSK.auditor.record("Pruned containers"), verbosity: :debug
       execute *MRSK.prune.containers
     end
   end

lib/mrsk/cli/traefik.rb

@@ -14,7 +14,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
   desc "start", "Start existing Traefik on servers"
   def start
     on(MRSK.traefik_hosts) do
-      execute *MRSK.auditor.record("traefik start"), verbosity: :debug
+      execute *MRSK.auditor.record("Started traefik"), verbosity: :debug
       execute *MRSK.traefik.start, raise_on_non_zero_exit: false
     end
   end
@@ -22,7 +22,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
   desc "stop", "Stop Traefik on servers"
   def stop
     on(MRSK.traefik_hosts) do
-      execute *MRSK.auditor.record("traefik stop"), verbosity: :debug
+      execute *MRSK.auditor.record("Stopped traefik"), verbosity: :debug
       execute *MRSK.traefik.stop, raise_on_non_zero_exit: false
     end
   end
@@ -72,7 +72,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
   desc "remove_container", "Remove Traefik container from servers"
   def remove_container
     on(MRSK.traefik_hosts) do
-      execute *MRSK.auditor.record("traefik remove container"), verbosity: :debug
+      execute *MRSK.auditor.record("Removed traefik container"), verbosity: :debug
       execute *MRSK.traefik.remove_container
     end
   end
@@ -80,7 +80,7 @@ class Mrsk::Cli::Traefik < Mrsk::Cli::Base
   desc "remove_container", "Remove Traefik image from servers"
   def remove_image
     on(MRSK.traefik_hosts) do
-      execute *MRSK.auditor.record("traefik remove image"), verbosity: :debug
+      execute *MRSK.auditor.record("Removed traefik image"), verbosity: :debug
       execute *MRSK.traefik.remove_image
     end
   end

lib/mrsk/commander.rb

@@ -73,6 +73,10 @@ class Mrsk::Commander
     @auditor ||= Mrsk::Commands::Auditor.new(config)
   end
 
+  def healthcheck
+    @healthcheck ||= Mrsk::Commands::Healthcheck.new(config)
+  end
+
 
   def with_verbosity(level)
     old_level = self.verbosity

lib/mrsk/commands/app.rb

@@ -75,10 +75,6 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
     docker :ps, "-q", *service_filter
   end
 
-  def container_id_for(container_name:)
-    docker :container, :ls, "-a", "-f", "name=#{container_name}", "-q"
-  end
-
   def current_running_version
     # FIXME: Find more graceful way to extract the version from "app-version" than using sed and tail!
     pipe \
@@ -93,11 +89,21 @@ class Mrsk::Commands::App < Mrsk::Commands::Base
       "head -n 1"
   end
 
+  def all_versions_from_available_containers
+    pipe \
+      docker(:image, :ls, "--format", '"{{.Tag}}"', config.repository),
+      "head -n 1"
+  end
+
 
   def list_containers
     docker :container, :ls, "-a", *service_filter
   end
 
+  def list_container_names
+    [ *list_containers, "--format", "'{{ .Names }}'" ]
+  end
+
   def remove_container(version:)
     pipe \
       container_id_for(container_name: service_with_version(version)),

lib/mrsk/commands/auditor.rb

@@ -1,12 +1,22 @@
 require "active_support/core_ext/time/conversions"
 
 class Mrsk::Commands::Auditor < Mrsk::Commands::Base
+  # Runs remotely
   def record(line)
     append \
       [ :echo, tagged_line(line) ],
       audit_log_file
   end
 
+  # Runs locally
+  def broadcast(line)
+    if broadcast_cmd = config.audit_broadcast_cmd
+      pipe \
+        [ :echo, tagged_line(line) ],
+        broadcast_cmd
+    end
+  end
+
   def reveal
     [ :tail, "-n", 50, audit_log_file ]
   end
@@ -21,14 +31,14 @@ class Mrsk::Commands::Auditor < Mrsk::Commands::Base
     end
 
     def tags
-      "[#{timestamp}] [#{performer}]"
+      "[#{recorded_at}] [#{performer}]"
     end
 
     def performer
-      `whoami`.strip
+      @performer ||= `whoami`.strip
     end
 
-    def timestamp
+    def recorded_at
       Time.now.to_fs(:db)
     end
 end

lib/mrsk/commands/base.rb

@@ -17,6 +17,10 @@ module Mrsk::Commands
       end
     end
 
+    def container_id_for(container_name:)
+      docker :container, :ls, "-a", "-f", "name=#{container_name}", "-q"
+    end
+
     private
       def combine(*commands, by: "&&")
         commands

lib/mrsk/commands/builder/base.rb

@@ -5,19 +5,27 @@ class Mrsk::Commands::Builder::Base < Mrsk::Commands::Base
     docker :pull, config.absolute_image
   end
 
-  def build_args
-    argumentize "--build-arg", args, redacted: true
-  end
-
-  def build_secrets
-    argumentize "--secret", secrets.collect { |secret| [ "id", secret ] }
-  end
-
-  def build_tags
-    [ "-t", config.absolute_image, "-t", config.latest_image ]
+  def build_options
+    [ *build_tags, *build_labels, *build_args, *build_secrets ]
   end
 
   private
+    def build_tags
+      [ "-t", config.absolute_image, "-t", config.latest_image ]
+    end
+
+    def build_labels
+      argumentize "--label", { service: config.service }
+    end
+
+    def build_args
+      argumentize "--build-arg", args, redacted: true
+    end
+
+    def build_secrets
+      argumentize "--secret", secrets.collect { |secret| [ "id", secret ] }
+    end
+
     def args
       (config.builder && config.builder["args"]) || {}
     end

lib/mrsk/commands/builder/multiarch.rb

@@ -12,9 +12,7 @@ class Mrsk::Commands::Builder::Multiarch < Mrsk::Commands::Builder::Base
       "--push",
       "--platform", "linux/amd64,linux/arm64",
       "--builder", builder_name,
-      *build_tags, 
-      *build_args,
-      *build_secrets,
+      *build_options,
       "."
   end
 

lib/mrsk/commands/builder/native.rb

@@ -9,7 +9,7 @@ class Mrsk::Commands::Builder::Native < Mrsk::Commands::Builder::Base
 
   def push
     combine \
-      docker(:build, *build_tags, *build_args, *build_secrets, "."),
+      docker(:build, *build_options, "."),
       docker(:push, config.absolute_image)
   end
 

lib/mrsk/commands/builder/native/remote.rb

@@ -16,9 +16,7 @@ class Mrsk::Commands::Builder::Native::Remote < Mrsk::Commands::Builder::Native
       "--push",
       "--platform", platform,
       "--builder", builder_name,
-      *build_tags,
-      *build_args,
-      *build_secrets,
+      *build_options,
       "."
   end
 

lib/mrsk/commands/healthcheck.rb

@@ -0,0 +1,46 @@
+class Mrsk::Commands::Healthcheck < Mrsk::Commands::Base
+  EXPOSED_PORT = 3999
+
+  def run
+    web = config.role(:web)
+
+    docker :run,
+      "-d",
+      "--name", container_name_with_version,
+      "-p", "#{EXPOSED_PORT}:#{config.healthcheck["port"]}",
+      "--label", "service=#{container_name}",
+      *web.env_args,
+      *config.volume_args,
+      config.absolute_image,
+      web.cmd
+  end
+
+  def curl
+    [ :curl, "--silent", "--output", "/dev/null", "--write-out", "'%{http_code}'", health_url ]
+  end
+
+  def stop
+    pipe \
+      container_id_for(container_name: container_name),
+      xargs(docker(:stop))
+  end
+
+  def remove
+    pipe \
+      container_id_for(container_name: container_name),
+      xargs(docker(:container, :rm))
+  end
+
+  private
+    def container_name
+      "healthcheck-#{config.service}"
+    end
+
+    def container_name_with_version
+      "healthcheck-#{config.service_with_version}"
+    end
+
+    def health_url
+      "http://localhost:#{EXPOSED_PORT}#{config.healthcheck["path"]}"
+    end
+end

lib/mrsk/commands/prune.rb

@@ -2,15 +2,11 @@ require "active_support/duration"
 require "active_support/core_ext/numeric/time"
 
 class Mrsk::Commands::Prune < Mrsk::Commands::Base
-  PRUNE_IMAGES_AFTER     = 7.days.in_hours.to_i
-  PRUNE_CONTAINERS_AFTER = 3.days.in_hours.to_i
-
-  def images
-    docker :image, :prune, "-f", "--filter", "until=#{PRUNE_IMAGES_AFTER}h"
+  def images(until_hours: 7.days.in_hours.to_i)
+    docker :image, :prune, "--all", "--force", "--filter", "label=service=#{config.service}", "--filter", "until=#{until_hours}h"
   end
 
-  def containers
-    docker :image, :prune, "-f", "--filter", "until=#{PRUNE_IMAGES_AFTER}h"
-    docker :container, :prune, "-f", "--filter", "label=service=#{config.service}", "--filter", "'until=#{PRUNE_CONTAINERS_AFTER}h'"
+  def containers(until_hours: 3.days.in_hours.to_i)
+    docker :container, :prune, "--force", "--filter", "label=service=#{config.service}", "--filter", "until=#{until_hours}h"
   end
 end

lib/mrsk/configuration.rb

@@ -107,6 +107,7 @@ class Mrsk::Configuration
     end
   end
 
+
   def ssh_user
     if raw_config.ssh.present?
       raw_config.ssh["user"] || "root"
@@ -127,6 +128,15 @@ class Mrsk::Configuration
   end
 
 
+  def audit_broadcast_cmd
+    raw_config.audit_broadcast_cmd
+  end
+
+  def healthcheck
+    { "path" => "/up", "port" => 3000 }.merge(raw_config.healthcheck || {})
+  end
+
+
   def valid?
     ensure_required_keys_present && ensure_env_available
   end
@@ -145,7 +155,8 @@ class Mrsk::Configuration
       volume_args: volume_args,
       ssh_options: ssh_options,
       builder: raw_config.builder,
-      accessories: raw_config.accessories
+      accessories: raw_config.accessories,
+      healthcheck: healthcheck
     }.compact
   end
 

lib/mrsk/configuration/role.rb

@@ -59,7 +59,7 @@ class Mrsk::Configuration::Role
       if running_traefik?
         {
           "traefik.http.routers.#{config.service}.rule" => "'PathPrefix(`/`)'",
-          "traefik.http.services.#{config.service}.loadbalancer.healthcheck.path" => "/up",
+          "traefik.http.services.#{config.service}.loadbalancer.healthcheck.path" => config.healthcheck["path"],
           "traefik.http.services.#{config.service}.loadbalancer.healthcheck.interval" => "1s",
           "traefik.http.middlewares.#{config.service}.retry.attempts" => "3",
           "traefik.http.middlewares.#{config.service}.retry.initialinterval" => "500ms"

lib/mrsk/version.rb

@@ -1,3 +1,3 @@
 module Mrsk
-  VERSION = "0.6.2"
+  VERSION = "0.7.0"
 end

test/cli/main_test.rb

@@ -5,4 +5,29 @@ class CliMainTest < CliTestCase
     version = stdouted { Mrsk::Cli::Main.new.version }
     assert_equal Mrsk::VERSION, version
   end
+
+  test "rollback bad version" do
+    run_command("details") # Preheat MRSK const
+
+    run_command("rollback", "nonsense").tap do |output|
+      assert_match /docker container ls -a --filter label=service=app --format '{{ .Names }}'/, output
+      assert_match /The app version 'nonsense' is not available as a container/, output
+    end
+  end
+
+  test "rollback good version" do
+    Mrsk::Cli::Main.any_instance.stubs(:container_name_available?).returns(true)
+
+    run_command("rollback", "123").tap do |output|
+      assert_match /Stop current version, then start version 123/, output
+      assert_match /docker ps -q --filter label=service=app | xargs docker stop/, output
+      assert_match /docker start app-123/, output
+    end
+  end
+
+
+  private
+    def run_command(*command)
+      stdouted { Mrsk::Cli::Main.start([*command, "-c", "test/fixtures/deploy_with_accessories.yml"]) }
+    end
 end

test/commands/app_test.rb

@@ -26,6 +26,14 @@ class CommandsAppTest < ActiveSupport::TestCase
       @app.run.join(" ")
   end
 
+  test "run with custom healthcheck path" do
+    @config[:healthcheck] = { "path" => "/healthz" }
+
+    assert_equal \
+      "docker run -d --restart unless-stopped --log-opt max-size=10m --name app-999 -e RAILS_MASTER_KEY=456 --label service=app --label role=web --label traefik.http.routers.app.rule='PathPrefix(`/`)' --label traefik.http.services.app.loadbalancer.healthcheck.path=/healthz --label traefik.http.services.app.loadbalancer.healthcheck.interval=1s --label traefik.http.middlewares.app.retry.attempts=3 --label traefik.http.middlewares.app.retry.initialinterval=500ms dhh/app:999",
+      @app.run.join(" ")
+  end
+
   test "start" do
     assert_equal \
       "docker start app-999",

test/commands/auditor_test.rb

@@ -0,0 +1,27 @@
+require "test_helper"
+
+class CommandsAuditorTest < ActiveSupport::TestCase
+  setup do
+    @config = {
+      service: "app", image: "dhh/app", registry: { "username" => "dhh", "password" => "secret" }, servers: [ "1.1.1.1" ],
+      audit_broadcast_cmd: "bin/audit_broadcast"
+    }
+  end
+
+  test "record" do
+    assert_match \
+      /echo '.* app removed container' >> mrsk-app-audit.log/,
+      new_command.record("app removed container").join(" ")
+  end
+
+  test "broadcast" do
+    assert_match \
+      /echo '.* app removed container' \| bin\/audit_broadcast/,
+      new_command.broadcast("app removed container").join(" ")
+  end
+
+  private
+    def new_command
+      Mrsk::Commands::Auditor.new(Mrsk::Configuration.new(@config, version: "123"))
+    end
+end

test/commands/builder_test.rb

@@ -8,50 +8,68 @@ class CommandsBuilderTest < ActiveSupport::TestCase
   test "target multiarch by default" do
     builder = new_builder_command
     assert_equal "multiarch", builder.name
-    assert_equal [:docker, :buildx, :build, "--push", "--platform", "linux/amd64,linux/arm64", "--builder", "mrsk-app-multiarch", "-t", "dhh/app:123", "-t", "dhh/app:latest", "."], builder.push
+    assert_equal \
+      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch -t dhh/app:123 -t dhh/app:latest --label service=app .",
+      builder.push.join(" ")
   end
 
   test "target native when multiarch is off" do
     builder = new_builder_command(builder: { "multiarch" => false })
     assert_equal "native", builder.name
-    assert_equal [:docker, :build, "-t", "dhh/app:123", "-t", "dhh/app:latest", ".", "&&", :docker, :push, "dhh/app:123"], builder.push
+    assert_equal \
+      "docker build -t dhh/app:123 -t dhh/app:latest --label service=app . && docker push dhh/app:123",
+      builder.push.join(" ")
   end
 
   test "target multiarch remote when local and remote is set" do
     builder = new_builder_command(builder: { "local" => { }, "remote" => { } })
     assert_equal "multiarch/remote", builder.name
-    assert_equal [:docker, :buildx, :build, "--push", "--platform", "linux/amd64,linux/arm64", "--builder", "mrsk-app-multiarch-remote", "-t", "dhh/app:123", "-t", "dhh/app:latest", "."], builder.push
+    assert_equal \
+      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch-remote -t dhh/app:123 -t dhh/app:latest --label service=app .",
+      builder.push.join(" ")
   end
 
   test "target native remote when only remote is set" do
     builder = new_builder_command(builder: { "remote" => { "arch" => "amd64" } })
     assert_equal "native/remote", builder.name
-    assert_equal [:docker, :buildx, :build, "--push", "--platform", "linux/amd64", "--builder", "mrsk-app-native-remote", "-t", "dhh/app:123", "-t", "dhh/app:latest", "."], builder.push
+    assert_equal \
+      "docker buildx build --push --platform linux/amd64 --builder mrsk-app-native-remote -t dhh/app:123 -t dhh/app:latest --label service=app .",
+      builder.push.join(" ")
   end
 
   test "build args" do
     builder = new_builder_command(builder: { "args" => { "a" => 1, "b" => 2 } })
-    assert_equal [ "--build-arg", "a=1", "--build-arg", "b=2" ], builder.target.build_args
+    assert_equal \
+      "-t dhh/app:123 -t dhh/app:latest --label service=app --build-arg a=1 --build-arg b=2",
+      builder.target.build_options.join(" ")
   end
 
   test "build secrets" do
     builder = new_builder_command(builder: { "secrets" => ["token_a", "token_b"] })
-    assert_equal [ "--secret", "id=token_a", "--secret", "id=token_b" ], builder.target.build_secrets
+    assert_equal \
+      "-t dhh/app:123 -t dhh/app:latest --label service=app --secret id=token_a --secret id=token_b",
+      builder.target.build_options.join(" ")
   end
 
   test "native push with build args" do
     builder = new_builder_command(builder: { "multiarch" => false, "args" => { "a" => 1, "b" => 2 } })
-    assert_equal [ :docker, :build, "-t", "dhh/app:123", "-t", "dhh/app:latest", "--build-arg", "a=1", "--build-arg", "b=2", ".", "&&", :docker, :push, "dhh/app:123" ], builder.push
+    assert_equal \
+      "docker build -t dhh/app:123 -t dhh/app:latest --label service=app --build-arg a=1 --build-arg b=2 . && docker push dhh/app:123",
+      builder.push.join(" ")
   end
 
   test "multiarch push with build args" do
     builder = new_builder_command(builder: { "args" => { "a" => 1, "b" => 2 } })
-    assert_equal [ :docker, :buildx, :build, "--push", "--platform", "linux/amd64,linux/arm64", "--builder", "mrsk-app-multiarch", "-t", "dhh/app:123", "-t", "dhh/app:latest", "--build-arg", "a=1", "--build-arg", "b=2", "." ], builder.push
+    assert_equal \
+      "docker buildx build --push --platform linux/amd64,linux/arm64 --builder mrsk-app-multiarch -t dhh/app:123 -t dhh/app:latest --label service=app --build-arg a=1 --build-arg b=2 .",
+      builder.push.join(" ")
   end
 
   test "native push with with build secrets" do
     builder = new_builder_command(builder: { "multiarch" => false, "secrets" => [ "a", "b" ] })
-    assert_equal [ :docker, :build, "-t", "dhh/app:123", "-t", "dhh/app:latest", "--secret", "id=a", "--secret", "id=b", ".", "&&", :docker, :push, "dhh/app:123" ], builder.push
+    assert_equal \
+      "docker build -t dhh/app:123 -t dhh/app:latest --label service=app --secret id=a --secret id=b . && docker push dhh/app:123",
+      builder.push.join(" ")
   end
 
   private

test/commands/healthcheck_test.rb

@@ -0,0 +1,55 @@
+require "test_helper"
+
+class CommandsHealthcheckTest < ActiveSupport::TestCase
+  setup do
+    @config = {
+      service: "app", image: "dhh/app", registry: { "username" => "dhh", "password" => "secret" }, servers: [ "1.1.1.1" ],
+      traefik: { "args" => { "accesslog.format" => "json", "metrics.prometheus.buckets" => "0.1,0.3,1.2,5.0" } }
+    }
+  end
+
+  test "run" do
+    assert_equal \
+      "docker run -d --name healthcheck-app-123 -p 3999:3000 --label service=healthcheck-app dhh/app:123",
+      new_command.run.join(" ")
+  end
+
+  test "run with custom port" do
+    @config[:healthcheck] = { "port" => 3001 }
+
+    assert_equal \
+      "docker run -d --name healthcheck-app-123 -p 3999:3001 --label service=healthcheck-app dhh/app:123",
+      new_command.run.join(" ")
+  end
+
+  test "curl" do
+    assert_equal \
+      "curl --silent --output /dev/null --write-out '%{http_code}' http://localhost:3999/up",
+      new_command.curl.join(" ")
+  end
+
+  test "curl with custom path" do
+    @config[:healthcheck] = { "path" => "/healthz" }
+
+    assert_equal \
+      "curl --silent --output /dev/null --write-out '%{http_code}' http://localhost:3999/healthz",
+      new_command.curl.join(" ")
+  end
+
+  test "stop" do
+    assert_equal \
+      "docker container ls -a -f name=healthcheck-app -q | xargs docker stop",
+      new_command.stop.join(" ")
+  end
+
+  test "remove" do
+    assert_equal \
+      "docker container ls -a -f name=healthcheck-app -q | xargs docker container rm",
+      new_command.remove.join(" ")
+  end
+
+  private
+    def new_command
+      Mrsk::Commands::Healthcheck.new(Mrsk::Configuration.new(@config, version: "123"))
+    end
+end

test/commands/prune_test.rb

@@ -0,0 +1,27 @@
+require "test_helper"
+
+class CommandsPruneTest < ActiveSupport::TestCase
+  setup do
+    @config = {
+      service: "app", image: "dhh/app", registry: { "username" => "dhh", "password" => "secret" }, servers: [ "1.1.1.1" ],
+      traefik: { "args" => { "accesslog.format" => "json", "metrics.prometheus.buckets" => "0.1,0.3,1.2,5.0" } }
+    }
+  end
+
+  test "images" do
+    assert_equal \
+      "docker image prune --all --force --filter label=service=app --filter until=168h",
+      new_command.images.join(" ")
+  end
+
+  test "containers" do
+    assert_equal \
+      "docker container prune --force --filter label=service=app --filter until=72h",
+      new_command.containers.join(" ")
+  end
+
+  private
+    def new_command
+      Mrsk::Commands::Prune.new(Mrsk::Configuration.new(@config, version: "123"))
+    end
+end