The cause message doesn't include the host the error occurred on.
Before:
```
$ mrsk deploy
Acquiring the deploy lock...
Finished all in 0.1 seconds
ERROR (SocketError): getaddrinfo: nodename nor servname provided, or not known
```
After:
```
$ mrsk deploy -d staging
Acquiring the deploy lock...
Finished all in 0.1 seconds
ERROR (SocketError): Exception while executing on host server-123: getaddrinfo: nodename nor servname provided, or not known
```
Add tests for main, app, accessory, traefik and lock commands.
Other commands are generally covered by the main tests.
Also adds some changes to speed up the integration specs:
- Use a persistent volume for the registry so we can push images to to
reuse between runs (also gets around docker hub rate limits)
- Use persistent volume for mrsk gem install, to avoid re-installing
between tests
- Shorter stop wait time
- Shorter connection timeouts on the load balancer
Takes just over 2 minutes to run all tests locally on an M1 Mac
after docker caches are primed.
The code in Mrsk::Cli::Main#rollback was very similar to
Mrsk::Cli::App#boot.
Modify Mrsk::Cli::App#boot so it can handle rollbacks by:
1. Only renaming running containers
2. Trying first to start then run the new container
If there are uncommitted changes in the app repository when building,
then append `_uncommitted_<random>` to it to distinguish the image
from one built from a clean checkout.
Also change the version used when renaming a container on redeploy to
distinguish and explain the version suffixes.
In the image prune command --all overrides --dangling=true. This removes
the image git sha image tag for the latest image which prevented
us from rolling back to it.
I've updated the integration test to now test deploy, redeploy and
rollback.
Time based container and image retention can have variable space
requirements depending on how often we deploy.
- Only prune stopped containers, retaining the 5 newest
- Then prune dangling images so we only keep images for the retained
containers.
Hosts could end up out of sync with each other if prune commands are run
manually or when new hosts are added.
Before rolling back confirm that the required container is available on
all hosts and roles.
Getting the lock status with invoke passes through any options from the
original command which will raise an exception if they are not also
valid for the lock status command.
Fixes https://github.com/mrsked/mrsk/issues/239
Rather than waiting 5 seconds and hoping for the best after we boot
docker compose, add docker healthchecks and wait for all the containers
to be healthy.
Adds a simple integration test to ensure that `mrsk deploy` works.
Everything required is spun up with docker compose:
- shared: a container that contains an ssh key and a self signed cert to
be shared between the images
- deployer: the image we will deploy from
- registry: a docker registry
- two vm images to deploy into
- load_balancer: an nginx load balancer to use between our images
The other images are in privileged mode so that we can run
docker-in-docker. We need to run docker inside the images - mapping in
the docker socket doesn't work because both VMs would share the host
daemon.
The docker registry requires a self signed cert as you cannot use basic
auth over HTTP except on localhost. It runs on port 4443 rather than 443
because docker refused to accept that "registry" is a docker host and
tries to push images to docker.io/registry. "registry:4443" works fine.
The shared container contains the ssh keys for the deployer and vms, and
the self signed cert for the registry. When the shared container boots,
it copies them into a shared volume.
The other deployer and vm images are built with soft links from the
shared volume to the require locations. Their boot scripts wait for the
files to be copied in before continuing.
The root mrsk folder is mapped into the deployer container. On boot it
builds the gem and installs it.
Right now there's just a single test. We confirm that the load balancer
is returning a 502, run `mrsk deploy` and then confirm it returns 200.
If we get an error we'll only hold the deploy lock if it occurs while
trying to switch the running containers.
We'll also move tagging the latest image from when the image is pulled
to just before the container switch. This ensures that earlier errors
don't leave the hosts with an updated latest tag while still running the
older version.
Rollbacks stopped working after https://github.com/mrsked/mrsk/pull/99.
We'll confirm that a container is available for the first role on the
primary host before attempting to rollback.
1. Don't raise lock error for non-lock issues during lock acquire
(see https://github.com/mrsked/mrsk/pull/181)
2. If there is an error while the lock is held, don't release the lock
and send a warning to stderr
Allow the hosts for accessories to be specified by host or role, or on
all app hosts by setting `daemon: true`.
```
# Single host
mysql:
host: 1.1.1.1
# Multiple hosts
redis:
hosts:
- 1.1.1.1
- 1.1.1.2
# By role
monitoring:
roles:
- web
- jobs
```
When deploying check if there is already a container with the existing
name. If there is rename it to "<version>_<random_hex_string>" to remove
the name clash with the new container we want to boot.
We can then do the normal zero downtime run/wait/stop.
While implementing this I discovered the --filter name=foo does a
substring match for foo, so I've updated those filters to do an exact
match instead.
Add a deploy lock for commands that are unsafe to run concurrently.
The lock is taken by creating a `mrsk_lock` directory on the primary
host. Details of who took the lock are added to a details file in that
directory.
Additional CLI commands have been added to manual release and acquire
the lock and to check its status.
```
Commands:
mrsk lock acquire -m, --message=MESSAGE # Acquire the deploy lock
mrsk lock help [COMMAND] # Describe subcommands or one specific subcommand
mrsk lock release # Release the deploy lock
mrsk lock status # Report lock status
Options:
-v, [--verbose], [--no-verbose] # Detailed logging
-q, [--quiet], [--no-quiet] # Minimal logging
[--version=VERSION] # Run commands against a specific app version
-p, [--primary], [--no-primary] # Run commands only on primary host instead of all
-h, [--hosts=HOSTS] # Run commands on these hosts instead of all (separate by comma)
-r, [--roles=ROLES] # Run commands on these roles instead of all (separate by comma)
-c, [--config-file=CONFIG_FILE] # Path to config file
# Default: config/deploy.yml
-d, [--destination=DESTINATION] # Specify destination to be used for config file (staging -> deploy.staging.yml)
-B, [--skip-broadcast], [--no-skip-broadcast] # Skip audit broadcasts
```
If we add support for running multiple deployments on a single server
we'll need to extend the locking to lock per deployment.
`docker image ls` doesn't tell us what the latest deployed image is (e.g
if we've rolled back). Pull the latest image tag through to the server
so we can use it instead.
If we don't supply a version when deploying we'll use the result of
docker image ls to decide which image to boot. But that doesn't
necessarily correspond to the one we have just built.
E.g. if you do something like:
```
mrsk deploy # deploys git sha AAAAAAAAAAAAAA
git commit --amend # update the commit message
mrsk deploy # deploys git sha BBBBBBBBBBBBBB
```
In this case running `docker image ls` will give you the same image
twice (because the contents are identical) with tags for both SHAs but
the image we have just built will not be returned first. Maybe the order
is random, but it always seems to come second as far as I have seen.
i.e you'll get something like:
```
REPOSITORY TAG IMAGE ID CREATED SIZE
foo/bar AAAAAAAAAAAAAA 6272349a9619 31 minutes ago 791MB
foo/bar BBBBBBBBBBBBBB 6272349a9619 31 minutes ago 791MB
```
Since we already know what version we want to deploy from the config,
let's just pass that through.
Donal McBreen