Instead of using `kamal envify` to generate the .env file, we now assume
that it will be in place for us.
Options in place of `kamal envify`:
1. Pre-generate the .env file
2. Create the env file in the `.pre-init` hook
3. Log into a secret store/check you are logged in in the pre-init hook
Then use .dotenv command and variable substitution to interpolate the
secrets.
The hook is run before the environment is loaded or the config is
parsed.
This makes it a bit of a special case - it doesn't have the usual
KAMAL_XYZ environment variables, as we haven't parsed the config.
The use case for this is to do auth checking or setup. So for example
we can confirm you are logged in to a secret manager, and then you
can directly call it to load your secrets in the .kamal/.env file
using .dotenv's
[command substitution](https://github.com/bkeepers/dotenv?tab=readme-ov-file#command-substitution).
To avoid conflicts with other tools that use .env files, read the files
from .kamal/ instead.
If there are no matching env files in .kamal/, we'll read from the
project root for now and emit a warning.
Validate the Kamal configuration giving useful warning on errors.
Each section of the configuration has its own config class and a YAML
file containing documented example configuration.
You can run `kamal docs` to see the example configuration, and
`kamal docs <section>` to see the example configuration for a specific
section.
The validation matches the configuration to the example configuration
checking that there are no unknown keys and that the values are of
matching types.
Where there is more complex validation - e.g for envs and servers, we
have custom validators that implement those rules.
Additonally the configuration examples are used to generate the
configuration documentation in the kamal-site repo.
You generate them by running:
```
bundle exec bin/docs <kamal-site-checkout>
```
To speed up deployments, we'll remove the healthcheck step.
This adds some risk to deployments for non-web roles - if they don't
have a Docker healthcheck configured then the only check we do is if
the container is running.
If there is a bad image we might see the container running before it
exits and deploy it. Previously the healthcheck step would have avoided
this by ensuring a web container could boot and serve traffic first.
To mitigate this, we'll add a deployment barrier. Until one of the
primary role containers passes its healthcheck, we'll keep the barrier
up and avoid stopping the containers on the non-primary roles.
It the primary role container fails its healthcheck, we'll close the
barrier and shut down the new containers on the waiting roles.
We also have a new integration test to check we correctly handle a
a broken image. This highlighted that SSHKit's default runner will
stop at the first error it encounters. We'll now have a custom runner
that waits for all threads to finish allowing them to clean up.
Allow hosts to be tagged so we can have host specific env variables.
We might want host specific env variables for things like datacenter
specific tags or testing GC settings on a specific host.
Right now you either need to set up a separate role, or have the app
be host aware.
Now you can define tag env variables and assign those to hosts.
For example:
```
servers:
- 1.1.1.1
- 1.1.1.2: tag1
- 1.1.1.2: tag2
- 1.1.1.3: [ tag1, tag2 ]
env_tags:
tag1:
ENV1: value1
tag2:
ENV2: value2
```
The tag env supports the full env format, allowing you to set secret and
clear values.
Secret and clear env variables have different lifecycles. The clear ones
are part of the repo, so it makes sense to always deploy them with the
rest of the repo.
The secret ones are external so we can't be sure that they are up to
date, therefore they require an explicit push via `envify` or `env push`.
We'll keep the env file, but now it just contains secrets. The clear
values are passed directly to `docker run`.
Add an app with roles to the integration tests. We'll deploy two web
containers and one worker. The worker just sleeps, so we are testing
that the container has booted.
The env check is not needded anymore as all the commands rely on the
env files having already been created remotely.
The only place the env is needed is when running `kamal env push` and
that will still raise an apropriate error.
During deployments both the old and new containers will be active for a
small period of time. There also may be lagging requests for older CSS
and JS after the deployment.
This can lead to 404s if a request for old assets hits a new container
or visa-versa.
This PR makes sure that both sets of assets are available throughout the
deployment from before the new version of the app is booted.
This can be configured by setting the asset path:
```yaml
asset_path: "/rails/public/assets"
```
The process is:
1. We extract the assets out of the container, with docker run, docker
cp, docker stop. Docker run sets the container command to "sleep" so
this needs to be available in the container.
2. We create an asset volume directory on the host for the new version
of the app on the host and copy the assets in there.
3. If there is a previous deployment we also copy the new assets into
its asset volume and copy the older assets into the new asset volume.
4. We start the new container mapping the asset volume over the top of
the container's asset path.
This means the both the old and new versions have replaced the asset
path with a volume containing both sets of assets and should be able
to serve any request during the deployment. The older assets will
continue to be available until the next deployment.
When replacing a container currently we:
1. Boot the new container
2. Wait for it to become healthy
3. Stop the old container
Traefik will send requests to the old container until it notices that it
is unhealthy. But it may have stopped serving requests before that point
which can result in errors.
To get round that the new boot process is:
1. Create a directory with a single file on the host
2. Boot the new container, mounting the cord file into /tmp and
including a check for the file in the docker healthcheck
3. Wait for it to become healthy
4. Delete the healthcheck file ("cut the cord") for the old container
5. Wait for it to become unhealthy and give Traefik a couple of seconds
to notice
6. Stop the old container
The extra steps ensure that Traefik stops sending requests before the
old container is shutdown.
Setting env variables in the docker arguments requires having them on
the deploy host.
Instead we'll add two new commands `kamal env push` and
`kamal env delete` which will manage copying the environment as .env
files to the remote host.
Docker will pick up the file with `--env-file <path-to-file>`. Env files
will be stored under `<kamal run directory>/env`.
Running `kamal env push` will create env files for each role and
accessory, and traefik if required.
`kamal envify` has been updated to also push the env files.
By avoiding using `kamal envify` and creating the local and remote
secrets manually, you can now avoid accessing secrets needed
for the docker runtime environment locally. You will still need build
secrets.
One thing to note - the Docker doesn't parse the environment variables
in the env file, one result of this is that you can't specify multi-line
values - see https://github.com/moby/moby/issues/12997.
We maybe need to look docker config or docker secrets longer term to get
around this.
Hattip to @kevinmcconnell - this was all his idea.
Set a high idle timeout on the sshkit connection pool. This will
reduce the incidence of re-connection storms when a deployment has been
idle for a while (e.g. when waiting for a docker build).
The default timeout was 30 seconds, so we'll enable keepalives at a
30s interval to match. This is to help prevent connections from being
killed during long idle periods.
Useful for checking the status of CI before deploying. Doing this at
this point in the deployment maximises the parallelisation of building
and running CI.
These replace the custom audit_broadcast_cmd code. An additional env
variable MRSK_RUNTIME is passed to them.
The audit broadcast after booting an accessory has been removed.
Adds hooks to MRSK. Currently just two hooks, pre-build and post-push.
We could break the build and push into two separate commands if we
found the need for post-build and/or pre-push hooks.
Hooks are stored in `.mrsk/hooks`. Running `mrsk init` will now create
that folder and add sample hook scripts.
Hooks returning non-zero exit codes will abort the current command.
Further potential work here:
- We could replace the audit broadcast command with a
post-deploy/post-rollback hook or similar
- Maybe provide pre-command/post-command hooks that run after every
mrsk invocation
- Also look for hooks in `~/.mrsk/hooks`
Add tests for main, app, accessory, traefik and lock commands.
Other commands are generally covered by the main tests.
Also adds some changes to speed up the integration specs:
- Use a persistent volume for the registry so we can push images to to
reuse between runs (also gets around docker hub rate limits)
- Use persistent volume for mrsk gem install, to avoid re-installing
between tests
- Shorter stop wait time
- Shorter connection timeouts on the load balancer
Takes just over 2 minutes to run all tests locally on an M1 Mac
after docker caches are primed.
