Use the --registry, --repository and --image_version options of
`kamal proxy boot_config set` to change the kamal-proxy image used.
We'll still insist that the image version is at least as high as the
minimum.
v3 was recently released which broke the integration tests. Update them
to use the correct config file.
Set the major version to prevent this from happening when v4 is
released.
We cat the options file, append the proxy image and then pass it
to xargs to ensure it handles spaces correctly.
Works better than using eval which can handle spaces but tries
to evaluate things like backticks.
Fixes: https://github.com/basecamp/kamal/issues/1448
Add two new hooks pre-app-boot and post-app-boot. They are analagous
to the pre/post proxy reboot hooks.
If the boot strategy deploys in groups, then the hooks are called once
per group of hosts and `KAMAL_HOSTS` contains a comma delimited list of
the hosts in that group.
If all hosts are deployed to at once, then they are called once with
`KAMAL_HOSTS` containing all the hosts.
It is possible to have pauses between groups of hosts in the boot config,
where this is the case the pause happens after the post-app-boot hook is
called.
Add commands for managing proxy boot config. Since the proxy can be
shared by multiple applications, the configuration doesn't belong in
`config/deploy.yml`.
Instead you can set the config with:
```
Usage:
kamal proxy boot_config <set|get|clear>
Options:
[--publish], [--no-publish], [--skip-publish] # Publish the proxy ports on the host
# Default: true
[--http-port=N] # HTTP port to publish on the host
# Default: 80
[--https-port=N] # HTTPS port to publish on the host
# Default: 443
[--docker-options=option=value option2=value2] # Docker options to pass to the proxy container
```
By default we boot the proxy with `--publish 80:80 --publish 443:443`.
You can stop it from publishing ports, specify different ports and pass
other docker options.
The config is stored in `.kamal/proxy/options` as arguments to be passed
verbatim to docker run.
Where someone wants to set the options in their application they can do
that by calling `kamal proxy boot_config set` in a pre-deploy hook.
There's an example in the integration tests showing how to use this to
front kamal-proxy with Traefik, using an accessory.
Use localhost for app_with_roles and 127.0.0.1 for app. Confirm we can
deploy both and the respond to requests. Ensure the proxy is removed
once both have been removed.
Remove `stop_wait_time` and `readiness_timeout` from the root config
and remove `deploy_timeout` and `drain_timeout` from the proxy config.
Instead we'll just have `deploy_timeout` and `drain_timeout` in the
root config.
For roles that run the proxy, they are passed to the kamal-proxy deploy
command. Once that returns we can assume the container is ready to
shut down.
For other roles, we'll use the `deploy_timeout` when polling the
container to see if it is ready and the `drain_timeout` when stopping
the container.
The proxy can be enabled via the config:
```
proxy:
enabled: true
hosts:
- 10.0.0.1
- 10.0.0.2
```
This will enable the proxy and cause it to be run on the hosts listed
under `hosts`, after running `kamal proxy reboot`.
Enabling the proxy disables `kamal traefik` commands and replaces them
with `kamal proxy` ones. However only the marked hosts will run the
kamal-proxy container, the rest will run Traefik as before.
Add a shared secrets file used across all destinations. Useful for
things Github tokens or registry passwords.
The secrets are added to a new file called `secrets-common` to highlight
they are shared, and to avoid acciedentally inheriting a secret from the
`secrets` file to `secrets.destination`.
Secrets should be interpolated at runtime so we do want the file in git.
But add a warning at the top to avoid adding secrets or git ignore the
file if you do.
Also provide examples of the three options for interpolating secrets.
By default look for the env file in .kamal/env to avoid clashes with
other tools using .env.
For now we'll still load .env and issue a deprecation warning, but in
future we'll stop reading those.
1. Add driver as an option, defaulting to `docker-container`. For a
"native" build you can set it to `docker`
2. Set arch as a array of architectures to build for, defaulting to
`[ "amd64", "arm64" ]` unless you are using the docker driver in
which case we default to not setting a platform
3. Remote is now just a connection string for the remote builder
4. If remote is set, we only use it for non-local arches, if we are
only building for the local arch, we'll ignore it.
Examples:
On arm64, build for arm64 locally, amd64 remotely or
On amd64, build for amd64 locally, arm64 remotely:
```yaml
builder:
remote: ssh://docker@docker-builder
```
On arm64, build amd64 on remote,
On amd64 build locally:
```yaml
builder:
arch:
- amd64
remote:
host: ssh://docker@docker-builder
```
Build amd64 on local:
```yaml
builder:
arch:
- amd64
```
Use docker driver, building for local arch:
```yaml
builder:
driver: docker
```
If you can have an alias like:
```
aliases:
rails: app exec -p rails
```
Then `kamal rails db:migrate:status` will execute
`kamal app exec -p rails db:migrate:status`.
So this works, we'll allow multiple arguments `app exec` and
`server exec` to accept multiple arguments.
The arguments are combined by simply joining them with a space. This
means that these are equivalent:
```
kamal app exec -p rails db:migrate:status
kamal app exec -p "rails db:migrate:status"
```
If you want to pass an argument with spaces, you'll need to quote it:
```
kamal app exec -p "git commit -am \"My comment\""
kamal app exec -p git commit -am "\"My comment\""
```
Aliases are defined in the configuration file under the `aliases` key.
The configuration is a map of alias name to command. When we run the
command the we just do a literal replacement of the alias with the
string.
So if we have:
```yaml
aliases:
console: app exec -r console -i --reuse "rails console"
```
Then running `kamal console -r workers` will run the command
```sh
$ kamal app exec -r console -i --reuse "rails console" -r workers
```
Because of the order Thor parses the arguments, this allows us to
override the role from the alias command.
There might be cases where we need to munge the command a bit more but
that would involve getting into Thor command parsing internals,
which are complicated and possibly subject to change.
There's a chance that your aliases could conflict with future built-in
commands, but there's not likely to be many of those and if it happens
you'll get a validation error when you upgrade.
Thanks to @dhnaranjo for the idea!
The integrations tests use their own registry so avoid hitting docker
hub rate limits.
This was using a self signed certificate but instead use
`--insecure-registry` to let the docker daemon use HTTP.
To speed up deployments, we'll remove the healthcheck step.
This adds some risk to deployments for non-web roles - if they don't
have a Docker healthcheck configured then the only check we do is if
the container is running.
If there is a bad image we might see the container running before it
exits and deploy it. Previously the healthcheck step would have avoided
this by ensuring a web container could boot and serve traffic first.
To mitigate this, we'll add a deployment barrier. Until one of the
primary role containers passes its healthcheck, we'll keep the barrier
up and avoid stopping the containers on the non-primary roles.
It the primary role container fails its healthcheck, we'll close the
barrier and shut down the new containers on the waiting roles.
We also have a new integration test to check we correctly handle a
a broken image. This highlighted that SSHKit's default runner will
stop at the first error it encounters. We'll now have a custom runner
that waits for all threads to finish allowing them to clean up.
Allow hosts to be tagged so we can have host specific env variables.
We might want host specific env variables for things like datacenter
specific tags or testing GC settings on a specific host.
Right now you either need to set up a separate role, or have the app
be host aware.
Now you can define tag env variables and assign those to hosts.
For example:
```
servers:
- 1.1.1.1
- 1.1.1.2: tag1
- 1.1.1.2: tag2
- 1.1.1.3: [ tag1, tag2 ]
env_tags:
tag1:
ENV1: value1
tag2:
ENV2: value2
```
The tag env supports the full env format, allowing you to set secret and
clear values.
Add an app with roles to the integration tests. We'll deploy two web
containers and one worker. The worker just sleeps, so we are testing
that the container has booted.
This allows the user to make any necessary configuration changes to
Docker before setting up any containers, allowing those configuration
changes to take effect from the outset.
Provide pre and post reboot hooks for Traefik, that can be used to
remove/add to an external load balancer to prevent requests from being
sent during the reboot.
Works best with the --rolling setting, where each hook is called once
per host.
During deployments both the old and new containers will be active for a
small period of time. There also may be lagging requests for older CSS
and JS after the deployment.
This can lead to 404s if a request for old assets hits a new container
or visa-versa.
This PR makes sure that both sets of assets are available throughout the
deployment from before the new version of the app is booted.
This can be configured by setting the asset path:
```yaml
asset_path: "/rails/public/assets"
```
The process is:
1. We extract the assets out of the container, with docker run, docker
cp, docker stop. Docker run sets the container command to "sleep" so
this needs to be available in the container.
2. We create an asset volume directory on the host for the new version
of the app on the host and copy the assets in there.
3. If there is a previous deployment we also copy the new assets into
its asset volume and copy the older assets into the new asset volume.
4. We start the new container mapping the asset volume over the top of
the container's asset path.
This means the both the old and new versions have replaced the asset
path with a volume containing both sets of assets and should be able
to serve any request during the deployment. The older assets will
continue to be available until the next deployment.
When replacing a container currently we:
1. Boot the new container
2. Wait for it to become healthy
3. Stop the old container
Traefik will send requests to the old container until it notices that it
is unhealthy. But it may have stopped serving requests before that point
which can result in errors.
To get round that the new boot process is:
1. Create a directory with a single file on the host
2. Boot the new container, mounting the cord file into /tmp and
including a check for the file in the docker healthcheck
3. Wait for it to become healthy
4. Delete the healthcheck file ("cut the cord") for the old container
5. Wait for it to become unhealthy and give Traefik a couple of seconds
to notice
6. Stop the old container
The extra steps ensure that Traefik stops sending requests before the
old container is shutdown.
Setting env variables in the docker arguments requires having them on
the deploy host.
Instead we'll add two new commands `kamal env push` and
`kamal env delete` which will manage copying the environment as .env
files to the remote host.
Docker will pick up the file with `--env-file <path-to-file>`. Env files
will be stored under `<kamal run directory>/env`.
Running `kamal env push` will create env files for each role and
accessory, and traefik if required.
`kamal envify` has been updated to also push the env files.
By avoiding using `kamal envify` and creating the local and remote
secrets manually, you can now avoid accessing secrets needed
for the docker runtime environment locally. You will still need build
secrets.
One thing to note - the Docker doesn't parse the environment variables
in the env file, one result of this is that you can't specify multi-line
values - see https://github.com/moby/moby/issues/12997.
We maybe need to look docker config or docker secrets longer term to get
around this.
Hattip to @kevinmcconnell - this was all his idea.
