Commit Graph

12 Commits

Author SHA1 Message Date
Jberczel
0e19ead37c Capture container health log when unhealthy 2023-05-03 15:03:05 -04:00
Kevin McConnell
df202d6ef4 Move health checks into Docker
Replaces our current host-based HTTP healthchecks with Docker
healthchecks, and adds a new `healthcheck.cmd` config option that can be
used to define a custom health check command. Also removes Traefik's
healthchecks, since they are no longer necessary.

When deploying a container that has a healthcheck defined, we wait for
it to report a healthy status before stopping the old container that it
replaces. Containers that don't have a healthcheck defined continue to
wait for `MRSK.config.readiness_delay`.

There are some pros and cons to using Docker healthchecks rather than
checking from the host. The main advantages are:

- Supports non-HTTP checks, and app-specific check scripts provided by a
  container.
- When booting a container, allows MRSK to wait for a container to be
  healthy before shutting down the old container it replaces. This
  should be safer than relying on a timeout.
- Containers with healthchecks won't be active in Traefik until they
  reach a healthy state, which prevents any traffic from being routed to
  them before they are ready.

The main _disadvantage_ is that containers are now required to provide
some way to check their health. Our default check assumes that `curl` is
available in the container which, while common, won't always be the
case.
2023-04-13 16:08:43 +01:00
Donal McBreen
05488e4c1e Zero downtime redeploys
When deploying check if there is already a container with the existing
name. If there is rename it to "<version>_<random_hex_string>" to remove
the name clash with the new container we want to boot.

We can then do the normal zero downtime run/wait/stop.

While implementing this I discovered the --filter name=foo does a
substring match for foo, so I've updated those filters to do an exact
match instead.
2023-03-24 17:09:20 +00:00
Richard Taylor
3e0b71b631 Fix healthcheck test
Looks like the tests started failing on the options healthcheck PR
after merging the container name env var PR.
2023-03-13 14:51:54 +00:00
David Heinemeier Hansson
20733a4493 Merge pull request #102 from moomerman/cmd-options-for-healthcheck
Use custom web options for healthcheck
2023-03-13 15:12:25 +01:00
Richard Taylor
bb241dea43 Add container name env var for containers
Because the container name is generated it isn't possible to
determine this inside the container.

This adds the MRSK_CONTAINER_NAME env var when running the
container so it can be read by the service running inside the
container.
2023-03-11 10:14:41 +00:00
Richard Taylor
e5c5e89232 Use custom web options for healthcheck
If the web role has custom options, ensure these are used for the
healthcheck.
2023-03-10 15:55:04 +00:00
Tobias Bühlmann
8b913068de Add destination to healthcheck containers names 2023-03-06 16:54:13 +01:00
David Heinemeier Hansson
9d6ccf9889 Changed to a timeout 2023-02-20 17:59:41 +01:00
David Heinemeier Hansson
5898fdd8f4 Expand arguments to be more self-explanatory in logs 2023-02-19 18:11:06 +01:00
David Heinemeier Hansson
e5c4cb0344 Retry healthcheck for up to 10 seconds (in case container wasnt ready) 2023-02-19 15:34:36 +01:00
David Heinemeier Hansson
933ece35ab Add healthcheck before deploy 2023-02-18 16:22:08 +01:00