Compare commits

..

821 Commits

Author SHA1 Message Date
Donal McBreen
6e60ab918a Bump version for 1.6.0 2024-06-03 08:34:12 +01:00
Donal McBreen
90ecb6a12a Merge pull request #821 from basecamp/retry-clone
Handle corrupt git clones
2024-05-28 15:23:36 +01:00
Donal McBreen
2c2053558a Handle corrupt git clones
When cloning the git repo:
1. Try to clone
2. If there's already a build directory reset it
3. Check the clone is valid

If anything goes wrong during that process:
1. Delete the clone directory
2. Clone it again
3. Check the clone is valid

Raise any errors after that
2024-05-27 11:17:34 +01:00
Donal McBreen
10b8c826d8 Merge pull request #755 from basecamp/lock-less
Lock less
2024-05-21 12:33:45 +01:00
Donal McBreen
187861fa60 Space not tab 2024-05-21 12:20:19 +01:00
Donal McBreen
5ff1203c80 Always lock before pre-deploy hook 2024-05-21 12:02:16 +01:00
Donal McBreen
0e73f02743 Split lock and connection setup
Allow run the pre-connect hook before the first SSH command is executed,
but only run the locking in `with_lock` blocks.
2024-05-21 12:02:16 +01:00
Donal McBreen
83d0078525 Confirm outside mutating 2024-05-21 12:02:16 +01:00
Donal McBreen
96ef0fbc4d Fix merge error 2024-05-21 12:02:16 +01:00
Donal McBreen
b12654ccd0 Don't lock until confirmed 2024-05-21 12:02:16 +01:00
Donal McBreen
64f5955444 Don't hold lock on error 2024-05-21 12:02:12 +01:00
Donal McBreen
d2a719998a Building doesn't need a deploy lock 2024-05-21 12:01:08 +01:00
Donal McBreen
6a7c90cf4d Only stopping containers locks 2024-05-21 12:01:07 +01:00
Donal McBreen
2c2d94c6d9 Merge pull request #740 from basecamp/remove-healthcheck-step
Remove the healthcheck step
2024-05-21 12:00:25 +01:00
Donal McBreen
c62bd1dc31 Merge pull request #815 from basecamp/envify-already-pushes-env
Envify already env pushes
2024-05-21 11:57:55 +01:00
Donal McBreen
a83df9e135 Merge pull request #620 from mlitwiniuk/allow_custom_ports_for_ssh
Allow custom user and port for builder host
2024-05-21 11:49:06 +01:00
Donal McBreen
7b55f4734e Envify already env pushes
`kamal envify` will do `kamal env push` for us, so no need to call it
ourselves during setup.
2024-05-21 11:47:51 +01:00
Donal McBreen
1e296c4140 Update sshkit_with_ext.rb 2024-05-21 11:38:30 +01:00
Donal McBreen
9700e2b3c4 Merge pull request #646 from nickhammond/server/exec
Add in a server exec command for running ad-hoc commands directly on the server
2024-05-21 11:34:20 +01:00
Donal McBreen
706b82baa1 Simplify messages and remove multiple execute error 2024-05-21 10:40:01 +01:00
Donal McBreen
fa7e941648 Make SSHKit::Runner::Parallel fail slow
Using a different SSHKit runner doesn't work well, because the group
runner uses the Parallel runner internally. So instead we'll patch its
behaviour to fail slow.

We'll also get it to return all the errors so we can report on all the
hosts that failed.
2024-05-21 09:33:22 +01:00
Donal McBreen
78c0a0ba4b Don't start other roles we have a healthy container
If a primary role container is unhealthy, we might take a while to
timeout the health check poller. In the meantime if we have started the
other roles, they'll be running tow containers.

This could be a problem, especially if they read run jobs as that
doubles the worker capacity which could cause exessive load.

We'll wait for the first primary role container to boot successfully
before starting the other containers from other roles.
2024-05-21 08:37:36 +01:00
Donal McBreen
060e5d2027 Update lib/kamal/cli/server.rb
Co-authored-by: Sijawusz Pur Rahnama <sija@sija.pl>
2024-05-21 08:22:20 +01:00
Nick Hammond
8a4f7163bb Apply suggestions from code review
Co-authored-by: Donal McBreen <dmcbreen@gmail.com>
2024-05-20 11:15:14 -07:00
Donal McBreen
ee758d951a Only use barrier when needed, more descriptive info 2024-05-20 12:18:30 +01:00
Donal McBreen
bb2ca81d87 Fix rebase method duplication 2024-05-20 12:18:30 +01:00
Donal McBreen
773ba3a5ab Show container logs and healthcheck status on failure 2024-05-20 12:18:30 +01:00
Donal McBreen
5be6fa3b4e Improve comments 2024-05-20 12:18:30 +01:00
Donal McBreen
07c5658396 Remove redundant method 2024-05-20 12:18:30 +01:00
Donal McBreen
0efb5ccfff Remove the healthcheck step
To speed up deployments, we'll remove the healthcheck step.

This adds some risk to deployments for non-web roles - if they don't
have a Docker healthcheck configured then the only check we do is if
the container is running.

If there is a bad image we might see the container running before it
exits and deploy it. Previously the healthcheck step would have avoided
this by ensuring a web container could boot and serve traffic first.

To mitigate this, we'll add a deployment barrier. Until one of the
primary role containers passes its healthcheck, we'll keep the barrier
up and avoid stopping the containers on the non-primary roles.

It the primary role container fails its healthcheck, we'll close the
barrier and shut down the new containers on the waiting roles.

We also have a new integration test to check we correctly handle a
a broken image. This highlighted that SSHKit's default runner will
stop at the first error it encounters. We'll now have a custom runner
that waits for all threads to finish allowing them to clean up.
2024-05-20 12:18:30 +01:00
Donal McBreen
990f1b4413 Merge pull request #798 from basecamp/git-clone
Build from within a git clone by default
2024-05-20 12:18:07 +01:00
Donal McBreen
da9428f64d Merge pull request #813 from basecamp/handle-no-env-tags
Don't blow up if there are no env tags
2024-05-20 10:58:11 +01:00
Donal McBreen
17dcaccb6a Don't blow up if there are no env tags 2024-05-20 10:50:07 +01:00
Donal McBreen
448349d0e5 Merge pull request #812 from basecamp/sshkit-1.22.2-minimum
Set sshkit minimum version to 1.22.2
2024-05-20 10:39:17 +01:00
Donal McBreen
b6dba57c7d Set sshkit minimum version to 1.22.2
This includes a fix for a bug in the eviction thread that could cause
this error:

```
[ERROR (IOError): Exception while executing on host foo: closed stream]
```

See https://github.com/capistrano/sshkit/pull/534
2024-05-20 10:06:53 +01:00
Donal McBreen
0ea2a2c509 Don't include destination in clone directory
Reusing the clone directory should allow caching of the build context
between deployments to different destinations.
2024-05-20 09:34:42 +01:00
Donal McBreen
307750ff70 Build from within a git clone by default
Docker does not respect the .dockerignore file when building from a tar.

Instead by default we'll make a local clone into a tmp directory and
build from there. Subsequent builds will reset the clone to match the
checkout.

Compared to building directly in the repo, we'll have reproducible
builds.

Compared to using a git archive:
1. .dockerignore is respected
2. We'll have faster builds - docker can be smarter about caching the
build context on subsequent builds from a directory

To build from the repo directly, set the build context to "." in the
config.

If there are uncommitted changes, we'll warn about them either being
included or ignored depending on whether we build from the clone.
2024-05-20 09:30:56 +01:00
Donal McBreen
88947b6a7b Merge pull request #805 from basecamp/env-under-tags
Move env_tags under env key
2024-05-15 11:09:47 +01:00
Donal McBreen
f48c227768 Move env_tags under env key
Instead of:

```
env:
  CLEAR_TAG: untagged
env_tags:
  tag1:
    CLEAR_TAG: tagged
```

We'll have:

```
env:
  clear:
    CLEAR_TAG: untagged
  tags:
    tag1:
      CLEAR_TAG: tagged
```
2024-05-15 10:19:22 +01:00
David Heinemeier Hansson
f98380ef0c Merge pull request #802 from basecamp/envify-during-setup
Envify during setup
2024-05-14 16:00:28 -07:00
David Heinemeier Hansson
0bc27c10cc Fix tests 2024-05-14 11:59:42 -07:00
David Heinemeier Hansson
e58d2f67f2 Fix env template path check and tests 2024-05-14 10:07:31 -07:00
David Heinemeier Hansson
938ac375a1 Only envify if there is a template file available 2024-05-13 17:08:53 -07:00
David Heinemeier Hansson
dc1f707a56 Fix test 2024-05-13 17:01:50 -07:00
David Heinemeier Hansson
033f2a3401 Correct invocation 2024-05-13 16:59:50 -07:00
David Heinemeier Hansson
7cac7e6fb0 Envify during setup 2024-05-13 15:18:11 -07:00
Nick Hammond
fb58fc0ba6 Add in a server exec command for running ad-hoc commands directly on the server 2024-05-13 14:17:06 -07:00
Donal McBreen
12cad5458a Merge pull request #762 from kryachkov/main
Trim long hostnames
2024-05-10 16:05:27 +01:00
Donal McBreen
f8b7f74543 Merge pull request #786 from hundredwatt/add-target-option-to-builder
Add --target option to Builder to support multi-stage Docker builds
2024-05-10 15:15:31 +01:00
Donal McBreen
489d6dbcbb Merge pull request #789 from basecamp/host-tags
Host specific env with tags
2024-05-10 08:08:29 +01:00
Donal McBreen
6d062ce271 Host specific env with tags
Allow hosts to be tagged so we can have host specific env variables.

We might want host specific env variables for things like datacenter
specific tags or testing GC settings on a specific host.

Right now you either need to set up a separate role, or have the app
be host aware.

Now you can define tag env variables and assign those to hosts.

For example:
```
servers:
  - 1.1.1.1
  - 1.1.1.2: tag1
  - 1.1.1.2: tag2
  - 1.1.1.3: [ tag1, tag2 ]
env_tags:
  tag1:
    ENV1: value1
  tag2:
    ENV2: value2
```

The tag env supports the full env format, allowing you to set secret and
clear values.
2024-05-09 16:02:45 +01:00
Jason Nochlin
1e44cc2597 fix rubocop violation 2024-05-08 19:22:25 -06:00
André Falk
63c47eca4c Trim long hostnames
Hostnames longer than 64 characters are not supported by docker
2024-05-07 19:06:39 +02:00
Donal McBreen
3c8428504d Bump version for 1.5.2 2024-05-07 09:44:11 +01:00
Donal McBreen
8e71c48747 Merge pull request #759 from basecamp/details-accessory-host
Output the host when running accessory details
2024-05-02 15:54:08 +01:00
Donal McBreen
67a86e1068 Merge pull request #790 from basecamp/warn-on-missing-builder
Warn on missing builder
2024-05-02 12:50:00 +01:00
Donal McBreen
b67f40bdf7 Warn on missing builder
We are going to try to create a builder if one is missing, so let's warn
rather than report it as an error.
2024-05-02 12:38:20 +01:00
Donal McBreen
375f0283c4 Merge pull request #785 from basecamp/filter-traefik-hosts
Apply --hosts and --roles filters to traefik hosts as well
2024-04-29 14:48:23 +01:00
Jason Nochlin
947be0877f add --target option for builder configuration 2024-04-27 10:24:47 -06:00
Matthew Kent
b8aaddb4c9 Apply --hosts and --roles filters to traefik hosts as well. 2024-04-26 17:08:57 -07:00
Donal McBreen
f48f528043 Bump version for 1.5.1 2024-04-26 14:26:02 +01:00
Donal McBreen
ec0a082542 Merge pull request #779 from basecamp/fix-log-following
Escape single quotes to fix log following
2024-04-26 14:25:27 +01:00
Donal McBreen
6c638a8a77 Merge pull request #778 from basecamp/glob-match-roles-and-hosts
Allow glob matches for roles and hosts
2024-04-26 14:20:17 +01:00
Donal McBreen
1f5b936fa2 Escape single quotes to fix log following
Fixes: https://github.com/basecamp/kamal/issues/777
2024-04-26 14:16:19 +01:00
Donal McBreen
f785451cc7 Allow glob matches for roles and hosts
This lets you do things like:

```
kamal details -h '1.1.1.[1-9]'
kamal details -r 'w{eb,orkers}'
```
2024-04-26 13:43:52 +01:00
Donal McBreen
d475e88dbe Bump version for 1.5.0 2024-04-25 13:39:06 +01:00
Donal McBreen
d551f044d6 Merge pull request #772 from aishek/take-accessory-hosts-into-account
Take accessory hosts into account for --hosts
2024-04-25 11:57:45 +01:00
Donal McBreen
2611179d5e Merge pull request #773 from xiaohui-zhangxh/docker-env-file-keep-non-ascii
don't escape non-ascii characters in docker env file
2024-04-25 11:57:14 +01:00
Donal McBreen
1a013b8d4b Merge pull request #770 from ttilberg/769-ensure-valid-service-name-with-capital-letters
Allow capital letters to match valid service name, such as in MyApp
2024-04-25 11:52:55 +01:00
Maciej Litwiniuk
2f912367ac Allow custom user and port for builder host
When ssh options are set, they overwrite username and password passed as ssh builder uri. Passing part of uri for ssh-kit is fine, as it then properly extracts username and password and forwards it as host.ssh_options (in which case it's no longer empty)
2024-04-17 17:49:50 +02:00
xiaohui
9a9a0914cd don't escape non-ascii characters in docker env file 2024-04-17 17:42:06 +08:00
Alexandr Borisov
12c518097f Take accessory hosts into account 2024-04-17 11:45:33 +03:00
Tim Tilberg
69f90387a8 Allow capital letters to match valid service name, such as in MyApp 2024-04-15 09:09:58 -05:00
Donal McBreen
e6d436f646 Output the host when running accessory details
We already do this for app and Traefik hosts.
2024-04-05 12:46:51 +01:00
Donal McBreen
31669d4dce Merge pull request #758 from basecamp/any-runtime-in-test
Accept any runtime in the hook tests
2024-04-03 16:27:59 +01:00
Donal McBreen
9d20c1466e Merge pull request #757 from basecamp/executable-sample-docker-setup-hook
Make the sample docker setup hook executable
2024-04-03 16:12:46 +01:00
Donal McBreen
ff1dabe7f8 Merge pull request #756 from basecamp/tidy-up-role-host-setup
Tidy up role and host commander setup
2024-04-03 16:09:12 +01:00
Donal McBreen
69aa422890 Accept any runtime in the hook tests
Occasionally in CI things run slowly and it takes more that 1 second
for a cli test to run, so let's allow any value for the runtime in the
hook checks.
2024-04-03 16:06:53 +01:00
Donal McBreen
f8b0883036 Make the sample docker setup hook executable
Fixes https://github.com/basecamp/kamal/issues/754
2024-04-03 15:50:47 +01:00
Donal McBreen
c8100d1f26 Tidy up role and host commander setup
Extract Kamal::Commander::Specifics to deal with host and role setup and
ensure that primary hosts and roles always come first. This means that
in a rolling deploy we deploy to the primary ones first.

This will be important when we remove the healthcheck step as we want
to confirm the primary host can be deployed to before completing a
deployment for other roles.

By setting the hosts and roles all together in one place we can sort
the primary ones to the front without creating infinite loops.
2024-04-03 15:46:30 +01:00
Donal McBreen
3628ecaa44 Merge pull request #753 from basecamp/dump-hook-output-on-failure
Include error message on failure
2024-04-03 10:52:49 +01:00
Donal McBreen
67a2d5e7ca Include error message on failure 2024-04-03 10:43:20 +01:00
Donal McBreen
5e492ecc4d Merge pull request #748 from basecamp/latest-by-tag
Latest by tag
2024-04-03 09:11:03 +01:00
Donal McBreen
77bad291a1 Merge pull request #751 from basecamp/app-exec-env
Set env variables when running kamal app exec
2024-04-02 11:03:38 +01:00
Donal McBreen
a0ce9f66c4 Merge pull request #752 from basecamp/dont-debug-hooks
Use default verbosity for hooks
2024-04-02 11:03:24 +01:00
Donal McBreen
82962c375d Use default verbosity for hooks 2024-04-02 10:47:05 +01:00
Donal McBreen
8a6a51977f Set env variables when running kamal app exec
Allow additional env variable to be set when running `kamal app exec`.
Works for both new and existing containers.
2024-04-01 15:01:32 +01:00
Donal McBreen
2562853ae3 Merge pull request #746 from igor-alexandrov/file-join
Replaced string interpolations with File.join to build paths
2024-03-29 13:47:24 +00:00
Donal McBreen
ed90b99f0d Add tag_latest_image tests 2024-03-29 10:51:57 +00:00
Donal McBreen
ba7a13f895 Only tag after deploying to all hosts 2024-03-29 10:29:58 +00:00
Donal McBreen
05ac808f2a Use image tag to determine stale containers
Use current_running_version to determine the latest version when finding
stale containers.
2024-03-29 10:23:50 +00:00
Donal McBreen
fb7d9077ff Use latest tag for the current destination 2024-03-29 09:48:09 +00:00
Donal McBreen
bade195e93 Redefine what the "latest" container means
Currently the latest container is the one that was created last. But if
we have had a failed deployment that left two containers running that
would not be the one we want. The second container could be in a
restart loop for example.

Instead we want the container that is running the image tagged as
latest. As we now tag as latest after a successful deployment we can
trust that that is a healthy container.

In the case that there is no container running the latest image tag,
we'll fall back to the latest container.

This could happen if the deploy was halted in between the old container
being stopped and the image being tagged as latest.
2024-03-29 08:51:50 +00:00
Donal McBreen
55dd2f49c1 Tag image after booting and include destination
If you are deploying more than one destination to a host, the latest
tags will conflict, so we'll append the destination to the tag.

The latest tag is used when booting the app or exec-ing a new container.

If a deploy doesn't complete on a host for all roles then we should
probably not be using it, so move the tagging to the end of the boot
process.
2024-03-29 08:51:50 +00:00
Igor Alexandrov
511a182539 Replaced string interpolations with to build paths 2024-03-28 20:25:24 +04:00
Donal McBreen
8bb596e216 Merge pull request #741 from igor-alexandrov/destination_in_lock
Added destination to the lock directory
2024-03-28 08:26:57 +00:00
Igor Alexandrov
699bcc0d27 Combined two methods and into one 2024-03-27 20:56:47 +04:00
Donal McBreen
6aacd1f9e2 Merge pull request #745 from basecamp/add-empty-destination-label
Label containers with empty destinations
2024-03-27 15:05:43 +00:00
Donal McBreen
20e71d91c0 Label containers with empty destinations
This will allow us to filter for containers that have no destination in
cases where we deploy an empty + a non empty destination to the same
host.

To note:

```
\# Containers with a destination label
$ docker ps --filter label=destination

\# Containers with an empty destination label
$ docker ps --filter label=destination=
```
2024-03-27 14:48:55 +00:00
Donal McBreen
866303a59b Merge pull request #700 from basecamp/git-archive-build
Build from a git archive
2024-03-27 09:23:00 +00:00
Donal McBreen
53bfefeb2f Make building from a git archive the default
If no context is specified and we are in a git repo, then we'll build
from a git archive by default. This means we don't need a separate
setting and gives us a safer default build.
2024-03-27 08:42:10 +00:00
Donal McBreen
f3b7569032 Build from a git archive
Building directly from a checkout will pull in uncommitted files to or
more sneakily files that are git ignored, but not docker ignored.

To avoid this, we'll add an option to build from a git archive of HEAD
instead. Docker doesn't provide a way to build directly from a git
repo, so instead we create a tarball of the current HEAD with git
archive and pipe it into the build command.

When building from a git archive, we'll still display the warning about
uncommitted changes, but we won't add the `_uncommitted_...` suffix to
the container name as they won't be included in the build.

Perhaps this should be the default, but we'll leave that decision for
now.
2024-03-27 08:38:56 +00:00
Donal McBreen
e5457cf7b4 Merge pull request #736 from tiramizoo/traefik-info
Add tip how to apply changes to traefik by "traefik reboot"
2024-03-27 08:38:15 +00:00
Igor Alexandrov
cee449c269 Put locks in a locks directory. Ensure that locks directory exits on a primary host. 2024-03-27 12:04:39 +04:00
Donal McBreen
786454f2ee Merge pull request #502 from latyshev/main
Fix accessory name checking that is passing to command `kamal accessory`
2024-03-26 13:58:26 +00:00
Donal McBreen
827e18480d Merge pull request #732 from basecamp/always-send-clear-env
Always send the clear env to the container
2024-03-26 11:01:59 +00:00
Donal McBreen
9f9c9ccbde Merge pull request #742 from igor-alexandrov/remove_service_role_dest
Removed unused method from Kamal::Commands::App
2024-03-26 08:10:36 +00:00
Evgeny Latyshev
981d391d4d Fix accessory name check in with_accessory 2024-03-26 09:29:34 +03:00
Igor Alexandrov
900041001a Removed unused method 2024-03-25 22:48:23 +04:00
Igor Alexandrov
43672ec9a5 Added destination to the lock directory 2024-03-25 22:42:22 +04:00
Donal McBreen
5481fbb973 Test that we pull in env host variables
Now that clear env variables specified on the command line we can check
that values specified as `${VAR}` are pulled in from the host.
2024-03-25 12:26:37 +00:00
Donal McBreen
49afdbb09a Always send the clear env to the container
Secret and clear env variables have different lifecycles. The clear ones
are part of the repo, so it makes sense to always deploy them with the
rest of the repo.

The secret ones are external so we can't be sure that they are up to
date, therefore they require an explicit push via `envify` or `env push`.

We'll keep the env file, but now it just contains secrets. The clear
values are passed directly to `docker run`.
2024-03-25 11:42:27 +00:00
Donal McBreen
5f58575b62 Merge pull request #730 from igor-alexandrov/confirming_dialogs
Added -y option to kamal traefik reboot command
2024-03-22 15:14:44 +00:00
Wojciech Wnętrzak
cb49d7dada Add tip how to apply changes to traefik by "traefik reboot"
Running "traefik restart" is not enough to apply changes
2024-03-22 13:50:54 +01:00
Igor Alexandrov
3d26fa8ddd Updated confirmation text for the traefik reboot command 2024-03-22 14:27:18 +04:00
Donal McBreen
ea9f8b488d Merge pull request #735 from basecamp/extract-app-boot-steps
Extract app boot steps
2024-03-22 09:35:04 +00:00
Donal McBreen
83472af32c Merge pull request #734 from basecamp/rubocop-rails-omakase
Switch to rubocop-rails-omakase rubocop rules
2024-03-22 09:33:25 +00:00
Donal McBreen
e99e1955b8 Extract app boot steps
The Kamal::Cli::App#boot has a lot to do, so extract the steps to make
things clearer.
2024-03-22 09:21:52 +00:00
Donal McBreen
30e0c44396 Switch to rubocop-rails-omakase rubocop rules
No code changes required
2024-03-21 13:47:20 +00:00
Donal McBreen
20d6e5365e Merge pull request #733 from basecamp/integration-test-roles
Integration test roles
2024-03-21 13:43:33 +00:00
Donal McBreen
72ace2bf0b Add an integration test for roles
Add an app with roles to the integration tests. We'll deploy two web
containers and one worker. The worker just sleeps, so we are testing
that the container has booted.
2024-03-21 13:30:53 +00:00
Donal McBreen
ba40d026d0 Make integration test app to deploy configurable 2024-03-21 12:09:59 +00:00
Igor Alexandrov
0f13600ba3 Fixed Traefik integration test 2024-03-21 09:25:07 +04:00
Igor Alexandrov
bbf952952d Added -y option to kamal traefik reboot command 2024-03-20 22:00:13 +04:00
Donal McBreen
474b76cf47 Merge pull request #701 from basecamp/rubocop
Add Rubocop
2024-03-20 10:59:35 +00:00
Donal McBreen
3ecfb3744f Add Rubocop
- Pull in the 37signals house style
- Autofix violations
- Add to CI
2024-03-20 10:23:02 +00:00
Donal McBreen
c985fa33d1 Bump version for 1.4.0 2024-03-20 09:27:23 +00:00
Donal McBreen
e8b9f8907f Merge pull request #715 from basecamp/use-role-not-string-in-config
Pass around Roles instead of Strings
2024-03-08 08:55:53 +00:00
Donal McBreen
4966d52919 Pass around Roles instead of Strings
Avoid looking up roles by names everywhere. This avoids the awkward
role/role_config naming as well.
2024-03-08 08:44:35 +00:00
Donal McBreen
52bb40add0 Merge pull request #656 from DanielJackson-Oslo/informative-error-message-on-lock
Informative message on lock error
2024-03-07 11:16:18 +00:00
Donal McBreen
73a9276cdd Fix up app command tests 2024-03-07 11:11:20 +00:00
Donal McBreen
8c0784ed4a Merge pull request #634 from alhafoudh/main
Allow lines option to be configured when following app logs
2024-03-07 11:11:08 +00:00
Donal McBreen
089a2d3bba Merge pull request #710 from basecamp/install-wget-or-curl
Install docker with curl or wget
2024-03-07 11:01:30 +00:00
Donal McBreen
bd76d23916 Merge pull request #593 from CleverFew/role_logging_config
Role specific logging configuration
2024-03-07 10:53:34 +00:00
Donal McBreen
fa37fcd10c Merge pull request #585 from tsvallender/docker-network
Add docker-setup hook
2024-03-07 10:51:08 +00:00
Donal McBreen
f5dc0858b0 Update error message to include wget 2024-03-07 10:49:32 +00:00
Donal McBreen
9dddb140b1 Merge pull request #558 from GeNiuS69/add-skip_push-to-setup
Add --skip_push option to setup
2024-03-07 10:26:41 +00:00
Donal McBreen
26b1d57c90 Install docker with curl or wget
If curl is not available to download the docker install script, try
with wget instead.

If neither is available or both fail, return a simple failing script
so that we don't carry on regardless.

Fixes: https://github.com/basecamp/kamal/issues/395
2024-03-07 10:16:03 +00:00
Donal McBreen
b94199415f Convert combine by: '||' to any 2024-03-07 09:10:49 +00:00
Trevor Vallender
f69c45b7ea Add docker-setup hook
This allows the user to make any necessary configuration changes to
Docker before setting up any containers, allowing those configuration
changes to take effect from the outset.
2024-03-06 19:01:48 +00:00
Donal McBreen
32a2ae5b2c Merge pull request #708 from nickhammond/valid_service_name
Remove warning for valid service name
2024-03-06 16:22:04 +00:00
Nick Hammond
37544a6383 Merge branch 'basecamp:main' into valid_service_name 2024-03-06 09:09:13 -07:00
Nick Hammond
a1bc6d61af Switch the regex ordering for hyphen and underscore for service name to remove warning 2024-03-06 09:08:17 -07:00
Donal McBreen
5c32be10f1 Merge pull request #707 from basecamp/boot-strategy-min-limit-1
Ensure a minimum limit of 1 for % boot strategy
2024-03-06 16:06:35 +00:00
Donal McBreen
dc5af03593 Update tests to match single quotes 2024-03-06 16:04:31 +00:00
Donal McBreen
1abd029ea0 Merge pull request #696 from dorianmariecom/patch-1
Replace \`service\` by 'service' so it doesn't get executed by bash
2024-03-06 16:04:11 +00:00
Donal McBreen
c4d0d3e5eb Merge pull request #704 from basecamp/escape-registry-username-password
Escape the docker registry username and password
2024-03-06 15:58:46 +00:00
Donal McBreen
46e7cf8e78 Merge pull request #706 from basecamp/kamal-remove-noop
Ensure `kamal remove` completes without setup
2024-03-06 15:58:34 +00:00
Donal McBreen
c7cfc074b6 Ensure a minimum limit of 1 for % boot strategy
Fixes: https://github.com/basecamp/kamal/issues/681
2024-03-06 15:51:35 +00:00
Donal McBreen
c10f43e365 Merge pull request #692 from nickhammond/valid_service_name
Add a simple validation to the service name to prevent setup issues
2024-03-06 15:24:39 +00:00
Donal McBreen
8e2184d65e Ensure kamal remove completes without setup
If `kamal setup` has not run or errored out part way through,
`kamal remove` should still complete.

Fixes: https://github.com/basecamp/kamal/issues/629
2024-03-06 14:59:26 +00:00
Donal McBreen
2be397b679 Escape the docker registry username and password
Fixes: https://github.com/basecamp/kamal/issues/278
2024-03-06 11:04:55 +00:00
Donal McBreen
cc8c508556 Merge branch 'main' into valid_service_name 2024-03-05 11:02:33 +00:00
Nick Hammond
3b16e047c5 Add hyphen to the allowed character list for service name 2024-03-04 10:03:22 -07:00
Donal McBreen
6563393d9a Merge pull request #627 from aishek/626-mention-sprockets-config-in-deploy-template
Mention Sprockets config in deploy template
2024-03-04 15:31:41 +00:00
Ahmed Al Hafoudh
91f350fcce Merge branch 'basecamp:main' into main 2024-03-04 16:22:28 +01:00
Nick Lozon
e4e9664049 use double quotes 2024-03-04 10:10:51 -05:00
Nick Lozon
1acef5221f test deep_merge 2024-03-04 10:06:30 -05:00
Nick Lozon
788a57e85e role logging_args method, use in app 2024-03-04 10:06:30 -05:00
Nick Lozon
f9a934a01f configuration logging accessor 2024-03-04 10:06:30 -05:00
Aleksandr Borisov
f286fdc374 Update lib/kamal/cli/templates/deploy.yml
Co-authored-by: Donal McBreen <dmcbreen@gmail.com>
2024-03-04 16:26:11 +03:00
Donal McBreen
828cca322b Merge pull request #650 from basecamp/retained-containers
Config the number of containers to keep
2024-03-04 12:05:35 +00:00
Donal McBreen
cb030e8751 Merge pull request #680 from igor-alexandrov/traefik-2.10
Bump default Traefik image to 2.10
2024-03-04 11:58:37 +00:00
Donal McBreen
6892abb4be Config the number of containers to keep
By default we keep 5 containers around for rollback. The containers
don't take much space, but the images for them can.

Make the number of containers to retain configurable, either in the
config with the `retain_containers` setting on the command line
with the `--retain` option.
2024-03-04 11:55:45 +00:00
Donal McBreen
bcfd0ca88a Merge pull request #645 from juan-apa/fix-missing-netscp-require
require missing net/scp dependency
2024-03-04 11:49:43 +00:00
Donal McBreen
2e8071a5b3 Merge pull request #608 from CleverFew/fix_accessory_cli_host_params
Accessory CLI respects `--hosts`
2024-03-04 11:31:50 +00:00
Donal McBreen
200e2686fd Merge pull request #506 from rience/custom-acc-service-name
Allow for Custom Accessory Service Name
2024-03-04 10:57:10 +00:00
Donal McBreen
db94789dc1 Merge pull request #434 from rience/ssh-agent-support
Supports Passing SSH Agent Socket to Build Options
2024-03-04 10:54:47 +00:00
Dorian Marié
2bffc3bc74 Replace \service\ by 'service' so it doesn't get executed by bash
Fixes #694
2024-03-01 09:54:06 +01:00
Aleksandr Nigomatulin
064ace0598 Rollback passing invoke_options 2024-02-24 21:36:20 +06:00
Nick Hammond
a02af74dda Add a simple validation to the service name to prevent setup issues 2024-02-22 09:47:48 -07:00
Aleksandr Nigomatulin
5ef384d666 Add test 2024-02-17 00:11:03 +06:00
Aleksandr Nigomatulin
b94dfe193b Remove unnecessary code 2024-02-16 12:52:07 +06:00
Aleksandr Nigomatulin
bc6c027315 Upds according remarks 2024-02-16 11:56:58 +06:00
Krzysztof Adamski
1c2a45817a Supports Passing SSH Args to Build Options 2024-02-15 14:20:20 +01:00
Krzysztof Adamski
b411356409 Allow for Custom Accessory Service Name 2024-02-15 11:12:18 +01:00
Igor Alexandrov
77e72e34ce Bumped default Traefik image to 2.10 2024-02-13 16:00:02 +04:00
Daniel Jackson
ad04bb7556 Show context for lock status message on raise_if_locked 2024-01-23 09:17:15 +01:00
Daniel Jackson
1ec69d3764 Tell user about 'kamal lock help' when deploy fails due to a lock 2024-01-23 09:16:09 +01:00
Daniel Jackson
2d1a0dc9ba Informative message on lock error 2024-01-22 09:11:17 +01:00
Juan Aparicio
c984db152f require missing net/scp dependency 2024-01-11 17:00:13 -03:00
David Heinemeier Hansson
aea55480ad Merge pull request #640 from basecamp/local-different-arch
Allow local builds using a different arch than native
2024-01-10 13:28:37 -08:00
dhh
5a09aa12ba Allow local builds using a different arch than native 2024-01-10 13:00:48 -08:00
Donal McBreen
aca7796e9d Bump version for 1.3.1 2024-01-10 08:56:34 +00:00
Donal McBreen
8b6d8306d1 Merge pull request #637 from basecamp/tests-wait-longer-for-health
Be a bit more patient during tests
2024-01-09 16:45:28 +00:00
Donal McBreen
bb50546467 Merge pull request #636 from basecamp/tests-clean-known-hosts
Fix Net::SSH::HostKeyMismatch between bin/test runs
2024-01-09 16:45:12 +00:00
Donal McBreen
acc6b9ad71 Merge pull request #635 from basecamp/missing-base64-require
Add a missing base64 require
2024-01-09 16:44:42 +00:00
Matthew Kent
9c681d4a38 Be a bit more patient during tests.
Seeing reasonably consistent local failures at 20 seconds.
2024-01-09 08:21:45 -08:00
Matthew Kent
2a8924b53c Address Net::SSH::HostKeyMismatch seen locally between bin/test runs. 2024-01-09 08:21:30 -08:00
Matthew Kent
c5ae54d7d4 Add a missing base64 require.
Also, prepare for the moving of base64 from default to a bundled gem in ruby 3.4.
2024-01-09 08:21:10 -08:00
Donal McBreen
4b05068493 Merge pull request #638 from basecamp/rails-7.2-compatible-rubies
Rails 7.2 compatible Rubies
2024-01-09 12:10:29 +00:00
Donal McBreen
68eb549795 Update to actions/checkout@v4 to silence node warning 2024-01-09 11:35:10 +00:00
Donal McBreen
1a3dd52af4 Rails 7.2 compatible Rubies
1. Add Ruby 2.7 specific Gemfile that uses an older version of nokogiri
2. Rails edge doesn't support Ruby 2.7.0, so exclude it.
3. Add Ruby 3.3
4. Update Gemfile.lock to test against Rails 7.1.2 as it's the latest
   version.
5. Remove continue-on-error from the matrix and always set to true
2024-01-09 11:13:11 +00:00
Ahmed Al Hafoudh
0d709a3fdb Allow lines option to be configured when following app logs 2024-01-08 09:34:38 +01:00
Alexandr Borisov
414d29ae4e Mention Sprockets config in deploy template 2024-01-04 09:18:38 +04:00
Nick Lozon
f8d8319c2f better test description 2023-12-12 15:37:12 -05:00
Nick Lozon
f6a9d54902 unit test 2023-12-12 15:07:29 -05:00
Nick Lozon
b2fd5744fb perform intersection on specified hosts 2023-12-12 14:39:33 -05:00
Donal McBreen
457f06da13 Merge pull request #598 from basecamp/fix-duplicate-role-env-vars
Fix duplicate role env vars
2023-11-29 10:09:34 +00:00
Matthew Kent
7fa53d90bd Merge hashes to de-dupe the app and role envs.
This is better then adding them together which confusingly results in
both ENV vars in the same file, though based on the load order, they
worked anyway.
2023-11-28 15:59:03 -08:00
Donal McBreen
a155b7baab Bump version for 1.3.0 2023-11-28 14:06:45 +00:00
Donal McBreen
175e3bc159 Merge pull request #507 from leonvogt/introduce-absolute-accessories-paths
Add option to set an absolute directory path
2023-11-28 10:15:27 +00:00
Donal McBreen
e3d8a2aa82 Merge pull request #594 from basecamp/match-primary-role-in-filters
Try to match primary_role when roles are filtered
2023-11-28 09:15:39 +00:00
Donal McBreen
0e067fb5e1 Merge pull request #595 from basecamp/error-on-filter-miss
Error out when roles or host filters don't match anything
2023-11-27 08:08:19 +00:00
Matthew Kent
63babecba7 Raise an error when either the filtered hosts or roles are empty.
Keeps us confusingly running things on the primary_host when nothing
matches.
2023-11-25 12:47:39 -08:00
Matthew Kent
79baa598fa Make an effort to match the primary_role from a list of specific roles.
This is less surprising than picking the first role and first host.
2023-11-24 17:41:58 -08:00
Donal McBreen
b1dc188841 Remove stray file 2023-11-23 09:22:36 +00:00
Donal McBreen
635876bdb9 Merge pull request #523 from rmacklin/fix-error-message-in-pre-build-sample-hook
Fix duplicate error message in pre-build.sample
2023-11-16 08:51:22 +00:00
Donal McBreen
11521517fa Merge pull request #550 from dmitrytrager/feature-name-all-for-accessory-reboot
feature: add NAME=all option for accessory reboot
2023-11-16 08:50:51 +00:00
Donal McBreen
610d9de3fd Merge pull request #580 from happyscribe/feat/no-web
Allow Kamal to run without traefik
2023-11-16 08:44:45 +00:00
Donal McBreen
bf79df0f72 Bump version for 1.2.0 2023-11-15 14:48:11 +00:00
Donal McBreen
a0959b5afd Merge pull request #573 from basecamp/pre-post-traefik-reboot-hooks
Pre and post Traefik reboot hooks
2023-11-15 14:01:40 +00:00
Yoel Cabo
7472e5dfa6 Merge remote-tracking branch 'origin/main' into feat/no-web 2023-11-14 12:11:18 +01:00
Yoel Cabo
887b7dd46d Do not invoke healthcheck on deploy when no web role 2023-11-14 11:34:32 +01:00
Donal McBreen
77a79b299a Merge pull request #583 from basecamp/wildcard-filters
Add wildcards to roles and hosts filters
2023-11-14 08:19:02 +00:00
Matthew Kent
efcb855db7 Advertise wildcard support. 2023-11-13 23:43:26 -08:00
Matthew Kent
7137850354 Add support for wildcard matches with '*' on roles and hosts.
eg:
  --roles=*_chicago,*_tokyo
  --hosts=app-*

Useful for targeted deploys.
2023-11-13 23:43:23 -08:00
Donal McBreen
8a85840a47 Merge pull request #582 from basecamp/allow-empty-roles
Add allow_empty_roles to control aborting on roles with no hosts.
2023-11-13 09:30:01 +00:00
Donal McBreen
80cc0c23d8 Merge pull request #578 from basecamp/enable-yaml-aliases
Enable yaml aliases
2023-11-13 09:28:40 +00:00
Donal McBreen
14a9129410 Merge pull request #577 from basecamp/set-primary-web-role
Support customizing the primary_web_role
2023-11-13 09:27:18 +00:00
Matthew Kent
60187cc3a4 Add allow_empty_roles to control aborting on roles with no hosts.
This added flexibility allows you to define base roles that might not
necessarily exist in each deploy destination.
2023-11-12 08:54:28 -08:00
Yoel Cabo
87cb8c1f71 fix: allow configurations without web roles 2023-11-12 09:39:07 +01:00
Matthew Kent
ed58ce6e61 Add test coverage with aliases. 2023-11-11 17:25:50 -08:00
Matthew Kent
263b4a4fb8 Enable aliases for more exotic templating situations.
This is super useful for DRY when configuring a number of roles and you
hit the limits of what's reasonable with ERB.
2023-11-11 17:25:50 -08:00
Matthew Kent
073f745677 Test for both undefined roles and missing traefik. 2023-11-11 12:57:52 -08:00
Matthew Kent
a9cc7c73d2 Handle an undefined primary_web_role. 2023-11-11 12:57:31 -08:00
Matthew Kent
6898e8789e Further test the override. 2023-11-10 17:17:16 -08:00
Matthew Kent
d0ac6507e7 Add test coverage. 2023-11-10 16:49:37 -08:00
Matthew Kent
628a47ad88 Background for the new option. 2023-11-10 16:39:06 -08:00
Matthew Kent
47f8725cf3 Support a dynamic primary_web_role instead of assuming it's 'web'.
This allows for more meaningful naming in roles.

The only caution here is that we don't support the renaming of roles, so
any migration is left to the user.
2023-11-10 16:35:25 -08:00
Donal McBreen
5fd4a28bf7 Pre and post Traefik reboot hooks
Provide pre and post reboot hooks for Traefik, that can be used to
remove/add to an external load balancer to prevent requests from being
sent during the reboot.

Works best with the --rolling setting, where each hook is called once
per host.
2023-11-08 15:11:26 +00:00
Donal McBreen
97ba6b746b Merge pull request #564 from basecamp/return-502-if-no-container
Return a 502 when container is down
2023-11-08 14:58:22 +00:00
Donal McBreen
9e25d8a012 Priority 2 for the main app 2023-11-08 14:12:45 +00:00
Donal McBreen
da161445fa Merge pull request #508 from leonvogt/ssh-port-option
Configurable SSH port
2023-11-06 08:48:26 +00:00
Leon
f339626667 Add option to set absolute directory path 2023-11-03 22:48:30 +01:00
Leon
2d86d4f7cc Add SSH port to run_over_ssh 2023-11-03 22:32:37 +01:00
Leon
792aa1dbdf Add SSH port option 2023-11-03 22:32:37 +01:00
Donal McBreen
24a2f51641 Return a 502 when container is down
If the app container is down or not responding then traefik will return
a 404 response code. This is not ideal as it suggests a client rather
than a server problem.

To fix this, we'll define a catch all route that always returns a 502.

This is not ideal as this route would take priority over a shorter route
with priorty 1.

TODO: up the priority of the app route.
2023-11-03 14:20:52 +00:00
Donal McBreen
8f53104d00 Bump version for 1.1.0 2023-11-01 09:20:45 +00:00
dmitrytrager
2d22143a24 feature: add NAME=all option for accessory reboot 2023-10-31 00:13:45 +01:00
Aleksandr Nigomatulin
cbd99306eb Add skip_push option to setup 2023-10-30 23:27:58 +06:00
Donal McBreen
78fc91f2ec Merge pull request #557 from basecamp/envify-reset-env-before-push
Reset the env before pushing
2023-10-30 11:54:00 +00:00
Donal McBreen
dd748fac8c Reset the env before pushing
Calling `load_envs` again does not load updated env variables, because
Dotenv does not overwrite existing values.

To fix this we'll store the original ENV and reset to it before
reloading.

https://github.com/basecamp/kamal/issues/512
2023-10-30 11:31:50 +00:00
Donal McBreen
b732b2dd55 Merge pull request #547 from nickhammond/envify/trim-lines
Enable trim mode with ERB
2023-10-30 08:57:55 +00:00
Donal McBreen
e3254b2aa8 Merge pull request #544 from nickhammond/bugfix-require-sshkit-sensitive-util
Require sshkit within the sshkit util
2023-10-30 08:57:08 +00:00
Donal McBreen
e9269d2ee8 Merge pull request #501 from rience/optional-envify-push
Optionally Skip Push for "envify"
2023-10-30 08:30:21 +00:00
Donal McBreen
d2214b43b7 Merge pull request #499 from basecamp/env-only-needed-for-push
Remove the env check
2023-10-30 08:22:56 +00:00
Donal McBreen
370481921e Merge pull request #498 from basecamp/app-exec-env-file
App exec with env file
2023-10-30 08:22:35 +00:00
Donal McBreen
aa23f26330 Merge pull request #479 from npezza93/main
Loosen superuser check to match docker-installs script check
2023-10-30 08:21:30 +00:00
Donal McBreen
f4933d83bf Merge pull request #477 from clintmiller/patch-1
Pass KAMAL_VERSION env var to container run
2023-10-30 08:19:20 +00:00
Nick Hammond
6c36c82153 Enable trim mode with ERB 2023-10-24 17:09:05 -07:00
Krzysztof Adamski
8ca04032a1 Optionally Skip Push for "envify" 2023-10-23 14:49:39 +02:00
Nick Hammond
2fb22c934b Require sshkit within the sshkit util 2023-10-22 22:34:22 -07:00
Richard Macklin
f96d071222 Fix copy-pasted error message in pre-build.sample
The "No git remote set" error message was appropriate for the previous
block (where it was presumably copy-pasted from), but in this line we
have failed the check that determines if we have a git branch checked
out, so we should output a corresponding error.
2023-10-08 15:14:40 -07:00
Donal McBreen
f6662c7a8f Remove the env check
The env check is not needded anymore as all the commands rely on the
env files having already been created remotely.

The only place the env is needed is when running `kamal env push` and
that will still raise an apropriate error.
2023-09-25 15:23:01 +01:00
Donal McBreen
645f5ab72d App exec with env file
When calling `kamal app exec` for new non interactive containers, run
the command per role on each server and include the role config
including the environment.

Fixes: https://github.com/basecamp/kamal/issues/492
2023-09-25 15:07:05 +01:00
Clint Miller
8dca65f48f Fix commands/app tests 2023-09-20 08:12:27 -05:00
dhh
83a2d52ff4 Bump version for 1.0.0 2023-09-18 17:39:01 -07:00
Nick Pezza
1a2796a7d0 Loosen superuser check to match docker-installs script check 2023-09-18 20:32:59 -04:00
Clint Miller
d80fdf8468 Pass KAMAL_VERSION env var to container run
In lieu of a general purpose mechanism to pass dynamically-evaluated env-vars at container execution time, we can pass the `config.version` as KAMAL_VERSION to avoid having to take apart the container name just to determine the SHA of the deployed version in the entrypoint.
2023-09-18 16:07:36 -05:00
dhh
90fefc419f Point to rolling restarts 2023-09-18 08:31:49 -07:00
dhh
8671963719 Explain asset bridging 2023-09-18 08:16:44 -07:00
Donal McBreen
a03ffd5b92 Merge pull request #476 from basecamp/exec-with-role
Run interactive commands with the correct host
2023-09-18 12:14:13 +01:00
Donal McBreen
0861730e0e Run interactive commands with the correct host
Fixes https://github.com/basecamp/kamal/issues/430
2023-09-18 12:00:36 +01:00
David Heinemeier Hansson
6b0f93a564 Update README.md 2023-09-16 16:02:54 -07:00
David Heinemeier Hansson
e6371faf4f Merge pull request #473 from basecamp/introduce-git-gateway
Extract Kamal::Git as gateway for all git usage
2023-09-16 11:47:18 -07:00
dhh
e95a9b4fa2 Fix tests 2023-09-16 11:35:29 -07:00
dhh
e5886a1a8e Merge branch 'main' into introduce-git-gateway
* main:
  Healthcheck polling is a CLI concern
2023-09-16 11:31:48 -07:00
David Heinemeier Hansson
ec8192b160 Merge pull request #472 from basecamp/move-healthcheck-poller-to-cli
Healthcheck polling is a CLI concern
2023-09-16 11:31:28 -07:00
dhh
2da03a220d Merge branch 'main' into introduce-git-gateway
* main:
  No longer used
  Fix env validation
  Fix tests
  Fix test
  Extract Kamal::EnvFile
2023-09-16 11:31:18 -07:00
dhh
cfbfb37e23 Extract Kamal::Git as gateway for all git usage 2023-09-16 11:30:29 -07:00
David Heinemeier Hansson
ff4d025840 Merge pull request #471 from basecamp/extract-env-writer
Extract Kamal::EnvFile
2023-09-16 11:29:43 -07:00
dhh
59ac59d351 Healthcheck polling is a CLI concern
Also, it has no instance variables, so let's just have it be a module.
2023-09-16 11:19:38 -07:00
dhh
3df87520db No longer used 2023-09-16 11:12:52 -07:00
dhh
85ce65a4ce Merge branch 'main' into extract-env-writer
* main:
  Inline util method only used in one place
2023-09-16 11:12:08 -07:00
dhh
12a82a6c58 Inline util method only used in one place 2023-09-16 11:11:24 -07:00
dhh
b2d2a254d7 Fix env validation 2023-09-16 11:05:47 -07:00
dhh
62cdf31ae2 Fix tests 2023-09-16 11:01:16 -07:00
dhh
0dcebe7d34 Fix test 2023-09-16 10:59:41 -07:00
dhh
32a5c157b9 Merge branch 'main' into extract-env-writer
* main:
  No longer used
2023-09-16 10:56:29 -07:00
dhh
97cea8950d No longer used 2023-09-16 10:56:00 -07:00
dhh
873be0b76b Extract Kamal::EnvFile
Cleaning up the Utils junk drawer.
2023-09-16 10:55:41 -07:00
David Heinemeier Hansson
3a8eb0cf7d Merge pull request #470 from basecamp/extract-app-concerns
Extract app concerns
2023-09-16 10:24:24 -07:00
dhh
e9ef13d06d Group configuration methods in logical sections 2023-09-16 10:20:08 -07:00
dhh
f648fe6c3f Grouping + ordering 2023-09-16 10:14:04 -07:00
dhh
46895d0b08 Better ordering and spacing 2023-09-16 10:11:42 -07:00
dhh
431ca9e809 Remind about env push 2023-09-16 10:09:42 -07:00
dhh
6b5c5f0650 Extract Logging too
Leave only the core essentials in App
2023-09-16 10:03:28 -07:00
dhh
d303fcc621 Extract Containers and Images concerns 2023-09-16 09:58:09 -07:00
dhh
3ae855ef28 Explain method better 2023-09-16 09:53:03 -07:00
dhh
76a3086569 Group related methods with spacing 2023-09-16 09:52:54 -07:00
dhh
07646bc020 Extract Cord, Assets, and Execution concerns from App
It was getting crowded!
2023-09-16 09:51:45 -07:00
dhh
880b8b267a Fix test 2023-09-16 09:38:30 -07:00
dhh
37e5c48a27 Setup run directory on accessory hosts as well
cc @djmb
2023-09-15 11:08:27 -07:00
dhh
deb67386fa No need to suggest use of erb 2023-09-15 10:54:50 -07:00
dhh
81d74e4a9d Record push of env files for audit on app servers 2023-09-15 10:20:31 -07:00
dhh
39c13dcc18 Push env files as part of setup 2023-09-15 10:20:31 -07:00
dhh
e7314a0eea Explain ensuring Docker is installed 2023-09-15 10:20:31 -07:00
Donal McBreen
168c6e2da3 Merge pull request #467 from basecamp/assets-copy-hidden-files
Copy all files into asset volume
2023-09-15 08:46:02 +01:00
Donal McBreen
564765862b Add hidden file check to integration tests 2023-09-15 08:37:41 +01:00
Donal McBreen
3c12d1799c Copy all files into asset volume
Adding -T to the copy command ensures that the files are copied at the
same level into the target directory whether it exists or not.

That allows us to drop the `/*` which was not picking up hidden files.

Fixes: https://github.com/basecamp/kamal/issues/465
2023-09-15 08:07:48 +01:00
Donal McBreen
60835d13a8 Merge pull request #444 from rience/custom-healthcheck-log-lines-count
Configurable Number of Lines in Healthcheck Log Output
2023-09-13 08:57:00 +01:00
Krzysztof Adamski
892cf0e66b Configurable Log Lines Number in Healthcheck Log Output 2023-09-12 21:06:36 +02:00
Krzysztof Adamski
8ddc484ce6 Configurable Lines Number in Healthcheck Log Output 2023-09-12 21:04:18 +02:00
Donal McBreen
0e021e3c57 Merge pull request #461 from basecamp/escape-newline-from-inspect-format
Escape the newline in the inspect query
2023-09-12 19:19:47 +01:00
Donal McBreen
fb0aeec27e Escape the newline in the inspect query 2023-09-12 19:10:39 +01:00
Donal McBreen
a367819a1c Merge pull request #460 from basecamp/traefik-wait-5s-after-unhealthy
Give Traefik 5s to drop old container
2023-09-12 17:12:20 +01:00
Donal McBreen
0afe289a20 Give Traefik 5s to drop old container 2023-09-12 17:03:51 +01:00
Donal McBreen
bf6af46ac3 Merge pull request #459 from basecamp/env-file-escape-newlines
Escape newlines in docker env files
2023-09-12 15:05:38 +01:00
Donal McBreen
df2b76aee1 Escape newlines in docker env files
When env variables were passed via `-e` newlines were escaped. This
updates the env file to do the same thing.
2023-09-12 14:57:19 +01:00
Donal McBreen
70a3c7195a Merge pull request #458 from basecamp/avoid-env-empty-file-warning
Fix empty file warning when uploading env files
2023-09-12 12:05:31 +01:00
Donal McBreen
c651de177f Fix empty file warning when uploading env files 2023-09-12 11:57:28 +01:00
Donal McBreen
7b42daa9fb Merge pull request #457 from basecamp/remove-dangling-image-filter
Remove the `dangling=true` filter
2023-09-12 11:21:50 +01:00
Donal McBreen
9d49b3e391 Merge pull request #456 from basecamp/validate-image
Validate the build image
2023-09-12 11:18:32 +01:00
Donal McBreen
2c5ab054db Remove the dangling=true filter
This has been removed from Docker Engine 24 and `docker image prune`
only deletes dangling images anyway.

Fixes https://github.com/basecamp/kamal/issues/410
2023-09-12 11:09:26 +01:00
Donal McBreen
66291a2aea Validate the build image
Kamal needs images to have the service label so it can track them for
pruning. Images built by Kamal will have the label, but externally built
ones may not.

Without it images will build up over time. The worst case is an outage
if all the hosts disks fill up at the same time.

We'll add a check for the label and halt if it is not there.
2023-09-12 10:45:01 +01:00
Donal McBreen
d96e086945 Merge pull request #452 from basecamp/preconnect-to-build-remote-host
Connect to remote host before creating builder
2023-09-12 09:21:57 +01:00
Donal McBreen
8424458174 Check protocol is SSH before connecting 2023-09-12 09:12:57 +01:00
Donal McBreen
6a3b0249fe Connect to remote host before creating builder
Connecting to the remote host will make any SSH configuration issues
obvious and add the host to known hosts if that is how SSHKit is
configured.
2023-09-12 09:12:57 +01:00
Donal McBreen
dfc2803714 Merge pull request #454 from basecamp/lts-ubuntu
Use LTS version of Ubuntu for integration tests
2023-09-12 09:12:31 +01:00
Donal McBreen
ade90bc051 Use LTS version of Ubuntu for integration tests 2023-09-12 08:59:54 +01:00
Donal McBreen
daa53f5831 Merge pull request #451 from basecamp/require-destinations
Add a require_destination setting
2023-09-12 08:26:36 +01:00
Donal McBreen
50a4f83db6 Merge pull request #450 from basecamp/stop-stale-container-when-deploying
Stop stale containers when deploying
2023-09-12 08:26:16 +01:00
Donal McBreen
00cb7d99d8 Merge pull request #449 from basecamp/asset-path
Asset paths
2023-09-12 08:26:07 +01:00
Donal McBreen
fb74910dc8 Merge pull request #425 from basecamp/prune-healthcheck-containers
Prune healthcheck containers
2023-09-12 08:25:50 +01:00
Donal McBreen
26dcd75423 Add a require_destination setting
If you always want to use a destination, and have a base deploy.yml file
that doesn't specify any hosts, then if you forget to specific the
destination you will get a cryptic error.

Add a "require_destination" setting you can use to avoid this.
2023-09-11 16:57:11 +01:00
Donal McBreen
afb9b0bbe2 Stop stale containers when deploying
An interrupted deployment can leave older containers lying around. To
ensure they are cleaned up subsequently, stop stale containers during
deployments instead of just reporting them.
2023-09-11 14:49:06 +01:00
Donal McBreen
718776eb72 Prune healthcheck containers
If a deployment is interrupted it could leave stale healthcheck
containers around that prevent dependent images from being pruned.
2023-09-11 14:36:25 +01:00
Donal McBreen
9d35793287 Merge pull request #440 from gf3/fix/ssh-auth-methods
fix: do not hardcode Net::SSH auth_methods
2023-09-11 14:32:37 +01:00
Donal McBreen
0b439362da Asset paths
During deployments both the old and new containers will be active for a
small period of time. There also may be lagging requests for older CSS
and JS after the deployment.

This can lead to 404s if a request for old assets hits a new container
or visa-versa.

This PR makes sure that both sets of assets are available throughout the
deployment from before the new version of the app is booted.

This can be configured by setting the asset path:

```yaml
asset_path: "/rails/public/assets"
```

The process is:
1. We extract the assets out of the container, with docker run, docker
cp, docker stop. Docker run sets the container command to "sleep" so
this needs to be available in the container.
2. We create an asset volume directory on the host for the new version
of the app on the host and copy the assets in there.
3. If there is a previous deployment we also copy the new assets into
its asset volume and copy the older assets into the new asset volume.
4. We start the new container mapping the asset volume over the top of
the container's asset path.

This means the both the old and new versions have replaced the asset
path with a volume containing both sets of assets and should be able
to serve any request during the deployment. The older assets will
continue to be available until the next deployment.
2023-09-11 12:18:18 +01:00
Donal McBreen
2962f545b9 Merge pull request #447 from basecamp/output-per-line-mounts
Output one mount per line
2023-09-07 15:30:03 +01:00
Donal McBreen
cd02510d0f Output one mount per line
The go template was concatenating all the mounts into one line. It
happened to work because the mount we are interested was always first.

Fix it to output one mount per line instead.
2023-09-07 15:20:50 +01:00
Donal McBreen
cccf79ed94 Merge branch 'main' into fix/ssh-auth-methods 2023-09-07 10:21:28 +01:00
Donal McBreen
aa9999809c Merge pull request #439 from basecamp/zero-downtime-deploy-file
Zero downtime deployment with cord file
2023-09-07 09:34:40 +01:00
Donal McBreen
6263bf96ba Merge pull request #438 from basecamp/remote-env-file
Copy env files to remote hosts
2023-09-07 09:34:22 +01:00
Gianni Chiappetta
9a539ffc86 chore: update tests to remove hardcoded ssh auth method 2023-09-06 10:59:17 -04:00
Donal McBreen
8a41d15b69 Zero downtime deployment with cord file
When replacing a container currently we:
1. Boot the new container
2. Wait for it to become healthy
3. Stop the old container

Traefik will send requests to the old container until it notices that it
is unhealthy. But it may have stopped serving requests before that point
which can result in errors.

To get round that the new boot process is:

1. Create a directory with a single file on the host
2. Boot the new container, mounting the cord file into /tmp and
including a check for the file in the docker healthcheck
3. Wait for it to become healthy
4. Delete the healthcheck file ("cut the cord") for the old container
5. Wait for it to become unhealthy and give Traefik a couple of seconds
to notice
6. Stop the old container

The extra steps ensure that Traefik stops sending requests before the
old container is shutdown.
2023-09-06 14:35:30 +01:00
Donal McBreen
94bf090657 Copy env files to remote hosts
Setting env variables in the docker arguments requires having them on
the deploy host.

Instead we'll add two new commands `kamal env push` and
`kamal env delete` which will manage copying the environment as .env
files to the remote host.

Docker will pick up the file with `--env-file <path-to-file>`. Env files
will be stored under `<kamal run directory>/env`.

Running `kamal env push` will create env files for each role and
accessory, and traefik if required.

`kamal envify` has been updated to also push the env files.

By avoiding using `kamal envify` and creating the local and remote
secrets manually, you can now avoid accessing secrets needed
for the docker runtime environment locally. You will still need build
secrets.

One thing to note - the Docker doesn't parse the environment variables
in the env file, one result of this is that you can't specify multi-line
values - see https://github.com/moby/moby/issues/12997.

We maybe need to look docker config or docker secrets longer term to get
around this.

Hattip to @kevinmcconnell - this was all his idea.
2023-09-06 14:33:13 +01:00
Donal McBreen
adc7173cf2 Merge pull request #437 from basecamp/kamal-run-directory
Configurable Kamal directory
2023-09-06 14:31:07 +01:00
Donal McBreen
fd6bf5324a Merge pull request #443 from rience/custom-healthcheck-port
Configurable Healthcheck Expose Port
2023-09-06 11:09:48 +01:00
Krzysztof Adamski
c2b2f7ea33 Fixing Tests 2023-09-06 10:16:59 +02:00
Krzysztof Adamski
bbcc90e4d1 Configurable Healthcheck Expose Port 2023-09-05 10:53:32 +02:00
Gianni Chiappetta
84f78cd9f9 fix: do not hardcode Net::SSH auth_methods 2023-09-01 15:11:12 -04:00
Donal McBreen
787688ea08 kamal -> .kamal 2023-08-28 17:13:52 +01:00
Donal McBreen
bcfa1d83e8 Configurable Kamal directory
To avoid polluting the default SSH directory with lots of Kamal config,
we'll default to putting them in a `kamal` sub directory.

But also make the directory configurable with the `run_directory` key,
so for example you can set it as `/var/run/kamal/`

The directory is created during bootstrap or before any command that
will need to access a file.
2023-08-28 16:32:18 +01:00
David Heinemeier Hansson
9363b6a464 Bump version for 0.16.1 2023-08-24 09:16:13 -07:00
David Heinemeier Hansson
338fd4e493 Merge pull request #428 from tbuehlmann/main
Fix picking the first available role on primary_host
2023-08-24 08:36:29 -07:00
David Heinemeier Hansson
eb3cb81a79 Merge pull request #368 from tsvallender/main 2023-08-24 06:12:48 -07:00
Tobias Bühlmann
556f7f5a37 Fix picking the first available role on primary_host 2023-08-24 13:50:24 +02:00
Trevor Vallender
c2ec04f8c1 Allow Traefik to run without publishing port
Adds the `publish` option which, if set to false, does not pass `--publish` to
`docker run` when starting Traefik. This is useful when running Traefik
behind a reverse proxy, for example.
2023-08-24 10:52:10 +01:00
David Heinemeier Hansson
519659b84c Merge pull request #422 from fig/fix-421
require ActiveSupport module to provide String#remove
2023-08-23 13:50:04 -07:00
David Heinemeier Hansson
560d0698ac Merge pull request #426 from northeastprince/fix-site-in-gemspec
Fix site URL in gemspec
2023-08-23 13:47:26 -07:00
fig
f40e8e9af1 Merge branch 'fix-421' of https://github.com/fig/mrsk into fix-421 2023-08-23 15:22:41 +01:00
fig
1ab7405e36 require ActiveSupport module to provide String#remove
fixes #421
2023-08-23 15:17:26 +01:00
Matt Almeida
aeadd7c11f Fix site URL in gemspec 2023-08-23 15:15:51 +02:00
Donal McBreen
d0fbf538d3 Add integration test hooks back in 2023-08-23 07:36:48 +01:00
David Heinemeier Hansson
cfe77934e8 Update README.md
Point all docs to the site so we don't duplicate everything.
2023-08-22 17:11:26 -07:00
David Heinemeier Hansson
3f6ca1648e Update docker-publish.yml
Require setting tag
2023-08-22 15:44:07 -07:00
David Heinemeier Hansson
7c6d302baa Update docker-publish.yml
Allow manual invocation
2023-08-22 15:20:02 -07:00
fig
b8eb50b982 require ActiveSupport module to provide String#remove
fixes #421
2023-08-22 20:58:48 +01:00
David Heinemeier Hansson
d981c3c968 Move hooks 2023-08-22 12:47:00 -07:00
David Heinemeier Hansson
416860d9b0 Update docker-publish.yml
Reflect rename
2023-08-22 12:34:57 -07:00
David Heinemeier Hansson
33d5d7e9a2 Update README.md
Point to name change.
2023-08-22 12:20:24 -07:00
David Heinemeier Hansson
99c1102a3a Update README.md
Will do a new video shortly.
2023-08-22 12:13:54 -07:00
David Heinemeier Hansson
ac11089c7a Bump version for 0.16.0 2023-08-22 11:42:32 -07:00
David Heinemeier Hansson
180ca219df Merge pull request #423 from basecamp/rename
Rename project to Kamal
2023-08-22 11:41:42 -07:00
David Heinemeier Hansson
dc1421a1fc Correct casing 2023-08-22 09:22:32 -07:00
David Heinemeier Hansson
c4a203e648 Rename to Kamal 2023-08-22 08:24:31 -07:00
Donal McBreen
e2c3709d74 Merge pull request #417 from manastyretskyi/main
Fix builder registry cache when using default registry
2023-08-17 14:08:05 +01:00
Liubomyr Manastyretskyi
f68a33465f Fix review comments 2023-08-17 11:58:14 +03:00
Donal McBreen
e7bc74d9ee Merge pull request #418 from mrsked/ssh-logging
Configurable log levels
2023-08-16 07:22:18 +01:00
Donal McBreen
1163c3de07 Configurable log levels
Allow ssh log_level to be set - this will help to debug connection
issues.
2023-08-15 16:51:56 +01:00
Donal McBreen
715cd94bbf Merge pull request #413 from mrsked/extract-version-from-container-name-correctly
Extract versions that contains dashes
2023-08-15 15:11:03 +01:00
Donal McBreen
dda7099b2f Merge pull request #414 from mrsked/traefik-start-stop-run-errors
Don't hide Traefik errors
2023-08-15 15:10:47 +01:00
Donal McBreen
4262fce863 Merge pull request #415 from igor-alexandrov/fix-builder-configuration-validation
Removed validation for remote and local builder params
2023-08-15 15:10:23 +01:00
Liubomyr Manastyretskyi
6774675547 Fix builder registry cache when using default registry 2023-08-13 12:04:03 +03:00
Igor Alexandrov
0c52a1053e Removed not needed configuration test 2023-08-08 19:14:03 +04:00
Igor Alexandrov
c24c7abb79 Fix for https://github.com/mrsked/mrsk/issues/407 2023-08-08 19:04:35 +04:00
Donal McBreen
c2d7fd775f Don't hide Traefik errors
When stopping or starting Traefik, don't hide important errors.

Docker doesn't return an error when starting a started container or
stopping a stopped container.

When rebooting we want to know about errors during run as we've just
stopped and removed the previous container.

When booting, we want to leave the running container if it exists,
restart a stopped container and run a new one if none exists.

We can implement this with `docker start ... || docker run ...`:
- if the container is started, `docker start` will exit with 0
- if the container is stopped, `docker start` will start it and exit with 0
- if the container doesn't exist, `docker start` will return a non zero
exit code and `docker run` will create a new container. Any errors in
`docker run` will be returned.
2023-08-08 15:41:16 +01:00
Donal McBreen
4dd8208290 Extract versions that contains dashes
The version extraction assumed that the version is everything after the
last `-` in the container name. This doesn't work if you deploy a
non-MRSK generated version that contains a `-`.

To fix we'll generate the non version prefix and strip it off. In some
places for this to work we need to make sure to pass the role through.

Fixes: https://github.com/mrsked/mrsk/issues/402
2023-08-08 14:16:32 +01:00
Donal McBreen
aa89ededde Merge pull request #399 from mrsked/manage-ssh-connection-starts
Manage SSH connection starts
2023-08-07 14:37:34 +01:00
David Heinemeier Hansson
299b166db7 Merge pull request #389 from brunoprietog/include-role-options-when-executing-commands
Include role options when executing commands
2023-07-26 14:04:28 +02:00
Donal McBreen
94d6a763a8 Extract ssh and sshkit configuration 2023-07-26 12:26:23 +01:00
Donal McBreen
752ff53458 Merge pull request #396 from igor-alexandrov/track-uncommitted-changes
Log uncommitted changes during deploy
2023-07-25 14:35:44 +01:00
Donal McBreen
eb8c97a417 Document new sshkit settings 2023-07-25 13:09:49 +01:00
Donal McBreen
f64b596907 Prevent SSH connection restarts
Set a high idle timeout on the sshkit connection pool. This will
reduce the incidence of re-connection storms when a deployment has been
idle for a while (e.g. when waiting for a docker build).

The default timeout was 30 seconds, so we'll enable keepalives at a
30s interval to match. This is to help prevent connections from being
killed during long idle periods.
2023-07-25 13:09:46 +01:00
Donal McBreen
b25cfa178b Limit SSH start concurrency
Starting many (90+) SSH connections has caused us some issues such as
failed DNS lookups and hitting process file descriptor limits.

To mitigate this, patch SSHKit::Backend::Netssh to limit concurrency of
connection starts. We'll default to 30 at a time which seems to work
without issue, but can be configured via:

```
sshkit:
  max_concurrent_starts: 10
```
2023-07-25 13:08:44 +01:00
Donal McBreen
edcfc77d95 Bump version for 0.15.1 2023-07-25 13:07:04 +01:00
Donal McBreen
a71e167a03 Merge pull request #400 from mrsked/revert-386-ssh-log-levels
Revert "Configurable SSH log levels"
2023-07-25 13:04:21 +01:00
Donal McBreen
2daaf442fa Revert "Configurable SSH log levels" 2023-07-25 12:53:45 +01:00
Igor Alexandrov
d414253393 Updated uncommitted notification text 2023-07-24 20:12:22 +04:00
Bruno Prieto
cbd180205d Include role options when executing commands 2023-07-24 17:45:24 +02:00
Donal McBreen
61b7dc90f2 Bump version for 0.15.0 2023-07-24 13:43:50 +01:00
David Heinemeier Hansson
f6442513ae Merge pull request #357 from igor-alexandrov/documentation-update
Updated README with info for Rails <7 usage
2023-07-24 14:39:48 +02:00
Igor Alexandrov
ea941f33f9 Moved uncommitted changes message out of run_locally block 2023-07-21 22:45:23 +04:00
Igor Alexandrov
9c2a1dc7cd Removed commented code in tests 2023-07-21 18:44:01 +04:00
Igor Alexandrov
0cfafd1d25 Log uncommitted changes during deploy 2023-07-21 18:37:45 +04:00
Donal McBreen
5e8df58e6b Merge pull request #393 from basecamp/rolling-traefik-restarts
Support a --rolling option for traefik reboots
2023-07-19 16:43:59 +01:00
Lewis Buckley
9d5a6d1321 Document the rolling option for traefik reboots 2023-07-19 15:03:15 +01:00
Lewis Buckley
ecfd258093 Document the rolling reboot option 2023-07-19 14:58:46 +01:00
Lewis Buckley
313f89a108 Merge branch 'main' into rolling-traefik-restarts
* main:
  Removed not needed MRSK.traefik.run command in Traefil reboot
  Updated README with locking directory name
  Include service name to lock details
  Configurable SSH log levels
  Add registry container output to debug
  Minor tweaks to hooks section in readme
  Update README.md
  Updated README.md to make setup examples consistent
  Login to the registry proactively before stoping Accessory and Traefik
2023-07-19 14:46:16 +01:00
Lewis Buckley
9ab448e186 Support a --rolling option for traefik reboots 2023-07-19 14:39:27 +01:00
Donal McBreen
e1433f3895 Merge pull request #349 from igor-alexandrov/login-to-registry-proactively
Login to the registry proactively before stoping Accessory and Traefik
2023-07-19 13:36:00 +01:00
Igor Alexandrov
a29e188c90 Removed not needed MRSK.traefik.run command in Traefil reboot 2023-07-19 15:08:42 +04:00
Donal McBreen
95e3915991 Merge pull request #386 from mrsked/ssh-log-levels
Configurable SSH log levels
2023-07-17 14:09:21 +01:00
Donal McBreen
30d342183d Merge pull request #387 from igor-alexandrov/lock-with-service-name
Include service name to lock details
2023-07-17 14:07:22 +01:00
Igor Alexandrov
83f5f3f053 Updated README with locking directory name 2023-07-17 10:39:50 +04:00
Igor Alexandrov
e6ca270537 Include service name to lock details 2023-07-15 21:50:39 +04:00
Donal McBreen
cd88c49c42 Configurable SSH log levels
Allow ssh log_level to be set to debug connection issues.
2023-07-14 16:08:47 +01:00
Donal McBreen
d03195ce1c Merge pull request #385 from mrsked/integration-test-registry-debug
Add registry container output to debug
2023-07-14 13:50:31 +01:00
Donal McBreen
da1c049829 Add registry container output to debug 2023-07-14 13:41:30 +01:00
Donal McBreen
4095e1853d Merge pull request #356 from helgeblod/fix-readme-username-typo
Update README.md to make setup examples consistent
2023-07-12 13:24:51 +01:00
Donal McBreen
dbc9989730 Merge pull request #364 from nickhammond/patch-2
Update README.md
2023-07-12 13:22:53 +01:00
Donal McBreen
e493369453 Merge pull request #379 from nickhammond/patch-3
Minor tweaks to hooks section in readme
2023-07-12 11:29:36 +01:00
Nick Hammond
e760cfa457 Minor tweaks to hooks section in readme 2023-07-08 10:59:55 -06:00
Nick Hammond
f8d651af0d Update README.md
Add a note about utilizing biometrics with the envify example.
2023-06-28 10:03:27 -06:00
Igor Alexandrov
08172be375 Updated README with info for Rails <7 usage 2023-06-26 16:05:24 +04:00
Jonas Helgemo
a3cc2317e2 Updated README.md to make setup examples consistent
- SSH and apt examples should use same username
2023-06-26 11:57:23 +02:00
Igor Alexandrov
2746a48e88 Login to the registry proactively before stoping Accessory and Traefik 2023-06-22 15:13:47 +04:00
David Heinemeier Hansson
9a501867b4 Bump version for 0.14.0 2023-06-20 17:02:42 +02:00
David Heinemeier Hansson
c5397ff51e Merge pull request #342 from mrsked/only-require-secrets-when-mutating
Only require secrets when mutating
2023-06-20 16:50:14 +02:00
Donal McBreen
4950f61a87 Only require secrets when mutating
Rename `with_lock` to more generic `mutating` and move the env_args
check to that point. This allows read-only actions to be run without
requiring secrets.
2023-06-20 15:39:51 +01:00
David Heinemeier Hansson
08d8790851 Merge pull request #337 from igor-alexandrov/feature/cache
Support for Docker multistage build cache
2023-06-20 11:38:46 +02:00
Igor Alexandrov
02256ac8fe More code style improvements 2023-06-19 18:22:07 +04:00
Igor Alexandrov
dadd8225da Various code style improvements 2023-06-18 23:39:44 +04:00
Igor Alexandrov
aa28ee0f3e Inroduce Native::Cached builder 2023-06-18 22:45:04 +04:00
David Heinemeier Hansson
2007ab475e Merge pull request #327 from bestfriendfinance/fix-run-over-ssh-with-proxy-command
Add support for proxy_command to run_over_ssh
2023-06-18 17:18:50 +02:00
Igor Alexandrov
4df3389d09 Added support for multistage build cache 2023-06-18 19:02:10 +04:00
Matt Robinson
21b13bf8d3 Add support for proxy_command to run_over_ssh 2023-06-16 08:22:10 -03:00
David Heinemeier Hansson
6e6f696717 Merge pull request #335 from mrsked/specify-min-version
Add a minimum version setting
2023-06-16 10:02:40 +02:00
Donal McBreen
98c12a254e Add a minimum version setting
Allow a minimum MRSK version to be specified in the config.
2023-06-15 14:53:03 +01:00
Donal McBreen
f0301d2007 Merge pull request #328 from igor-alexandrov/319-override-default-traefik-args
Added ability to override default Traefik command line arguments
2023-06-15 14:49:56 +01:00
Igor Alexandrov
d3f5e9efe8 Updated Traefik CLI test 2023-06-15 17:11:20 +04:00
Igor Alexandrov
d9b3fac17a Added ability to override default Traefik command line arguments 2023-06-15 15:41:20 +04:00
Donal McBreen
cd5c41ddbe Merge pull request #334 from basecamp/fix-ssh-symlink
Fix ssh symlink
2023-06-15 12:14:30 +01:00
Donal McBreen
a14c6141e5 Dump container logs on failure 2023-06-15 12:05:50 +01:00
Donal McBreen
95d6ee5031 Remove /root/.ssh before symlinking
Ensure the symlinks are created correctly whether or not /root/.ssh
already exists.
2023-06-15 12:02:56 +01:00
David Heinemeier Hansson
80a4ca4f8a Merge pull request #331 from basecamp/fix-sample-pre-deploy-hook
Fix up the sample pre-deploy hook
2023-06-12 14:09:29 +02:00
Donal McBreen
12ca865e71 Fix up the sample pre-deploy hook
- Fix the shebang
- Extract a GithubStatusChecks class
2023-06-12 12:47:51 +01:00
David Heinemeier Hansson
66b4a0ea40 Bump version for 0.13.2 2023-06-01 12:54:06 +02:00
David Heinemeier Hansson
04b39ea798 Fix spelling 2023-05-31 18:15:21 +02:00
David Heinemeier Hansson
ae55a7b5d8 Merge pull request #325 from mrsked/revert-324-fix-filename-typo
Revert "Fix typo in fixture filename"
2023-05-31 18:14:51 +02:00
David Heinemeier Hansson
601cfbd95e Revert "Fix typo in fixture filename" 2023-05-31 18:14:43 +02:00
David Heinemeier Hansson
9fdc85c2e6 Merge pull request #324 from basecamp/fix-filename-typo
Fix typo in fixture filename
2023-05-31 18:13:11 +02:00
Donal McBreen
222eda6085 Fix typo in fixture filename
Following on from https://github.com/mrsked/mrsk/pull/320, fix the
fixture filename as well
2023-05-31 17:11:13 +01:00
David Heinemeier Hansson
504a09ef1d Merge pull request #318 from basecamp/pre-deploy-hook
Add a pre-deploy hook
2023-05-31 17:59:46 +02:00
David Heinemeier Hansson
5a25f073f7 Merge pull request #320 from jsoref/spelling
Spelling
2023-05-31 17:59:18 +02:00
David Heinemeier Hansson
c8f521c0e8 Merge pull request #323 from basecamp/prefix-docker-host-with-real-host
Prefix container hostname with the underlying one
2023-05-31 17:58:55 +02:00
Donal McBreen
28d6a131a9 Prefix container hostname with the underlying one
To make it easier to identity where a docker container is running,
prefix its hostname with the underlying one from the host.

Docker chooses a 12 character random hex string by default, so we'll
keep that as the suffix.
2023-05-31 16:22:25 +01:00
David Heinemeier Hansson
3a9075b8ba Merge pull request #321 from basecamp/more-robust-image-pruning 2023-05-31 13:02:48 +02:00
Donal McBreen
079d9538bb Improve image pruning robustness
If you different images with the same git SHA, on the second deploy the
tag is moved and the first image becomes untagged. It may however still
be attached to an existing container.

To handle this:
1. Initially prune dangling images - this will remove any untagged
images that are not attached to an existing image
2. Then filter out the untagged images when deleting tagged images - any
that remain will be attached to a container.

The second issue is that `docker container ls -a --format '{{.Image}}`
will sometimes return the image id rather than a tag. This means that
the image doesn't get filtered out when we grep to remove the active
images.

To fix that we'll grep against both the image id and repo:tag.
2023-05-31 10:17:52 +01:00
Josh Soref
8e94c21729 spelling: with
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-05-29 20:46:34 -04:00
Josh Soref
b536fcfa43 spelling: percentage
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-05-29 20:46:34 -04:00
Josh Soref
85005be07f spelling: message
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-05-29 20:46:34 -04:00
Josh Soref
fc00392d68 spelling: installed
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-05-29 20:46:34 -04:00
Josh Soref
fe9affa349 spelling: healthchecks
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-05-29 20:46:34 -04:00
Josh Soref
3ecb3a4bfc spelling: guidelines
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-05-29 20:46:34 -04:00
Josh Soref
787812cdc2 spelling: every time
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-05-29 20:46:34 -04:00
Josh Soref
91fb85d6b5 spelling: etc.
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-05-29 20:46:34 -04:00
Donal McBreen
db0bf6bb16 Add a pre-deploy hook
Useful for checking the status of CI before deploying. Doing this at
this point in the deployment maximises the parallelisation of building
and running CI.
2023-05-29 16:06:41 +01:00
David Heinemeier Hansson
de2de19434 Merge pull request #315 from basecamp/prune-unused-images
Prune unused images correctly
2023-05-29 11:42:49 +02:00
David Heinemeier Hansson
f9fbebaa72 Merge pull request #316 from f440/fix-typo
Fix typo
2023-05-29 11:42:26 +02:00
Donal McBreen
1e300f3798 Wait longer for app to come up 2023-05-29 08:31:19 +01:00
f440
0373f6c4de Fix typo 2023-05-27 16:27:19 +09:00
Donal McBreen
9037088f99 Increase nginx timeouts in load balancer 2023-05-25 17:31:20 +01:00
Donal McBreen
ff7a1e6726 Prune unused images correctly
dangling=true doesn't prune any images, as we are not creating dangling
images.

Using --all should remove unused images, but it considers the Git SHA
tag on the latest image to be unused (presumably because there are two
tags, the SHA and latest and the running container is only considered to
be using "latest"). As a result it deletes the tag, which means that we
can't rollback to that SHA later.

Its a bit more complicated to only remove images that are not referenced
by any containers.

First we find the tags we want to keep from the containers (running and
stopped).

Then we append the latest tag to that list.

Then we get a full list of image tags and remove those tags from that
list (using `grep -v -w`).

Finally we pass the tags to `docker rmi`. That either deletes the tag if
there are other references to the image or both the tag and the image if
it is the only one.
2023-05-25 17:16:46 +01:00
David Heinemeier Hansson
602aa43496 Bump version for 0.13.1 2023-05-25 14:04:29 +02:00
David Heinemeier Hansson
e35334e5fe Merge pull request #313 from basecamp/stop-restarting-containers
Stop containers with restarting status
2023-05-25 14:04:09 +02:00
Donal McBreen
cedb8d900f Stop containers with restarting status
When stopping the old container we need to also look for ones with a
restarting status.
2023-05-25 12:10:26 +01:00
David Heinemeier Hansson
8f0b7829ce Bump version for 0.13.0 2023-05-25 12:05:04 +02:00
David Heinemeier Hansson
57e4f08c4c Merge pull request #308 from tannakartikey/hooks_small_fix
Hooks sample files typo fix
2023-05-25 09:02:25 +02:00
David Heinemeier Hansson
a8bfe90fbe Merge pull request #312 from shafy/docs_docker_setup
docs: change intro command to mrsk setup
2023-05-25 09:01:04 +02:00
David Heinemeier Hansson
f114dd71f6 Merge pull request #311 from basecamp/pre-connect-hook
Add a pre-connect hook
2023-05-25 08:58:19 +02:00
Can Olcer
d1b5b9cf7a docs: change intro command to mrsk setup 2023-05-24 20:32:41 +02:00
Donal McBreen
66f9ce0e90 Add a pre-connect hook
This can be used for hooks that should run before connecting to remote
hosts. An example use case is pre-warming DNS.
2023-05-24 14:39:30 +01:00
Kartikey Tanna
956ab3560b Hooks typo fix 2023-05-23 22:12:49 +05:30
David Heinemeier Hansson
483b893018 Merge pull request #291 from basecamp/hooks
MRSK Hooks
2023-05-23 17:07:04 +02:00
Donal McBreen
19f0f40adf Add skip_hooks option 2023-05-23 15:56:47 +01:00
Donal McBreen
f9cb87e55a Fixup rebase issues 2023-05-23 14:10:38 +01:00
Donal McBreen
cc2b321d93 Combine post-deploy and post-rollback 2023-05-23 13:57:24 +01:00
Donal McBreen
004f1b04e6 Remove the skip_broadcast option 2023-05-23 13:57:00 +01:00
Donal McBreen
3b695ae127 Add service_version and add running hook message 2023-05-23 13:56:19 +01:00
Donal McBreen
258887a451 Set sample hook permissions and preserve when copying 2023-05-23 13:56:19 +01:00
Donal McBreen
9fd184dc32 Add post-deploy and post-rollback hooks
These replace the custom audit_broadcast_cmd code. An additional env
variable MRSK_RUNTIME is passed to them.

The audit broadcast after booting an accessory has been removed.
2023-05-23 13:56:16 +01:00
Donal McBreen
38023fe538 Remove post push hook 2023-05-23 13:55:05 +01:00
Donal McBreen
0bc1fbfb74 Set max-concurrent-downloads to 1 to prevent timeouts 2023-05-23 13:55:05 +01:00
David Heinemeier Hansson
5ab630cb03 Style 2023-05-23 13:55:04 +01:00
Donal McBreen
910f14e9c0 Add configuration for hooks_path 2023-05-23 13:55:04 +01:00
Donal McBreen
f3ec9f19c8 Add debug for failed version checks 2023-05-23 13:55:04 +01:00
Donal McBreen
58c1096a90 MRSK hooks
Adds hooks to MRSK. Currently just two hooks, pre-build and post-push.

We could break the build and push into two separate commands if we
found the need for post-build and/or pre-push hooks.

Hooks are stored in `.mrsk/hooks`. Running `mrsk init` will now create
that folder and add sample hook scripts.

Hooks returning non-zero exit codes will abort the current command.

Further potential work here:
- We could replace the audit broadcast command with a
post-deploy/post-rollback hook or similar
- Maybe provide pre-command/post-command hooks that run after every
mrsk invocation
- Also look for hooks in `~/.mrsk/hooks`
2023-05-23 13:55:04 +01:00
Donal McBreen
340ed94fa9 Make verify_local_dependencies private
We don't need to what it returns, it raises if there is a problem.

Move it out of the run_locally block to make it easier to add hooks.
2023-05-23 13:55:04 +01:00
David Heinemeier Hansson
4e9c39f26d Merge pull request #271 from basecamp/app-boot-for-rollback
Call app:boot to rollback
2023-05-23 13:17:30 +02:00
David Heinemeier Hansson
d08aacadac Merge pull request #287 from Novtopro/traefik-inject-environment-variables
Allow to inject environment variables to traefik
2023-05-22 10:10:34 +02:00
David Heinemeier Hansson
702490d10f Merge pull request #305 from johnmcdowall/update_readme_for_aws_ecr
Update the README with info on AWS ECR
2023-05-22 09:58:33 +02:00
David Heinemeier Hansson
13079dd2a3 Merge pull request #299 from basecamp/report-host-with-error
Report the host an error occurred on
2023-05-22 09:57:22 +02:00
John McDowall
7daee9a0df Update the README on the use of shelling out to the aws cli command to obtain the token for ECR automatically 2023-05-20 13:46:05 -07:00
Donal McBreen
f7c5840473 Report the host an error occurred on
The cause message doesn't include the host the error occurred on.

Before:

```
$ mrsk deploy
Acquiring the deploy lock...
  Finished all in 0.1 seconds
  ERROR (SocketError): getaddrinfo: nodename nor servname provided, or not known
```

After:

```
$ mrsk deploy -d staging
Acquiring the deploy lock...
  Finished all in 0.1 seconds
  ERROR (SocketError): Exception while executing on host server-123: getaddrinfo: nodename nor servname provided, or not known
```
2023-05-17 08:51:01 +01:00
David Heinemeier Hansson
a7d869ad40 Merge pull request #298 from basecamp/more-integration-tests 2023-05-17 09:39:00 +02:00
Donal McBreen
7cd25fd163 Add more integration tests
Add tests for main, app, accessory, traefik and lock commands.
Other commands are generally covered by the main tests.

Also adds some changes to speed up the integration specs:
- Use a persistent volume for the registry so we can push images to to
reuse between runs (also gets around docker hub rate limits)
- Use persistent volume for mrsk gem install, to avoid re-installing
between tests
- Shorter stop wait time
- Shorter connection timeouts on the load balancer

Takes just over 2 minutes to run all tests locally on an M1 Mac
after docker caches are primed.
2023-05-16 10:35:35 +01:00
Donal McBreen
ee25f200d7 Call app:boot to rollback
The code in Mrsk::Cli::Main#rollback was very similar to
Mrsk::Cli::App#boot.

Modify Mrsk::Cli::App#boot so it can handle rollbacks by:
1. Only renaming running containers
2. Trying first to start then run the new container
2023-05-16 08:59:07 +01:00
David Heinemeier Hansson
059388cb02 Merge pull request #292 from basecamp/unique-uncommited-changes-version
Highlight uncommitted changes in version
2023-05-15 14:05:31 +02:00
Donal McBreen
a5ef1f254f Highlight uncommitted changes in version
If there are uncommitted changes in the app repository when building,
then append `_uncommitted_<random>` to it to distinguish the image
from one built from a clean checkout.

Also change the version used when renaming a container on redeploy to
distinguish and explain the version suffixes.
2023-05-12 11:08:48 +01:00
David Heinemeier Hansson
15e8ac0ced Merge pull request #290 from acidtib/check-interval
add healthcheck interval option
2023-05-11 14:13:13 +02:00
acidtib
9a31c20321 add healthcheck interval option 2023-05-10 20:27:21 -06:00
River He
44b83151e3 Allow to inject environment variables to traefik 2023-05-10 03:18:26 +00:00
David Heinemeier Hansson
0defcbb640 Merge pull request #283 from basecamp/better-lock-messages
Better lock messages
2023-05-09 16:06:50 +02:00
Donal McBreen
5d33fb6c33 Better lock messages
- Debug verbosity commands
- Show lock status when we fail to acquire it
- Include lock acquire/release in runtime
2023-05-09 14:17:58 +01:00
David Heinemeier Hansson
e9d838ec46 Update README.md 2023-05-09 14:32:02 +02:00
David Heinemeier Hansson
ee319fee1c Merge pull request #277 from xiaohui-zhangxh/bugfix/readme
Fix readme bug on traefik volumes option
2023-05-09 12:37:38 +02:00
xiaohui
5646f6cc64 fix readme bug on traefik volumes option 2023-05-07 23:58:38 +08:00
David Heinemeier Hansson
31aaa82991 Merge pull request #272 from olimart/patch-1
Fix typo mesasge --> message
2023-05-07 10:58:11 +02:00
Olivier
5ea552be40 Fix typo in message 2023-05-05 10:43:21 -04:00
David Heinemeier Hansson
625be70e4d Bump version for 0.12.1 2023-05-05 14:33:25 +02:00
David Heinemeier Hansson
aafaee7ac8 Merge pull request #223 from basecamp/customizable-audit-broadcast
Allow customizing audit broadcast with env
2023-05-05 14:30:04 +02:00
David Heinemeier Hansson
97a190300d Merge pull request #270 from basecamp/fix-aggressive-prune-breaking-rollback
Fix aggressive prune breaking rollback
2023-05-05 14:28:22 +02:00
Donal McBreen
326711a3e0 Fix aggressive prune breaking rollback
In the image prune command --all overrides --dangling=true. This removes
the image git sha image tag for the latest image which prevented
us from rolling back to it.

I've updated the integration test to now test deploy, redeploy and
rollback.
2023-05-05 12:13:14 +01:00
Kevin McConnell
82be521e66 Merge branch 'main' into customizable-audit-broadcast
* main:
  Fix staging label bug
  Fix typo
  Capture container health log when unhealthy
  Bump version for 0.12.0
2023-05-05 11:40:29 +01:00
David Heinemeier Hansson
21110080d5 Merge pull request #267 from danthegoodman1/patch-1
Fix staging label bug in README
2023-05-05 11:25:22 +02:00
David Heinemeier Hansson
ef107c41b6 Merge pull request #265 from Jberczel/improve-healthcheck-logging
Improve healthcheck logging
2023-05-05 11:24:55 +02:00
Dan Goodman
1bf4b6b76f Fix staging label bug
I think this is the correct fix based on the `service-role-destination` format, but seeing as it wasn't changed I assumed it was incorrect.
2023-05-04 17:47:17 -04:00
Jeremy Daer
36a3b13bf4 Fix SSHKit #command override args mangling 2023-05-04 08:58:18 -07:00
Jberczel
01483140f5 Fix typo 2023-05-03 15:03:05 -04:00
Jberczel
0e19ead37c Capture container health log when unhealthy 2023-05-03 15:03:05 -04:00
Jeremy Daer
048aecf352 Audit details (#1)
Audit details

* Audit logs and broadcasts accept `details` whose values are included as log tags and MRSK_* env vars passed to the broadcast command
* Commands may return execution options to the CLI in their args list
* Introduce `mrsk broadcast` helper for sending audit broadcasts
* Report UTC time, not local time, in audit logs. Standardize on ISO 8601 format
2023-05-02 11:42:05 -07:00
David Heinemeier Hansson
38c85e8021 Bump version for 0.12.0 2023-05-02 17:23:10 +02:00
David Heinemeier Hansson
88a7413b3e Merge branch 'main' into pr/223
* main:
  Don't run actions twice on PRs
  Further distinguish dependency verification
  Naming
  Reveal configured dockerfile path
  Style
  Distinguish from server dependencies
  Distinguish from local dependency verification
  Improve clarity and intent
  Style
  Style
  Style
  Add local dependencies check
  Bootstrap: use multi-platform installer
2023-05-02 14:44:16 +02:00
David Heinemeier Hansson
9cc73fed9a Merge branch 'main' into pr/223
* main:
  Simplify domain language to just "boot" and unscoped config keys
  Retain a fixed number of containers when pruning
  Don't assume rolling back in message
  Check all hosts before rolling back
  Ensure Traefik service name is consistent
  Extend traefik delay by 1 second
  Include traefik access logs
  Check if we are still getting a 404
  Also dump load balancer logs
  Dump traefik logs when app not booted
  Fix missing for apt-get
  Report on container health after failure
  Fix the integration test healthcheck
  Allow percentage-based rolling deployments
  Move `group_limit` & `group_wait` under `boot`
  Limit rolling deployment to boot operation
  Allow performing boot & start operations in groups
2023-05-02 14:43:17 +02:00
David Heinemeier Hansson
787ef96639 Don't run actions twice on PRs 2023-05-02 14:41:18 +02:00
David Heinemeier Hansson
1e8edc25e2 Merge pull request #205 from basecamp/docker-readiness
Bootstrap: multi-OS Docker install
2023-05-02 14:35:26 +02:00
David Heinemeier Hansson
b7877c59b4 Merge branch 'main' into docker-readiness 2023-05-02 14:30:35 +02:00
David Heinemeier Hansson
35b5b317af Merge branch 'main' into pr/205
* main:
  Simplify domain language to just "boot" and unscoped config keys
  Retain a fixed number of containers when pruning
  Don't assume rolling back in message
  Check all hosts before rolling back
  Ensure Traefik service name is consistent
  Extend traefik delay by 1 second
  Include traefik access logs
  Check if we are still getting a 404
  Also dump load balancer logs
  Dump traefik logs when app not booted
  Fix missing for apt-get
  Report on container health after failure
  Fix the integration test healthcheck
  Allow percentage-based rolling deployments
  Move `group_limit` & `group_wait` under `boot`
  Limit rolling deployment to boot operation
  Allow performing boot & start operations in groups
2023-05-02 14:29:06 +02:00
David Heinemeier Hansson
4c448f7eb1 Merge pull request #256 from Jberczel/check-local-dependencies
Add local dependencies check
2023-05-02 14:13:23 +02:00
David Heinemeier Hansson
263a24afe3 Further distinguish dependency verification 2023-05-02 14:09:10 +02:00
David Heinemeier Hansson
a2d99e48bf Naming 2023-05-02 14:08:29 +02:00
David Heinemeier Hansson
a22e27dbf8 Reveal configured dockerfile path 2023-05-02 14:07:47 +02:00
David Heinemeier Hansson
bb74a74dc4 Style 2023-05-02 14:07:30 +02:00
David Heinemeier Hansson
c611a1616a Distinguish from server dependencies 2023-05-02 14:06:06 +02:00
David Heinemeier Hansson
98e7b995d5 Distinguish from local dependency verification 2023-05-02 14:04:37 +02:00
David Heinemeier Hansson
ae2effb80c Improve clarity and intent 2023-05-02 14:04:23 +02:00
David Heinemeier Hansson
f719540e0c Style 2023-05-02 13:35:05 +02:00
David Heinemeier Hansson
cbda851436 Style 2023-05-02 13:34:56 +02:00
David Heinemeier Hansson
8854bb63a1 Merge pull request #254 from basecamp/retain-last-5-containers
Retain a fixed number of containers when pruning
2023-05-02 13:16:49 +02:00
David Heinemeier Hansson
35ea9f3c81 Merge pull request #255 from basecamp/check-all-hosts-for-rollback-container
Check all hosts before rolling back
2023-05-02 13:16:03 +02:00
David Heinemeier Hansson
18312f5191 Merge pull request #253 from basecamp/ensure-consistent-service-name
Ensure Traefik service name is consistent
2023-05-02 13:15:36 +02:00
David Heinemeier Hansson
71bc9bcf54 Merge pull request #222 from basecamp/deploy-groups
Allow booting containers in groups for rolling restarts
2023-05-02 13:14:32 +02:00
David Heinemeier Hansson
c83b74dcb7 Simplify domain language to just "boot" and unscoped config keys 2023-05-02 13:11:31 +02:00
Donal McBreen
971a91da15 Retain a fixed number of containers when pruning
Time based container and image retention can have variable space
requirements depending on how often we deploy.

- Only prune stopped containers, retaining the 5 newest
- Then prune dangling images so we only keep images for the retained
containers.
2023-05-02 10:15:08 +01:00
Donal McBreen
86d6f8d674 Don't assume rolling back in message 2023-05-02 10:14:50 +01:00
Donal McBreen
7fe24d5048 Check all hosts before rolling back
Hosts could end up out of sync with each other if prune commands are run
manually or when new hosts are added.

Before rolling back confirm that the required container is available on
all hosts and roles.
2023-05-02 10:14:50 +01:00
Kevin McConnell
a72f95f44d Ensure Traefik service name is consistent
If we don't specify any service properties when labelling containers,
the generated service will be named according to the container. However,
we change the container name on every deployment (as it is versioned),
which means that the auto-generated service name will be different in
each container.

That is a problem for two reasons:

- Multiple containers share a common router while a deployment is
  happening. At this point, the router configuration will be different
  between the containers; Traefik flags this as an error, and stops
  routing to the containers until it's resolved.
- We allow custom labels to be set in an app's config. In order to
  define custom configuration on the service, we'll need to know what
  it will be called.

Changed to force the service name by setting one of its properties.
2023-05-02 09:43:04 +01:00
David Heinemeier Hansson
dc3be30b16 Style 2023-05-02 10:29:49 +02:00
David Heinemeier Hansson
54881a0298 Merge pull request #250 from basecamp/integration-test-healthcheck-wget
Integration test healthcheck wget
2023-05-02 10:27:49 +02:00
David Heinemeier Hansson
19527b4f65 Merge branch 'main' into customizable-audit-broadcast 2023-05-02 10:25:25 +02:00
Jberczel
bfb70b2118 Add local dependencies check
Add checks for:

* Docker installed locally
* Docker buildx plugin installed locally
* Dockerfile exists

If checks fail, it will halt deployment and provide more specific error messages.

Also adds a cli subcommand:
`mrsk build dependencies`

Fixes: #109 and #237
2023-05-01 16:32:41 -04:00
Jeremy Daer
e85bd5ff63 Bootstrap: use multi-platform installer
* Limit auto-install to root users; otherwise, give manual install guidance
* Support non-Debian/Ubuntu with the multi-OS get.docker.com installer
2023-05-01 13:26:00 -07:00
Donal McBreen
d0f66db33c Extend traefik delay by 1 second 2023-05-01 18:58:46 +01:00
Donal McBreen
650f9b1fbf Include traefik access logs 2023-05-01 18:55:10 +01:00
Donal McBreen
1170e2311e Check if we are still getting a 404 2023-05-01 18:32:07 +01:00
Donal McBreen
94f87edded Also dump load balancer logs 2023-05-01 18:27:08 +01:00
Donal McBreen
548a1019c1 Dump traefik logs when app not booted 2023-05-01 18:21:22 +01:00
Donal McBreen
ca2e2bac2e Fix missing for apt-get 2023-05-01 12:50:45 +01:00
Donal McBreen
494a1ae089 Report on container health after failure 2023-05-01 12:13:12 +01:00
Donal McBreen
a77428143f Fix the integration test healthcheck
The alpine nginx container doesn't contain curl, so let's override the
healthcheck command to use wget.
2023-05-01 12:11:24 +01:00
David Heinemeier Hansson
4fa6a6c06d Merge pull request #219 from basecamp/docker-health-checks 2023-04-28 11:43:33 +02:00
David Heinemeier Hansson
2ad0dc0703 Merge pull request #241 from Jberczel/fix-traefik-subcommand-typo 2023-04-28 11:38:46 +02:00
David Heinemeier Hansson
df067e4893 Merge pull request #244 from basecamp/get-lock-status-without-invoke 2023-04-25 18:57:05 +02:00
Donal McBreen
cd668066ff Get lock status by executing directly
Getting the lock status with invoke passes through any options from the
original command which will raise an exception if they are not also
valid for the lock status command.

Fixes https://github.com/mrsked/mrsk/issues/239
2023-04-25 16:57:02 +01:00
David Heinemeier Hansson
1a7d123746 Merge pull request #245 from basecamp/integration-test-wait-for-healthy
Wait for healthy containers in integration test
2023-04-25 16:54:22 +02:00
Donal McBreen
52ca5b846a Wait for healthy containers in integration test
Rather than waiting 5 seconds and hoping for the best after we boot
docker compose, add docker healthchecks and wait for all the containers
to be healthy.
2023-04-25 15:41:25 +01:00
Jberczel
126e0bbd06 Fix traefik remove_image desc typo 2023-04-24 17:40:28 -04:00
David Heinemeier Hansson
9ec3895dab Merge pull request #216 from dmrty/add-ssh-client-to-dockerised-mrsk 2023-04-15 08:29:49 +02:00
David Heinemeier Hansson
a6245a6bc9 Merge pull request #221 from iamFIREcracker/patch-1 2023-04-15 08:29:03 +02:00
David Heinemeier Hansson
0d80709e2d Merge pull request #224 from basecamp/integration-tests 2023-04-15 08:25:47 +02:00
Kevin McConnell
aceabb3824 Update README with env name change 2023-04-14 16:13:59 +01:00
Kevin McConnell
99fe31d4b4 Rename MRSK_EVENT -> MRSK_MESSAGE
It's a better name, and frees up `MRSK_EVENT` to be used later.
2023-04-14 16:11:42 +01:00
Donal McBreen
bcf8a927f5 Run a mrsk deploy integration test
Adds a simple integration test to ensure that `mrsk deploy` works.

Everything required is spun up with docker compose:
- shared: a container that contains an ssh key and a self signed cert to
be shared between the images
- deployer: the image we will deploy from
- registry: a docker registry
- two vm images to deploy into
- load_balancer: an nginx load balancer to use between our images

The other images are in privileged mode so that we can run
docker-in-docker. We need to run docker inside the images - mapping in
the docker socket doesn't work because both VMs would share the host
daemon.

The docker registry requires a self signed cert as you cannot use basic
auth over HTTP except on localhost. It runs on port 4443 rather than 443
because docker refused to accept that "registry" is a docker host and
tries to push images to docker.io/registry. "registry:4443" works fine.

The shared container contains the ssh keys for the deployer and vms, and
the self signed cert for the registry. When the shared container boots,
it copies them into a shared volume.

The other deployer and vm images are built with soft links from the
shared volume to the require locations. Their boot scripts wait for the
files to be copied in before continuing.

The root mrsk folder is mapped into the deployer container. On boot it
builds the gem and installs it.

Right now there's just a single test. We confirm that the load balancer
is returning a 502, run `mrsk deploy` and then confirm it returns 200.
2023-04-14 15:49:43 +01:00
Kevin McConnell
f055766918 Allow percentage-based rolling deployments 2023-04-14 12:46:14 +01:00
Kevin McConnell
a8726be20e Move group_limit & group_wait under boot
Also make formatting the group strategy the responsibility of the
commander.
2023-04-14 11:31:51 +01:00
Kevin McConnell
100b72e4b4 Limit rolling deployment to boot operation 2023-04-14 10:41:07 +01:00
Kevin McConnell
828e56912e Allow customizing audit broadcast with env
When invoking the audit broadcast command, provide a few environment
variables so that people can customize the format of the message if they
want.

We currently provide `MRSK_PERFORMER`, `MRSK_ROLE`, `MRSK_DESTINATION` and
`MRSK_EVENT`.

Also adds the destination to the default message, which we continue to
send as the first argument as before.
2023-04-13 17:54:25 +01:00
Kevin McConnell
df202d6ef4 Move health checks into Docker
Replaces our current host-based HTTP healthchecks with Docker
healthchecks, and adds a new `healthcheck.cmd` config option that can be
used to define a custom health check command. Also removes Traefik's
healthchecks, since they are no longer necessary.

When deploying a container that has a healthcheck defined, we wait for
it to report a healthy status before stopping the old container that it
replaces. Containers that don't have a healthcheck defined continue to
wait for `MRSK.config.readiness_delay`.

There are some pros and cons to using Docker healthchecks rather than
checking from the host. The main advantages are:

- Supports non-HTTP checks, and app-specific check scripts provided by a
  container.
- When booting a container, allows MRSK to wait for a container to be
  healthy before shutting down the old container it replaces. This
  should be safer than relying on a timeout.
- Containers with healthchecks won't be active in Traefik until they
  reach a healthy state, which prevents any traffic from being routed to
  them before they are ready.

The main _disadvantage_ is that containers are now required to provide
some way to check their health. Our default check assumes that `curl` is
available in the container which, while common, won't always be the
case.
2023-04-13 16:08:43 +01:00
Kevin McConnell
f530009a6e Allow performing boot & start operations in groups
Adds top-level configuration options for `group_limit` and `group_wait`.
When a `group_limit` is present, we'll perform app boot & start
operations on no more than `group_limit` hosts at a time, optionally
sleeping for `group_wait` seconds after each batch.

We currently only do this batching on boot & start operations (including
when they are part of a deployment). Other commands, like `app stop` or
`app details` still work on all hosts in parallel.
2023-04-13 15:58:27 +01:00
Matteo Landi
4b36df5dab Configure git to trust /workdir
Resolves: #220
2023-04-13 15:13:13 +02:00
Gilles Demarty
79d46ceb16 Add OpenSSH Client to the alpine server 2023-04-12 19:20:09 +02:00
David Heinemeier Hansson
bc8875e020 Merge pull request #183 from basecamp/cleanup-excessive-containers-running
Clear stale containers
2023-04-12 15:58:59 +02:00
David Heinemeier Hansson
d4a72da9d8 Merge pull request #213 from ncreuschling/fix-spelling-of-label
fix spelling of label
2023-04-12 15:58:46 +02:00
David Heinemeier Hansson
04a04c05e0 Merge branch 'main' into fix-spelling-of-label 2023-04-12 15:58:41 +02:00
David Heinemeier Hansson
cff8b058af Merge pull request #214 from tannakartikey/traefik_lables_readme_example_fix
Traefik label example typo fix
2023-04-12 15:58:08 +02:00
David Heinemeier Hansson
b6f7d94ac3 Merge pull request #144 from monorkin/shell-escape-dollar-signs
Shell escape dollar signs
2023-04-12 15:57:37 +02:00
Stanko K.R
3ab16c8994 Shell escape dollar signs
But allow for shell expansion using curly braces e.g. ${PWD}
2023-04-12 15:55:54 +02:00
Kartikey Tanna
b6743e5e1c Traefik label example typo fix 2023-04-12 19:21:20 +05:30
Jacopo
9ddb181f50 Merge branch 'main' into cleanup-excessive-containers-running
* main:
  Pull the primary host from the role
  Minimise holding the deploy lock
2023-04-12 15:19:19 +02:00
Nicolai Reuschling
fbe1458478 fix spelling of label 2023-04-12 14:56:39 +02:00
David Heinemeier Hansson
2f1393cd92 Merge pull request #212 from basecamp/role-primary-hosts
Pull the primary host from the role
2023-04-12 14:09:38 +02:00
David Heinemeier Hansson
76673c0c1b Merge pull request #211 from basecamp/minimise-lock-retention
Minimise holding the deploy lock
2023-04-12 14:08:05 +02:00
Donal McBreen
fb62f2e6e1 Pull the primary host from the role
So commands like this run on a host with the specified role:
```
mrsk app exec -r=console -i "/bin/bash`
mrsk app logs -f -r=workers
```
2023-04-12 13:03:02 +01:00
Donal McBreen
051556674f Minimise holding the deploy lock
If we get an error we'll only hold the deploy lock if it occurs while
trying to switch the running containers.

We'll also move tagging the latest image from when the image is pulled
to just before the container switch. This ensures that earlier errors
don't leave the hosts with an updated latest tag while still running the
older version.
2023-04-12 12:09:56 +01:00
Jacopo
3cbf4aea46 Make method private method and use :send 2023-04-12 11:53:49 +02:00
Jacopo
5ed431b807 Merge branch 'main' into cleanup-excessive-containers-running
* main: (24 commits)
  Bump version for 0.11.0
  Labels can be added to Traefik
  Make rollbacks role-aware
  fix typo role to roles
  Explained the latest modifications of Traefik container labels
  Remove .idea folder
  Updated README.md with new healthcheck.max_attempts option
  Fix test case: console output message was not updated to display the current/total attempts
  Require net-ssh ~> 7.0 for SHA-2 support
  Improved deploy lock acquisition
  Excess CR
  Style
  Simpler
  Make it explicit, focus on Ubuntu
  More explicit
  Not that --bundle is a Rails 7+ option
  Update README.md
  Update README.md
  Improved: configurable max_attempts for healthcheck
  Traefik service name to be derived from role and destination
  ...
2023-04-12 11:52:47 +02:00
David Heinemeier Hansson
60a19f0b30 Bump version for 0.11.0 2023-04-12 11:45:33 +02:00
David Heinemeier Hansson
2d0a7e1b67 Merge pull request #208 from tannakartikey/add_labels_to_traefik
Labels can be added to Traefik
2023-04-12 11:35:28 +02:00
David Heinemeier Hansson
49df19fb0d Merge pull request #209 from ncreuschling/fix-roles-documentation
fix typo role to roles
2023-04-12 11:34:02 +02:00
David Heinemeier Hansson
cef8fddfb4 Merge pull request #210 from basecamp/role-aware-rollbacks
Make rollbacks role-aware
2023-04-12 11:33:45 +02:00
Kartikey Tanna
c59eb00dd0 Labels can be added to Traefik 2023-04-12 14:53:48 +05:30
Donal McBreen
43f7409de0 Make rollbacks role-aware
Rollbacks stopped working after https://github.com/mrsked/mrsk/pull/99.

We'll confirm that a container is available for the first role on the
primary host before attempting to rollback.
2023-04-12 09:59:39 +01:00
Nicolai Reuschling
448ea7719f fix typo role to roles 2023-04-12 10:53:10 +02:00
Jacopo
72b70e3e9e More compact 2023-04-11 16:22:47 +02:00
Jacopo
e8697327fa Use no_commands block 2023-04-11 16:20:16 +02:00
Jacopo
0bfd4ca780 Use cli = self approach 2023-04-11 16:04:46 +02:00
Jacopo
12e3a562c4 Extract helper 2023-04-11 15:26:55 +02:00
David Heinemeier Hansson
ab54dbdb8b Merge pull request #206 from tannakartikey/traefik_rule_docs
Explained the latest modifications of Traefik container labels
2023-04-11 14:18:31 +02:00
David Heinemeier Hansson
ac3771447a Merge pull request #203 from matharvard/main
Require net-ssh ~> 7.0 for SHA-2 support
2023-04-11 14:17:52 +02:00
David Heinemeier Hansson
daa0c9b5be Merge pull request #196 from handy-la/main
Configurable max_attempts for healthcheck
2023-04-11 14:17:17 +02:00
Jacopo
c3393c8213 Remove dot 2023-04-11 11:03:11 +02:00
Jacopo
03d933d10b Add Role to the message 2023-04-11 10:59:25 +02:00
Jacopo
579b4cd9aa Simplify
By using and ad-hoc command to detect and stop stale containers.
By default stale containers are only detected.
2023-04-11 10:22:03 +02:00
Jacopo
f9436d5673 Style 2023-04-11 08:53:33 +02:00
Jacopo
8ae5331d97 Boot stop all the old containers 2023-04-11 08:53:33 +02:00
Jacopo
4d47fbdf41 Merge stop and stop_stale_containers 2023-04-11 08:53:33 +02:00
Jacopo
e980f1164e Avoid using GNU-only Perl Regepx Grep 2023-04-11 08:53:33 +02:00
Jacopo
e2f6db5cae Clear stale containers
By stopping all the older containers with matching /#{service}-#{role}-#{dest}-.*/ running on the same host.
2023-04-11 08:53:33 +02:00
Kartikey Tanna
d3936363d0 Explained the latest modifications of Traefik container labels 2023-04-11 10:20:16 +05:30
Arturo Ojeda
cfc8fa0590 Remove .idea folder 2023-04-10 22:33:20 -06:00
Arturo Ojeda
161ebe4bc1 Updated README.md with new healthcheck.max_attempts option 2023-04-10 22:26:10 -06:00
Arturo Ojeda
514b2aa243 Fix test case: console output message was not updated to display the current/total attempts 2023-04-10 09:29:19 -06:00
David Heinemeier Hansson
18031bc552 Merge pull request #202 from basecamp/deploy-lock-acquisition
Improved deploy lock acquisition
2023-04-10 16:42:03 +02:00
Mat Harvard
d8c61004e4 Require net-ssh ~> 7.0 for SHA-2 support
Versions of net-ssh before 7.0 do not support the SHA-2 algorithm and result in mrsk not being able to connect to hosts using keys generated with it. net-ssh is also a dependency of sshkit, however, sshkit has a version requirement of >= 2.8.0 for net-ssh, so is not effective at ensuring mrsk has the version it needs to be the most compatible.
2023-04-10 07:29:07 -07:00
Donal McBreen
c4df440c79 Improved deploy lock acquisition
1. Don't raise lock error for non-lock issues during lock acquire
  (see https://github.com/mrsked/mrsk/pull/181)
2. If there is an error while the lock is held, don't release the lock
  and send a warning to stderr
2023-04-10 15:23:00 +01:00
David Heinemeier Hansson
fb1718ca6d Merge pull request #197 from tannakartikey/traefik_rules_with_destination
Traefik service name to be derived from role and destination
2023-04-10 15:11:07 +02:00
David Heinemeier Hansson
7d17a6c3b5 Excess CR 2023-04-10 15:10:08 +02:00
David Heinemeier Hansson
f4133de896 Merge pull request #176 from dilpreet92/enable_ssh_over_proxy_command
Enable ssh over proxy command
2023-04-10 14:41:45 +02:00
David Heinemeier Hansson
a9488e935d Style 2023-04-10 14:39:18 +02:00
David Heinemeier Hansson
ac61528dfc Merge pull request #189 from basecamp/traefik-image
Traefik image config for version pinning, upgrades, and custom images
2023-04-10 14:35:30 +02:00
David Heinemeier Hansson
0eb7a8d087 Merge branch 'main' into pr/176
* main:
  Simpler
  Make it explicit, focus on Ubuntu
  More explicit
  Not that --bundle is a Rails 7+ option
  Update README.md
  Update README.md
  Add github discussions link to readme
  Bump debug to fix missing deps in CI
  Only redact the non-sensitive bits of build args and env vars.
  improve code sample (traefik configuration)
2023-04-10 14:31:43 +02:00
David Heinemeier Hansson
7559f439e9 Merge pull request #195 from nickhammond/patch-1
Add github discussions link to readme
2023-04-10 14:28:59 +02:00
David Heinemeier Hansson
54a5b90d8f Simpler 2023-04-10 14:28:52 +02:00
David Heinemeier Hansson
a245adfad2 Merge pull request #200 from huksley/main
Add sample commands to bootstrap non-root ssh server
2023-04-10 14:27:13 +02:00
David Heinemeier Hansson
f386c3bdab Make it explicit, focus on Ubuntu 2023-04-10 14:26:49 +02:00
David Heinemeier Hansson
2a3e576182 More explicit 2023-04-10 14:24:51 +02:00
David Heinemeier Hansson
f3e3196ce5 Not that --bundle is a Rails 7+ option 2023-04-10 14:22:58 +02:00
Ruslan Gainutdinov
fca5b11682 Update README.md
Use docker.io on Ubuntu
2023-04-10 12:26:57 +03:00
Ruslan Gainutdinov
d09cddde8d Update README.md
Add sample commands to bootstrap non-root ssh server.
2023-04-10 12:23:06 +03:00
Arturo Ojeda
3969f56fa6 Improved: configurable max_attempts for healthcheck 2023-04-09 12:07:27 -06:00
Kartikey Tanna
c60cc92dfe Traefik service name to be derived from role and destination 2023-04-09 13:44:57 +05:30
Arturo Ojeda
cb3c5a53f4 Configurable max_attempts for healthcheck 2023-04-08 19:52:53 -06:00
Nick Hammond
ef04410d77 Add github discussions link to readme
I realize that there's a discussions link on github but I didn't realize mrsk actually utilized it until I saw it mentioned on Discord. I was thinking adding it to the readme would help push people there.
2023-04-08 13:33:31 -07:00
Jeremy Daer
bd8f13dd5e Traefik image config for version pinning, upgrades, and custom images
Accounts for the 2.9.10 security release and allows testing Traefik 3 betas.

* Use `image` to configure a specific Traefik Docker image.
* Default to `traefik:v2.9` to track future 2.9.x minor releases rather
  than tightly pinning to `v2.9.9`.
* Support images from the configured registry.

References #165
2023-04-07 14:15:25 -07:00
David Heinemeier Hansson
2146f6d0ec Merge pull request #182 from basecamp/sensitive-args
Only redact the non-sensitive bits of build args and env vars.
2023-04-06 16:19:41 +02:00
David Heinemeier Hansson
52d8c112d3 Merge branch 'main' into pr/182
* main:
  Bump debug to fix missing deps in CI
2023-04-06 16:18:22 +02:00
David Heinemeier Hansson
c9afd66222 Merge pull request #184 from basecamp/fix-ci 2023-04-06 13:21:26 +02:00
Jeremy Daer
36c458407f Bump debug to fix missing deps in CI 2023-04-05 12:00:15 -07:00
Jeremy Daer
c137b38c87 Only redact the non-sensitive bits of build args and env vars.
* `-e [REDACTED]` → `-e SOME_SECRET=[REDACTED]`
* Replaces `Utils.redact` with `Utils.sensitive` to clarify that we're
  indicating redactability, not actually performing redaction.
* Redacts from YAML output, including `mrsk config` (fixes #96)
2023-04-05 09:45:28 -07:00
David Heinemeier Hansson
f851d6528d Merge pull request #169 from ncreuschling/patch-1
improve code sample (traefik configuration)
2023-04-05 16:31:10 +02:00
Dilpreet Singh
12632aa7f9 Enable ssh over proxy command 2023-04-03 17:14:06 +05:30
Nicolai Reuschling
2f97bc488f improve code sample (traefik configuration)
fixed yaml format (code sample traefik configuration)
2023-03-31 11:50:43 +02:00
David Heinemeier Hansson
032266a76a Bump version for 0.10.1 2023-03-29 16:23:58 +02:00
David Heinemeier Hansson
33cc6c8bae Merge pull request #166 from calmyournerves/exit-code
Set proper exit code on failure
2023-03-29 16:21:54 +02:00
Samuel Sieg
5638ab8594 Set proper exit code on failure 2023-03-29 13:47:34 +02:00
David Heinemeier Hansson
60916cdac3 Bump version for 0.10.0 2023-03-28 18:05:46 +02:00
David Heinemeier Hansson
1f83b5f6be Fix failure to pass on class options to subcommands 2023-03-28 18:04:16 +02:00
David Heinemeier Hansson
070c6e8e75 Merge pull request #165 from basecamp/pin-traefik-version
Pin Traefik to v2.9.9
2023-03-28 16:27:49 +02:00
Kevin McConnell
2957388bf6 Pin Traefik to v2.9.9 2023-03-28 14:59:03 +01:00
David Heinemeier Hansson
7f178101f7 Merge pull request #164 from basecamp/accessory-hosts-or-roles
Run accessories on multiple hosts or roles
2023-03-28 14:31:24 +02:00
David Heinemeier Hansson
aed345466f Dropped "all" 2023-03-28 14:28:54 +02:00
Donal McBreen
c06585fef4 Daemon/host/role accessories
Allow the hosts for accessories to be specified by host or role, or on
all app hosts by setting `daemon: true`.

```
  # Single host
  mysql:
    host: 1.1.1.1
  # Multiple hosts
  redis:
    hosts:
      - 1.1.1.1
      - 1.1.1.2
  # By role
  monitoring:
    roles:
      - web
      - jobs
```
2023-03-28 13:26:27 +01:00
David Heinemeier Hansson
fd5313ec3e Merge pull request #163 from milk1000cc/rolify-app-logs
Rolify app logs cli/command
2023-03-28 14:13:02 +02:00
David Heinemeier Hansson
4184d3204e Merge pull request #161 from tbuehlmann/push-latest-image
Push <image>:latest in addition to <image>:<git-ref>
2023-03-28 14:09:32 +02:00
milk1000cc
15a41d3fd8 Follow web role logs when no roles are specified 2023-03-28 09:02:42 +09:00
milk1000cc
03614bfb79 Rolify app logs cli/command 2023-03-27 23:08:46 +09:00
Tobias Bühlmann
078d68b170 Push <image>:latest in addition to <image>:<git-ref> 2023-03-27 12:52:11 +02:00
David Heinemeier Hansson
cec82ac641 Merge pull request #158 from basecamp/zero-downtime-redeploys 2023-03-24 18:27:29 +01:00
Donal McBreen
05488e4c1e Zero downtime redeploys
When deploying check if there is already a container with the existing
name. If there is rename it to "<version>_<random_hex_string>" to remove
the name clash with the new container we want to boot.

We can then do the normal zero downtime run/wait/stop.

While implementing this I discovered the --filter name=foo does a
substring match for foo, so I've updated those filters to do an exact
match instead.
2023-03-24 17:09:20 +00:00
David Heinemeier Hansson
01a2b678d7 Merge pull request #154 from basecamp/lock-deploys
Deploy locks
2023-03-24 15:50:33 +01:00
David Heinemeier Hansson
84540cee7b Merge branch 'main' into pr/154
* main: (32 commits)
  Inline default as with other options
  Symbols!
  Fix tests
  test stop with custom stop wait time
  No need to replicate Docker default
  Describe purpose rather than elements
  Style and ordering
  Customizable stop wait time
  Fix tests
  Ensure it also works when configuring just log options without setting a driver
  Add accessory test
  Undo change
  Improve test
  Update README
  Ensure default log option `max-size=10m`
  #142 Allow to customize container options in accessories
  Fix flaky test
  Fix tests
  More resilient tests
  Fix other tests
  ...
2023-03-24 15:43:17 +01:00
David Heinemeier Hansson
5bbb4aeb58 Merge pull request #131 from calmyournerves/global-logging-config
Global logging configuration
2023-03-24 15:36:11 +01:00
David Heinemeier Hansson
6a27a46e5f Inline default as with other options 2023-03-24 15:34:34 +01:00
David Heinemeier Hansson
b5ccc1fa5d Merge branch 'main' into global-logging-config 2023-03-24 15:32:41 +01:00
David Heinemeier Hansson
e2e5e18af9 Merge pull request #155 from basecamp/gracefully-shut-down-containers
Customizable stop wait time
2023-03-24 15:31:14 +01:00
David Heinemeier Hansson
4fa71834ad Symbols! 2023-03-24 15:27:11 +01:00
David Heinemeier Hansson
65663ae2ea Merge branch 'main' into pr/155
* main:
  Describe purpose rather than elements
  Style and ordering
  #142 Allow to customize container options in accessories
2023-03-24 15:25:45 +01:00
Samuel Sieg
4044abdde1 Fix tests 2023-03-24 15:25:29 +01:00
Samuel Sieg
bc64a07a95 Merge branch 'main' into global-logging-config 2023-03-24 15:24:06 +01:00
David Heinemeier Hansson
fdb2502216 test stop with custom stop wait time 2023-03-24 15:22:34 +01:00
David Heinemeier Hansson
a9bb8d7376 No need to replicate Docker default 2023-03-24 15:18:18 +01:00
David Heinemeier Hansson
53095a053e Describe purpose rather than elements 2023-03-24 15:16:38 +01:00
David Heinemeier Hansson
4ab5199853 Style and ordering 2023-03-24 15:16:15 +01:00
David Heinemeier Hansson
348f5844d5 Merge pull request #153 from javierav/feature/accessory-options
#142 Allow to customize container options in accessories
2023-03-24 15:09:12 +01:00
Jacopo
9b43a6b23b Customizable stop wait time
Configurable via a global `stop_wait_time` option.
The default is `10` which matches Docker defaults.
2023-03-24 15:04:45 +01:00
David Heinemeier Hansson
1f196045a9 Merge pull request #99 from tbuehlmann/role-awareness
Role aware container names
2023-03-24 15:01:34 +01:00
Samuel Sieg
86e99fb079 Merge branch 'main' into global-logging-config 2023-03-24 14:40:27 +01:00
David Heinemeier Hansson
494e29d672 Fix tests 2023-03-24 14:35:17 +01:00
David Heinemeier Hansson
93423f2f20 Merge branch 'main' into pr/99
* main:
  Wording
  Remove accessory images using tags rather than labels
  Update readme to point to ghcr.io/mrsked/mrsk
  Validate that all roles have hosts
  Commander needn't accumulate configuration
  Pull latest image tag, so we can identity it
  Default to deploying the config version
  Remove unneeded Dockerfile.dind, update Readme
  add D-in-D dockerfile, update Readme
2023-03-24 14:26:31 +01:00
Donal McBreen
8d8f9f6ada Deploy locks
Add a deploy lock for commands that are unsafe to run concurrently.

The lock is taken by creating a `mrsk_lock` directory on the primary
host. Details of who took the lock are added to a details file in that
directory.

Additional CLI commands have been added to manual release and acquire
the lock and to check its status.

```
Commands:
  mrsk lock acquire -m, --message=MESSAGE  # Acquire the deploy lock
  mrsk lock help [COMMAND]                 # Describe subcommands or one specific subcommand
  mrsk lock release                        # Release the deploy lock
  mrsk lock status                         # Report lock status

Options:
  -v, [--verbose], [--no-verbose]                # Detailed logging
  -q, [--quiet], [--no-quiet]                    # Minimal logging
      [--version=VERSION]                        # Run commands against a specific app version
  -p, [--primary], [--no-primary]                # Run commands only on primary host instead of all
  -h, [--hosts=HOSTS]                            # Run commands on these hosts instead of all (separate by comma)
  -r, [--roles=ROLES]                            # Run commands on these roles instead of all (separate by comma)
  -c, [--config-file=CONFIG_FILE]                # Path to config file
                                                 # Default: config/deploy.yml
  -d, [--destination=DESTINATION]                # Specify destination to be used for config file (staging -> deploy.staging.yml)
  -B, [--skip-broadcast], [--no-skip-broadcast]  # Skip audit broadcasts
```

If we add support for running multiple deployments on a single server
we'll need to extend the locking to lock per deployment.
2023-03-24 12:28:08 +00:00
David Heinemeier Hansson
17e74910e4 Merge pull request #150 from basecamp/remove-accessory-image
Remove accessory images using tags rather than labels
2023-03-24 13:21:15 +01:00
David Heinemeier Hansson
8ebcafd3d8 Wording 2023-03-24 13:20:52 +01:00
David Heinemeier Hansson
89b4b909db Merge pull request #118 from kumulustech/kumulus/docker-in-docker
Add docker in docker to Dockerfile for container dev
2023-03-24 13:19:33 +01:00
David Heinemeier Hansson
c89b77127b Merge pull request #143 from djmb/default-to-deploying-config-version
Default to deploying the config version
2023-03-24 12:36:20 +01:00
Samuel Sieg
9c27ead21f Ensure it also works when configuring just log options without setting a driver 2023-03-24 09:38:02 +01:00
Samuel Sieg
c3de89bb59 Add accessory test 2023-03-24 09:19:13 +01:00
Samuel Sieg
20a6bc31cd Undo change 2023-03-24 09:15:37 +01:00
Samuel Sieg
ba5bdf95ec Improve test 2023-03-24 09:15:30 +01:00
Samuel Sieg
3392fc6c1b Update README 2023-03-24 09:15:03 +01:00
Samuel Sieg
7369be48ff Ensure default log option max-size=10m 2023-03-24 09:10:36 +01:00
Samuel Sieg
4670db7f6d Merge branch 'main' into global-logging-config 2023-03-24 08:35:43 +01:00
Jeremy Daer
e859a581ab Remove accessory images using tags rather than labels 2023-03-23 15:59:28 -07:00
Javier Aranda
5d5d58a4ec #142 Allow to customize container options in accessories 2023-03-23 23:56:59 +01:00
Robert Starmer
cf38feb1d6 Update readme to point to ghcr.io/mrsked/mrsk 2023-03-23 12:35:15 -07:00
David Heinemeier Hansson
e2d10ec5a9 Merge pull request #145 from basecamp/config-version
Commander needn't accumulate configuration
2023-03-23 17:51:30 +01:00
Jeremy Daer
035e4afff7 Validate that all roles have hosts 2023-03-23 08:57:34 -07:00
Jeremy Daer
1887a6518e Commander needn't accumulate configuration
Commander had version/destination solely to incrementally accumulate CLI
options. Simpler to configure in one shot.

Clarifies responsibility and lets us introduce things like
`abbreviated_version` in one spot - Configuration.
2023-03-23 08:57:32 -07:00
Donal McBreen
1ed4a37da2 Pull latest image tag, so we can identity it
`docker image ls` doesn't tell us what the latest deployed image is (e.g
if we've rolled back). Pull the latest image tag through to the server
so we can use it instead.
2023-03-23 14:39:32 +00:00
David Heinemeier Hansson
7e1596e722 Fix flaky test 2023-03-23 15:36:02 +01:00
David Heinemeier Hansson
e7e3cd98eb Fix tests 2023-03-23 15:16:10 +01:00
David Heinemeier Hansson
a1fc00347b Merge branch 'main' into pr/99
* main:
  Ask for access token
  Style
  Style
  config.traefik is already nil safe
  Update README.md
  Bump dev deps and consolidate platform matches
  Deploys mention the released service@version
  Accessories aren't required to publish a port
  Accessories may be pulled from authenticated registries
  Polish destination config loading
  Allow arbitrary docker options for traefik
  Fixed typos
  Fixed readme
  Rebased on main
  Added volume configuration in response to issue coments
  Modified in response to PR comments
  Added the additional_ports configuration
2023-03-23 14:48:13 +01:00
David Heinemeier Hansson
f73c526890 Ask for access token 2023-03-23 14:46:41 +01:00
David Heinemeier Hansson
65b90dd5c8 Merge branch 'main' into default-to-deploying-config-version 2023-03-23 14:42:31 +01:00
David Heinemeier Hansson
9648721ce7 Merge pull request #146 from basecamp/tell-me-more
Deploys mention the service and version
2023-03-23 14:38:31 +01:00
David Heinemeier Hansson
e409281bb2 Merge pull request #147 from basecamp/destination-config-polish
Polish destination config loading
2023-03-23 14:35:29 +01:00
David Heinemeier Hansson
bab8e42965 Merge pull request #151 from basecamp/portless-accessories
Accessories aren't required to publish a port
2023-03-23 14:32:58 +01:00
David Heinemeier Hansson
110df5244b Merge pull request #152 from basecamp/deps
Bump dev deps and consolidate platform matches
2023-03-23 14:31:22 +01:00
David Heinemeier Hansson
01d684746e Merge pull request #100 from stepbeekio/feature/multiple-traefik-entrypoints
Added the docker options override configuration for traefik
2023-03-23 14:28:40 +01:00
David Heinemeier Hansson
951a71f38e Style 2023-03-23 14:26:12 +01:00
David Heinemeier Hansson
8b755c6973 Style 2023-03-23 14:24:34 +01:00
David Heinemeier Hansson
9a909ba7eb config.traefik is already nil safe 2023-03-23 14:06:15 +01:00
David Heinemeier Hansson
14512fe409 Update README.md 2023-03-23 12:10:56 +01:00
David Heinemeier Hansson
e97216b0ea Merge pull request #149 from basecamp/private-accessories
Private accessory images
2023-03-23 09:57:39 +01:00
Jeremy Daer
f3d93d3899 Bump dev deps and consolidate platform matches 2023-03-23 01:40:05 -07:00
Jeremy Daer
53d7f9d528 Deploys mention the released service@version
Less work for broadcast commands to take on.

Also fixes a bug where rollback on hosts without a running container
would stop the container they had just started.
2023-03-23 01:09:25 -07:00
Jeremy Daer
c870e560c1 Accessories aren't required to publish a port
Allows for background accessories like schedulers that don't act
as typical network service dependencies and have no port to expose.
2023-03-23 00:10:30 -07:00
Jeremy Daer
04b1d5e49e Accessories may be pulled from authenticated registries 2023-03-22 23:48:22 -07:00
Robert Starmer
714960f184 Merge branch 'main' into kumulus/docker-in-docker 2023-03-22 11:27:28 -07:00
Jeremy Daer
c0d5b48f22 Polish destination config loading
* `Pathname#sub_ext` to munge .yml ext to .destination.yml
* Extract multi-file config merge
2023-03-22 10:38:37 -07:00
Donal McBreen
fb3353084f Default to deploying the config version
If we don't supply a version when deploying we'll use the result of
docker image ls to decide which image to boot. But that doesn't
necessarily correspond to the one we have just built.

E.g. if you do something like:

```
mrsk deploy        # deploys git sha AAAAAAAAAAAAAA
git commit --amend # update the commit message
mrsk deploy        # deploys git sha BBBBBBBBBBBBBB
```

In this case running `docker image ls` will give you the same image
twice (because the contents are identical) with tags for both SHAs but
the image we have just built will not be returned first. Maybe the order
is random, but it always seems to come second as far as I have seen.

i.e you'll get something like:

```
REPOSITORY    TAG              IMAGE ID       CREATED          SIZE
foo/bar       AAAAAAAAAAAAAA   6272349a9619   31 minutes ago   791MB
foo/bar       BBBBBBBBBBBBBB   6272349a9619   31 minutes ago   791MB
```

Since we already know what version we want to deploy from the config,
let's just pass that through.
2023-03-22 16:14:50 +00:00
David Heinemeier Hansson
19104cafb4 Merge branch 'main' into role-awareness 2023-03-21 08:20:26 -04:00
Samuel Sieg
1bdfc217c4 Merge branch 'main' into global-logging-config 2023-03-21 13:20:12 +01:00
David Heinemeier Hansson
83dc82661b Merge pull request #125 from calmyournerves/fix-destination-filter
Fix label filters when destination is passed
2023-03-21 07:44:59 -04:00
David Heinemeier Hansson
790be0f5f3 Style 2023-03-21 12:42:04 +01:00
David Heinemeier Hansson
49d60a045a Style 2023-03-21 12:41:28 +01:00
David Heinemeier Hansson
60faf27a05 More resilient tests 2023-03-20 17:40:36 +01:00
David Heinemeier Hansson
43d1ecc94b Fix other tests 2023-03-20 17:33:13 +01:00
David Heinemeier Hansson
00b970323b Merge branch 'main' into pr/99
* main:
  Add another assertion for `escape_shell_value`
  Add tests for `Mrsk::Utils`
  Fix indentation
  Don't report exception here too
  Don't report exception
  Add CLI tests for remaining commands that are not tested yet
  Minor: Properly require active_support
2023-03-20 17:31:50 +01:00
David Heinemeier Hansson
d0c4030257 Merge pull request #128 from calmyournerves/utils-tests
Tests for `Mrsk::Utils`
2023-03-20 02:28:42 -04:00
Robert Starmer
9591096131 Merge branch 'main' into kumulus/docker-in-docker 2023-03-19 12:34:32 -07:00
Samuel Sieg
b635b3198f Fix 2023-03-19 09:49:23 +01:00
Samuel Sieg
662873de49 Add logging to README 2023-03-19 09:48:54 +01:00
Samuel Sieg
b5372988f7 Add global logging configuration 2023-03-19 09:21:08 +01:00
Samuel Sieg
c3d0382935 Add another assertion for escape_shell_value 2023-03-17 16:31:10 +01:00
Samuel Sieg
2de5250486 Add tests for Mrsk::Utils 2023-03-17 16:29:25 +01:00
Samuel Sieg
491777221f Fix destination label filter 2023-03-16 16:15:31 +01:00
David Heinemeier Hansson
d167e48584 Merge pull request #122 from calmyournerves/add-cli-tests
Add CLI tests for remaining commands that are not tested yet
2023-03-16 09:31:28 -04:00
David Heinemeier Hansson
d071246865 Merge pull request #119 from ylecuyer/active_support-yle
Minor: Properly require active_support
2023-03-16 09:29:34 -04:00
Samuel Sieg
dae8b14469 Fix indentation 2023-03-16 08:35:12 +01:00
Samuel Sieg
b166f3fbf4 Don't report exception here too 2023-03-16 08:29:10 +01:00
Samuel Sieg
d33b723afb Don't report exception 2023-03-16 08:24:54 +01:00
Samuel Sieg
aae290cefc Add CLI tests for remaining commands that are not tested yet 2023-03-15 16:48:12 +01:00
Stephen van Beek
4c542930c5 Allow arbitrary docker options for traefik 2023-03-15 15:37:10 +00:00
Tobias Bühlmann
a15603655c Adapt test for single host 2023-03-15 09:28:10 +01:00
Robert Starmer
11af999800 Remove unneeded Dockerfile.dind, update Readme 2023-03-14 16:27:19 -07:00
David Heinemeier Hansson
cb824bdc42 Merge branch 'main' into role-awareness 2023-03-14 19:11:10 -04:00
Yoann Lecuyer
85a0267447 Minor: Properly require active_support 2023-03-14 23:29:00 +01:00
Robert Starmer
886914c82e Merge branch 'main' into kumulus/docker-in-docker 2023-03-14 14:14:07 -07:00
Robert Starmer
5b506a2daa add D-in-D dockerfile, update Readme 2023-03-14 14:14:02 -07:00
Stephen van Beek
9843c5e1ce Fixed typos 2023-03-14 20:13:13 +00:00
Stephen van Beek
c2ca269eb6 Fixed readme 2023-03-14 20:12:11 +00:00
Stephen van Beek
53046efad4 Rebased on main 2023-03-14 20:11:09 +00:00
Stephen van Beek
2db1bfde00 Added volume configuration in response to issue coments 2023-03-14 19:59:19 +00:00
Stephen van Beek
2cea12c56b Modified in response to PR comments 2023-03-14 19:59:19 +00:00
Stephen van Beek
43a1b42f8c Added the additional_ports configuration
ISSUE: https://github.com/mrsked/mrsk/issues/98
2023-03-14 19:59:19 +00:00
David Heinemeier Hansson
c282461265 Merge pull request #116 from tbuehlmann/traefik-command-options
Properly pass traefik command options
2023-03-14 15:08:27 -04:00
David Heinemeier Hansson
dcbe038555 Merge pull request #117 from calmyournerves/cli-main-tests
Add tests for main CLI commands
2023-03-14 15:07:07 -04:00
Samuel Sieg
3fd2f3f2c5 Improve comments 2023-03-14 16:05:57 +01:00
Samuel Sieg
46dad1ee6c Add tests for main CLI commands 2023-03-14 15:58:12 +01:00
Tobias Bühlmann
3ca5bc50b6 Properly pass traefik command options
Traefik command options need to be passed as `--key=value`, not `--key value`.
2023-03-14 15:04:33 +01:00
David Heinemeier Hansson
b668ce3f25 Merge pull request #111 from calmyournerves/deploy-without-build-push 2023-03-14 07:32:27 -04:00
David Heinemeier Hansson
253d4ac37b Merge pull request #115 from intrip/fix-traefik-default-middleware 2023-03-14 07:31:20 -04:00
Jacopo
50ee954ca9 Fix Traefik retry middleware
As per [Traefik docs](https://doc.traefik.io/traefik/middlewares/overview/#configuration-example)
a middleware to be activated needs to be applied to a route. Change the default settings
to apply the `retry` middleware on every role with Traefik enabled.
2023-03-14 12:15:00 +01:00
Samuel Sieg
0ac2cd2a4b Add tests for deploy/redeploy commands 2023-03-14 11:49:31 +01:00
Tobias Bühlmann
72e0184e9f Fix failing tests 2023-03-13 17:36:02 +01:00
Samuel Sieg
577cf2cec9 Merge branch 'main' into deploy-without-build-push 2023-03-13 16:11:38 +01:00
Samuel Sieg
5010850b86 Merge branch 'main' into deploy-without-build-push 2023-03-13 16:10:31 +01:00
David Heinemeier Hansson
fa07c2403c Merge pull request #113 from moomerman/fix-healthcheck-test
Fix healthcheck test
2023-03-13 16:10:17 +01:00
Samuel Sieg
c29d1ddeba Fix 2023-03-13 16:05:21 +01:00
Samuel Sieg
cb15800d25 Move option to deploy/redeploy, rename to skip-push 2023-03-13 16:02:24 +01:00
Richard Taylor
3e0b71b631 Fix healthcheck test
Looks like the tests started failing on the options healthcheck PR
after merging the container name env var PR.
2023-03-13 14:51:54 +00:00
David Heinemeier Hansson
9b666e54f3 Update README.md 2023-03-13 10:43:44 -04:00
David Heinemeier Hansson
d2f76dac6b Merge branch 'main' into role-awareness 2023-03-13 15:16:44 +01:00
David Heinemeier Hansson
bf3d3f3ba7 Merge pull request #101 from davegudge/fix-docker-publish
fix: GitHub Workflow: Docker Publish
2023-03-13 15:14:06 +01:00
David Heinemeier Hansson
20733a4493 Merge pull request #102 from moomerman/cmd-options-for-healthcheck
Use custom web options for healthcheck
2023-03-13 15:12:25 +01:00
David Heinemeier Hansson
a267c1e835 Merge pull request #103 from 99linesofcode/fix-dockerfile-buildx
Install buildx inside container
2023-03-13 15:11:37 +01:00
David Heinemeier Hansson
c1c26a154d Merge pull request #104 from moomerman/add-container-name-env-var
Add container name env var for containers
2023-03-13 15:10:02 +01:00
David Heinemeier Hansson
5969ff66d5 Merge pull request #107 from clowder/order-options-dig
Avoid `[ActiveSupport::OrderedOptions#dig]`
2023-03-13 15:08:31 +01:00
David Heinemeier Hansson
b1f5165dc0 Merge pull request #108 from clowder/patch-1
Update `accessory remove` description and warning
2023-03-13 15:07:47 +01:00
David Heinemeier Hansson
cce0fafdc4 Merge pull request #110 from kjellberg/patch-1
Update README.md to reflect backtick escaping in Utils.optionize
2023-03-13 15:07:09 +01:00
Samuel Sieg
6232175ef8 Undo changes from experimenting 2023-03-12 10:56:12 +01:00
Samuel Sieg
47af6d9483 Is a global option better? 2023-03-12 10:53:29 +01:00
Samuel Sieg
ff0170076e Simplify 2023-03-12 10:44:33 +01:00
Samuel Sieg
9b39f2f3ab Keep it simple for the proposal 2023-03-12 10:41:04 +01:00
Rasmus Kjellberg
600902ef5e Update README.md
Backticks are handled by `Utils.optionize`
2023-03-12 07:39:07 +01:00
Richard Taylor
bb241dea43 Add container name env var for containers
Because the container name is generated it isn't possible to
determine this inside the container.

This adds the MRSK_CONTAINER_NAME env var when running the
container so it can be read by the service running inside the
container.
2023-03-11 10:14:41 +00:00
Chris Lowder
f26beeaa9f Update accessory remove description and warning
Make it clear the accessory's data directory will also be removed.
2023-03-10 20:51:14 +00:00
Chris Lowder
41a5cb2a04 Avoid [ActiveSupport::OrderedOptions#dig]
The implementation has been updated upstream[^1] to expect symbolized
keys. MRSK relies heavily on the fact that nested keys are strings, so
we're removing existing uses of `#dig`.

[^1]: 5c15b586aa
2023-03-10 19:45:35 +00:00
Chris Lowder
643cb2c520 Include edge Rails in the build matrix
Highlighting an incompatibility with the new implementation of
`[ActiveSupport::OrderedOptions#dig]`.

[^1]: 5c15b586aa
2023-03-10 19:40:57 +00:00
Jordy Schreuders
b2c819fe32 Add README section on running MRSK from Docker 2023-03-10 19:23:14 +02:00
Jordy Schreuders
439b681308 Neglected to install buildx inside container 2023-03-10 18:13:32 +02:00
Richard Taylor
e5c5e89232 Use custom web options for healthcheck
If the web role has custom options, ensure these are used for the
healthcheck.
2023-03-10 15:55:04 +00:00
Samuel Sieg
4bf77ccd1b Allow deploy/deliver without building and pushing the image 2023-03-10 11:26:35 +01:00
Dave Gudge
57e9231c5e fix: Github Workflow: Docker Publish
The workflow was failing with:

```
The workflow is not valid. .github/workflows/docker-publish.yml (Line: 22, Col: 14): Unexpected symbol: '|'. Located at position 12 within expression: github.ref | replace('refs/tags/', '')
```

The `set-output` command is deprecated, so the issue has been fixed by utilising the `github.ref_name` context to retrieve the version tag that triggered the workflow.

> `github.ref_name`: The short ref name of the branch or tag that triggered the workflow run. This value matches the branch or tag name shown on GitHub. For example, `feature-branch-1`.

https://docs.github.com/en/actions/learn-github-actions/contexts
2023-03-10 10:02:33 +00:00
Tobias Bühlmann
ccf8762c98 Reuse web container per default 2023-03-10 10:50:26 +01:00
Tobias Bühlmann
418bc13ae7 Apply filters correctly 2023-03-10 10:33:55 +01:00
Tobias Bühlmann
7d4dfc4c86 Pass role names for simplicity 2023-03-10 09:18:47 +01:00
Tobias Bühlmann
fdb0c8ee91 Rolify app cli/command 2023-03-10 08:50:26 +01:00
Tobias Bühlmann
6b11303230 Prepare auditor to print a present role 2023-03-09 20:55:37 +01:00
Tobias Bühlmann
901484d75d Filter roles and hosts by their respective counterpart 2023-03-09 18:21:39 +01:00
Tobias Bühlmann
e178907a21 Don't list duplicate hosts 2023-03-09 16:46:57 +01:00
David Heinemeier Hansson
3026a92c98 Merge pull request #71 from tbuehlmann/destination-awareness
Destination aware container names
2023-03-09 13:25:49 +00:00
David Heinemeier Hansson
ab7c6c6540 Use compact.join strategy here too 2023-03-09 14:24:19 +01:00
David Heinemeier Hansson
11f4dbfc5f Bump version for 0.9.1 2023-03-09 14:11:42 +01:00
David Heinemeier Hansson
15e879e83c Merge pull request #97 from martinbjeldbak/syntax-error-docker-install
Fix syntax error in dependency install step
2023-03-09 13:11:22 +00:00
Martin Bjeldbak Madsen
96180f9bd0 Fix syntax error in docker install exec 2023-03-09 22:34:11 +11:00
Tobias Bühlmann
8b913068de Add destination to healthcheck containers names 2023-03-06 16:54:13 +01:00
Tobias Bühlmann
170562c7e7 Let App be aware of destination 2023-03-03 15:29:00 +01:00
216 changed files with 9699 additions and 3270 deletions

View File

@@ -1,6 +1,25 @@
name: CI
on: [push, pull_request]
on:
push:
branches:
- main
pull_request:
jobs:
rubocop:
name: RuboCop
runs-on: ubuntu-latest
env:
BUNDLE_ONLY: rubocop
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Ruby and install gems
uses: ruby/setup-ruby@v1
with:
ruby-version: 3.3.0
bundler-cache: true
- name: Run Rubocop
run: bundle exec rubocop --parallel
tests:
strategy:
matrix:
@@ -8,14 +27,29 @@ jobs:
- "2.7"
- "3.1"
- "3.2"
continue-on-error: [false]
- "3.3"
gemfile:
- Gemfile
- gemfiles/ruby_2.7.gemfile
- gemfiles/rails_edge.gemfile
exclude:
- ruby-version: "2.7"
gemfile: Gemfile
- ruby-version: "2.7"
gemfile: gemfiles/rails_edge.gemfile
- ruby-version: "3.1"
gemfile: gemfiles/ruby_2.7.gemfile
- ruby-version: "3.2"
gemfile: gemfiles/ruby_2.7.gemfile
- ruby-version: "3.3"
gemfile: gemfiles/ruby_2.7.gemfile
name: ${{ format('Tests (Ruby {0})', matrix.ruby-version) }}
runs-on: ubuntu-latest
continue-on-error: ${{ matrix.continue-on-error }}
continue-on-error: true
env:
BUNDLE_GEMFILE: ${{ github.workspace }}/${{ matrix.gemfile }}
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: Install Ruby
uses: ruby/setup-ruby@v1

View File

@@ -1,6 +1,12 @@
name: Docker
on:
workflow_dispatch:
inputs:
tagInput:
description: 'Tag'
required: true
release:
types: [created]
tags:
@@ -16,10 +22,6 @@ jobs:
-
name: Checkout
uses: actions/checkout@v3
-
name: Extract Version Number
id: extract_version
run: echo "::set-output name=version::${{ github.ref | replace('refs/tags/', '') }}"
-
name: Set up QEMU
uses: docker/setup-qemu-action@v2
@@ -33,6 +35,14 @@ jobs:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Determine version tag
id: version-tag
run: |
INPUT_VALUE="${{ github.event.inputs.tagInput }}"
if [ -z "$INPUT_VALUE" ]; then
INPUT_VALUE="${{ github.ref_name }}"
fi
echo "::set-output name=value::$INPUT_VALUE"
-
name: Build and push
uses: docker/build-push-action@v3
@@ -41,5 +51,5 @@ jobs:
platforms: linux/amd64,linux/arm64
push: true
tags: |
ghcr.io/mrsked/mrsk:latest
ghcr.io/mrsked/mrsk:${{ steps.extract_version.outputs.version }}
ghcr.io/basecamp/kamal:latest
ghcr.io/basecamp/kamal:${{ steps.version-tag.outputs.value }}

3
.gitignore vendored
View File

@@ -1,4 +1,5 @@
.byebug_history
*.gem
coverage/*
.DS_Store
.DS_Store
gemfiles/*.lock

2
.rubocop.yml Normal file
View File

@@ -0,0 +1,2 @@
inherit_gem:
rubocop-rails-omakase: rubocop.yml

View File

@@ -1,10 +1,10 @@
# Contributor Code of Conduct
As contributors and maintainers of the MRSK project, we pledge to create a welcoming and inclusive environment for everyone. We value the participation of each member of our community and want all contributors to feel respected and valued.
As contributors and maintainers of the Kamal project, we pledge to create a welcoming and inclusive environment for everyone. We value the participation of each member of our community and want all contributors to feel respected and valued.
We are committed to providing a harassment-free experience for everyone, regardless of gender, gender identity and expression, sexual orientation, disability, physical appearance, body size, race, age, or religion (or lack thereof). We do not tolerate harassment of participants in any form.
This code of conduct applies to all MRSK project spaces, including but not limited to project code, issue trackers, chat rooms, and mailing lists. Violations of this code of conduct may result in removal from the project community.
This code of conduct applies to all Kamal project spaces, including but not limited to project code, issue trackers, chat rooms, and mailing lists. Violations of this code of conduct may result in removal from the project community.
## Our standards

View File

@@ -1,18 +1,18 @@
# Contributing to MRSK development
# Contributing to Kamal development
Thank you for considering contributing to MRSK! This document outlines some guidelines for contributing to this open source project.
Thank you for considering contributing to Kamal! This document outlines some guidelines for contributing to this open source project.
Please make sure to review our [Code of Conduct](CODE_OF_CONDUCT.md) before contributing to MRSK.
Please make sure to review our [Code of Conduct](CODE_OF_CONDUCT.md) before contributing to Kamal.
There are several ways you can contribute to the betterment of the project:
- **Report an issue?** - If the issue isnt reported, we cant fix it. Please report any bugs, feature, and/or improvement requests on the [MRSK GitHub Issues tracker](https://github.com/mrsked/mrsk/issues).
- **Submit patches** - Do you have a new feature or a fix you'd like to share? [Submit a pull request](https://github.com/mrsked/mrsk/pulls)!
- **Write blog articles** - Are you using MRSK? We'd love to hear how you're using it with your projects. Write a tutorial and post it on your blog!
- **Report an issue?** - If the issue isnt reported, we cant fix it. Please report any bugs, feature, and/or improvement requests on the [Kamal GitHub Issues tracker](https://github.com/basecamp/kamal/issues).
- **Submit patches** - Do you have a new feature or a fix you'd like to share? [Submit a pull request](https://github.com/basecamp/kamal/pulls)!
- **Write blog articles** - Are you using Kamal? We'd love to hear how you're using it with your projects. Write a tutorial and post it on your blog!
## Issues
If you encounter any issues with the project, please check the [existing issues](https://github.com/mrsked/mrsk/issues) first to see if the issue has already been reported. If the issue hasn't been reported, please open a new issue with a clear description of the problem and steps to reproduce it.
If you encounter any issues with the project, please check the [existing issues](https://github.com/basecamp/kamal/issues) first to see if the issue has already been reported. If the issue hasn't been reported, please open a new issue with a clear description of the problem and steps to reproduce it.
## Pull Requests
@@ -27,23 +27,23 @@ Please keep the following guidelines in mind when opening a pull request:
- Add tests for your changes, if possible.
- Ensure that your changes don't break existing functionality.
#### Commit message guidline
#### Commit message guidelines
A good commit message should describe what changed and why.
## Development
The `main` branch is regularly built and tested, but it is not guaranteed to be completely stable. Tags are created regularly from release branches to indicate new official, stable release versions of MRSK.
The `main` branch is regularly built and tested, but it is not guaranteed to be completely stable. Tags are created regularly from release branches to indicate new official, stable release versions of Kamal.
MRSK is written in Ruby. You should have Ruby 3.2+ installed on your machine in order to work on MRSK. If that's already setup, run `bundle` in the root directory to install all dependencies. Then you can run `bin/test` to run all tests.
Kamal is written in Ruby. You should have Ruby 3.2+ installed on your machine in order to work on Kamal. If that's already setup, run `bundle` in the root directory to install all dependencies. Then you can run `bin/test` to run all tests.
1. Fork the project repository.
2. Create a new branch for your contribution.
3. Write your code or make the desired changes.
4. **Ensure that your code passes the project's minitests by running ./bin/test.**
5. Commit your changes and push them to your forked repository.
6. [Open a pull request](https://github.com/mrsked/mrsk/pulls) to the main project repository with a detailed description of your changes.
6. [Open a pull request](https://github.com/basecamp/kamal/pulls) to the main project repository with a detailed description of your changes.
## License
MRSK is released under the MIT License. By contributing to this project, you agree to license your contributions under the same license.
Kamal is released under the MIT License. By contributing to this project, you agree to license your contributions under the same license.

View File

@@ -1,33 +1,40 @@
# Use the official Ruby 3.2.0 Alpine image as the base image
FROM ruby:3.2.0-alpine
# Set the working directory to /mrsk
WORKDIR /mrsk
# Install docker/buildx-bin
COPY --from=docker/buildx-bin /buildx /usr/libexec/docker/cli-plugins/docker-buildx
# Set the working directory to /kamal
WORKDIR /kamal
# Copy the Gemfile, Gemfile.lock into the container
COPY Gemfile Gemfile.lock mrsk.gemspec ./
COPY Gemfile Gemfile.lock kamal.gemspec ./
# Required in mrsk.gemspec
COPY lib/mrsk/version.rb /mrsk/lib/mrsk/version.rb
# Required in kamal.gemspec
COPY lib/kamal/version.rb /kamal/lib/kamal/version.rb
# Install system dependencies
RUN apk add --no-cache --update build-base git docker openrc \
RUN apk add --no-cache --update build-base git docker openrc openssh-client-default \
&& rc-update add docker boot \
&& gem install bundler --version=2.4.3 \
&& bundle install
# Copy the rest of our application code into the container.
# We do this after bundle install, to avoid having to run bundle
# everytime we do small fixes in the source code.
# every time we do small fixes in the source code.
COPY . .
# Install the gem locally from the project folder
RUN gem build mrsk.gemspec && \
gem install ./mrsk-*.gem --no-document
RUN gem build kamal.gemspec && \
gem install ./kamal-*.gem --no-document
# Set the working directory to /workdir
WORKDIR /workdir
# Tell git it's safe to access /workdir/.git even if
# the directory is owned by a different user
RUN git config --global --add safe.directory /workdir
# Set the entrypoint to run the installed binary in /workdir
# Example: docker run -it -v "$PWD:/workdir" mrsk init
ENTRYPOINT ["mrsk"]
# Example: docker run -it -v "$PWD:/workdir" kamal init
ENTRYPOINT ["kamal"]

View File

@@ -1,8 +1,8 @@
source 'https://rubygems.org'
source "https://rubygems.org"
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
gemspec
gem "debug"
gem "mocha"
gem "railties"
group :rubocop do
gem "rubocop-rails-omakase", require: false
end

View File

@@ -1,109 +1,183 @@
PATH
remote: .
specs:
mrsk (0.9.0)
kamal (1.6.0)
activesupport (>= 7.0)
base64 (~> 0.2)
bcrypt_pbkdf (~> 1.0)
concurrent-ruby (~> 1.2)
dotenv (~> 2.8)
ed25519 (~> 1.2)
sshkit (~> 1.21)
net-ssh (~> 7.0)
sshkit (>= 1.22.2, < 2.0)
thor (~> 1.2)
zeitwerk (~> 2.5)
GEM
remote: https://rubygems.org/
specs:
actionpack (7.0.4)
actionview (= 7.0.4)
activesupport (= 7.0.4)
rack (~> 2.0, >= 2.2.0)
actionpack (7.1.2)
actionview (= 7.1.2)
activesupport (= 7.1.2)
nokogiri (>= 1.8.5)
racc
rack (>= 2.2.4)
rack-session (>= 1.0.1)
rack-test (>= 0.6.3)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.0, >= 1.2.0)
actionview (7.0.4)
activesupport (= 7.0.4)
rails-dom-testing (~> 2.2)
rails-html-sanitizer (~> 1.6)
actionview (7.1.2)
activesupport (= 7.1.2)
builder (~> 3.1)
erubi (~> 1.4)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.1, >= 1.2.0)
activesupport (7.0.4)
erubi (~> 1.11)
rails-dom-testing (~> 2.2)
rails-html-sanitizer (~> 1.6)
activesupport (7.1.2)
base64
bigdecimal
concurrent-ruby (~> 1.0, >= 1.0.2)
connection_pool (>= 2.2.5)
drb
i18n (>= 1.6, < 2)
minitest (>= 5.1)
mutex_m
tzinfo (~> 2.0)
ast (2.4.2)
base64 (0.2.0)
bcrypt_pbkdf (1.1.0)
bigdecimal (3.1.5)
builder (3.2.4)
concurrent-ruby (1.1.10)
concurrent-ruby (1.2.2)
connection_pool (2.4.1)
crass (1.0.6)
debug (1.7.1)
irb (>= 1.5.0)
reline (>= 0.3.1)
debug (1.9.1)
irb (~> 1.10)
reline (>= 0.3.8)
dotenv (2.8.1)
drb (2.2.0)
ruby2_keywords
ed25519 (1.3.0)
erubi (1.12.0)
i18n (1.12.0)
i18n (1.14.1)
concurrent-ruby (~> 1.0)
io-console (0.6.0)
irb (1.6.2)
reline (>= 0.3.0)
loofah (2.19.1)
io-console (0.7.1)
irb (1.11.0)
rdoc
reline (>= 0.3.8)
json (2.7.1)
language_server-protocol (3.17.0.3)
loofah (2.22.0)
crass (~> 1.0.2)
nokogiri (>= 1.5.9)
method_source (1.0.0)
minitest (5.17.0)
mocha (2.0.2)
nokogiri (>= 1.12.0)
minitest (5.20.0)
mocha (2.1.0)
ruby2_keywords (>= 0.0.5)
mutex_m (0.2.0)
net-scp (4.0.0)
net-ssh (>= 2.6.5, < 8.0.0)
net-ssh (7.0.1)
nokogiri (1.14.0-arm64-darwin)
net-sftp (4.0.0)
net-ssh (>= 5.0.0, < 8.0.0)
net-ssh (7.2.1)
nokogiri (1.16.0-arm64-darwin)
racc (~> 1.4)
nokogiri (1.14.0-x86_64-darwin)
nokogiri (1.16.0-x86_64-darwin)
racc (~> 1.4)
nokogiri (1.14.0-x86_64-linux)
nokogiri (1.16.0-x86_64-linux)
racc (~> 1.4)
racc (1.6.2)
rack (2.2.5)
rack-test (2.0.2)
parallel (1.24.0)
parser (3.3.0.5)
ast (~> 2.4.1)
racc
psych (5.1.2)
stringio
racc (1.7.3)
rack (3.0.8)
rack-session (2.0.0)
rack (>= 3.0.0)
rack-test (2.1.0)
rack (>= 1.3)
rails-dom-testing (2.0.3)
activesupport (>= 4.2.0)
rackup (2.1.0)
rack (>= 3)
webrick (~> 1.8)
rails-dom-testing (2.2.0)
activesupport (>= 5.0.0)
minitest
nokogiri (>= 1.6)
rails-html-sanitizer (1.4.4)
loofah (~> 2.19, >= 2.19.1)
railties (7.0.4)
actionpack (= 7.0.4)
activesupport (= 7.0.4)
method_source
rails-html-sanitizer (1.6.0)
loofah (~> 2.21)
nokogiri (~> 1.14)
railties (7.1.2)
actionpack (= 7.1.2)
activesupport (= 7.1.2)
irb
rackup (>= 1.0.0)
rake (>= 12.2)
thor (~> 1.0)
zeitwerk (~> 2.5)
rake (13.0.6)
reline (0.3.2)
thor (~> 1.0, >= 1.2.2)
zeitwerk (~> 2.6)
rainbow (3.1.1)
rake (13.1.0)
rdoc (6.6.2)
psych (>= 4.0.0)
regexp_parser (2.9.0)
reline (0.4.2)
io-console (~> 0.5)
rexml (3.2.6)
rubocop (1.62.1)
json (~> 2.3)
language_server-protocol (>= 3.17.0)
parallel (~> 1.10)
parser (>= 3.3.0.2)
rainbow (>= 2.2.2, < 4.0)
regexp_parser (>= 1.8, < 3.0)
rexml (>= 3.2.5, < 4.0)
rubocop-ast (>= 1.31.1, < 2.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 2.4.0, < 3.0)
rubocop-ast (1.31.2)
parser (>= 3.3.0.4)
rubocop-minitest (0.35.0)
rubocop (>= 1.61, < 2.0)
rubocop-ast (>= 1.31.1, < 2.0)
rubocop-performance (1.20.2)
rubocop (>= 1.48.1, < 2.0)
rubocop-ast (>= 1.30.0, < 2.0)
rubocop-rails (2.24.0)
activesupport (>= 4.2.0)
rack (>= 1.1)
rubocop (>= 1.33.0, < 2.0)
rubocop-ast (>= 1.31.1, < 2.0)
rubocop-rails-omakase (1.0.0)
rubocop
rubocop-minitest
rubocop-performance
rubocop-rails
ruby-progressbar (1.13.0)
ruby2_keywords (0.0.5)
sshkit (1.21.3)
sshkit (1.22.2)
base64
mutex_m
net-scp (>= 1.1.2)
net-sftp (>= 2.1.2)
net-ssh (>= 2.8.0)
thor (1.2.1)
tzinfo (2.0.5)
stringio (3.1.0)
thor (1.3.0)
tzinfo (2.0.6)
concurrent-ruby (~> 1.0)
zeitwerk (2.6.6)
unicode-display_width (2.5.0)
webrick (1.8.1)
zeitwerk (2.6.12)
PLATFORMS
arm64-darwin-20
arm64-darwin-21
arm64-darwin-22
x86_64-darwin-20
x86_64-darwin-21
x86_64-darwin-22
arm64-darwin
x86_64-darwin
x86_64-linux
DEPENDENCIES
debug
kamal!
mocha
mrsk!
railties
rubocop-rails-omakase
BUNDLED WITH
2.4.3

624
README.md
View File

@@ -1,625 +1,13 @@
# MRSK
# Kamal: Deploy web apps anywhere
MRSK deploys web apps anywhere from bare metal to cloud VMs using Docker with zero downtime. It uses the dynamic reverse-proxy Traefik to hold requests while the new application container is started and the old one is stopped. It works seamlessly across multiple hosts, using SSHKit to execute commands. It was built for Rails applications, but works with any type of web app that can be containerized with Docker.
From bare metal to cloud VMs, deploy web apps anywhere with zero downtime. Kamal has the dynamic reverse-proxy Traefik hold requests while a new app container is started and the old one is stopped. Works seamlessly across multiple hosts, using SSHKit to execute commands. Originally built for Rails apps, Kamal will work with any type of web app that can be containerized with Docker.
Watch the screencast: https://www.youtube.com/watch?v=LL1cV2FXZ5I
➡️ See [kamal-deploy.org](https://kamal-deploy.org) for documentation on [installation](https://kamal-deploy.org/docs/installation), [configuration](https://kamal-deploy.org/docs/configuration), and [commands](https://kamal-deploy.org/docs/commands).
## Installation
## Contributing to the documentation
Install MRSK globally with `gem install mrsk`. Then, inside your app directory, run `mrsk init` (or `mrsk init --bundle` within Rails apps where you want a bin/mrsk binstub). Now edit the new file `config/deploy.yml`. It could look as simple as this:
```yaml
service: hey
image: 37s/hey
servers:
- 192.168.0.1
- 192.168.0.2
registry:
username: registry-user-name
password:
- MRSK_REGISTRY_PASSWORD
env:
secret:
- RAILS_MASTER_KEY
```
Then edit your `.env` file to add your registry password as `MRSK_REGISTRY_PASSWORD` (and your `RAILS_MASTER_KEY` for production with a Rails app).
Now you're ready to deploy to the servers:
```
mrsk deploy
```
This will:
1. Connect to the servers over SSH (using root by default, authenticated by your ssh key)
2. Install Docker on any server that might be missing it (using apt-get)
3. Log into the registry both locally and remotely
4. Build the image using the standard Dockerfile in the root of the application.
5. Push the image to the registry.
6. Pull the image from the registry onto the servers.
7. Ensure Traefik is running and accepting traffic on port 80.
8. Ensure your app responds with `200 OK` to `GET /up`.
9. Start a new container with the version of the app that matches the current git version hash.
10. Stop the old container running the previous version of the app.
11. Prune unused images and stopped containers to ensure servers don't fill up.
Voila! All the servers are now serving the app on port 80. If you're just running a single server, you're ready to go. If you're running multiple servers, you need to put a load balancer in front of them.
## Vision
In the past decade+, there's been an explosion in commercial offerings that make deploying web apps easier. Heroku kicked it off with an incredible offering that stayed ahead of the competition seemingly forever. These days we have excellent alternatives like Fly.io and Render. And hosted Kubernetes is making things easier too on AWS, GCP, Digital Ocean, and elsewhere. But these are all offerings that have you renting computers in the cloud at a premium. If you want to run on your own hardware, or even just have a clear migration path to do so in the future, you need to carefully consider how locked in you get to these commercial platforms. Preferably before the bills swallow your business whole!
MRSK seeks to bring the advance in ergonomics pioneered by these commercial offerings to deploying web apps anywhere. Whether that's low-cost cloud options without the managed-service markup from the likes of Digital Ocean, Hetzner, OVH, etc, or it's your own colocated bare metal. To MRSK, it's all the same. Feed the config file a list of IP addresses with vanilla Ubuntu servers that have seen no prep beyond an added SSH key, and you'll be running in literally minutes.
This approach gives you enormous portability. You can have your web app deployed on several clouds at ease like this. Or you can buy the baseline with your own hardware, then deploy to a cloud before a big seasonal spike to get more capacity. When you're not locked into a single provider from a tooling perspective, there are a lot of compelling options available.
Ultimately, MRSK is meant to compress the complexity of going to production using open source tooling that isn't tied to any commercial offering. Not to zero, mind you. You're probably still better off with a fully managed service if basic Linux or Docker is still difficult, but as soon as those concepts are familiar, you'll be ready to go with MRSK.
## Why not just run Capistrano, Kubernetes or Docker Swarm?
MRSK basically is Capistrano for Containers, without the need to carefully prepare servers in advance. No need to ensure that the servers have just the right version of Ruby or other dependencies you need. That all lives in the Docker image now. You can boot a brand new Ubuntu (or whatever) server, add it to the list of servers in MRSK, and it'll be auto-provisioned with Docker, and run right away. Docker's layer caching also speeds up deployments with less mucking about on the server. And the images built for MRSK can be used for CI or later introspection.
Kubernetes is a beast. Running it yourself on your own hardware is not for the faint of heart. It's a fine option if you want to run on someone else's platform, either transparently [like Render](https://thenewstack.io/render-cloud-deployment-with-less-engineering/) or explicitly on AWS/GCP, but if you'd like the freedom to move between cloud and your own hardware, or even mix the two, MRSK is much simpler. You can see everything that's going on, it's just basic Docker commands being called.
Docker Swarm is much simpler than Kubernetes, but it's still built on the same declarative model that uses state reconciliation. MRSK is intentionally designed around imperative commands, like Capistrano.
Ultimately, there are a myriad of ways to deploy web apps, but this is the toolkit we're using at [37signals](https://37signals.com) to bring [HEY](https://www.hey.com) [home from the cloud](https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0) without losing the advantages of modern containerization tooling.
## Configuration
### Using .env file to load required environment variables
MRSK uses [dotenv](https://github.com/bkeepers/dotenv) to automatically load environment variables set in the `.env` file present in the application root. This file can be used to set variables like `MRSK_REGISTRY_PASSWORD` or database passwords. But for this reason you must ensure that .env files are not checked into Git or included in your Dockerfile! The format is just key-value like:
```bash
MRSK_REGISTRY_PASSWORD=pw
DB_PASSWORD=secret123
```
### Using a generated .env file
#### 1Password as a secret store
If you're using a centralized secret store, like 1Password, you can create `.env.erb` as a template which looks up the secrets. Example of a .env.erb file:
```erb
<% if (session_token = `op signin --account my-one-password-account --raw`.strip) != "" %># Generated by mrsk envify
GITHUB_TOKEN=<%= `gh config get -h github.com oauth_token`.strip %>
MRSK_REGISTRY_PASSWORD=<%= `op read "op://Vault/Docker Hub/password" -n --session #{session_token}` %>
RAILS_MASTER_KEY=<%= `op read "op://Vault/My App/RAILS_MASTER_SECRET" -n --session #{session_token}` %>
MYSQL_ROOT_PASSWORD=<%= `op read "op://Vault/My App/MYSQL_ROOT_PASSWORD" -n --session #{session_token}` %>
<% else raise ArgumentError, "Session token missing" end %>
```
This template can safely be checked into git. Then everyone deploying the app can run `mrsk envify` when they setup the app for the first time or passwords change to get the correct `.env` file.
If you need separate env variables for different destinations, you can set them with `.env.destination.erb` for the template, which will generate `.env.staging` when run with `mrsk envify -d staging`.
#### Bitwarden as a secret store
If you are using open source secret store like bitwarden, you can create `.env.erb` as a template which looks up the secrets.
You can store `SOME_SECRET` in a secure note in bitwarden vault.
```
$ bw list items --search SOME_SECRET | jq
? Master password: [hidden]
[
{
"object": "item",
"id": "123123123-1232-4224-222f-234234234234",
"organizationId": null,
"folderId": null,
"type": 2,
"reprompt": 0,
"name": "SOME_SECRET",
"notes": "yyy",
"favorite": false,
"secureNote": {
"type": 0
},
"collectionIds": [],
"revisionDate": "2023-02-28T23:54:47.868Z",
"creationDate": "2022-11-07T03:16:05.828Z",
"deletedDate": null
}
]
```
and extract the `id` of `SOME_SECRET` from the `json` above and use in the `erb` below.
Example `.env.erb` file:
```erb
<% if (session_token=`bw unlock --raw`.strip) != "" %># Generated by mrsk envify
SOME_SECRET=<%= `bw get notes 123123123-1232-4224-222f-234234234234 --session #{session_token}` %>
<% else raise ArgumentError, "session_token token missing" end %>
```
Then everyone deploying the app can run `mrsk envify` and mrsk will generate `.env`
### Using another registry than Docker Hub
The default registry is Docker Hub, but you can change it using `registry/server`:
```yaml
registry:
server: registry.digitalocean.com
username:
- DOCKER_REGISTRY_TOKEN
password:
- DOCKER_REGISTRY_TOKEN
```
A reference to secret `DOCKER_REGISTRY_TOKEN` will look for `ENV["DOCKER_REGISTRY_TOKEN"]` on the machine running MRSK.
### Using a different SSH user than root
The default SSH user is root, but you can change it using `ssh/user`:
```yaml
ssh:
user: app
```
### Using a proxy SSH host
If you need to connect to server through a proxy host, you can use `ssh/proxy`:
```yaml
ssh:
proxy: "192.168.0.1" # defaults to root as the user
```
Or with specific user:
```yaml
ssh:
proxy: "app@192.168.0.1"
```
### Using env variables
You can inject env variables into the app containers using `env`:
```yaml
env:
DATABASE_URL: mysql2://db1/hey_production/
REDIS_URL: redis://redis1:6379/1
```
### Using secret env variables
If you have env variables that are secret, you can divide the `env` block into `clear` and `secret`:
```yaml
env:
clear:
DATABASE_URL: mysql2://db1/hey_production/
REDIS_URL: redis://redis1:6379/1
secret:
- DATABASE_PASSWORD
- REDIS_PASSWORD
```
The list of secret env variables will be expanded at run time from your local machine. So a reference to a secret `DATABASE_PASSWORD` will look for `ENV["DATABASE_PASSWORD"]` on the machine running MRSK. Just like with build secrets.
If the referenced secret ENVs are missing, the configuration will be halted with a `KeyError` exception.
Note: Marking an ENV as secret currently only redacts its value in the output for MRSK. The ENV is still injected in the clear into the container at runtime.
### Using volumes
You can add custom volumes into the app containers using `volumes`:
```yaml
volumes:
- "/local/path:/container/path"
```
### Using different roles for servers
If your application uses separate hosts for running jobs or other roles beyond the default web running, you can specify these hosts in a dedicated role with a new entrypoint command like so:
```yaml
servers:
web:
- 192.168.0.1
- 192.168.0.2
job:
hosts:
- 192.168.0.3
- 192.168.0.4
cmd: bin/jobs
```
Note: Traefik will only by default be installed and run on the servers in the `web` role (and on all servers if no roles are defined). If you need Traefik on hosts in other roles than `web`, add `traefik: true`:
```yaml
servers:
web:
- 192.168.0.1
- 192.168.0.2
web2:
traefik: true
hosts:
- 192.168.0.3
- 192.168.0.4
```
### Using container labels
You can specialize the default Traefik rules by setting labels on the containers that are being started:
```
labels:
traefik.http.routers.hey.rule: Host(\`app.hey.com\`)
```
Note: The escaped backticks are needed to ensure the rule is passed in correctly and not treated as command substitution by Bash!
This allows you to run multiple applications on the same server sharing the same Traefik instance and port.
See https://doc.traefik.io/traefik/routing/routers/#rule for a full list of available routing rules.
The labels can also be applied on a per-role basis:
```yaml
servers:
web:
- 192.168.0.1
- 192.168.0.2
job:
hosts:
- 192.168.0.3
- 192.168.0.4
cmd: bin/jobs
labels:
my-label: "50"
```
### Using container options
You can specialize the options used to start containers using the `options` definitions:
```yaml
servers:
web:
- 192.168.0.1
- 192.168.0.2
job:
hosts:
- 192.168.0.3
- 192.168.0.4
cmd: bin/jobs
options:
cap-add: true
cpu-count: 4
```
That'll start the job containers with `docker run ... --cap-add --cpu-count 4 ...`.
### Using remote builder for native multi-arch
If you're developing on ARM64 (like Apple Silicon), but you want to deploy on AMD64 (x86 64-bit), you can use multi-architecture images. By default, MRSK will setup a local buildx configuration that does this through QEMU emulation. But this can be quite slow, especially on the first build.
If you want to speed up this process by using a remote AMD64 host to natively build the AMD64 part of the image, while natively building the ARM64 part locally, you can do so using builder options:
```yaml
builder:
local:
arch: arm64
host: unix:///Users/<%= `whoami`.strip %>/.docker/run/docker.sock
remote:
arch: amd64
host: ssh://root@192.168.0.1
```
Note: You must have Docker running on the remote host being used as a builder. This instance should only be shared for builds using the same registry and credentials.
### Using remote builder for single-arch
If you're developing on ARM64 (like Apple Silicon), want to deploy on AMD64 (x86 64-bit), but don't need to run the image locally (or on other ARM64 hosts), you can configure a remote builder that just targets AMD64. This is a bit faster than building with multi-arch, as there's nothing to build locally.
```yaml
builder:
remote:
arch: amd64
host: ssh://root@192.168.0.1
```
### Using native builder when multi-arch isn't needed
If you're developing on the same architecture as the one you're deploying on, you can speed up the build by forgoing both multi-arch and remote building:
```yaml
builder:
multiarch: false
```
This is also a good option if you're running MRSK from a CI server that shares architecture with the deployment servers.
### Using a different Dockerfile or context when building
If you need to pass a different Dockerfile or context to the build command (e.g. if you're using a monorepo or you have
different Dockerfiles), you can do so in the builder options:
```yaml
# Use a different Dockerfile
builder:
dockerfile: Dockerfile.xyz
# Set context
builder:
context: ".."
# Set Dockerfile and context
builder:
dockerfile: "../Dockerfile.xyz"
context: ".."
```
### Using build secrets for new images
Some images need a secret passed in during build time, like a GITHUB_TOKEN, to give access to private gem repositories. This can be done by having the secret in ENV, then referencing it in the builder configuration:
```yaml
builder:
secrets:
- GITHUB_TOKEN
```
This build secret can then be referenced in the Dockerfile:
```dockerfile
# Copy Gemfiles
COPY Gemfile Gemfile.lock ./
# Install dependencies, including private repositories via access token (then remove bundle cache with exposed GITHUB_TOKEN)
RUN --mount=type=secret,id=GITHUB_TOKEN \
BUNDLE_GITHUB__COM=x-access-token:$(cat /run/secrets/GITHUB_TOKEN) \
bundle install && \
rm -rf /usr/local/bundle/cache
```
### Using command arguments for Traefik
You can customize the traefik command line:
```yaml
traefik:
args:
accesslog: true
accesslog.format: json
```
This will start the traefik container with `--accesslog=true accesslog.format=json`.
### Traefik's host port binding
By default Traefik binds to port 80 of the host machine, it can be configured to use an alternative port:
```yaml
traefik:
host_port: 8080
```
### Configuring build args for new images
Build arguments that aren't secret can also be configured:
```yaml
builder:
args:
RUBY_VERSION: 3.2.0
```
This build argument can then be used in the Dockerfile:
```
ARG RUBY_VERSION
FROM ruby:$RUBY_VERSION-slim as base
```
### Using accessories for database, cache, search services
You can manage your accessory services via MRSK as well. The services will build off public images, and will not be automatically updated when you deploy:
```yaml
accessories:
mysql:
image: mysql:5.7
host: 1.1.1.3
port: 3306
env:
clear:
MYSQL_ROOT_HOST: '%'
secret:
- MYSQL_ROOT_PASSWORD
volumes:
- /var/lib/mysql:/var/lib/mysql
redis:
image: redis:latest
host: 1.1.1.4
port: "36379:6379"
volumes:
- /var/lib/redis:/data
```
Now run `mrsk accessory start mysql` to start the MySQL server on 1.1.1.3. See `mrsk accessory` for all the commands possible.
### Using Cron
You can use a specific container to run your Cron jobs:
```yaml
servers:
cron:
hosts:
- 192.168.0.1
cmd:
bash -c "cat config/crontab | crontab - && cron -f"
```
This assumes the Cron settings are stored in `config/crontab`.
### Using audit broadcasts
If you'd like to broadcast audits of deploys, rollbacks, etc to a chatroom or elsewhere, you can configure the `audit_broadcast_cmd` setting with the path to a bin file that will be passed the audit line as the first argument:
```yaml
audit_broadcast_cmd:
bin/audit_broadcast
```
The broadcast command could look something like:
```bash
#!/usr/bin/env bash
curl -q -d content="[My App] ${1}" https://3.basecamp.com/XXXXX/integrations/XXXXX/buckets/XXXXX/chats/XXXXX/lines
```
That'll post a line like follows to a preconfigured chatbot in Basecamp:
```
[My App] [dhh] Rolled back to version d264c4e92470ad1bd18590f04466787262f605de
```
### Using custom healthcheck path or port
MRSK defaults to checking the health of your application again `/up` on port 3000. You can tailor both with the `healthcheck` setting:
```yaml
healthcheck:
path: /healthz
port: 4000
```
This will ensure your application is configured with a traefik label for the healthcheck against `/healthz` and that the pre-deploy healthcheck that MRSK performs is done against the same path on port 4000.
## Commands
### Running commands on servers
You can execute one-off commands on the servers:
```bash
# Runs command on all servers
mrsk app exec 'ruby -v'
App Host: 192.168.0.1
ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]
App Host: 192.168.0.2
ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]
# Runs command on primary server
mrsk app exec --primary 'cat .ruby-version'
App Host: 192.168.0.1
3.1.3
# Runs Rails command on all servers
mrsk app exec 'bin/rails about'
App Host: 192.168.0.1
About your application's environment
Rails version 7.1.0.alpha
Ruby version ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]
RubyGems version 3.3.26
Rack version 2.2.5
Middleware ActionDispatch::HostAuthorization, Rack::Sendfile, ActionDispatch::Static, ActionDispatch::Executor, Rack::Runtime, Rack::MethodOverride, ActionDispatch::RequestId, ActionDispatch::RemoteIp, Rails::Rack::Logger, ActionDispatch::ShowExceptions, ActionDispatch::DebugExceptions, ActionDispatch::Callbacks, ActionDispatch::Cookies, ActionDispatch::Session::CookieStore, ActionDispatch::Flash, ActionDispatch::ContentSecurityPolicy::Middleware, ActionDispatch::PermissionsPolicy::Middleware, Rack::Head, Rack::ConditionalGet, Rack::ETag, Rack::TempfileReaper
Application root /rails
Environment production
Database adapter sqlite3
Database schema version 20221231233303
App Host: 192.168.0.2
About your application's environment
Rails version 7.1.0.alpha
Ruby version ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]
RubyGems version 3.3.26
Rack version 2.2.5
Middleware ActionDispatch::HostAuthorization, Rack::Sendfile, ActionDispatch::Static, ActionDispatch::Executor, Rack::Runtime, Rack::MethodOverride, ActionDispatch::RequestId, ActionDispatch::RemoteIp, Rails::Rack::Logger, ActionDispatch::ShowExceptions, ActionDispatch::DebugExceptions, ActionDispatch::Callbacks, ActionDispatch::Cookies, ActionDispatch::Session::CookieStore, ActionDispatch::Flash, ActionDispatch::ContentSecurityPolicy::Middleware, ActionDispatch::PermissionsPolicy::Middleware, Rack::Head, Rack::ConditionalGet, Rack::ETag, Rack::TempfileReaper
Application root /rails
Environment production
Database adapter sqlite3
Database schema version 20221231233303
# Run Rails runner on primary server
mrsk app exec -p 'bin/rails runner "puts Rails.application.config.time_zone"'
UTC
```
### Running interactive commands over SSH
You can run interactive commands, like a Rails console or a bash session, on a server (default is primary, use `--hosts` to connect to another):
```bash
# Starts a bash session in a new container made from the most recent app image
mrsk app exec -i bash
# Starts a bash session in the currently running container for the app
mrsk app exec -i --reuse bash
# Starts a Rails console in a new container made from the most recent app image
mrsk app exec -i 'bin/rails console'
```
### Running details to show state of containers
You can see the state of your servers by running `mrsk details`:
```
Traefik Host: 192.168.0.1
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6195b2a28c81 traefik "/entrypoint.sh --pr…" 30 minutes ago Up 19 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp traefik
Traefik Host: 192.168.0.2
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
de14a335d152 traefik "/entrypoint.sh --pr…" 30 minutes ago Up 19 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp traefik
App Host: 192.168.0.1
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
badb1aa51db3 registry.digitalocean.com/user/app:6ef8a6a84c525b123c5245345a8483f86d05a123 "/rails/bin/docker-e…" 13 minutes ago Up 13 minutes 3000/tcp chat-6ef8a6a84c525b123c5245345a8483f86d05a123
App Host: 192.168.0.2
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1d3c91ed1f55 registry.digitalocean.com/user/app:6ef8a6a84c525b123c5245345a8483f86d05a123 "/rails/bin/docker-e…" 13 minutes ago Up 13 minutes 3000/tcp chat-6ef8a6a84c525b123c5245345a8483f86d05a123
```
You can also see just info for app containers with `mrsk app details` or just for Traefik with `mrsk traefik details`.
### Running rollback to fix a bad deploy
If you've discovered a bad deploy, you can quickly rollback by reactivating the old, paused container image. You can see what old containers are available for rollback by running `mrsk app containers`. It'll give you a presentation similar to `mrsk app details`, but include all the old containers as well. Showing something like this:
```
App Host: 192.168.0.1
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1d3c91ed1f51 registry.digitalocean.com/user/app:6ef8a6a84c525b123c5245345a8483f86d05a123 "/rails/bin/docker-e…" 19 minutes ago Up 19 minutes 3000/tcp chat-6ef8a6a84c525b123c5245345a8483f86d05a123
539f26b28369 registry.digitalocean.com/user/app:e5d9d7c2b898289dfbc5f7f1334140d984eedae4 "/rails/bin/docker-e…" 31 minutes ago Exited (1) 27 minutes ago chat-e5d9d7c2b898289dfbc5f7f1334140d984eedae4
App Host: 192.168.0.2
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
badb1aa51db4 registry.digitalocean.com/user/app:6ef8a6a84c525b123c5245345a8483f86d05a123 "/rails/bin/docker-e…" 19 minutes ago Up 19 minutes 3000/tcp chat-6ef8a6a84c525b123c5245345a8483f86d05a123
6f170d1172ae registry.digitalocean.com/user/app:e5d9d7c2b898289dfbc5f7f1334140d984eedae4 "/rails/bin/docker-e…" 31 minutes ago Exited (1) 27 minutes ago chat-e5d9d7c2b898289dfbc5f7f1334140d984eedae4
```
From the example above, we can see that `e5d9d7c2b898289dfbc5f7f1334140d984eedae4` was the last version, so it's available as a rollback target. We can perform this rollback by running `mrsk rollback e5d9d7c2b898289dfbc5f7f1334140d984eedae4`. That'll stop `6ef8a6a84c525b123c5245345a8483f86d05a123` and then start `e5d9d7c2b898289dfbc5f7f1334140d984eedae4`. Because the old container is still available, this is very quick. Nothing to download from the registry.
Note that by default old containers are pruned after 3 days when you run `mrsk deploy`.
### Running removal to clean up servers
If you wish to remove the entire application, including Traefik, containers, images, and registry session, you can run `mrsk remove`. This will leave the servers clean.
## Stage of development
This is beta software. Commands may still move around. But we're live in production at [37signals](https://37signals.com).
Please help us improve Kamal's documentation on the [the basecamp/kamal-site repository](https://github.com/basecamp/kamal-site).
## License
MRSK is released under the [MIT License](https://opensource.org/licenses/MIT).
Kamal is released under the [MIT License](https://opensource.org/licenses/MIT).

View File

@@ -3,14 +3,16 @@
# Prevent failures from being reported twice.
Thread.report_on_exception = false
require "mrsk"
require "kamal"
begin
Mrsk::Cli::Main.start(ARGV)
Kamal::Cli::Main.start(ARGV)
rescue SSHKit::Runner::ExecuteError => e
puts " \e[31mERROR (#{e.cause.class}): #{e.cause.message}\e[0m"
puts " \e[31mERROR (#{e.cause.class}): #{e.message}\e[0m"
puts e.cause.backtrace if ENV["VERBOSE"]
exit 1
rescue => e
puts " \e[31mERROR (#{e.class}): #{e.message}\e[0m"
puts e.backtrace if ENV["VERBOSE"]
exit 1
end

View File

@@ -2,13 +2,13 @@
VERSION=$1
printf "module Mrsk\n VERSION = \"$VERSION\"\nend\n" > ./lib/mrsk/version.rb
printf "module Kamal\n VERSION = \"$VERSION\"\nend\n" > ./lib/kamal/version.rb
bundle
git add Gemfile.lock lib/mrsk/version.rb
git add Gemfile.lock lib/kamal/version.rb
git commit -m "Bump version for $VERSION"
git push
git tag v$VERSION
git push --tags
gem build mrsk.gemspec
gem push "mrsk-$VERSION.gem" --host https://rubygems.org
rm "mrsk-$VERSION.gem"
gem build kamal.gemspec
gem push "kamal-$VERSION.gem" --host https://rubygems.org
rm "kamal-$VERSION.gem"

View File

@@ -0,0 +1,9 @@
source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
git "https://github.com/rails/rails.git" do
gem "railties"
gem "activesupport"
end
gemspec path: "../"

View File

@@ -0,0 +1,6 @@
source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
gemspec path: "../"
gem "nokogiri", "~> 1.15.0"

View File

@@ -1,22 +1,28 @@
require_relative "lib/mrsk/version"
require_relative "lib/kamal/version"
Gem::Specification.new do |spec|
spec.name = "mrsk"
spec.version = Mrsk::VERSION
spec.name = "kamal"
spec.version = Kamal::VERSION
spec.authors = [ "David Heinemeier Hansson" ]
spec.email = "dhh@hey.com"
spec.homepage = "https://github.com/rails/mrsk"
spec.homepage = "https://github.com/basecamp/kamal"
spec.summary = "Deploy web apps in containers to servers running Docker with zero downtime."
spec.license = "MIT"
spec.files = Dir["lib/**/*", "MIT-LICENSE", "README.md"]
spec.executables = %w[ mrsk ]
spec.executables = %w[ kamal ]
spec.add_dependency "activesupport", ">= 7.0"
spec.add_dependency "sshkit", "~> 1.21"
spec.add_dependency "sshkit", ">= 1.22.2", "< 2.0"
spec.add_dependency "net-ssh", "~> 7.0"
spec.add_dependency "thor", "~> 1.2"
spec.add_dependency "dotenv", "~> 2.8"
spec.add_dependency "zeitwerk", "~> 2.5"
spec.add_dependency "ed25519", "~> 1.2"
spec.add_dependency "bcrypt_pbkdf", "~> 1.0"
spec.add_dependency "concurrent-ruby", "~> 1.2"
spec.add_dependency "base64", "~> 0.2"
spec.add_development_dependency "debug"
spec.add_development_dependency "mocha"
spec.add_development_dependency "railties"
end

View File

@@ -1,9 +1,10 @@
module Mrsk
module Kamal
end
require "active_support"
require "zeitwerk"
loader = Zeitwerk::Loader.for_gem
loader.ignore("#{__dir__}/mrsk/sshkit_with_ext.rb")
loader.ignore(File.join(__dir__, "kamal", "sshkit_with_ext.rb"))
loader.setup
loader.eager_load # We need all commands loaded.

7
lib/kamal/cli.rb Normal file
View File

@@ -0,0 +1,7 @@
module Kamal::Cli
class LockError < StandardError; end
class HookError < StandardError; end
end
# SSHKit uses instance eval, so we need a global const for ergonomics
KAMAL = Kamal::Commander.new

257
lib/kamal/cli/accessory.rb Normal file
View File

@@ -0,0 +1,257 @@
class Kamal::Cli::Accessory < Kamal::Cli::Base
desc "boot [NAME]", "Boot new accessory service on host (use NAME=all to boot all accessories)"
def boot(name, login: true)
with_lock do
if name == "all"
KAMAL.accessory_names.each { |accessory_name| boot(accessory_name) }
else
with_accessory(name) do |accessory, hosts|
directories(name)
upload(name)
on(hosts) do
execute *KAMAL.registry.login if login
execute *KAMAL.auditor.record("Booted #{name} accessory"), verbosity: :debug
execute *accessory.run
end
end
end
end
end
desc "upload [NAME]", "Upload accessory files to host", hide: true
def upload(name)
with_lock do
with_accessory(name) do |accessory, hosts|
on(hosts) do
accessory.files.each do |(local, remote)|
accessory.ensure_local_file_present(local)
execute *accessory.make_directory_for(remote)
upload! local, remote
execute :chmod, "755", remote
end
end
end
end
end
desc "directories [NAME]", "Create accessory directories on host", hide: true
def directories(name)
with_lock do
with_accessory(name) do |accessory, hosts|
on(hosts) do
accessory.directories.keys.each do |host_path|
execute *accessory.make_directory(host_path)
end
end
end
end
end
desc "reboot [NAME]", "Reboot existing accessory on host (stop container, remove container, start new container; use NAME=all to boot all accessories)"
def reboot(name)
with_lock do
if name == "all"
KAMAL.accessory_names.each { |accessory_name| reboot(accessory_name) }
else
with_accessory(name) do |accessory, hosts|
on(hosts) do
execute *KAMAL.registry.login
end
stop(name)
remove_container(name)
boot(name, login: false)
end
end
end
end
desc "start [NAME]", "Start existing accessory container on host"
def start(name)
with_lock do
with_accessory(name) do |accessory, hosts|
on(hosts) do
execute *KAMAL.auditor.record("Started #{name} accessory"), verbosity: :debug
execute *accessory.start
end
end
end
end
desc "stop [NAME]", "Stop existing accessory container on host"
def stop(name)
with_lock do
with_accessory(name) do |accessory, hosts|
on(hosts) do
execute *KAMAL.auditor.record("Stopped #{name} accessory"), verbosity: :debug
execute *accessory.stop, raise_on_non_zero_exit: false
end
end
end
end
desc "restart [NAME]", "Restart existing accessory container on host"
def restart(name)
with_lock do
with_accessory(name) do
stop(name)
start(name)
end
end
end
desc "details [NAME]", "Show details about accessory on host (use NAME=all to show all accessories)"
def details(name)
if name == "all"
KAMAL.accessory_names.each { |accessory_name| details(accessory_name) }
else
type = "Accessory #{name}"
with_accessory(name) do |accessory, hosts|
on(hosts) { puts_by_host host, capture_with_info(*accessory.info), type: type }
end
end
end
desc "exec [NAME] [CMD]", "Execute a custom command on servers (use --help to show options)"
option :interactive, aliases: "-i", type: :boolean, default: false, desc: "Execute command over ssh for an interactive shell (use for console/bash)"
option :reuse, type: :boolean, default: false, desc: "Reuse currently running container instead of starting a new one"
def exec(name, cmd)
with_accessory(name) do |accessory, hosts|
case
when options[:interactive] && options[:reuse]
say "Launching interactive command with via SSH from existing container...", :magenta
run_locally { exec accessory.execute_in_existing_container_over_ssh(cmd) }
when options[:interactive]
say "Launching interactive command via SSH from new container...", :magenta
run_locally { exec accessory.execute_in_new_container_over_ssh(cmd) }
when options[:reuse]
say "Launching command from existing container...", :magenta
on(hosts) do
execute *KAMAL.auditor.record("Executed cmd '#{cmd}' on #{name} accessory"), verbosity: :debug
capture_with_info(*accessory.execute_in_existing_container(cmd))
end
else
say "Launching command from new container...", :magenta
on(hosts) do
execute *KAMAL.auditor.record("Executed cmd '#{cmd}' on #{name} accessory"), verbosity: :debug
capture_with_info(*accessory.execute_in_new_container(cmd))
end
end
end
end
desc "logs [NAME]", "Show log lines from accessory on host (use --help to show options)"
option :since, aliases: "-s", desc: "Show logs since timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes)"
option :lines, type: :numeric, aliases: "-n", desc: "Number of log lines to pull from each server"
option :grep, aliases: "-g", desc: "Show lines with grep match only (use this to fetch specific requests by id)"
option :follow, aliases: "-f", desc: "Follow logs on primary server (or specific host set by --hosts)"
def logs(name)
with_accessory(name) do |accessory, hosts|
grep = options[:grep]
if options[:follow]
run_locally do
info "Following logs on #{hosts}..."
info accessory.follow_logs(grep: grep)
exec accessory.follow_logs(grep: grep)
end
else
since = options[:since]
lines = options[:lines].presence || ((since || grep) ? nil : 100) # Default to 100 lines if since or grep isn't set
on(hosts) do
puts capture_with_info(*accessory.logs(since: since, lines: lines, grep: grep))
end
end
end
end
desc "remove [NAME]", "Remove accessory container, image and data directory from host (use NAME=all to remove all accessories)"
option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
def remove(name)
confirming "This will remove all containers, images and data directories for #{name}. Are you sure?" do
with_lock do
if name == "all"
KAMAL.accessory_names.each { |accessory_name| remove_accessory(accessory_name) }
else
remove_accessory(name)
end
end
end
end
desc "remove_container [NAME]", "Remove accessory container from host", hide: true
def remove_container(name)
with_lock do
with_accessory(name) do |accessory, hosts|
on(hosts) do
execute *KAMAL.auditor.record("Remove #{name} accessory container"), verbosity: :debug
execute *accessory.remove_container
end
end
end
end
desc "remove_image [NAME]", "Remove accessory image from host", hide: true
def remove_image(name)
with_lock do
with_accessory(name) do |accessory, hosts|
on(hosts) do
execute *KAMAL.auditor.record("Removed #{name} accessory image"), verbosity: :debug
execute *accessory.remove_image
end
end
end
end
desc "remove_service_directory [NAME]", "Remove accessory directory used for uploaded files and data directories from host", hide: true
def remove_service_directory(name)
with_lock do
with_accessory(name) do |accessory, hosts|
on(hosts) do
execute *accessory.remove_service_directory
end
end
end
end
private
def with_accessory(name)
if KAMAL.config.accessory(name)
accessory = KAMAL.accessory(name)
yield accessory, accessory_hosts(accessory)
else
error_on_missing_accessory(name)
end
end
def error_on_missing_accessory(name)
options = KAMAL.accessory_names.presence
error \
"No accessory by the name of '#{name}'" +
(options ? " (options: #{options.to_sentence})" : "")
end
def accessory_hosts(accessory)
if KAMAL.specific_hosts&.any?
KAMAL.specific_hosts & accessory.hosts
else
accessory.hosts
end
end
def remove_accessory(name)
with_accessory(name) do
stop(name)
remove_container(name)
remove_image(name)
remove_service_directory(name)
end
end
end

295
lib/kamal/cli/app.rb Normal file
View File

@@ -0,0 +1,295 @@
class Kamal::Cli::App < Kamal::Cli::Base
desc "boot", "Boot app on servers (or reboot app if already running)"
def boot
with_lock do
say "Get most recent version available as an image...", :magenta unless options[:version]
using_version(version_or_latest) do |version|
say "Start container with version #{version} using a #{KAMAL.config.readiness_delay}s readiness delay (or reboot if already running)...", :magenta
# Assets are prepared in a separate step to ensure they are on all hosts before booting
on(KAMAL.hosts) do
KAMAL.roles_on(host).each do |role|
Kamal::Cli::App::PrepareAssets.new(host, role, self).run
end
end
# Primary hosts and roles are returned first, so they can open the barrier
barrier = Kamal::Cli::Healthcheck::Barrier.new if KAMAL.roles.many?
on(KAMAL.hosts, **KAMAL.boot_strategy) do |host|
KAMAL.roles_on(host).each do |role|
Kamal::Cli::App::Boot.new(host, role, self, version, barrier).run
end
end
# Tag once the app booted on all hosts
on(KAMAL.hosts) do |host|
execute *KAMAL.auditor.record("Tagging #{KAMAL.config.absolute_image} as the latest image"), verbosity: :debug
execute *KAMAL.app.tag_latest_image
end
end
end
end
desc "start", "Start existing app container on servers"
def start
with_lock do
on(KAMAL.hosts) do |host|
roles = KAMAL.roles_on(host)
roles.each do |role|
execute *KAMAL.auditor.record("Started app version #{KAMAL.config.version}"), verbosity: :debug
execute *KAMAL.app(role: role, host: host).start, raise_on_non_zero_exit: false
end
end
end
end
desc "stop", "Stop app container on servers"
def stop
with_lock do
on(KAMAL.hosts) do |host|
roles = KAMAL.roles_on(host)
roles.each do |role|
execute *KAMAL.auditor.record("Stopped app", role: role), verbosity: :debug
execute *KAMAL.app(role: role, host: host).stop, raise_on_non_zero_exit: false
end
end
end
end
# FIXME: Drop in favor of just containers?
desc "details", "Show details about app containers"
def details
on(KAMAL.hosts) do |host|
roles = KAMAL.roles_on(host)
roles.each do |role|
puts_by_host host, capture_with_info(*KAMAL.app(role: role, host: host).info)
end
end
end
desc "exec [CMD]", "Execute a custom command on servers within the app container (use --help to show options)"
option :interactive, aliases: "-i", type: :boolean, default: false, desc: "Execute command over ssh for an interactive shell (use for console/bash)"
option :reuse, type: :boolean, default: false, desc: "Reuse currently running container instead of starting a new one"
option :env, aliases: "-e", type: :hash, desc: "Set environment variables for the command"
def exec(cmd)
env = options[:env]
case
when options[:interactive] && options[:reuse]
say "Get current version of running container...", :magenta unless options[:version]
using_version(options[:version] || current_running_version) do |version|
say "Launching interactive command with version #{version} via SSH from existing container on #{KAMAL.primary_host}...", :magenta
run_locally { exec KAMAL.app(role: KAMAL.primary_role, host: KAMAL.primary_host).execute_in_existing_container_over_ssh(cmd, env: env) }
end
when options[:interactive]
say "Get most recent version available as an image...", :magenta unless options[:version]
using_version(version_or_latest) do |version|
say "Launching interactive command with version #{version} via SSH from new container on #{KAMAL.primary_host}...", :magenta
run_locally do
exec KAMAL.app(role: KAMAL.primary_role, host: KAMAL.primary_host).execute_in_new_container_over_ssh(cmd, env: env)
end
end
when options[:reuse]
say "Get current version of running container...", :magenta unless options[:version]
using_version(options[:version] || current_running_version) do |version|
say "Launching command with version #{version} from existing container...", :magenta
on(KAMAL.hosts) do |host|
roles = KAMAL.roles_on(host)
roles.each do |role|
execute *KAMAL.auditor.record("Executed cmd '#{cmd}' on app version #{version}", role: role), verbosity: :debug
puts_by_host host, capture_with_info(*KAMAL.app(role: role, host: host).execute_in_existing_container(cmd, env: env))
end
end
end
else
say "Get most recent version available as an image...", :magenta unless options[:version]
using_version(version_or_latest) do |version|
say "Launching command with version #{version} from new container...", :magenta
on(KAMAL.hosts) do |host|
roles = KAMAL.roles_on(host)
roles.each do |role|
execute *KAMAL.auditor.record("Executed cmd '#{cmd}' on app version #{version}"), verbosity: :debug
puts_by_host host, capture_with_info(*KAMAL.app(role: role, host: host).execute_in_new_container(cmd, env: env))
end
end
end
end
end
desc "containers", "Show app containers on servers"
def containers
on(KAMAL.hosts) { |host| puts_by_host host, capture_with_info(*KAMAL.app.list_containers) }
end
desc "stale_containers", "Detect app stale containers"
option :stop, aliases: "-s", type: :boolean, default: false, desc: "Stop the stale containers found"
def stale_containers
stop = options[:stop]
with_lock_if_stopping do
on(KAMAL.hosts) do |host|
roles = KAMAL.roles_on(host)
roles.each do |role|
app = KAMAL.app(role: role, host: host)
versions = capture_with_info(*app.list_versions, raise_on_non_zero_exit: false).split("\n")
versions -= [ capture_with_info(*app.current_running_version, raise_on_non_zero_exit: false).strip ]
versions.each do |version|
if stop
puts_by_host host, "Stopping stale container for role #{role} with version #{version}"
execute *app.stop(version: version), raise_on_non_zero_exit: false
else
puts_by_host host, "Detected stale container for role #{role} with version #{version} (use `kamal app stale_containers --stop` to stop)"
end
end
end
end
end
end
desc "images", "Show app images on servers"
def images
on(KAMAL.hosts) { |host| puts_by_host host, capture_with_info(*KAMAL.app.list_images) }
end
desc "logs", "Show log lines from app on servers (use --help to show options)"
option :since, aliases: "-s", desc: "Show lines since timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes)"
option :lines, type: :numeric, aliases: "-n", desc: "Number of lines to show from each server"
option :grep, aliases: "-g", desc: "Show lines with grep match only (use this to fetch specific requests by id)"
option :follow, aliases: "-f", desc: "Follow log on primary server (or specific host set by --hosts)"
def logs
# FIXME: Catch when app containers aren't running
grep = options[:grep]
since = options[:since]
if options[:follow]
lines = options[:lines].presence || ((since || grep) ? nil : 10) # Default to 10 lines if since or grep isn't set
run_locally do
info "Following logs on #{KAMAL.primary_host}..."
KAMAL.specific_roles ||= [ "web" ]
role = KAMAL.roles_on(KAMAL.primary_host).first
app = KAMAL.app(role: role, host: host)
info app.follow_logs(host: KAMAL.primary_host, lines: lines, grep: grep)
exec app.follow_logs(host: KAMAL.primary_host, lines: lines, grep: grep)
end
else
lines = options[:lines].presence || ((since || grep) ? nil : 100) # Default to 100 lines if since or grep isn't set
on(KAMAL.hosts) do |host|
roles = KAMAL.roles_on(host)
roles.each do |role|
begin
puts_by_host host, capture_with_info(*KAMAL.app(role: role, host: host).logs(since: since, lines: lines, grep: grep))
rescue SSHKit::Command::Failed
puts_by_host host, "Nothing found"
end
end
end
end
end
desc "remove", "Remove app containers and images from servers"
def remove
with_lock do
stop
remove_containers
remove_images
end
end
desc "remove_container [VERSION]", "Remove app container with given version from servers", hide: true
def remove_container(version)
with_lock do
on(KAMAL.hosts) do |host|
roles = KAMAL.roles_on(host)
roles.each do |role|
execute *KAMAL.auditor.record("Removed app container with version #{version}", role: role), verbosity: :debug
execute *KAMAL.app(role: role, host: host).remove_container(version: version)
end
end
end
end
desc "remove_containers", "Remove all app containers from servers", hide: true
def remove_containers
with_lock do
on(KAMAL.hosts) do |host|
roles = KAMAL.roles_on(host)
roles.each do |role|
execute *KAMAL.auditor.record("Removed all app containers", role: role), verbosity: :debug
execute *KAMAL.app(role: role, host: host).remove_containers
end
end
end
end
desc "remove_images", "Remove all app images from servers", hide: true
def remove_images
with_lock do
on(KAMAL.hosts) do
execute *KAMAL.auditor.record("Removed all app images"), verbosity: :debug
execute *KAMAL.app.remove_images
end
end
end
desc "version", "Show app version currently running on servers"
def version
on(KAMAL.hosts) do |host|
role = KAMAL.roles_on(host).first
puts_by_host host, capture_with_info(*KAMAL.app(role: role, host: host).current_running_version).strip
end
end
private
def using_version(new_version)
if new_version
begin
old_version = KAMAL.config.version
KAMAL.config.version = new_version
yield new_version
ensure
KAMAL.config.version = old_version
end
else
yield KAMAL.config.version
end
end
def current_running_version(host: KAMAL.primary_host)
version = nil
on(host) do
role = KAMAL.roles_on(host).first
version = capture_with_info(*KAMAL.app(role: role, host: host).current_running_version).strip
end
version.presence
end
def version_or_latest
options[:version] || KAMAL.config.latest_tag
end
def with_lock_if_stopping
if options[:stop]
with_lock { yield }
else
yield
end
end
end

119
lib/kamal/cli/app/boot.rb Normal file
View File

@@ -0,0 +1,119 @@
class Kamal::Cli::App::Boot
attr_reader :host, :role, :version, :barrier, :sshkit
delegate :execute, :capture_with_info, :capture_with_pretty_json, :info, :error, to: :sshkit
delegate :uses_cord?, :assets?, :running_traefik?, to: :role
def initialize(host, role, sshkit, version, barrier)
@host = host
@role = role
@version = version
@barrier = barrier
@sshkit = sshkit
end
def run
old_version = old_version_renamed_if_clashing
wait_at_barrier if queuer?
begin
start_new_version
rescue => e
close_barrier if gatekeeper?
stop_new_version
raise
end
release_barrier if gatekeeper?
if old_version
stop_old_version(old_version)
end
end
private
def old_version_renamed_if_clashing
if capture_with_info(*app.container_id_for_version(version), raise_on_non_zero_exit: false).present?
renamed_version = "#{version}_replaced_#{SecureRandom.hex(8)}"
info "Renaming container #{version} to #{renamed_version} as already deployed on #{host}"
audit("Renaming container #{version} to #{renamed_version}")
execute *app.rename_container(version: version, new_version: renamed_version)
end
capture_with_info(*app.current_running_version, raise_on_non_zero_exit: false).strip.presence
end
def start_new_version
audit "Booted app version #{version}"
execute *app.tie_cord(role.cord_host_file) if uses_cord?
hostname = "#{host.to_s[0...51].gsub(/\.+$/, '')}-#{SecureRandom.hex(6)}"
execute *app.run(hostname: hostname)
Kamal::Cli::Healthcheck::Poller.wait_for_healthy(pause_after_ready: true) { capture_with_info(*app.status(version: version)) }
end
def stop_new_version
execute *app.stop(version: version), raise_on_non_zero_exit: false
end
def stop_old_version(version)
if uses_cord?
cord = capture_with_info(*app.cord(version: version), raise_on_non_zero_exit: false).strip
if cord.present?
execute *app.cut_cord(cord)
Kamal::Cli::Healthcheck::Poller.wait_for_unhealthy(pause_after_ready: true) { capture_with_info(*app.status(version: version)) }
end
end
execute *app.stop(version: version), raise_on_non_zero_exit: false
execute *app.clean_up_assets if assets?
end
def release_barrier
if barrier.open
info "First #{KAMAL.primary_role} container is healthy on #{host}, booting other roles"
end
end
def wait_at_barrier
info "Waiting for the first healthy #{KAMAL.primary_role} container before booting #{role} on #{host}..."
barrier.wait
info "First #{KAMAL.primary_role} container is healthy, booting #{role} on #{host}..."
rescue Kamal::Cli::Healthcheck::Error
info "First #{KAMAL.primary_role} container is unhealthy, not booting #{role} on #{host}"
raise
end
def close_barrier
if barrier.close
info "First #{KAMAL.primary_role} container is unhealthy on #{host}, not booting other roles"
error capture_with_info(*app.logs(version: version))
error capture_with_info(*app.container_health_log(version: version))
end
end
def barrier_role?
role == KAMAL.primary_role
end
def app
@app ||= KAMAL.app(role: role, host: host)
end
def auditor
@auditor = KAMAL.auditor(role: role)
end
def audit(message)
execute *auditor.record(message), verbosity: :debug
end
def gatekeeper?
barrier && barrier_role?
end
def queuer?
barrier && !barrier_role?
end
end

View File

@@ -0,0 +1,24 @@
class Kamal::Cli::App::PrepareAssets
attr_reader :host, :role, :sshkit
delegate :execute, :capture_with_info, :info, to: :sshkit
delegate :assets?, to: :role
def initialize(host, role, sshkit)
@host = host
@role = role
@sshkit = sshkit
end
def run
if assets?
execute *app.extract_assets
old_version = capture_with_info(*app.current_running_version, raise_on_non_zero_exit: false).strip
execute *app.sync_asset_volumes(old_version: old_version)
end
end
private
def app
@app ||= KAMAL.app(role: role, host: host)
end
end

197
lib/kamal/cli/base.rb Normal file
View File

@@ -0,0 +1,197 @@
require "thor"
require "dotenv"
require "kamal/sshkit_with_ext"
module Kamal::Cli
class Base < Thor
include SSHKit::DSL
def self.exit_on_failure?() true end
class_option :verbose, type: :boolean, aliases: "-v", desc: "Detailed logging"
class_option :quiet, type: :boolean, aliases: "-q", desc: "Minimal logging"
class_option :version, desc: "Run commands against a specific app version"
class_option :primary, type: :boolean, aliases: "-p", desc: "Run commands only on primary host instead of all"
class_option :hosts, aliases: "-h", desc: "Run commands on these hosts instead of all (separate by comma, supports wildcards with *)"
class_option :roles, aliases: "-r", desc: "Run commands on these roles instead of all (separate by comma, supports wildcards with *)"
class_option :config_file, aliases: "-c", default: "config/deploy.yml", desc: "Path to config file"
class_option :destination, aliases: "-d", desc: "Specify destination to be used for config file (staging -> deploy.staging.yml)"
class_option :skip_hooks, aliases: "-H", type: :boolean, default: false, desc: "Don't run hooks"
def initialize(*)
super
@original_env = ENV.to_h.dup
load_envs
initialize_commander(options_with_subcommand_class_options)
end
private
def load_envs
if destination = options[:destination]
Dotenv.load(".env.#{destination}", ".env")
else
Dotenv.load(".env")
end
end
def reload_envs
ENV.clear
ENV.update(@original_env)
load_envs
end
def options_with_subcommand_class_options
options.merge(@_initializer.last[:class_options] || {})
end
def initialize_commander(options)
KAMAL.tap do |commander|
if options[:verbose]
ENV["VERBOSE"] = "1" # For backtraces via cli/start
commander.verbosity = :debug
end
if options[:quiet]
commander.verbosity = :error
end
commander.configure \
config_file: Pathname.new(File.expand_path(options[:config_file])),
destination: options[:destination],
version: options[:version]
commander.specific_hosts = options[:hosts]&.split(",")
commander.specific_roles = options[:roles]&.split(",")
commander.specific_primary! if options[:primary]
end
end
def print_runtime
started_at = Time.now
yield
Time.now - started_at
ensure
runtime = Time.now - started_at
puts " Finished all in #{sprintf("%.1f seconds", runtime)}"
end
def with_lock
if KAMAL.holding_lock?
yield
else
ensure_run_and_locks_directory
acquire_lock
begin
yield
rescue
begin
release_lock
rescue => e
say "Error releasing the deploy lock: #{e.message}", :red
end
raise
end
release_lock
end
end
def confirming(question)
return yield if options[:confirmed]
if ask(question, limited_to: %w[ y N ], default: "N") == "y"
yield
else
say "Aborted", :red
end
end
def acquire_lock
raise_if_locked do
say "Acquiring the deploy lock...", :magenta
on(KAMAL.primary_host) { execute *KAMAL.lock.acquire("Automatic deploy lock", KAMAL.config.version), verbosity: :debug }
end
KAMAL.holding_lock = true
end
def release_lock
say "Releasing the deploy lock...", :magenta
on(KAMAL.primary_host) { execute *KAMAL.lock.release, verbosity: :debug }
KAMAL.holding_lock = false
end
def raise_if_locked
yield
rescue SSHKit::Runner::ExecuteError => e
if e.message =~ /cannot create directory/
say "Deploy lock already in place!", :red
on(KAMAL.primary_host) { puts capture_with_debug(*KAMAL.lock.status) }
raise LockError, "Deploy lock found. Run 'kamal lock help' for more information"
else
raise e
end
end
def run_hook(hook, **extra_details)
if !options[:skip_hooks] && KAMAL.hook.hook_exists?(hook)
details = { hosts: KAMAL.hosts.join(","), command: command, subcommand: subcommand }
say "Running the #{hook} hook...", :magenta
run_locally do
execute *KAMAL.hook.run(hook, **details, **extra_details)
rescue SSHKit::Command::Failed => e
raise HookError.new("Hook `#{hook}` failed:\n#{e.message}")
end
end
end
def on(*args, &block)
if !KAMAL.connected?
run_hook "pre-connect"
KAMAL.connected = true
end
super
end
def command
@kamal_command ||= begin
invocation_class, invocation_commands = *first_invocation
if invocation_class == Kamal::Cli::Main
invocation_commands[0]
else
Kamal::Cli::Main.subcommand_classes.find { |command, clazz| clazz == invocation_class }[0]
end
end
end
def subcommand
@kamal_subcommand ||= begin
invocation_class, invocation_commands = *first_invocation
invocation_commands[0] if invocation_class != Kamal::Cli::Main
end
end
def first_invocation
instance_variable_get("@_invocations").first
end
def ensure_run_and_locks_directory
on(KAMAL.hosts) do
execute(*KAMAL.server.ensure_run_directory)
end
on(KAMAL.primary_host) do
execute(*KAMAL.lock.ensure_locks_directory)
end
end
end
end

131
lib/kamal/cli/build.rb Normal file
View File

@@ -0,0 +1,131 @@
require "uri"
class Kamal::Cli::Build < Kamal::Cli::Base
class BuildError < StandardError; end
desc "deliver", "Build app and push app image to registry then pull image on servers"
def deliver
push
pull
end
desc "push", "Build and push app image to registry"
def push
cli = self
verify_local_dependencies
run_hook "pre-build"
uncommitted_changes = Kamal::Git.uncommitted_changes
if KAMAL.config.builder.git_clone?
if uncommitted_changes.present?
say "Building from a local git clone, so ignoring these uncommitted changes:\n #{uncommitted_changes}", :yellow
end
run_locally do
Clone.new(self).prepare
end
elsif uncommitted_changes.present?
say "Building with uncommitted changes:\n #{uncommitted_changes}", :yellow
end
# Get the command here to ensure the Dir.chdir doesn't interfere with it
push = KAMAL.builder.push
run_locally do
begin
KAMAL.with_verbosity(:debug) do
Dir.chdir(KAMAL.config.builder.build_directory) { execute *push }
end
rescue SSHKit::Command::Failed => e
if e.message =~ /(no builder)|(no such file or directory)/
warn "Missing compatible builder, so creating a new one first"
if cli.create
KAMAL.with_verbosity(:debug) do
Dir.chdir(KAMAL.config.builder.build_directory) { execute *push }
end
end
else
raise
end
end
end
end
desc "pull", "Pull app image from registry onto servers"
def pull
on(KAMAL.hosts) do
execute *KAMAL.auditor.record("Pulled image with version #{KAMAL.config.version}"), verbosity: :debug
execute *KAMAL.builder.clean, raise_on_non_zero_exit: false
execute *KAMAL.builder.pull
execute *KAMAL.builder.validate_image
end
end
desc "create", "Create a build setup"
def create
if (remote_host = KAMAL.config.builder.remote_host)
connect_to_remote_host(remote_host)
end
run_locally do
begin
debug "Using builder: #{KAMAL.builder.name}"
execute *KAMAL.builder.create
rescue SSHKit::Command::Failed => e
if e.message =~ /stderr=(.*)/
error "Couldn't create remote builder: #{$1}"
false
else
raise
end
end
end
end
desc "remove", "Remove build setup"
def remove
run_locally do
debug "Using builder: #{KAMAL.builder.name}"
execute *KAMAL.builder.remove
end
end
desc "details", "Show build setup"
def details
run_locally do
puts "Builder: #{KAMAL.builder.name}"
puts capture(*KAMAL.builder.info)
end
end
private
def verify_local_dependencies
run_locally do
begin
execute *KAMAL.builder.ensure_local_dependencies_installed
rescue SSHKit::Command::Failed => e
build_error = e.message =~ /command not found/ ?
"Docker is not installed locally" :
"Docker buildx plugin is not installed locally"
raise BuildError, build_error
end
end
end
def connect_to_remote_host(remote_host)
remote_uri = URI.parse(remote_host)
if remote_uri.scheme == "ssh"
host = SSHKit::Host.new(
hostname: remote_uri.host,
ssh_options: { user: remote_uri.user, port: remote_uri.port }.compact
)
on(host, options) do
execute "true"
end
end
end
end

View File

@@ -0,0 +1,61 @@
require "uri"
class Kamal::Cli::Build::Clone
attr_reader :sshkit
delegate :info, :error, :execute, :capture_with_info, to: :sshkit
def initialize(sshkit)
@sshkit = sshkit
end
def prepare
begin
clone_repo
rescue SSHKit::Command::Failed => e
if e.message =~ /already exists and is not an empty directory/
reset
else
raise Kamal::Cli::Build::BuildError, "Failed to clone repo: #{e.message}"
end
end
validate!
rescue Kamal::Cli::Build::BuildError => e
error "Error preparing clone: #{e.message}, deleting and retrying..."
FileUtils.rm_rf KAMAL.config.builder.clone_directory
clone_repo
validate!
end
private
def clone_repo
info "Cloning repo into build directory `#{KAMAL.config.builder.build_directory}`..."
FileUtils.mkdir_p KAMAL.config.builder.clone_directory
execute *KAMAL.builder.clone
end
def reset
info "Resetting local clone as `#{KAMAL.config.builder.build_directory}` already exists..."
KAMAL.builder.clone_reset_steps.each { |step| execute *step }
rescue SSHKit::Command::Failed => e
raise Kamal::Cli::Build::BuildError, "Failed to clone repo: #{e.message}"
end
def validate!
status = capture_with_info(*KAMAL.builder.clone_status).strip
unless status.empty?
raise Kamal::Cli::Build::BuildError, "Clone in #{KAMAL.config.builder.build_directory} is dirty, #{status}"
end
revision = capture_with_info(*KAMAL.builder.clone_revision).strip
if revision != Kamal::Git.revision
raise Kamal::Cli::Build::BuildError, "Clone in #{KAMAL.config.builder.build_directory} is not on the correct revision, expected `#{Kamal::Git.revision}` but got `#{revision}`"
end
rescue SSHKit::Command::Failed => e
raise Kamal::Cli::Build::BuildError, "Failed to validate clone: #{e.message}"
end
end

54
lib/kamal/cli/env.rb Normal file
View File

@@ -0,0 +1,54 @@
require "tempfile"
class Kamal::Cli::Env < Kamal::Cli::Base
desc "push", "Push the env file to the remote hosts"
def push
with_lock do
on(KAMAL.hosts) do
execute *KAMAL.auditor.record("Pushed env files"), verbosity: :debug
KAMAL.roles_on(host).each do |role|
execute *KAMAL.app(role: role, host: host).make_env_directory
upload! role.env(host).secrets_io, role.env(host).secrets_file, mode: 400
end
end
on(KAMAL.traefik_hosts) do
execute *KAMAL.traefik.make_env_directory
upload! KAMAL.traefik.env.secrets_io, KAMAL.traefik.env.secrets_file, mode: 400
end
on(KAMAL.accessory_hosts) do
KAMAL.accessories_on(host).each do |accessory|
accessory_config = KAMAL.config.accessory(accessory)
execute *KAMAL.accessory(accessory).make_env_directory
upload! accessory_config.env.secrets_io, accessory_config.env.secrets_file, mode: 400
end
end
end
end
desc "delete", "Delete the env file from the remote hosts"
def delete
with_lock do
on(KAMAL.hosts) do
execute *KAMAL.auditor.record("Deleted env files"), verbosity: :debug
KAMAL.roles_on(host).each do |role|
execute *KAMAL.app(role: role, host: host).remove_env_file
end
end
on(KAMAL.traefik_hosts) do
execute *KAMAL.traefik.remove_env_file
end
on(KAMAL.accessory_hosts) do
KAMAL.accessories_on(host).each do |accessory|
accessory_config = KAMAL.config.accessory(accessory)
execute *KAMAL.accessory(accessory).remove_env_file
end
end
end
end
end

View File

@@ -0,0 +1,31 @@
class Kamal::Cli::Healthcheck::Barrier
def initialize
@ivar = Concurrent::IVar.new
end
def close
set(false)
end
def open
set(true)
end
def wait
unless opened?
raise Kamal::Cli::Healthcheck::Error.new("Halted at barrier")
end
end
private
def opened?
@ivar.value
end
def set(value)
@ivar.set(value)
true
rescue Concurrent::MultipleAssignmentError
false
end
end

View File

@@ -0,0 +1,2 @@
class Kamal::Cli::Healthcheck::Error < StandardError
end

View File

@@ -0,0 +1,63 @@
module Kamal::Cli::Healthcheck::Poller
extend self
TRAEFIK_UPDATE_DELAY = 5
def wait_for_healthy(pause_after_ready: false, &block)
attempt = 1
max_attempts = KAMAL.config.healthcheck["max_attempts"]
begin
case status = block.call
when "healthy"
sleep TRAEFIK_UPDATE_DELAY if pause_after_ready
when "running" # No health check configured
sleep KAMAL.config.readiness_delay if pause_after_ready
else
raise Kamal::Cli::Healthcheck::Error, "container not ready (#{status})"
end
rescue Kamal::Cli::Healthcheck::Error => e
if attempt <= max_attempts
info "#{e.message}, retrying in #{attempt}s (attempt #{attempt}/#{max_attempts})..."
sleep attempt
attempt += 1
retry
else
raise
end
end
info "Container is healthy!"
end
def wait_for_unhealthy(pause_after_ready: false, &block)
attempt = 1
max_attempts = KAMAL.config.healthcheck["max_attempts"]
begin
case status = block.call
when "unhealthy"
sleep TRAEFIK_UPDATE_DELAY if pause_after_ready
else
raise Kamal::Cli::Healthcheck::Error, "container not unhealthy (#{status})"
end
rescue Kamal::Cli::Healthcheck::Error => e
if attempt <= max_attempts
info "#{e.message}, retrying in #{attempt}s (attempt #{attempt}/#{max_attempts})..."
sleep attempt
attempt += 1
retry
else
raise
end
end
info "Container is unhealthy!"
end
private
def info(message)
SSHKit.config.output.info(message)
end
end

46
lib/kamal/cli/lock.rb Normal file
View File

@@ -0,0 +1,46 @@
class Kamal::Cli::Lock < Kamal::Cli::Base
desc "status", "Report lock status"
def status
handle_missing_lock do
on(KAMAL.primary_host) do
execute *KAMAL.server.ensure_run_directory
puts capture_with_debug(*KAMAL.lock.status)
end
end
end
desc "acquire", "Acquire the deploy lock"
option :message, aliases: "-m", type: :string, desc: "A lock message", required: true
def acquire
message = options[:message]
raise_if_locked do
on(KAMAL.primary_host) do
execute *KAMAL.server.ensure_run_directory
execute *KAMAL.lock.acquire(message, KAMAL.config.version), verbosity: :debug
end
say "Acquired the deploy lock"
end
end
desc "release", "Release the deploy lock"
def release
handle_missing_lock do
on(KAMAL.primary_host) do
execute *KAMAL.server.ensure_run_directory
execute *KAMAL.lock.release, verbosity: :debug
end
say "Released the deploy lock"
end
end
private
def handle_missing_lock
yield
rescue SSHKit::Runner::ExecuteError => e
if e.message =~ /No such file or directory/
say "There is no deploy lock"
else
raise
end
end
end

261
lib/kamal/cli/main.rb Normal file
View File

@@ -0,0 +1,261 @@
class Kamal::Cli::Main < Kamal::Cli::Base
desc "setup", "Setup all accessories, push the env, and deploy app to servers"
option :skip_push, aliases: "-P", type: :boolean, default: false, desc: "Skip image build and push"
def setup
print_runtime do
with_lock do
invoke_options = deploy_options
say "Ensure Docker is installed...", :magenta
invoke "kamal:cli:server:bootstrap", [], invoke_options
say "Evaluate and push env files...", :magenta
invoke "kamal:cli:main:envify", [], invoke_options
invoke "kamal:cli:accessory:boot", [ "all" ], invoke_options
deploy
end
end
end
desc "deploy", "Deploy app to servers"
option :skip_push, aliases: "-P", type: :boolean, default: false, desc: "Skip image build and push"
def deploy
runtime = print_runtime do
invoke_options = deploy_options
say "Log into image registry...", :magenta
invoke "kamal:cli:registry:login", [], invoke_options
if options[:skip_push]
say "Pull app image...", :magenta
invoke "kamal:cli:build:pull", [], invoke_options
else
say "Build and push app image...", :magenta
invoke "kamal:cli:build:deliver", [], invoke_options
end
with_lock do
run_hook "pre-deploy"
say "Ensure Traefik is running...", :magenta
invoke "kamal:cli:traefik:boot", [], invoke_options
say "Detect stale containers...", :magenta
invoke "kamal:cli:app:stale_containers", [], invoke_options.merge(stop: true)
invoke "kamal:cli:app:boot", [], invoke_options
say "Prune old containers and images...", :magenta
invoke "kamal:cli:prune:all", [], invoke_options
end
end
run_hook "post-deploy", runtime: runtime.round
end
desc "redeploy", "Deploy app to servers without bootstrapping servers, starting Traefik, pruning, and registry login"
option :skip_push, aliases: "-P", type: :boolean, default: false, desc: "Skip image build and push"
def redeploy
runtime = print_runtime do
invoke_options = deploy_options
if options[:skip_push]
say "Pull app image...", :magenta
invoke "kamal:cli:build:pull", [], invoke_options
else
say "Build and push app image...", :magenta
invoke "kamal:cli:build:deliver", [], invoke_options
end
with_lock do
run_hook "pre-deploy"
say "Detect stale containers...", :magenta
invoke "kamal:cli:app:stale_containers", [], invoke_options.merge(stop: true)
invoke "kamal:cli:app:boot", [], invoke_options
end
end
run_hook "post-deploy", runtime: runtime.round
end
desc "rollback [VERSION]", "Rollback app to VERSION"
def rollback(version)
rolled_back = false
runtime = print_runtime do
with_lock do
invoke_options = deploy_options
KAMAL.config.version = version
old_version = nil
if container_available?(version)
run_hook "pre-deploy"
invoke "kamal:cli:app:boot", [], invoke_options.merge(version: version)
rolled_back = true
else
say "The app version '#{version}' is not available as a container (use 'kamal app containers' for available versions)", :red
end
end
end
run_hook "post-deploy", runtime: runtime.round if rolled_back
end
desc "details", "Show details about all containers"
def details
invoke "kamal:cli:traefik:details"
invoke "kamal:cli:app:details"
invoke "kamal:cli:accessory:details", [ "all" ]
end
desc "audit", "Show audit log from servers"
def audit
on(KAMAL.hosts) do |host|
puts_by_host host, capture_with_info(*KAMAL.auditor.reveal)
end
end
desc "config", "Show combined config (including secrets!)"
def config
run_locally do
puts Kamal::Utils.redacted(KAMAL.config.to_h).to_yaml
end
end
desc "init", "Create config stub in config/deploy.yml and env stub in .env"
option :bundle, type: :boolean, default: false, desc: "Add Kamal to the Gemfile and create a bin/kamal binstub"
def init
require "fileutils"
if (deploy_file = Pathname.new(File.expand_path("config/deploy.yml"))).exist?
puts "Config file already exists in config/deploy.yml (remove first to create a new one)"
else
FileUtils.mkdir_p deploy_file.dirname
FileUtils.cp_r Pathname.new(File.expand_path("templates/deploy.yml", __dir__)), deploy_file
puts "Created configuration file in config/deploy.yml"
end
unless (deploy_file = Pathname.new(File.expand_path(".env"))).exist?
FileUtils.cp_r Pathname.new(File.expand_path("templates/template.env", __dir__)), deploy_file
puts "Created .env file"
end
unless (hooks_dir = Pathname.new(File.expand_path(".kamal/hooks"))).exist?
hooks_dir.mkpath
Pathname.new(File.expand_path("templates/sample_hooks", __dir__)).each_child do |sample_hook|
FileUtils.cp sample_hook, hooks_dir, preserve: true
end
puts "Created sample hooks in .kamal/hooks"
end
if options[:bundle]
if (binstub = Pathname.new(File.expand_path("bin/kamal"))).exist?
puts "Binstub already exists in bin/kamal (remove first to create a new one)"
else
puts "Adding Kamal to Gemfile and bundle..."
run_locally do
execute :bundle, :add, :kamal
execute :bundle, :binstubs, :kamal
end
puts "Created binstub file in bin/kamal"
end
end
end
desc "envify", "Create .env by evaluating .env.erb (or .env.staging.erb -> .env.staging when using -d staging)"
option :skip_push, aliases: "-P", type: :boolean, default: false, desc: "Skip .env file push"
def envify
if destination = options[:destination]
env_template_path = ".env.#{destination}.erb"
env_path = ".env.#{destination}"
else
env_template_path = ".env.erb"
env_path = ".env"
end
if Pathname.new(File.expand_path(env_template_path)).exist?
File.write(env_path, ERB.new(File.read(env_template_path), trim_mode: "-").result, perm: 0600)
unless options[:skip_push]
reload_envs
invoke "kamal:cli:env:push", options
end
else
puts "Skipping envify (no #{env_template_path} exist)"
end
end
desc "remove", "Remove Traefik, app, accessories, and registry session from servers"
option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
def remove
confirming "This will remove all containers and images. Are you sure?" do
with_lock do
invoke "kamal:cli:traefik:remove", [], options.without(:confirmed)
invoke "kamal:cli:app:remove", [], options.without(:confirmed)
invoke "kamal:cli:accessory:remove", [ "all" ], options
invoke "kamal:cli:registry:logout", [], options.without(:confirmed)
end
end
end
desc "version", "Show Kamal version"
def version
puts Kamal::VERSION
end
desc "accessory", "Manage accessories (db/redis/search)"
subcommand "accessory", Kamal::Cli::Accessory
desc "app", "Manage application"
subcommand "app", Kamal::Cli::App
desc "build", "Build application image"
subcommand "build", Kamal::Cli::Build
desc "env", "Manage environment files"
subcommand "env", Kamal::Cli::Env
desc "lock", "Manage the deploy lock"
subcommand "lock", Kamal::Cli::Lock
desc "prune", "Prune old application images and containers"
subcommand "prune", Kamal::Cli::Prune
desc "registry", "Login and -out of the image registry"
subcommand "registry", Kamal::Cli::Registry
desc "server", "Bootstrap servers with curl and Docker"
subcommand "server", Kamal::Cli::Server
desc "traefik", "Manage Traefik load balancer"
subcommand "traefik", Kamal::Cli::Traefik
private
def container_available?(version)
begin
on(KAMAL.hosts) do
KAMAL.roles_on(host).each do |role|
container_id = capture_with_info(*KAMAL.app(role: role, host: host).container_id_for_version(version))
raise "Container not found" unless container_id.present?
end
end
rescue SSHKit::Runner::ExecuteError, SSHKit::Runner::MultipleExecuteError => e
if e.message =~ /Container not found/
say "Error looking for container version #{version}: #{e.message}"
return false
else
raise
end
end
true
end
def deploy_options
{ "version" => KAMAL.config.version }.merge(options.without("skip_push"))
end
end

35
lib/kamal/cli/prune.rb Normal file
View File

@@ -0,0 +1,35 @@
class Kamal::Cli::Prune < Kamal::Cli::Base
desc "all", "Prune unused images and stopped containers"
def all
with_lock do
containers
images
end
end
desc "images", "Prune unused images"
def images
with_lock do
on(KAMAL.hosts) do
execute *KAMAL.auditor.record("Pruned images"), verbosity: :debug
execute *KAMAL.prune.dangling_images
execute *KAMAL.prune.tagged_images
end
end
end
desc "containers", "Prune all stopped containers, except the last n (default 5)"
option :retain, type: :numeric, default: nil, desc: "Number of containers to retain"
def containers
retain = options.fetch(:retain, KAMAL.config.retain_containers)
raise "retain must be at least 1" if retain < 1
with_lock do
on(KAMAL.hosts) do
execute *KAMAL.auditor.record("Pruned containers"), verbosity: :debug
execute *KAMAL.prune.app_containers(retain: retain)
execute *KAMAL.prune.healthcheck_containers
end
end
end
end

View File

@@ -1,8 +1,8 @@
class Mrsk::Cli::Registry < Mrsk::Cli::Base
class Kamal::Cli::Registry < Kamal::Cli::Base
desc "login", "Log in to registry locally and remotely"
def login
run_locally { execute *MRSK.registry.login }
on(MRSK.hosts) { execute *MRSK.registry.login }
run_locally { execute *KAMAL.registry.login }
on(KAMAL.hosts) { execute *KAMAL.registry.login }
# FIXME: This rescue needed?
rescue ArgumentError => e
puts e.message
@@ -10,7 +10,7 @@ class Mrsk::Cli::Registry < Mrsk::Cli::Base
desc "logout", "Log out of registry remotely"
def logout
on(MRSK.hosts) { execute *MRSK.registry.logout }
on(KAMAL.hosts) { execute *KAMAL.registry.logout }
# FIXME: This rescue needed?
rescue ArgumentError => e
puts e.message

49
lib/kamal/cli/server.rb Normal file
View File

@@ -0,0 +1,49 @@
class Kamal::Cli::Server < Kamal::Cli::Base
desc "exec", "Run a custom command on the server (use --help to show options)"
option :interactive, type: :boolean, aliases: "-i", default: false, desc: "Run the command interactively (use for console/bash)"
def exec(cmd)
hosts = KAMAL.hosts | KAMAL.accessory_hosts
case
when options[:interactive]
host = KAMAL.primary_host
say "Running '#{cmd}' on #{host} interactively...", :magenta
run_locally { exec KAMAL.server.run_over_ssh(cmd, host: host) }
else
say "Running '#{cmd}' on #{hosts.join(', ')}...", :magenta
on(hosts) do |host|
execute *KAMAL.auditor.record("Executed cmd '#{cmd}' on #{host}"), verbosity: :debug
puts_by_host host, capture_with_info(cmd)
end
end
end
desc "bootstrap", "Set up Docker to run Kamal apps"
def bootstrap
with_lock do
missing = []
on(KAMAL.hosts | KAMAL.accessory_hosts) do |host|
unless execute(*KAMAL.docker.installed?, raise_on_non_zero_exit: false)
if execute(*KAMAL.docker.superuser?, raise_on_non_zero_exit: false)
info "Missing Docker on #{host}. Installing…"
execute *KAMAL.docker.install
else
missing << host
end
end
execute(*KAMAL.server.ensure_run_directory)
end
if missing.any?
raise "Docker is not installed on #{missing.join(", ")} and can't be automatically installed without having root access and either `wget` or `curl`. Install Docker manually: https://docs.docker.com/engine/install/"
end
run_hook "docker-setup"
end
end
end

View File

@@ -13,20 +13,19 @@ registry:
# Specify the registry server, if you're not using Docker Hub
# server: registry.digitalocean.com / ghcr.io / ...
username: my-user
# Always use an access token rather than real password when possible.
password:
- MRSK_REGISTRY_PASSWORD
- KAMAL_REGISTRY_PASSWORD
# Inject ENV variables into containers (secrets come from .env).
# Remember to run `kamal env push` after making changes!
# env:
# clear:
# DB_HOST: 192.168.0.2
# secret:
# - RAILS_MASTER_KEY
# Call a broadcast command on deploys.
# audit_broadcast_cmd:
# bin/broadcast_to_bc
# Use a different ssh user than root
# ssh:
# user: app
@@ -54,7 +53,7 @@ registry:
# - MYSQL_ROOT_PASSWORD
# files:
# - config/mysql/production.cnf:/etc/mysql/my.cnf
# - db/production.sql.erb:/docker-entrypoint-initdb.d/setup.sql
# - db/production.sql:/docker-entrypoint-initdb.d/setup.sql
# directories:
# - data:/var/lib/mysql
# redis:
@@ -64,7 +63,7 @@ registry:
# directories:
# - data:/data
# Configure custom arguments for Traefik
# Configure custom arguments for Traefik. Be sure to reboot traefik when you modify it.
# traefik:
# args:
# accesslog: true
@@ -74,3 +73,29 @@ registry:
# healthcheck:
# path: /healthz
# port: 4000
# Bridge fingerprinted assets, like JS and CSS, between versions to avoid
# hitting 404 on in-flight requests. Combines all files from new and old
# version inside the asset_path.
#
# If your app is using the Sprockets gem, ensure it sets `config.assets.manifest`.
# See https://github.com/basecamp/kamal/issues/626 for details
#
# asset_path: /rails/public/assets
# Configure rolling deploys by setting a wait time between batches of restarts.
# boot:
# limit: 10 # Can also specify as a percentage of total hosts, such as "25%"
# wait: 2
# Configure the role used to determine the primary_host. This host takes
# deploy locks, runs health checks during the deploy, and follow logs, etc.
#
# Caution: there's no support for role renaming yet, so be careful to cleanup
# the previous role on the deployed hosts.
# primary_role: web
# Controls if we abort when see a role with no hosts. Disabling this may be
# useful for more complex deploy configurations.
#
# allow_empty_roles: false

View File

@@ -0,0 +1,7 @@
#!/usr/bin/env ruby
# A sample docker-setup hook
#
# Sets up a Docker network which can then be used by the applications containers
ssh user@example.com docker network create kamal

View File

@@ -0,0 +1,14 @@
#!/bin/sh
# A sample post-deploy hook
#
# These environment variables are available:
# KAMAL_RECORDED_AT
# KAMAL_PERFORMER
# KAMAL_VERSION
# KAMAL_HOSTS
# KAMAL_ROLE (if set)
# KAMAL_DESTINATION (if set)
# KAMAL_RUNTIME
echo "$KAMAL_PERFORMER deployed $KAMAL_VERSION to $KAMAL_DESTINATION in $KAMAL_RUNTIME seconds"

View File

@@ -0,0 +1,3 @@
#!/bin/sh
echo "Rebooted Traefik on $KAMAL_HOSTS"

View File

@@ -0,0 +1,51 @@
#!/bin/sh
# A sample pre-build hook
#
# Checks:
# 1. We have a clean checkout
# 2. A remote is configured
# 3. The branch has been pushed to the remote
# 4. The version we are deploying matches the remote
#
# These environment variables are available:
# KAMAL_RECORDED_AT
# KAMAL_PERFORMER
# KAMAL_VERSION
# KAMAL_HOSTS
# KAMAL_ROLE (if set)
# KAMAL_DESTINATION (if set)
if [ -n "$(git status --porcelain)" ]; then
echo "Git checkout is not clean, aborting..." >&2
git status --porcelain >&2
exit 1
fi
first_remote=$(git remote)
if [ -z "$first_remote" ]; then
echo "No git remote set, aborting..." >&2
exit 1
fi
current_branch=$(git branch --show-current)
if [ -z "$current_branch" ]; then
echo "Not on a git branch, aborting..." >&2
exit 1
fi
remote_head=$(git ls-remote $first_remote --tags $current_branch | cut -f1)
if [ -z "$remote_head" ]; then
echo "Branch not pushed to remote, aborting..." >&2
exit 1
fi
if [ "$KAMAL_VERSION" != "$remote_head" ]; then
echo "Version ($KAMAL_VERSION) does not match remote HEAD ($remote_head), aborting..." >&2
exit 1
fi
exit 0

View File

@@ -0,0 +1,47 @@
#!/usr/bin/env ruby
# A sample pre-connect check
#
# Warms DNS before connecting to hosts in parallel
#
# These environment variables are available:
# KAMAL_RECORDED_AT
# KAMAL_PERFORMER
# KAMAL_VERSION
# KAMAL_HOSTS
# KAMAL_ROLE (if set)
# KAMAL_DESTINATION (if set)
# KAMAL_RUNTIME
hosts = ENV["KAMAL_HOSTS"].split(",")
results = nil
max = 3
elapsed = Benchmark.realtime do
results = hosts.map do |host|
Thread.new do
tries = 1
begin
Socket.getaddrinfo(host, 0, Socket::AF_UNSPEC, Socket::SOCK_STREAM, nil, Socket::AI_CANONNAME)
rescue SocketError
if tries < max
puts "Retrying DNS warmup: #{host}"
tries += 1
sleep rand
retry
else
puts "DNS warmup failed: #{host}"
host
end
end
tries
end
end.map(&:value)
end
retries = results.sum - hosts.size
nopes = results.count { |r| r == max }
puts "Prewarmed %d DNS lookups in %.2f sec: %d retries, %d failures" % [ hosts.size, elapsed, retries, nopes ]

View File

@@ -0,0 +1,109 @@
#!/usr/bin/env ruby
# A sample pre-deploy hook
#
# Checks the Github status of the build, waiting for a pending build to complete for up to 720 seconds.
#
# Fails unless the combined status is "success"
#
# These environment variables are available:
# KAMAL_RECORDED_AT
# KAMAL_PERFORMER
# KAMAL_VERSION
# KAMAL_HOSTS
# KAMAL_COMMAND
# KAMAL_SUBCOMMAND
# KAMAL_ROLE (if set)
# KAMAL_DESTINATION (if set)
# Only check the build status for production deployments
if ENV["KAMAL_COMMAND"] == "rollback" || ENV["KAMAL_DESTINATION"] != "production"
exit 0
end
require "bundler/inline"
# true = install gems so this is fast on repeat invocations
gemfile(true, quiet: true) do
source "https://rubygems.org"
gem "octokit"
gem "faraday-retry"
end
MAX_ATTEMPTS = 72
ATTEMPTS_GAP = 10
def exit_with_error(message)
$stderr.puts message
exit 1
end
class GithubStatusChecks
attr_reader :remote_url, :git_sha, :github_client, :combined_status
def initialize
@remote_url = `git config --get remote.origin.url`.strip.delete_prefix("https://github.com/")
@git_sha = `git rev-parse HEAD`.strip
@github_client = Octokit::Client.new(access_token: ENV["GITHUB_TOKEN"])
refresh!
end
def refresh!
@combined_status = github_client.combined_status(remote_url, git_sha)
end
def state
combined_status[:state]
end
def first_status_url
first_status = combined_status[:statuses].find { |status| status[:state] == state }
first_status && first_status[:target_url]
end
def complete_count
combined_status[:statuses].count { |status| status[:state] != "pending"}
end
def total_count
combined_status[:statuses].count
end
def current_status
if total_count > 0
"Completed #{complete_count}/#{total_count} checks, see #{first_status_url} ..."
else
"Build not started..."
end
end
end
$stdout.sync = true
puts "Checking build status..."
attempts = 0
checks = GithubStatusChecks.new
begin
loop do
case checks.state
when "success"
puts "Checks passed, see #{checks.first_status_url}"
exit 0
when "failure"
exit_with_error "Checks failed, see #{checks.first_status_url}"
when "pending"
attempts += 1
end
exit_with_error "Checks are still pending, gave up after #{MAX_ATTEMPTS * ATTEMPTS_GAP} seconds" if attempts == MAX_ATTEMPTS
puts checks.current_status
sleep(ATTEMPTS_GAP)
checks.refresh!
end
rescue Octokit::NotFound
exit_with_error "Build status could not be found"
end

View File

@@ -0,0 +1,3 @@
#!/bin/sh
echo "Rebooting Traefik on $KAMAL_HOSTS..."

View File

@@ -0,0 +1,2 @@
KAMAL_REGISTRY_PASSWORD=change-this
RAILS_MASTER_KEY=another-env

120
lib/kamal/cli/traefik.rb Normal file
View File

@@ -0,0 +1,120 @@
class Kamal::Cli::Traefik < Kamal::Cli::Base
desc "boot", "Boot Traefik on servers"
def boot
with_lock do
on(KAMAL.traefik_hosts) do
execute *KAMAL.registry.login
execute *KAMAL.traefik.start_or_run
end
end
end
desc "reboot", "Reboot Traefik on servers (stop container, remove container, start new container)"
option :rolling, type: :boolean, default: false, desc: "Reboot traefik on hosts in sequence, rather than in parallel"
option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
def reboot
confirming "This will cause a brief outage on each host. Are you sure?" do
with_lock do
host_groups = options[:rolling] ? KAMAL.traefik_hosts : [ KAMAL.traefik_hosts ]
host_groups.each do |hosts|
host_list = Array(hosts).join(",")
run_hook "pre-traefik-reboot", hosts: host_list
on(hosts) do
execute *KAMAL.auditor.record("Rebooted traefik"), verbosity: :debug
execute *KAMAL.registry.login
execute *KAMAL.traefik.stop, raise_on_non_zero_exit: false
execute *KAMAL.traefik.remove_container
execute *KAMAL.traefik.run
end
run_hook "post-traefik-reboot", hosts: host_list
end
end
end
end
desc "start", "Start existing Traefik container on servers"
def start
with_lock do
on(KAMAL.traefik_hosts) do
execute *KAMAL.auditor.record("Started traefik"), verbosity: :debug
execute *KAMAL.traefik.start
end
end
end
desc "stop", "Stop existing Traefik container on servers"
def stop
with_lock do
on(KAMAL.traefik_hosts) do
execute *KAMAL.auditor.record("Stopped traefik"), verbosity: :debug
execute *KAMAL.traefik.stop, raise_on_non_zero_exit: false
end
end
end
desc "restart", "Restart existing Traefik container on servers"
def restart
with_lock do
stop
start
end
end
desc "details", "Show details about Traefik container from servers"
def details
on(KAMAL.traefik_hosts) { |host| puts_by_host host, capture_with_info(*KAMAL.traefik.info), type: "Traefik" }
end
desc "logs", "Show log lines from Traefik on servers"
option :since, aliases: "-s", desc: "Show logs since timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes)"
option :lines, type: :numeric, aliases: "-n", desc: "Number of log lines to pull from each server"
option :grep, aliases: "-g", desc: "Show lines with grep match only (use this to fetch specific requests by id)"
option :follow, aliases: "-f", desc: "Follow logs on primary server (or specific host set by --hosts)"
def logs
grep = options[:grep]
if options[:follow]
run_locally do
info "Following logs on #{KAMAL.primary_host}..."
info KAMAL.traefik.follow_logs(host: KAMAL.primary_host, grep: grep)
exec KAMAL.traefik.follow_logs(host: KAMAL.primary_host, grep: grep)
end
else
since = options[:since]
lines = options[:lines].presence || ((since || grep) ? nil : 100) # Default to 100 lines if since or grep isn't set
on(KAMAL.traefik_hosts) do |host|
puts_by_host host, capture(*KAMAL.traefik.logs(since: since, lines: lines, grep: grep)), type: "Traefik"
end
end
end
desc "remove", "Remove Traefik container and image from servers"
def remove
with_lock do
stop
remove_container
remove_image
end
end
desc "remove_container", "Remove Traefik container from servers", hide: true
def remove_container
with_lock do
on(KAMAL.traefik_hosts) do
execute *KAMAL.auditor.record("Removed traefik container"), verbosity: :debug
execute *KAMAL.traefik.remove_container
end
end
end
desc "remove_image", "Remove Traefik image from servers", hide: true
def remove_image
with_lock do
on(KAMAL.traefik_hosts) do
execute *KAMAL.auditor.record("Removed traefik image"), verbosity: :debug
execute *KAMAL.traefik.remove_image
end
end
end
end

160
lib/kamal/commander.rb Normal file
View File

@@ -0,0 +1,160 @@
require "active_support/core_ext/enumerable"
require "active_support/core_ext/module/delegation"
class Kamal::Commander
attr_accessor :verbosity, :holding_lock, :connected
delegate :hosts, :roles, :primary_host, :primary_role, :roles_on, :traefik_hosts, :accessory_hosts, to: :specifics
def initialize
self.verbosity = :info
self.holding_lock = false
self.connected = false
@specifics = nil
end
def config
@config ||= Kamal::Configuration.create_from(**@config_kwargs).tap do |config|
@config_kwargs = nil
configure_sshkit_with(config)
end
end
def configure(**kwargs)
@config, @config_kwargs = nil, kwargs
end
attr_reader :specific_roles, :specific_hosts
def specific_primary!
@specifics = nil
self.specific_hosts = [ config.primary_host ]
end
def specific_roles=(role_names)
@specifics = nil
if role_names.present?
@specific_roles = Kamal::Utils.filter_specific_items(role_names, config.roles)
if @specific_roles.empty?
raise ArgumentError, "No --roles match for #{role_names.join(',')}"
end
@specific_roles
end
end
def specific_hosts=(hosts)
@specifics = nil
if hosts.present?
@specific_hosts = Kamal::Utils.filter_specific_items(hosts, config.all_hosts)
if @specific_hosts.empty?
raise ArgumentError, "No --hosts match for #{hosts.join(',')}"
end
@specific_hosts
end
end
def accessory_names
config.accessories&.collect(&:name) || []
end
def accessories_on(host)
config.accessories.select { |accessory| accessory.hosts.include?(host.to_s) }.map(&:name)
end
def app(role: nil, host: nil)
Kamal::Commands::App.new(config, role: role, host: host)
end
def accessory(name)
Kamal::Commands::Accessory.new(config, name: name)
end
def auditor(**details)
Kamal::Commands::Auditor.new(config, **details)
end
def builder
@builder ||= Kamal::Commands::Builder.new(config)
end
def docker
@docker ||= Kamal::Commands::Docker.new(config)
end
def healthcheck
@healthcheck ||= Kamal::Commands::Healthcheck.new(config)
end
def hook
@hook ||= Kamal::Commands::Hook.new(config)
end
def lock
@lock ||= Kamal::Commands::Lock.new(config)
end
def prune
@prune ||= Kamal::Commands::Prune.new(config)
end
def registry
@registry ||= Kamal::Commands::Registry.new(config)
end
def server
@server ||= Kamal::Commands::Server.new(config)
end
def traefik
@traefik ||= Kamal::Commands::Traefik.new(config)
end
def with_verbosity(level)
old_level = self.verbosity
self.verbosity = level
SSHKit.config.output_verbosity = level
yield
ensure
self.verbosity = old_level
SSHKit.config.output_verbosity = old_level
end
def boot_strategy
if config.boot.limit.present?
{ in: :groups, limit: config.boot.limit, wait: config.boot.wait }
else
{}
end
end
def holding_lock?
self.holding_lock
end
def connected?
self.connected
end
private
# Lazy setup of SSHKit
def configure_sshkit_with(config)
SSHKit::Backend::Netssh.pool.idle_timeout = config.sshkit.pool_idle_timeout
SSHKit::Backend::Netssh.configure do |sshkit|
sshkit.max_concurrent_starts = config.sshkit.max_concurrent_starts
sshkit.ssh_options = config.ssh.options
end
SSHKit.config.command_map[:docker] = "docker" # No need to use /usr/bin/env, just clogs up the logs
SSHKit.config.output_verbosity = verbosity
end
def specifics
@specifics ||= Kamal::Commander::Specifics.new(config, specific_hosts, specific_roles)
end
end

View File

@@ -0,0 +1,49 @@
class Kamal::Commander::Specifics
attr_reader :primary_host, :primary_role, :hosts, :roles
delegate :stable_sort!, to: Kamal::Utils
def initialize(config, specific_hosts, specific_roles)
@config, @specific_hosts, @specific_roles = config, specific_hosts, specific_roles
@roles, @hosts = specified_roles, specified_hosts
@primary_host = specific_hosts&.first || primary_specific_role&.primary_host || config.primary_host
@primary_role = primary_or_first_role(roles_on(primary_host))
stable_sort!(roles) { |role| role == primary_role ? 0 : 1 }
stable_sort!(hosts) { |host| roles_on(host).any? { |role| role == primary_role } ? 0 : 1 }
end
def roles_on(host)
roles.select { |role| role.hosts.include?(host.to_s) }
end
def traefik_hosts
config.traefik_hosts & specified_hosts
end
def accessory_hosts
specific_hosts || config.accessories.flat_map(&:hosts)
end
private
attr_reader :config, :specific_hosts, :specific_roles
def primary_specific_role
primary_or_first_role(specific_roles) if specific_roles.present?
end
def primary_or_first_role(roles)
roles.detect { |role| role == config.primary_role } || roles.first
end
def specified_roles
(specific_roles || config.roles) \
.select { |role| ((specific_hosts || config.all_hosts) & role.hosts).any? }
end
def specified_hosts
(specific_hosts || config.all_hosts) \
.select { |host| (specific_roles || config.roles).flat_map(&:hosts).include?(host) }
end
end

2
lib/kamal/commands.rb Normal file
View File

@@ -0,0 +1,2 @@
module Kamal::Commands
end

View File

@@ -1,6 +1,7 @@
class Mrsk::Commands::Accessory < Mrsk::Commands::Base
class Kamal::Commands::Accessory < Kamal::Commands::Base
attr_reader :accessory_config
delegate :service_name, :image, :host, :port, :files, :directories, :env_args, :volume_args, :label_args, to: :accessory_config
delegate :service_name, :image, :hosts, :port, :files, :directories, :cmd,
:publish_args, :env_args, :volume_args, :label_args, :option_args, to: :accessory_config
def initialize(config, name:)
super(config)
@@ -12,12 +13,14 @@ class Mrsk::Commands::Accessory < Mrsk::Commands::Base
"--name", service_name,
"--detach",
"--restart", "unless-stopped",
"--log-opt", "max-size=#{MAX_LOG_SIZE}",
"--publish", port,
*config.logging_args,
*publish_args,
*env_args,
*volume_args,
*label_args,
image
*option_args,
image,
cmd
end
def start
@@ -73,7 +76,7 @@ class Mrsk::Commands::Accessory < Mrsk::Commands::Base
end
def run_over_ssh(command)
super command, host: host
super command, host: hosts.first
end
@@ -83,14 +86,6 @@ class Mrsk::Commands::Accessory < Mrsk::Commands::Base
end
end
def make_directory_for(remote_file)
make_directory Pathname.new(remote_file).dirname.to_s
end
def make_directory(path)
[ :mkdir, "-p", path ]
end
def remove_service_directory
[ :rm, "-rf", service_name ]
end
@@ -100,7 +95,15 @@ class Mrsk::Commands::Accessory < Mrsk::Commands::Base
end
def remove_image
docker :image, :prune, "--all", "--force", *service_filter
docker :image, :rm, "--force", image
end
def make_env_directory
make_directory accessory_config.env.secrets_directory
end
def remove_env_file
[ :rm, "-f", accessory_config.env.secrets_file ]
end
private

123
lib/kamal/commands/app.rb Normal file
View File

@@ -0,0 +1,123 @@
class Kamal::Commands::App < Kamal::Commands::Base
include Assets, Containers, Cord, Execution, Images, Logging
ACTIVE_DOCKER_STATUSES = [ :running, :restarting ]
attr_reader :role, :host
def initialize(config, role: nil, host: nil)
super(config)
@role = role
@host = host
end
def run(hostname: nil)
docker :run,
"--detach",
"--restart unless-stopped",
"--name", container_name,
*([ "--hostname", hostname ] if hostname),
"-e", "KAMAL_CONTAINER_NAME=\"#{container_name}\"",
"-e", "KAMAL_VERSION=\"#{config.version}\"",
*role.env_args(host),
*role.health_check_args,
*role.logging_args,
*config.volume_args,
*role.asset_volume_args,
*role.label_args,
*role.option_args,
config.absolute_image,
role.cmd
end
def start
docker :start, container_name
end
def status(version:)
pipe container_id_for_version(version), xargs(docker(:inspect, "--format", DOCKER_HEALTH_STATUS_FORMAT))
end
def stop(version: nil)
pipe \
version ? container_id_for_version(version) : current_running_container_id,
xargs(config.stop_wait_time ? docker(:stop, "-t", config.stop_wait_time) : docker(:stop))
end
def info
docker :ps, *filter_args
end
def current_running_container_id
current_running_container(format: "--quiet")
end
def container_id_for_version(version, only_running: false)
container_id_for(container_name: container_name(version), only_running: only_running)
end
def current_running_version
pipe \
current_running_container(format: "--format '{{.Names}}'"),
extract_version_from_name
end
def list_versions(*docker_args, statuses: nil)
pipe \
docker(:ps, *filter_args(statuses: statuses), *docker_args, "--format", '"{{.Names}}"'),
extract_version_from_name
end
def make_env_directory
make_directory role.env(host).secrets_directory
end
def remove_env_file
[ :rm, "-f", role.env(host).secrets_file ]
end
private
def container_name(version = nil)
[ role.container_prefix, version || config.version ].compact.join("-")
end
def latest_image_id
docker :image, :ls, *argumentize("--filter", "reference=#{config.latest_image}"), "--format", "'{{.ID}}'"
end
def current_running_container(format:)
pipe \
shell(chain(latest_image_container(format: format), latest_container(format: format))),
[ :head, "-1" ]
end
def latest_image_container(format:)
latest_container format: format, filters: [ "ancestor=$(#{latest_image_id.join(" ")})" ]
end
def latest_container(format:, filters: nil)
docker :ps, "--latest", *format, *filter_args(statuses: ACTIVE_DOCKER_STATUSES), argumentize("--filter", filters)
end
def filter_args(statuses: nil)
argumentize "--filter", filters(statuses: statuses)
end
def extract_version_from_name
# Extract SHA from "service-role-dest-SHA"
%(while read line; do echo ${line##{role.container_prefix}-}; done)
end
def filters(statuses: nil)
[ "label=service=#{config.service}" ].tap do |filters|
filters << "label=destination=#{config.destination}" if config.destination
filters << "label=role=#{role}" if role
statuses&.each do |status|
filters << "status=#{status}"
end
end
end
end

View File

@@ -0,0 +1,51 @@
module Kamal::Commands::App::Assets
def extract_assets
asset_container = "#{role.container_prefix}-assets"
combine \
make_directory(role.asset_extracted_path),
[ *docker(:stop, "-t 1", asset_container, "2> /dev/null"), "|| true" ],
docker(:run, "--name", asset_container, "--detach", "--rm", config.absolute_image, "sleep 1000000"),
docker(:cp, "-L", "#{asset_container}:#{role.asset_path}/.", role.asset_extracted_path),
docker(:stop, "-t 1", asset_container),
by: "&&"
end
def sync_asset_volumes(old_version: nil)
new_extracted_path, new_volume_path = role.asset_extracted_path(config.version), role.asset_volume.host_path
if old_version.present?
old_extracted_path, old_volume_path = role.asset_extracted_path(old_version), role.asset_volume(old_version).host_path
end
commands = [ make_directory(new_volume_path), copy_contents(new_extracted_path, new_volume_path) ]
if old_version.present?
commands << copy_contents(new_extracted_path, old_volume_path, continue_on_error: true)
commands << copy_contents(old_extracted_path, new_volume_path, continue_on_error: true)
end
chain *commands
end
def clean_up_assets
chain \
find_and_remove_older_siblings(role.asset_extracted_path),
find_and_remove_older_siblings(role.asset_volume_path)
end
private
def find_and_remove_older_siblings(path)
[
:find,
Pathname.new(path).dirname.to_s,
"-maxdepth 1",
"-name", "'#{role.container_prefix}-*'",
"!", "-name", Pathname.new(path).basename.to_s,
"-exec rm -rf \"{}\" +"
]
end
def copy_contents(source, destination, continue_on_error: false)
[ :cp, "-rnT", "#{source}", destination, *("|| true" if continue_on_error) ]
end
end

View File

@@ -0,0 +1,31 @@
module Kamal::Commands::App::Containers
DOCKER_HEALTH_LOG_FORMAT = "'{{json .State.Health}}'"
def list_containers
docker :container, :ls, "--all", *filter_args
end
def list_container_names
[ *list_containers, "--format", "'{{ .Names }}'" ]
end
def remove_container(version:)
pipe \
container_id_for(container_name: container_name(version)),
xargs(docker(:container, :rm))
end
def rename_container(version:, new_version:)
docker :rename, container_name(version), container_name(new_version)
end
def remove_containers
docker :container, :prune, "--force", *filter_args
end
def container_health_log(version:)
pipe \
container_id_for(container_name: container_name(version)),
xargs(docker(:inspect, "--format", DOCKER_HEALTH_LOG_FORMAT))
end
end

View File

@@ -0,0 +1,22 @@
module Kamal::Commands::App::Cord
def cord(version:)
pipe \
docker(:inspect, "-f '{{ range .Mounts }}{{printf \"%s %s\\n\" .Source .Destination}}{{ end }}'", container_name(version)),
[ :awk, "'$2 == \"#{role.cord_volume.container_path}\" {print $1}'" ]
end
def tie_cord(cord)
create_empty_file(cord)
end
def cut_cord(cord)
remove_directory(cord)
end
private
def create_empty_file(file)
chain \
make_directory_for(file),
[ :touch, file ]
end
end

View File

@@ -0,0 +1,29 @@
module Kamal::Commands::App::Execution
def execute_in_existing_container(*command, interactive: false, env:)
docker :exec,
("-it" if interactive),
*argumentize("--env", env),
container_name,
*command
end
def execute_in_new_container(*command, interactive: false, env:)
docker :run,
("-it" if interactive),
"--rm",
*role&.env_args(host),
*argumentize("--env", env),
*config.volume_args,
*role&.option_args,
config.absolute_image,
*command
end
def execute_in_existing_container_over_ssh(*command, env:)
run_over_ssh execute_in_existing_container(*command, interactive: true, env: env), host: host
end
def execute_in_new_container_over_ssh(*command, env:)
run_over_ssh execute_in_new_container(*command, interactive: true, env: env), host: host
end
end

View File

@@ -0,0 +1,13 @@
module Kamal::Commands::App::Images
def list_images
docker :image, :ls, config.repository
end
def remove_images
docker :image, :prune, "--all", "--force", *filter_args
end
def tag_latest_image
docker :tag, config.absolute_image, config.latest_image
end
end

View File

@@ -0,0 +1,18 @@
module Kamal::Commands::App::Logging
def logs(version: nil, since: nil, lines: nil, grep: nil)
pipe \
version ? container_id_for_version(version) : current_running_container_id,
"xargs docker logs#{" --since #{since}" if since}#{" --tail #{lines}" if lines} 2>&1",
("grep '#{grep}'" if grep)
end
def follow_logs(host:, lines: nil, grep: nil)
run_over_ssh \
pipe(
current_running_container_id,
"xargs docker logs --timestamps#{" --tail #{lines}" if lines} --follow 2>&1",
(%(grep "#{grep}") if grep)
),
host: host
end
end

View File

@@ -0,0 +1,30 @@
class Kamal::Commands::Auditor < Kamal::Commands::Base
attr_reader :details
def initialize(config, **details)
super(config)
@details = details
end
# Runs remotely
def record(line, **details)
append \
[ :echo, audit_tags(**details).except(:version, :service_version).to_s, line ],
audit_log_file
end
def reveal
[ :tail, "-n", 50, audit_log_file ]
end
private
def audit_log_file
file = [ config.service, config.destination, "audit.log" ].compact.join("-")
File.join(config.run_directory, file)
end
def audit_tags(**details)
tags(**self.details, **details)
end
end

View File

@@ -0,0 +1,88 @@
module Kamal::Commands
class Base
delegate :sensitive, :argumentize, to: Kamal::Utils
DOCKER_HEALTH_STATUS_FORMAT = "'{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'"
attr_accessor :config
def initialize(config)
@config = config
end
def run_over_ssh(*command, host:)
"ssh".tap do |cmd|
if config.ssh.proxy && config.ssh.proxy.is_a?(Net::SSH::Proxy::Jump)
cmd << " -J #{config.ssh.proxy.jump_proxies}"
elsif config.ssh.proxy && config.ssh.proxy.is_a?(Net::SSH::Proxy::Command)
cmd << " -o ProxyCommand='#{config.ssh.proxy.command_line_template}'"
end
cmd << " -t #{config.ssh.user}@#{host} -p #{config.ssh.port} '#{command.join(" ").gsub("'", "'\\\\''")}'"
end
end
def container_id_for(container_name:, only_running: false)
docker :container, :ls, *("--all" unless only_running), "--filter", "name=^#{container_name}$", "--quiet"
end
def make_directory_for(remote_file)
make_directory Pathname.new(remote_file).dirname.to_s
end
def make_directory(path)
[ :mkdir, "-p", path ]
end
def remove_directory(path)
[ :rm, "-r", path ]
end
private
def combine(*commands, by: "&&")
commands
.compact
.collect { |command| Array(command) + [ by ] }.flatten # Join commands
.tap { |commands| commands.pop } # Remove trailing combiner
end
def chain(*commands)
combine *commands, by: ";"
end
def pipe(*commands)
combine *commands, by: "|"
end
def append(*commands)
combine *commands, by: ">>"
end
def write(*commands)
combine *commands, by: ">"
end
def any(*commands)
combine *commands, by: "||"
end
def xargs(command)
[ :xargs, command ].flatten
end
def shell(command)
[ :sh, "-c", "'#{command.flatten.join(" ").gsub("'", "'\\\\''")}'" ]
end
def docker(*args)
args.compact.unshift :docker
end
def git(*args, path: nil)
[ :git, *([ "-C", path ] if path), *args.compact ]
end
def tags(**details)
Kamal::Tags.from_config(config, **details)
end
end
end

View File

@@ -0,0 +1,66 @@
require "active_support/core_ext/string/filters"
class Kamal::Commands::Builder < Kamal::Commands::Base
delegate :create, :remove, :push, :clean, :pull, :info, :validate_image, to: :target
include Clone
def name
target.class.to_s.remove("Kamal::Commands::Builder::").underscore.inquiry
end
def target
case
when !config.builder.multiarch? && !config.builder.cached?
native
when !config.builder.multiarch? && config.builder.cached?
native_cached
when config.builder.local? && config.builder.remote?
multiarch_remote
when config.builder.remote?
native_remote
else
multiarch
end
end
def native
@native ||= Kamal::Commands::Builder::Native.new(config)
end
def native_cached
@native ||= Kamal::Commands::Builder::Native::Cached.new(config)
end
def native_remote
@native ||= Kamal::Commands::Builder::Native::Remote.new(config)
end
def multiarch
@multiarch ||= Kamal::Commands::Builder::Multiarch.new(config)
end
def multiarch_remote
@multiarch_remote ||= Kamal::Commands::Builder::Multiarch::Remote.new(config)
end
def ensure_local_dependencies_installed
if name.native?
ensure_local_docker_installed
else
combine \
ensure_local_docker_installed,
ensure_local_buildx_installed
end
end
private
def ensure_local_docker_installed
docker "--version"
end
def ensure_local_buildx_installed
docker :buildx, "version"
end
end

View File

@@ -0,0 +1,77 @@
class Kamal::Commands::Builder::Base < Kamal::Commands::Base
class BuilderError < StandardError; end
delegate :argumentize, to: Kamal::Utils
delegate :args, :secrets, :dockerfile, :target, :local_arch, :local_host, :remote_arch, :remote_host, :cache_from, :cache_to, :ssh, to: :builder_config
def clean
docker :image, :rm, "--force", config.absolute_image
end
def pull
docker :pull, config.absolute_image
end
def build_options
[ *build_tags, *build_cache, *build_labels, *build_args, *build_secrets, *build_dockerfile, *build_target, *build_ssh ]
end
def build_context
config.builder.context
end
def validate_image
pipe \
docker(:inspect, "-f", "'{{ .Config.Labels.service }}'", config.absolute_image),
any(
[ :grep, "-x", config.service ],
"(echo \"Image #{config.absolute_image} is missing the 'service' label\" && exit 1)"
)
end
private
def build_tags
[ "-t", config.absolute_image, "-t", config.latest_image ]
end
def build_cache
if cache_to && cache_from
[ "--cache-to", cache_to,
"--cache-from", cache_from ]
end
end
def build_labels
argumentize "--label", { service: config.service }
end
def build_args
argumentize "--build-arg", args, sensitive: true
end
def build_secrets
argumentize "--secret", secrets.collect { |secret| [ "id", secret ] }
end
def build_dockerfile
if Pathname.new(File.expand_path(dockerfile)).exist?
argumentize "--file", dockerfile
else
raise BuilderError, "Missing #{dockerfile}"
end
end
def build_target
argumentize "--target", target if target.present?
end
def build_ssh
argumentize "--ssh", ssh if ssh.present?
end
def builder_config
config.builder
end
end

View File

@@ -0,0 +1,28 @@
module Kamal::Commands::Builder::Clone
extend ActiveSupport::Concern
included do
delegate :clone_directory, :build_directory, to: :"config.builder"
end
def clone
git :clone, Kamal::Git.root, path: clone_directory
end
def clone_reset_steps
[
git(:remote, "set-url", :origin, Kamal::Git.root, path: build_directory),
git(:fetch, :origin, path: build_directory),
git(:reset, "--hard", Kamal::Git.revision, path: build_directory),
git(:clean, "-fdx", path: build_directory)
]
end
def clone_status
git :status, "--porcelain", path: build_directory
end
def clone_revision
git :"rev-parse", :HEAD, path: build_directory
end
end

View File

@@ -1,4 +1,4 @@
class Mrsk::Commands::Builder::Multiarch < Mrsk::Commands::Builder::Base
class Kamal::Commands::Builder::Multiarch < Kamal::Commands::Builder::Base
def create
docker :buildx, :create, "--use", "--name", builder_name
end
@@ -7,23 +7,31 @@ class Mrsk::Commands::Builder::Multiarch < Mrsk::Commands::Builder::Base
docker :buildx, :rm, builder_name
end
def push
docker :buildx, :build,
"--push",
"--platform", "linux/amd64,linux/arm64",
"--builder", builder_name,
*build_options,
build_context
end
def info
combine \
docker(:context, :ls),
docker(:buildx, :ls)
end
def push
docker :buildx, :build,
"--push",
"--platform", platform_names,
"--builder", builder_name,
*build_options,
build_context
end
private
def builder_name
"mrsk-#{config.service}-multiarch"
"kamal-#{config.service}-multiarch"
end
def platform_names
if local_arch
"linux/#{local_arch}"
else
"linux/amd64,linux/arm64"
end
end
end

View File

@@ -1,4 +1,4 @@
class Mrsk::Commands::Builder::Multiarch::Remote < Mrsk::Commands::Builder::Multiarch
class Kamal::Commands::Builder::Multiarch::Remote < Kamal::Commands::Builder::Multiarch
def create
combine \
create_contexts,
@@ -22,17 +22,17 @@ class Mrsk::Commands::Builder::Multiarch::Remote < Mrsk::Commands::Builder::Mult
end
def create_local_buildx
docker :buildx, :create, "--name", builder_name, builder_name_with_arch(local["arch"]), "--platform", "linux/#{local["arch"]}"
docker :buildx, :create, "--name", builder_name, builder_name_with_arch(local_arch), "--platform", "linux/#{local_arch}"
end
def append_remote_buildx
docker :buildx, :create, "--append", "--name", builder_name, builder_name_with_arch(remote["arch"]), "--platform", "linux/#{remote["arch"]}"
docker :buildx, :create, "--append", "--name", builder_name, builder_name_with_arch(remote_arch), "--platform", "linux/#{remote_arch}"
end
def create_contexts
combine \
create_context(local["arch"], local["host"]),
create_context(remote["arch"], remote["host"])
create_context(local_arch, local_host),
create_context(remote_arch, remote_host)
end
def create_context(arch, host)
@@ -41,19 +41,11 @@ class Mrsk::Commands::Builder::Multiarch::Remote < Mrsk::Commands::Builder::Mult
def remove_contexts
combine \
remove_context(local["arch"]),
remove_context(remote["arch"])
remove_context(local_arch),
remove_context(remote_arch)
end
def remove_context(arch)
docker :context, :rm, builder_name_with_arch(arch)
end
def local
config.builder["local"]
end
def remote
config.builder["remote"]
end
end

View File

@@ -0,0 +1,20 @@
class Kamal::Commands::Builder::Native < Kamal::Commands::Builder::Base
def create
# No-op on native without cache
end
def remove
# No-op on native without cache
end
def info
# No-op on native
end
def push
combine \
docker(:build, *build_options, build_context),
docker(:push, config.absolute_image),
docker(:push, config.latest_image)
end
end

View File

@@ -0,0 +1,16 @@
class Kamal::Commands::Builder::Native::Cached < Kamal::Commands::Builder::Native
def create
docker :buildx, :create, "--use", "--driver=docker-container"
end
def remove
docker :buildx, :rm, builder_name
end
def push
docker :buildx, :build,
"--push",
*build_options,
build_context
end
end

View File

@@ -1,4 +1,4 @@
class Mrsk::Commands::Builder::Native::Remote < Mrsk::Commands::Builder::Native
class Kamal::Commands::Builder::Native::Remote < Kamal::Commands::Builder::Native
def create
chain \
create_context,
@@ -11,46 +11,38 @@ class Mrsk::Commands::Builder::Native::Remote < Mrsk::Commands::Builder::Native
remove_buildx
end
def push
docker :buildx, :build,
"--push",
"--platform", platform,
"--builder", builder_name,
*build_options,
build_context
end
def info
chain \
docker(:context, :ls),
docker(:buildx, :ls)
end
def push
docker :buildx, :build,
"--push",
"--platform", platform,
"--builder", builder_name,
*build_options,
build_context
end
private
def arch
config.builder["remote"]["arch"]
end
def host
config.builder["remote"]["host"]
end
def builder_name
"mrsk-#{config.service}-native-remote"
"kamal-#{config.service}-native-remote"
end
def builder_name_with_arch
"#{builder_name}-#{arch}"
"#{builder_name}-#{remote_arch}"
end
def platform
"linux/#{arch}"
"linux/#{remote_arch}"
end
def create_context
docker :context, :create,
builder_name_with_arch, "--description", "'#{builder_name} #{arch} native host'", "--docker", "'host=#{host}'"
builder_name_with_arch, "--description", "'#{builder_name} #{remote_arch} native host'", "--docker", "'host=#{remote_host}'"
end
def remove_context

View File

@@ -0,0 +1,30 @@
class Kamal::Commands::Docker < Kamal::Commands::Base
# Install Docker using the https://github.com/docker/docker-install convenience script.
def install
pipe get_docker, :sh
end
# Checks the Docker client version. Fails if Docker is not installed.
def installed?
docker "-v"
end
# Checks the Docker server version. Fails if Docker is not running.
def running?
docker :version
end
# Do we have superuser access to install Docker and start system services?
def superuser?
[ '[ "${EUID:-$(id -u)}" -eq 0 ] || command -v sudo >/dev/null || command -v su >/dev/null' ]
end
private
def get_docker
shell \
any \
[ :curl, "-fsSL", "https://get.docker.com" ],
[ :wget, "-O -", "https://get.docker.com" ],
[ :echo, "\"exit 1\"" ]
end
end

View File

@@ -0,0 +1,14 @@
class Kamal::Commands::Hook < Kamal::Commands::Base
def run(hook, **details)
[ hook_file(hook), env: tags(**details).env ]
end
def hook_exists?(hook)
Pathname.new(hook_file(hook)).exist?
end
private
def hook_file(hook)
File.join(config.hooks_path, hook)
end
end

View File

@@ -0,0 +1,74 @@
require "active_support/duration"
require "time"
require "base64"
class Kamal::Commands::Lock < Kamal::Commands::Base
def acquire(message, version)
combine \
[ :mkdir, lock_dir ],
write_lock_details(message, version)
end
def release
combine \
[ :rm, lock_details_file ],
[ :rm, "-r", lock_dir ]
end
def status
combine \
stat_lock_dir,
read_lock_details
end
def ensure_locks_directory
[ :mkdir, "-p", locks_dir ]
end
private
def write_lock_details(message, version)
write \
[ :echo, "\"#{Base64.encode64(lock_details(message, version))}\"" ],
lock_details_file
end
def read_lock_details
pipe \
[ :cat, lock_details_file ],
[ :base64, "-d" ]
end
def stat_lock_dir
write \
[ :stat, lock_dir ],
"/dev/null"
end
def locks_dir
File.join(config.run_directory, "locks")
end
def lock_dir
dir_name = [ config.service, config.destination ].compact.join("-")
File.join(locks_dir, dir_name)
end
def lock_details_file
File.join(lock_dir, "details")
end
def lock_details(message, version)
<<~DETAILS.strip
Locked by: #{locked_by} at #{Time.now.utc.iso8601}
Version: #{version}
Message: #{message}
DETAILS
end
def locked_by
Kamal::Git.user_name
rescue Errno::ENOENT
"Unknown"
end
end

View File

@@ -0,0 +1,46 @@
require "active_support/duration"
require "active_support/core_ext/numeric/time"
class Kamal::Commands::Prune < Kamal::Commands::Base
def dangling_images
docker :image, :prune, "--force", "--filter", "label=service=#{config.service}"
end
def tagged_images
pipe \
docker(:image, :ls, *service_filter, "--format", "'{{.ID}} {{.Repository}}:{{.Tag}}'"),
"grep -v -w \"#{active_image_list}\"",
"while read image tag; do docker rmi $tag; done"
end
def app_containers(retain:)
pipe \
docker(:ps, "-q", "-a", *service_filter, *stopped_containers_filters),
"tail -n +#{retain + 1}",
"while read container_id; do docker rm $container_id; done"
end
def healthcheck_containers
docker :container, :prune, "--force", *healthcheck_service_filter
end
private
def stopped_containers_filters
[ "created", "exited", "dead" ].flat_map { |status| [ "--filter", "status=#{status}" ] }
end
def active_image_list
# Pull the images that are used by any containers
# Append repo:latest - to avoid deleting the latest tag
# Append repo:<none> - to avoid deleting dangling images that are in use. Unused dangling images are deleted separately
"$(docker container ls -a --format '{{.Image}}\\|' --filter label=service=#{config.service} | tr -d '\\n')#{config.latest_image}\\|#{config.repository}:<none>"
end
def service_filter
[ "--filter", "label=service=#{config.service}" ]
end
def healthcheck_service_filter
[ "--filter", "label=service=#{config.healthcheck_service}" ]
end
end

View File

@@ -1,8 +1,11 @@
class Mrsk::Commands::Registry < Mrsk::Commands::Base
class Kamal::Commands::Registry < Kamal::Commands::Base
delegate :registry, to: :config
def login
docker :login, registry["server"], "-u", redact(lookup("username")), "-p", redact(lookup("password"))
docker :login,
registry["server"],
"-u", sensitive(Kamal::Utils.escape_shell_value(lookup("username"))),
"-p", sensitive(Kamal::Utils.escape_shell_value(lookup("password")))
end
def logout

View File

@@ -0,0 +1,5 @@
class Kamal::Commands::Server < Kamal::Commands::Base
def ensure_run_directory
[ :mkdir, "-p", config.run_directory ]
end
end

View File

@@ -0,0 +1,124 @@
class Kamal::Commands::Traefik < Kamal::Commands::Base
delegate :argumentize, :optionize, to: Kamal::Utils
DEFAULT_IMAGE = "traefik:v2.10"
CONTAINER_PORT = 80
DEFAULT_ARGS = {
"log.level" => "DEBUG"
}
DEFAULT_LABELS = {
# These ensure we serve a 502 rather than a 404 if no containers are available
"traefik.http.routers.catchall.entryPoints" => "http",
"traefik.http.routers.catchall.rule" => "PathPrefix(`/`)",
"traefik.http.routers.catchall.service" => "unavailable",
"traefik.http.routers.catchall.priority" => 1,
"traefik.http.services.unavailable.loadbalancer.server.port" => "0"
}
def run
docker :run, "--name traefik",
"--detach",
"--restart", "unless-stopped",
*publish_args,
"--volume", "/var/run/docker.sock:/var/run/docker.sock",
*env_args,
*config.logging_args,
*label_args,
*docker_options_args,
image,
"--providers.docker",
*cmd_option_args
end
def start
docker :container, :start, "traefik"
end
def stop
docker :container, :stop, "traefik"
end
def start_or_run
any start, run
end
def info
docker :ps, "--filter", "name=^traefik$"
end
def logs(since: nil, lines: nil, grep: nil)
pipe \
docker(:logs, "traefik", (" --since #{since}" if since), (" --tail #{lines}" if lines), "--timestamps", "2>&1"),
("grep '#{grep}'" if grep)
end
def follow_logs(host:, grep: nil)
run_over_ssh pipe(
docker(:logs, "traefik", "--timestamps", "--tail", "10", "--follow", "2>&1"),
(%(grep "#{grep}") if grep)
).join(" "), host: host
end
def remove_container
docker :container, :prune, "--force", "--filter", "label=org.opencontainers.image.title=Traefik"
end
def remove_image
docker :image, :prune, "--all", "--force", "--filter", "label=org.opencontainers.image.title=Traefik"
end
def port
"#{host_port}:#{CONTAINER_PORT}"
end
def env
Kamal::Configuration::Env.from_config \
config: config.traefik.fetch("env", {}),
secrets_file: File.join(config.host_env_directory, "traefik", "traefik.env")
end
def make_env_directory
make_directory(env.secrets_directory)
end
def remove_env_file
[ :rm, "-f", env.secrets_file ]
end
private
def publish_args
argumentize "--publish", port unless config.traefik["publish"] == false
end
def label_args
argumentize "--label", labels
end
def env_args
env.args
end
def labels
DEFAULT_LABELS.merge(config.traefik["labels"] || {})
end
def image
config.traefik.fetch("image") { DEFAULT_IMAGE }
end
def docker_options_args
optionize(config.traefik["options"] || {})
end
def cmd_option_args
if args = config.traefik["args"]
optionize DEFAULT_ARGS.merge(args), with: "="
else
optionize DEFAULT_ARGS, with: "="
end
end
def host_port
config.traefik["host_port"] || CONTAINER_PORT
end
end

351
lib/kamal/configuration.rb Normal file
View File

@@ -0,0 +1,351 @@
require "active_support/ordered_options"
require "active_support/core_ext/string/inquiry"
require "active_support/core_ext/module/delegation"
require "pathname"
require "erb"
require "net/ssh/proxy/jump"
class Kamal::Configuration
delegate :service, :image, :servers, :labels, :registry, :stop_wait_time, :hooks_path, :logging, to: :raw_config, allow_nil: true
delegate :argumentize, :optionize, to: Kamal::Utils
attr_reader :destination, :raw_config
class << self
def create_from(config_file:, destination: nil, version: nil)
raw_config = load_config_files(config_file, *destination_config_file(config_file, destination))
new raw_config, destination: destination, version: version
end
private
def load_config_files(*files)
files.inject({}) { |config, file| config.deep_merge! load_config_file(file) }
end
def load_config_file(file)
if file.exist?
# Newer Psych doesn't load aliases by default
load_method = YAML.respond_to?(:unsafe_load) ? :unsafe_load : :load
YAML.send(load_method, ERB.new(IO.read(file)).result).symbolize_keys
else
raise "Configuration file not found in #{file}"
end
end
def destination_config_file(base_config_file, destination)
base_config_file.sub_ext(".#{destination}.yml") if destination
end
end
def initialize(raw_config, destination: nil, version: nil, validate: true)
@raw_config = ActiveSupport::InheritableOptions.new(raw_config)
@destination = destination
@declared_version = version
valid? if validate
end
def version=(version)
@declared_version = version
end
def version
@declared_version.presence || ENV["VERSION"] || git_version
end
def abbreviated_version
if version
# Don't abbreviate <sha>_uncommitted_<etc>
if version.include?("_")
version
else
version[0...7]
end
end
end
def minimum_version
raw_config.minimum_version
end
def roles
@roles ||= role_names.collect { |role_name| Role.new(role_name, config: self) }
end
def role(name)
roles.detect { |r| r.name == name.to_s }
end
def accessories
@accessories ||= raw_config.accessories&.keys&.collect { |name| Kamal::Configuration::Accessory.new(name, config: self) } || []
end
def accessory(name)
accessories.detect { |a| a.name == name.to_s }
end
def all_hosts
(roles + accessories).flat_map(&:hosts).uniq
end
def primary_host
primary_role&.primary_host
end
def primary_role_name
raw_config.primary_role || "web"
end
def primary_role
role(primary_role_name)
end
def allow_empty_roles?
raw_config.allow_empty_roles
end
def traefik_roles
roles.select(&:running_traefik?)
end
def traefik_role_names
traefik_roles.flat_map(&:name)
end
def traefik_hosts
traefik_roles.flat_map(&:hosts).uniq
end
def repository
[ raw_config.registry["server"], image ].compact.join("/")
end
def absolute_image
"#{repository}:#{version}"
end
def latest_image
"#{repository}:#{latest_tag}"
end
def latest_tag
[ "latest", *destination ].join("-")
end
def service_with_version
"#{service}-#{version}"
end
def require_destination?
raw_config.require_destination
end
def retain_containers
raw_config.retain_containers || 5
end
def volume_args
if raw_config.volumes.present?
argumentize "--volume", raw_config.volumes
else
[]
end
end
def logging_args
if logging.present?
optionize({ "log-driver" => logging["driver"] }.compact) +
argumentize("--log-opt", logging["options"])
else
argumentize("--log-opt", { "max-size" => "10m" })
end
end
def boot
Kamal::Configuration::Boot.new(config: self)
end
def builder
Kamal::Configuration::Builder.new(config: self)
end
def traefik
raw_config.traefik || {}
end
def ssh
Kamal::Configuration::Ssh.new(config: self)
end
def sshkit
Kamal::Configuration::Sshkit.new(config: self)
end
def healthcheck
{ "path" => "/up", "port" => 3000, "max_attempts" => 7, "cord" => "/tmp/kamal-cord", "log_lines" => 50 }.merge(raw_config.healthcheck || {})
end
def healthcheck_service
[ "healthcheck", service, destination ].compact.join("-")
end
def readiness_delay
raw_config.readiness_delay || 7
end
def run_id
@run_id ||= SecureRandom.hex(16)
end
def run_directory
raw_config.run_directory || ".kamal"
end
def run_directory_as_docker_volume
if Pathname.new(run_directory).absolute?
run_directory
else
File.join "$(pwd)", run_directory
end
end
def hooks_path
raw_config.hooks_path || ".kamal/hooks"
end
def asset_path
raw_config.asset_path
end
def host_env_directory
File.join(run_directory, "env")
end
def env
raw_config.env || {}
end
def env_tags
@env_tags ||= if (tags = raw_config.env["tags"])
tags.collect { |name, config| Kamal::Configuration::Env::Tag.new(name, config: config) }
else
[]
end
end
def env_tag(name)
env_tags.detect { |t| t.name == name.to_s }
end
def valid?
ensure_destination_if_required && ensure_required_keys_present && ensure_valid_kamal_version && ensure_retain_containers_valid && ensure_valid_service_name
end
def to_h
{
roles: role_names,
hosts: all_hosts,
primary_host: primary_host,
version: version,
repository: repository,
absolute_image: absolute_image,
service_with_version: service_with_version,
volume_args: volume_args,
ssh_options: ssh.to_h,
sshkit: sshkit.to_h,
builder: builder.to_h,
accessories: raw_config.accessories,
logging: logging_args,
healthcheck: healthcheck
}.compact
end
private
# Will raise ArgumentError if any required config keys are missing
def ensure_destination_if_required
if require_destination? && destination.nil?
raise ArgumentError, "You must specify a destination"
end
true
end
def ensure_required_keys_present
%i[ service image registry servers ].each do |key|
raise ArgumentError, "Missing required configuration for #{key}" unless raw_config[key].present?
end
if raw_config.registry["username"].blank?
raise ArgumentError, "You must specify a username for the registry in config/deploy.yml"
end
if raw_config.registry["password"].blank?
raise ArgumentError, "You must specify a password for the registry in config/deploy.yml (or set the ENV variable if that's used)"
end
unless role_names.include?(primary_role_name)
raise ArgumentError, "The primary_role #{primary_role_name} isn't defined"
end
if primary_role.hosts.empty?
raise ArgumentError, "No servers specified for the #{primary_role.name} primary_role"
end
unless allow_empty_roles?
roles.each do |role|
if role.hosts.empty?
raise ArgumentError, "No servers specified for the #{role.name} role. You can ignore this with allow_empty_roles: true"
end
end
end
true
end
def ensure_valid_service_name
raise ArgumentError, "Service name can only include alphanumeric characters, hyphens, and underscores" unless raw_config[:service] =~ /^[a-z0-9_-]+$/i
true
end
def ensure_valid_kamal_version
if minimum_version && Gem::Version.new(minimum_version) > Gem::Version.new(Kamal::VERSION)
raise ArgumentError, "Current version is #{Kamal::VERSION}, minimum required is #{minimum_version}"
end
true
end
def ensure_retain_containers_valid
raise ArgumentError, "Must retain at least 1 container" if retain_containers < 1
true
end
def role_names
raw_config.servers.is_a?(Array) ? [ "web" ] : raw_config.servers.keys.sort
end
def git_version
@git_version ||=
if Kamal::Git.used?
if Kamal::Git.uncommitted_changes.present? && !builder.git_clone?
uncommitted_suffix = "_uncommitted_#{SecureRandom.hex(8)}"
end
[ Kamal::Git.revision, uncommitted_suffix ].compact.join
else
raise "Can't use commit hash as version, no git repository found in #{Dir.pwd}"
end
end
end

View File

@@ -0,0 +1,175 @@
class Kamal::Configuration::Accessory
delegate :argumentize, :optionize, to: Kamal::Utils
attr_accessor :name, :specifics
def initialize(name, config:)
@name, @config, @specifics = name.inquiry, config, config.raw_config["accessories"][name]
end
def service_name
specifics["service"] || "#{config.service}-#{name}"
end
def image
specifics["image"]
end
def hosts
if (specifics.keys & [ "host", "hosts", "roles" ]).size != 1
raise ArgumentError, "Specify one of `host`, `hosts` or `roles` for accessory `#{name}`"
end
hosts_from_host || hosts_from_hosts || hosts_from_roles
end
def port
if port = specifics["port"]&.to_s
port.include?(":") ? port : "#{port}:#{port}"
end
end
def publish_args
argumentize "--publish", port if port
end
def labels
default_labels.merge(specifics["labels"] || {})
end
def label_args
argumentize "--label", labels
end
def env
Kamal::Configuration::Env.from_config \
config: specifics.fetch("env", {}),
secrets_file: File.join(config.host_env_directory, "accessories", "#{service_name}.env")
end
def env_args
env.args
end
def files
specifics["files"]&.to_h do |local_to_remote_mapping|
local_file, remote_file = local_to_remote_mapping.split(":")
[ expand_local_file(local_file), expand_remote_file(remote_file) ]
end || {}
end
def directories
specifics["directories"]&.to_h do |host_to_container_mapping|
host_path, container_path = host_to_container_mapping.split(":")
[ expand_host_path(host_path), container_path ]
end || {}
end
def volumes
specific_volumes + remote_files_as_volumes + remote_directories_as_volumes
end
def volume_args
argumentize "--volume", volumes
end
def option_args
if args = specifics["options"]
optionize args
else
[]
end
end
def cmd
specifics["cmd"]
end
private
attr_accessor :config
def default_labels
{ "service" => service_name }
end
def expand_local_file(local_file)
if local_file.end_with?("erb")
with_clear_env_loaded { read_dynamic_file(local_file) }
else
Pathname.new(File.expand_path(local_file)).to_s
end
end
def with_clear_env_loaded
env.clear.each { |k, v| ENV[k] = v }
yield
ensure
env.clear.each { |k, v| ENV.delete(k) }
end
def read_dynamic_file(local_file)
StringIO.new(ERB.new(IO.read(local_file)).result)
end
def expand_remote_file(remote_file)
service_name + remote_file
end
def specific_volumes
specifics["volumes"] || []
end
def remote_files_as_volumes
specifics["files"]&.collect do |local_to_remote_mapping|
_, remote_file = local_to_remote_mapping.split(":")
"#{service_data_directory + remote_file}:#{remote_file}"
end || []
end
def remote_directories_as_volumes
specifics["directories"]&.collect do |host_to_container_mapping|
host_path, container_path = host_to_container_mapping.split(":")
[ expand_host_path(host_path), container_path ].join(":")
end || []
end
def expand_host_path(host_path)
absolute_path?(host_path) ? host_path : File.join(service_data_directory, host_path)
end
def absolute_path?(path)
Pathname.new(path).absolute?
end
def service_data_directory
"$PWD/#{service_name}"
end
def hosts_from_host
if specifics.key?("host")
host = specifics["host"]
if host
[ host ]
else
raise ArgumentError, "Missing host for accessory `#{name}`"
end
end
end
def hosts_from_hosts
if specifics.key?("hosts")
hosts = specifics["hosts"]
if hosts.is_a?(Array)
hosts
else
raise ArgumentError, "Hosts should be an Array for accessory `#{name}`"
end
end
end
def hosts_from_roles
if specifics.key?("roles")
specifics["roles"].flat_map { |role| config.role(role).hosts }
end
end
end

View File

@@ -0,0 +1,20 @@
class Kamal::Configuration::Boot
def initialize(config:)
@options = config.raw_config.boot || {}
@host_count = config.all_hosts.count
end
def limit
limit = @options["limit"]
if limit.to_s.end_with?("%")
[ @host_count * limit.to_i / 100, 1 ].max
else
limit
end
end
def wait
@options["wait"]
end
end

View File

@@ -0,0 +1,153 @@
class Kamal::Configuration::Builder
def initialize(config:)
@options = config.raw_config.builder || {}
@image = config.image
@server = config.registry["server"]
@service = config.service
@destination = config.destination
valid?
end
def to_h
@options
end
def multiarch?
@options["multiarch"] != false
end
def local?
!!@options["local"]
end
def remote?
!!@options["remote"]
end
def cached?
!!@options["cache"]
end
def args
@options["args"] || {}
end
def secrets
@options["secrets"] || []
end
def dockerfile
@options["dockerfile"] || "Dockerfile"
end
def target
@options["target"]
end
def context
@options["context"] || "."
end
def local_arch
@options["local"]["arch"] if local?
end
def local_host
@options["local"]["host"] if local?
end
def remote_arch
@options["remote"]["arch"] if remote?
end
def remote_host
@options["remote"]["host"] if remote?
end
def cache_from
if cached?
case @options["cache"]["type"]
when "gha"
cache_from_config_for_gha
when "registry"
cache_from_config_for_registry
end
end
end
def cache_to
if cached?
case @options["cache"]["type"]
when "gha"
cache_to_config_for_gha
when "registry"
cache_to_config_for_registry
end
end
end
def ssh
@options["ssh"]
end
def git_clone?
Kamal::Git.used? && @options["context"].nil?
end
def clone_directory
@clone_directory ||= File.join Dir.tmpdir, "kamal-clones", [ @service, pwd_sha ].compact.join("-")
end
def build_directory
@build_directory ||=
if git_clone?
File.join clone_directory, repo_basename, repo_relative_pwd
else
"."
end
end
private
def valid?
if @options["cache"] && @options["cache"]["type"]
raise ArgumentError, "Invalid cache type: #{@options["cache"]["type"]}" unless [ "gha", "registry" ].include?(@options["cache"]["type"])
end
end
def cache_image
@options["cache"]&.fetch("image", nil) || "#{@image}-build-cache"
end
def cache_image_ref
[ @server, cache_image ].compact.join("/")
end
def cache_from_config_for_gha
"type=gha"
end
def cache_from_config_for_registry
[ "type=registry", "ref=#{cache_image_ref}" ].compact.join(",")
end
def cache_to_config_for_gha
[ "type=gha", @options["cache"]&.fetch("options", nil) ].compact.join(",")
end
def cache_to_config_for_registry
[ "type=registry", @options["cache"]&.fetch("options", nil), "ref=#{cache_image_ref}" ].compact.join(",")
end
def repo_basename
File.basename(Kamal::Git.root)
end
def repo_relative_pwd
Dir.pwd.delete_prefix(Kamal::Git.root)
end
def pwd_sha
Digest::SHA256.hexdigest(Dir.pwd)[0..12]
end
end

View File

@@ -0,0 +1,40 @@
class Kamal::Configuration::Env
attr_reader :secrets_keys, :clear, :secrets_file
delegate :argumentize, to: Kamal::Utils
def self.from_config(config:, secrets_file: nil)
secrets_keys = config.fetch("secret", [])
clear = config.fetch("clear", config.key?("secret") || config.key?("tags") ? {} : config)
new clear: clear, secrets_keys: secrets_keys, secrets_file: secrets_file
end
def initialize(clear:, secrets_keys:, secrets_file:)
@clear = clear
@secrets_keys = secrets_keys
@secrets_file = secrets_file
end
def args
[ "--env-file", secrets_file, *argumentize("--env", clear) ]
end
def secrets_io
StringIO.new(Kamal::EnvFile.new(secrets).to_s)
end
def secrets
@secrets ||= secrets_keys.to_h { |key| [ key, ENV.fetch(key) ] }
end
def secrets_directory
File.dirname(secrets_file)
end
def merge(other)
self.class.new \
clear: @clear.merge(other.clear),
secrets_keys: @secrets_keys | other.secrets_keys,
secrets_file: secrets_file
end
end

12
lib/kamal/configuration/env/tag.rb vendored Normal file
View File

@@ -0,0 +1,12 @@
class Kamal::Configuration::Env::Tag
attr_reader :name, :config
def initialize(name, config:)
@name = name
@config = config
end
def env
Kamal::Configuration::Env.from_config(config: config)
end
end

View File

@@ -0,0 +1,263 @@
class Kamal::Configuration::Role
CORD_FILE = "cord"
delegate :argumentize, :optionize, to: Kamal::Utils
attr_accessor :name
alias to_s name
def initialize(name, config:)
@name, @config = name.inquiry, config
@tagged_hosts ||= extract_tagged_hosts_from_config
end
def primary_host
hosts.first
end
def hosts
tagged_hosts.keys
end
def env_tags(host)
tagged_hosts.fetch(host).collect { |tag| config.env_tag(tag) }
end
def cmd
specializations["cmd"]
end
def option_args
if args = specializations["options"]
optionize args
else
[]
end
end
def labels
default_labels.merge(traefik_labels).merge(custom_labels)
end
def label_args
argumentize "--label", labels
end
def logging_args
args = config.logging || {}
args.deep_merge!(specializations["logging"]) if specializations["logging"].present?
if args.any?
optionize({ "log-driver" => args["driver"] }.compact) +
argumentize("--log-opt", args["options"])
else
config.logging_args
end
end
def env(host)
@envs ||= {}
@envs[host] ||= [ base_env, specialized_env, *env_tags(host).map(&:env) ].reduce(:merge)
end
def env_args(host)
env(host).args
end
def asset_volume_args
asset_volume&.docker_args
end
def health_check_args(cord: true)
if health_check_cmd.present?
if cord && uses_cord?
optionize({ "health-cmd" => health_check_cmd_with_cord, "health-interval" => health_check_interval })
.concat(cord_volume.docker_args)
else
optionize({ "health-cmd" => health_check_cmd, "health-interval" => health_check_interval })
end
else
[]
end
end
def health_check_cmd
health_check_options["cmd"] || http_health_check(port: health_check_options["port"], path: health_check_options["path"])
end
def health_check_cmd_with_cord
"(#{health_check_cmd}) && (stat #{cord_container_file} > /dev/null || exit 1)"
end
def health_check_interval
health_check_options["interval"] || "1s"
end
def running_traefik?
if specializations["traefik"].nil?
primary?
else
specializations["traefik"]
end
end
def primary?
self == @config.primary_role
end
def uses_cord?
running_traefik? && cord_volume && health_check_cmd.present?
end
def cord_host_directory
File.join config.run_directory_as_docker_volume, "cords", [ container_prefix, config.run_id ].join("-")
end
def cord_volume
if (cord = health_check_options["cord"])
@cord_volume ||= Kamal::Configuration::Volume.new \
host_path: File.join(config.run_directory, "cords", [ container_prefix, config.run_id ].join("-")),
container_path: cord
end
end
def cord_host_file
File.join cord_volume.host_path, CORD_FILE
end
def cord_container_directory
health_check_options.fetch("cord", nil)
end
def cord_container_file
File.join cord_volume.container_path, CORD_FILE
end
def container_name(version = nil)
[ container_prefix, version || config.version ].compact.join("-")
end
def container_prefix
[ config.service, name, config.destination ].compact.join("-")
end
def asset_path
specializations["asset_path"] || config.asset_path
end
def assets?
asset_path.present? && running_traefik?
end
def asset_volume(version = nil)
if assets?
Kamal::Configuration::Volume.new \
host_path: asset_volume_path(version), container_path: asset_path
end
end
def asset_extracted_path(version = nil)
File.join config.run_directory, "assets", "extracted", container_name(version)
end
def asset_volume_path(version = nil)
File.join config.run_directory, "assets", "volumes", container_name(version)
end
private
attr_accessor :config, :tagged_hosts
def extract_tagged_hosts_from_config
{}.tap do |tagged_hosts|
extract_hosts_from_config.map do |host_config|
if host_config.is_a?(Hash)
raise ArgumentError, "Multiple hosts found: #{host_config.inspect}" unless host_config.size == 1
host, tags = host_config.first
tagged_hosts[host] = Array(tags)
elsif host_config.is_a?(String) || host_config.is_a?(Symbol)
tagged_hosts[host_config] = []
else
raise ArgumentError, "Invalid host config: #{host_config.inspect}"
end
end
end
end
def extract_hosts_from_config
if config.servers.is_a?(Array)
config.servers
else
servers = config.servers[name]
servers.is_a?(Array) ? servers : Array(servers["hosts"])
end
end
def default_labels
{ "service" => config.service, "role" => name, "destination" => config.destination }
end
def traefik_labels
if running_traefik?
{
# Setting a service property ensures that the generated service name will be consistent between versions
"traefik.http.services.#{traefik_service}.loadbalancer.server.scheme" => "http",
"traefik.http.routers.#{traefik_service}.rule" => "PathPrefix(`/`)",
"traefik.http.routers.#{traefik_service}.priority" => "2",
"traefik.http.middlewares.#{traefik_service}-retry.retry.attempts" => "5",
"traefik.http.middlewares.#{traefik_service}-retry.retry.initialinterval" => "500ms",
"traefik.http.routers.#{traefik_service}.middlewares" => "#{traefik_service}-retry@docker"
}
else
{}
end
end
def traefik_service
container_prefix
end
def custom_labels
Hash.new.tap do |labels|
labels.merge!(config.labels) if config.labels.present?
labels.merge!(specializations["labels"]) if specializations["labels"].present?
end
end
def specializations
if config.servers.is_a?(Array) || config.servers[name].is_a?(Array)
{}
else
config.servers[name].except("hosts")
end
end
def specialized_env
Kamal::Configuration::Env.from_config config: specializations.fetch("env", {})
end
# Secrets are stored in an array, which won't merge by default, so have to do it by hand.
def base_env
Kamal::Configuration::Env.from_config \
config: config.env,
secrets_file: File.join(config.host_env_directory, "roles", "#{container_prefix}.env")
end
def http_health_check(port:, path:)
"curl -f #{URI.join("http://localhost:#{port}", path)} || exit 1" if path.present? || port.present?
end
def health_check_options
@health_check_options ||= begin
options = specializations["healthcheck"] || {}
options = config.healthcheck.merge(options) if running_traefik?
options
end
end
end

View File

@@ -0,0 +1,42 @@
class Kamal::Configuration::Ssh
LOGGER = ::Logger.new(STDERR)
def initialize(config:)
@config = config.raw_config.ssh || {}
end
def user
config.fetch("user", "root")
end
def port
config.fetch("port", 22)
end
def proxy
if (proxy = config["proxy"])
Net::SSH::Proxy::Jump.new(proxy.include?("@") ? proxy : "root@#{proxy}")
elsif (proxy_command = config["proxy_command"])
Net::SSH::Proxy::Command.new(proxy_command)
end
end
def options
{ user: user, port: port, proxy: proxy, logger: logger, keepalive: true, keepalive_interval: 30 }.compact
end
def to_h
options.except(:logger).merge(log_level: log_level)
end
private
attr_accessor :config
def logger
LOGGER.tap { |logger| logger.level = log_level }
end
def log_level
config.fetch("log_level", :fatal)
end
end

View File

@@ -0,0 +1,20 @@
class Kamal::Configuration::Sshkit
def initialize(config:)
@options = config.raw_config.sshkit || {}
end
def max_concurrent_starts
options.fetch("max_concurrent_starts", 30)
end
def pool_idle_timeout
options.fetch("pool_idle_timeout", 900)
end
def to_h
options
end
private
attr_accessor :options
end

View File

@@ -0,0 +1,22 @@
class Kamal::Configuration::Volume
attr_reader :host_path, :container_path
delegate :argumentize, to: Kamal::Utils
def initialize(host_path:, container_path:)
@host_path = host_path
@container_path = container_path
end
def docker_args
argumentize "--volume", "#{host_path_for_docker_volume}:#{container_path}"
end
private
def host_path_for_docker_volume
if Pathname.new(host_path).absolute?
host_path
else
File.join "$(pwd)", host_path
end
end
end

38
lib/kamal/env_file.rb Normal file
View File

@@ -0,0 +1,38 @@
# Encode an env hash as a string where secret values have been looked up and all values escaped for Docker.
class Kamal::EnvFile
def initialize(env)
@env = env
end
def to_s
env_file = StringIO.new.tap do |contents|
@env.each do |key, value|
contents << docker_env_file_line(key, value)
end
end.string
# Ensure the file has some contents to avoid the SSHKIT empty file warning
env_file.presence || "\n"
end
alias to_str to_s
private
def docker_env_file_line(key, value)
"#{key}=#{escape_docker_env_file_value(value)}\n"
end
# Escape a value to make it safe to dump in a docker file.
def escape_docker_env_file_value(value)
# keep non-ascii(UTF-8) characters as it is
value.to_s.scan(/[\x00-\x7F]+|[^\x00-\x7F]+/).map do |part|
part.ascii_only? ? escape_docker_env_file_ascii_value(part) : part
end.join
end
def escape_docker_env_file_ascii_value(value)
# Doublequotes are treated literally in docker env files
# so remove leading and trailing ones and unescape any others
value.to_s.dump[1..-2].gsub(/\\"/, "\"")
end
end

23
lib/kamal/git.rb Normal file
View File

@@ -0,0 +1,23 @@
module Kamal::Git
extend self
def used?
system("git rev-parse")
end
def user_name
`git config user.name`.strip
end
def revision
`git rev-parse HEAD`.strip
end
def uncommitted_changes
`git status --porcelain`.strip
end
def root
`git rev-parse --show-toplevel`.strip
end
end

View File

@@ -0,0 +1,141 @@
require "sshkit"
require "sshkit/dsl"
require "net/scp"
require "active_support/core_ext/hash/deep_merge"
require "json"
class SSHKit::Backend::Abstract
def capture_with_info(*args, **kwargs)
capture(*args, **kwargs, verbosity: Logger::INFO)
end
def capture_with_debug(*args, **kwargs)
capture(*args, **kwargs, verbosity: Logger::DEBUG)
end
def capture_with_pretty_json(*args, **kwargs)
JSON.pretty_generate(JSON.parse(capture(*args, **kwargs)))
end
def puts_by_host(host, output, type: "App")
puts "#{type} Host: #{host}\n#{output}\n\n"
end
# Our execution pattern is for the CLI execute args lists returned
# from commands, but this doesn't support returning execution options
# from the command.
#
# Support this by using kwargs for CLI options and merging with the
# args-extracted options.
module CommandEnvMerge
private
# Override to merge options returned by commands in the args list with
# options passed by the CLI and pass them along as kwargs.
def command(args, options)
more_options, args = args.partition { |a| a.is_a? Hash }
more_options << options
build_command(args, **more_options.reduce(:deep_merge))
end
# Destructure options to pluck out env for merge
def build_command(args, env: nil, **options)
# Rely on native Ruby kwargs precedence rather than explicit Hash merges
SSHKit::Command.new(*args, **default_command_options, **options, env: env_for(env))
end
def default_command_options
{ in: pwd_path, host: @host, user: @user, group: @group }
end
def env_for(env)
@env.to_h.merge(env.to_h)
end
end
prepend CommandEnvMerge
end
class SSHKit::Backend::Netssh::Configuration
attr_accessor :max_concurrent_starts
end
class SSHKit::Backend::Netssh
module LimitConcurrentStartsClass
attr_reader :start_semaphore
def configure(&block)
super &block
# Create this here to avoid lazy creation by multiple threads
if config.max_concurrent_starts
@start_semaphore = Concurrent::Semaphore.new(config.max_concurrent_starts)
end
end
end
class << self
prepend LimitConcurrentStartsClass
end
module LimitConcurrentStartsInstance
private
def with_ssh(&block)
host.ssh_options = self.class.config.ssh_options.merge(host.ssh_options || {})
self.class.pool.with(
method(:start_with_concurrency_limit),
String(host.hostname),
host.username,
host.netssh_options,
&block
)
end
def start_with_concurrency_limit(*args)
if self.class.start_semaphore
self.class.start_semaphore.acquire do
Net::SSH.start(*args)
end
else
Net::SSH.start(*args)
end
end
end
prepend LimitConcurrentStartsInstance
end
class SSHKit::Runner::Parallel
# SSHKit joins the threads in sequence and fails on the first error it encounters, which means that we wait threads
# before the first failure to complete but not for ones after.
#
# We'll patch it to wait for them all to complete, and to record all the threads that errored so we can see when a
# problem occurs on multiple hosts.
module CompleteAll
def execute
threads = hosts.map do |host|
Thread.new(host) do |h|
backend(h, &block).run
rescue ::StandardError => e
e2 = SSHKit::Runner::ExecuteError.new e
raise e2, "Exception while executing #{host.user ? "as #{host.user}@" : "on host "}#{host}: #{e.message}"
end
end
exceptions = []
threads.each do |t|
begin
t.join
rescue SSHKit::Runner::ExecuteError => e
exceptions << e
end
end
if exceptions.one?
raise exceptions.first
elsif exceptions.many?
raise exceptions.first, [ "Exceptions on #{exceptions.count} hosts:", exceptions.map(&:message) ].join("\n")
end
end
end
prepend CompleteAll
end

39
lib/kamal/tags.rb Normal file
View File

@@ -0,0 +1,39 @@
require "time"
class Kamal::Tags
attr_reader :config, :tags
class << self
def from_config(config, **extra)
new(**default_tags(config), **extra)
end
def default_tags(config)
{ recorded_at: Time.now.utc.iso8601,
performer: `whoami`.chomp,
destination: config.destination,
version: config.version,
service_version: service_version(config) }
end
def service_version(config)
[ config.service, config.abbreviated_version ].compact.join("@")
end
end
def initialize(**tags)
@tags = tags.compact
end
def env
tags.transform_keys { |detail| "KAMAL_#{detail.upcase}" }
end
def to_s
tags.values.map { |value| "[#{value}]" }.join(" ")
end
def except(*tags)
self.class.new(**self.tags.except(*tags))
end
end

80
lib/kamal/utils.rb Normal file
View File

@@ -0,0 +1,80 @@
module Kamal::Utils
extend self
DOLLAR_SIGN_WITHOUT_SHELL_EXPANSION_REGEX = /\$(?!{[^\}]*\})/
# Return a list of escaped shell arguments using the same named argument against the passed attributes (hash or array).
def argumentize(argument, attributes, sensitive: false)
Array(attributes).flat_map do |key, value|
if value.present?
attr = "#{key}=#{escape_shell_value(value)}"
attr = self.sensitive(attr, redaction: "#{key}=[REDACTED]") if sensitive
[ argument, attr ]
else
[ argument, key ]
end
end
end
# Returns a list of shell-dashed option arguments. If the value is true, it's treated like a value-less option.
def optionize(args, with: nil)
options = if with
flatten_args(args).collect { |(key, value)| value == true ? "--#{key}" : "--#{key}#{with}#{escape_shell_value(value)}" }
else
flatten_args(args).collect { |(key, value)| [ "--#{key}", value == true ? nil : escape_shell_value(value) ] }
end
options.flatten.compact
end
# Flattens a one-to-many structure into an array of two-element arrays each containing a key-value pair
def flatten_args(args)
args.flat_map { |key, value| value.try(:map) { |entry| [ key, entry ] } || [ [ key, value ] ] }
end
# Marks sensitive values for redaction in logs and human-visible output.
# Pass `redaction:` to change the default `"[REDACTED]"` redaction, e.g.
# `sensitive "#{arg}=#{secret}", redaction: "#{arg}=xxxx"
def sensitive(...)
Kamal::Utils::Sensitive.new(...)
end
def redacted(value)
case
when value.respond_to?(:redaction)
value.redaction
when value.respond_to?(:transform_values)
value.transform_values { |value| redacted value }
when value.respond_to?(:map)
value.map { |element| redacted element }
else
value
end
end
# Escape a value to make it safe for shell use.
def escape_shell_value(value)
value.to_s.dump
.gsub(/`/, '\\\\`')
.gsub(DOLLAR_SIGN_WITHOUT_SHELL_EXPANSION_REGEX, '\$')
end
# Apply a list of host or role filters, including wildcard matches
def filter_specific_items(filters, items)
matches = []
Array(filters).select do |filter|
matches += Array(items).select do |item|
# Only allow * for a wildcard
# items are roles or hosts
File.fnmatch(filter, item.to_s, File::FNM_EXTGLOB)
end
end
matches.uniq
end
def stable_sort!(elements, &block)
elements.sort_by!.with_index { |element, index| [ block.call(element), index ] }
end
end

View File

@@ -0,0 +1,20 @@
require "active_support/core_ext/module/delegation"
require "sshkit"
class Kamal::Utils::Sensitive
# So SSHKit knows to redact these values.
include SSHKit::Redaction
attr_reader :unredacted, :redaction
delegate :to_s, to: :unredacted
delegate :inspect, to: :redaction
def initialize(value, redaction: "[REDACTED]")
@unredacted, @redaction = value, redaction
end
# Sensitive values won't leak into YAML output.
def encode_with(coder)
coder.represent_scalar nil, redaction
end
end

3
lib/kamal/version.rb Normal file
View File

@@ -0,0 +1,3 @@
module Kamal
VERSION = "1.6.0"
end

View File

@@ -1,5 +0,0 @@
module Mrsk::Cli
end
# SSHKit uses instance eval, so we need a global const for ergonomics
MRSK = Mrsk::Commander.new

View File

@@ -1,214 +0,0 @@
class Mrsk::Cli::Accessory < Mrsk::Cli::Base
desc "boot [NAME]", "Boot new accessory service on host (use NAME=all to boot all accessories)"
def boot(name)
if name == "all"
MRSK.accessory_names.each { |accessory_name| boot(accessory_name) }
else
with_accessory(name) do |accessory|
directories(name)
upload(name)
on(accessory.host) do
execute *MRSK.auditor.record("Booted #{name} accessory"), verbosity: :debug
execute *accessory.run
end
audit_broadcast "Booted accessory #{name}" unless options[:skip_broadcast]
end
end
end
desc "upload [NAME]", "Upload accessory files to host", hide: true
def upload(name)
with_accessory(name) do |accessory|
on(accessory.host) do
accessory.files.each do |(local, remote)|
accessory.ensure_local_file_present(local)
execute *accessory.make_directory_for(remote)
upload! local, remote
execute :chmod, "755", remote
end
end
end
end
desc "directories [NAME]", "Create accessory directories on host", hide: true
def directories(name)
with_accessory(name) do |accessory|
on(accessory.host) do
accessory.directories.keys.each do |host_path|
execute *accessory.make_directory(host_path)
end
end
end
end
desc "reboot [NAME]", "Reboot existing accessory on host (stop container, remove container, start new container)"
def reboot(name)
with_accessory(name) do |accessory|
stop(name)
remove_container(name)
boot(name)
end
end
desc "start [NAME]", "Start existing accessory container on host"
def start(name)
with_accessory(name) do |accessory|
on(accessory.host) do
execute *MRSK.auditor.record("Started #{name} accessory"), verbosity: :debug
execute *accessory.start
end
end
end
desc "stop [NAME]", "Stop existing accessory container on host"
def stop(name)
with_accessory(name) do |accessory|
on(accessory.host) do
execute *MRSK.auditor.record("Stopped #{name} accessory"), verbosity: :debug
execute *accessory.stop, raise_on_non_zero_exit: false
end
end
end
desc "restart [NAME]", "Restart existing accessory container on host"
def restart(name)
with_accessory(name) do
stop(name)
start(name)
end
end
desc "details [NAME]", "Show details about accessory on host (use NAME=all to show all accessories)"
def details(name)
if name == "all"
MRSK.accessory_names.each { |accessory_name| details(accessory_name) }
else
with_accessory(name) do |accessory|
on(accessory.host) { puts capture_with_info(*accessory.info) }
end
end
end
desc "exec [NAME] [CMD]", "Execute a custom command on servers (use --help to show options)"
option :interactive, aliases: "-i", type: :boolean, default: false, desc: "Execute command over ssh for an interactive shell (use for console/bash)"
option :reuse, type: :boolean, default: false, desc: "Reuse currently running container instead of starting a new one"
def exec(name, cmd)
with_accessory(name) do |accessory|
case
when options[:interactive] && options[:reuse]
say "Launching interactive command with via SSH from existing container...", :magenta
run_locally { exec accessory.execute_in_existing_container_over_ssh(cmd) }
when options[:interactive]
say "Launching interactive command via SSH from new container...", :magenta
run_locally { exec accessory.execute_in_new_container_over_ssh(cmd) }
when options[:reuse]
say "Launching command from existing container...", :magenta
on(accessory.host) do
execute *MRSK.auditor.record("Executed cmd '#{cmd}' on #{name} accessory"), verbosity: :debug
capture_with_info(*accessory.execute_in_existing_container(cmd))
end
else
say "Launching command from new container...", :magenta
on(accessory.host) do
execute *MRSK.auditor.record("Executed cmd '#{cmd}' on #{name} accessory"), verbosity: :debug
capture_with_info(*accessory.execute_in_new_container(cmd))
end
end
end
end
desc "logs [NAME]", "Show log lines from accessory on host (use --help to show options)"
option :since, aliases: "-s", desc: "Show logs since timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes)"
option :lines, type: :numeric, aliases: "-n", desc: "Number of log lines to pull from each server"
option :grep, aliases: "-g", desc: "Show lines with grep match only (use this to fetch specific requests by id)"
option :follow, aliases: "-f", desc: "Follow logs on primary server (or specific host set by --hosts)"
def logs(name)
with_accessory(name) do |accessory|
grep = options[:grep]
if options[:follow]
run_locally do
info "Following logs on #{accessory.host}..."
info accessory.follow_logs(grep: grep)
exec accessory.follow_logs(grep: grep)
end
else
since = options[:since]
lines = options[:lines].presence || ((since || grep) ? nil : 100) # Default to 100 lines if since or grep isn't set
on(accessory.host) do
puts capture_with_info(*accessory.logs(since: since, lines: lines, grep: grep))
end
end
end
end
desc "remove [NAME]", "Remove accessory container and image from host (use NAME=all to remove all accessories)"
option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
def remove(name)
if name == "all"
MRSK.accessory_names.each { |accessory_name| remove(accessory_name) }
else
if options[:confirmed] || ask("This will remove all containers and images for #{name}. Are you sure?", limited_to: %w( y N ), default: "N") == "y"
with_accessory(name) do
stop(name)
remove_container(name)
remove_image(name)
remove_service_directory(name)
end
end
end
end
desc "remove_container [NAME]", "Remove accessory container from host", hide: true
def remove_container(name)
with_accessory(name) do |accessory|
on(accessory.host) do
execute *MRSK.auditor.record("Remove #{name} accessory container"), verbosity: :debug
execute *accessory.remove_container
end
end
end
desc "remove_image [NAME]", "Remove accessory image from host", hide: true
def remove_image(name)
with_accessory(name) do |accessory|
on(accessory.host) do
execute *MRSK.auditor.record("Removed #{name} accessory image"), verbosity: :debug
execute *accessory.remove_image
end
end
end
desc "remove_service_directory [NAME]", "Remove accessory directory used for uploaded files and data directories from host", hide: true
def remove_service_directory(name)
with_accessory(name) do |accessory|
on(accessory.host) do
execute *accessory.remove_service_directory
end
end
end
private
def with_accessory(name)
if accessory = MRSK.accessory(name)
yield accessory
else
error_on_missing_accessory(name)
end
end
def error_on_missing_accessory(name)
options = MRSK.accessory_names.presence
error \
"No accessory by the name of '#{name}'" +
(options ? " (options: #{options.to_sentence})" : "")
end
end

View File

@@ -1,208 +0,0 @@
class Mrsk::Cli::App < Mrsk::Cli::Base
desc "boot", "Boot app on servers (or reboot app if already running)"
def boot
say "Get most recent version available as an image...", :magenta unless options[:version]
using_version(options[:version] || most_recent_version_available) do |version|
say "Start container with version #{version} using a #{MRSK.config.readiness_delay}s readiness delay (or reboot if already running)...", :magenta
cli = self
MRSK.config.roles.each do |role|
on(role.hosts) do |host|
execute *MRSK.auditor.record("Booted app version #{version}"), verbosity: :debug
begin
old_version = capture_with_info(*MRSK.app.current_running_version).strip
execute *MRSK.app.run(role: role.name)
sleep MRSK.config.readiness_delay
execute *MRSK.app.stop(version: old_version), raise_on_non_zero_exit: false if old_version.present?
rescue SSHKit::Command::Failed => e
if e.message =~ /already in use/
error "Rebooting container with same version #{version} already deployed on #{host} (may cause gap in zero-downtime promise!)"
execute *MRSK.auditor.record("Rebooted app version #{version}"), verbosity: :debug
execute *MRSK.app.stop(version: version)
execute *MRSK.app.remove_container(version: version)
execute *MRSK.app.run(role: role.name)
else
raise
end
end
end
end
end
end
desc "start", "Start existing app container on servers"
def start
on(MRSK.hosts) do
execute *MRSK.auditor.record("Started app version #{MRSK.version}"), verbosity: :debug
execute *MRSK.app.start, raise_on_non_zero_exit: false
end
end
desc "stop", "Stop app container on servers"
def stop
on(MRSK.hosts) do
execute *MRSK.auditor.record("Stopped app"), verbosity: :debug
execute *MRSK.app.stop, raise_on_non_zero_exit: false
end
end
# FIXME: Drop in favor of just containers?
desc "details", "Show details about app containers"
def details
on(MRSK.hosts) { |host| puts_by_host host, capture_with_info(*MRSK.app.info) }
end
desc "exec [CMD]", "Execute a custom command on servers (use --help to show options)"
option :interactive, aliases: "-i", type: :boolean, default: false, desc: "Execute command over ssh for an interactive shell (use for console/bash)"
option :reuse, type: :boolean, default: false, desc: "Reuse currently running container instead of starting a new one"
def exec(cmd)
case
when options[:interactive] && options[:reuse]
say "Get current version of running container...", :magenta unless options[:version]
using_version(options[:version] || current_running_version) do |version|
say "Launching interactive command with version #{version} via SSH from existing container on #{MRSK.primary_host}...", :magenta
run_locally { exec MRSK.app.execute_in_existing_container_over_ssh(cmd, host: MRSK.primary_host) }
end
when options[:interactive]
say "Get most recent version available as an image...", :magenta unless options[:version]
using_version(options[:version] || most_recent_version_available) do |version|
say "Launching interactive command with version #{version} via SSH from new container on #{MRSK.primary_host}...", :magenta
run_locally { exec MRSK.app.execute_in_new_container_over_ssh(cmd, host: MRSK.primary_host) }
end
when options[:reuse]
say "Get current version of running container...", :magenta unless options[:version]
using_version(options[:version] || current_running_version) do |version|
say "Launching command with version #{version} from existing container...", :magenta
on(MRSK.hosts) do |host|
execute *MRSK.auditor.record("Executed cmd '#{cmd}' on app version #{version}"), verbosity: :debug
puts_by_host host, capture_with_info(*MRSK.app.execute_in_existing_container(cmd))
end
end
else
say "Get most recent version available as an image...", :magenta unless options[:version]
using_version(options[:version] || most_recent_version_available) do |version|
say "Launching command with version #{version} from new container...", :magenta
on(MRSK.hosts) do |host|
execute *MRSK.auditor.record("Executed cmd '#{cmd}' on app version #{version}"), verbosity: :debug
puts_by_host host, capture_with_info(*MRSK.app.execute_in_new_container(cmd))
end
end
end
end
desc "containers", "Show app containers on servers"
def containers
on(MRSK.hosts) { |host| puts_by_host host, capture_with_info(*MRSK.app.list_containers) }
end
desc "images", "Show app images on servers"
def images
on(MRSK.hosts) { |host| puts_by_host host, capture_with_info(*MRSK.app.list_images) }
end
desc "logs", "Show log lines from app on servers (use --help to show options)"
option :since, aliases: "-s", desc: "Show lines since timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes)"
option :lines, type: :numeric, aliases: "-n", desc: "Number of lines to show from each server"
option :grep, aliases: "-g", desc: "Show lines with grep match only (use this to fetch specific requests by id)"
option :follow, aliases: "-f", desc: "Follow log on primary server (or specific host set by --hosts)"
def logs
# FIXME: Catch when app containers aren't running
grep = options[:grep]
if options[:follow]
run_locally do
info "Following logs on #{MRSK.primary_host}..."
info MRSK.app.follow_logs(host: MRSK.primary_host, grep: grep)
exec MRSK.app.follow_logs(host: MRSK.primary_host, grep: grep)
end
else
since = options[:since]
lines = options[:lines].presence || ((since || grep) ? nil : 100) # Default to 100 lines if since or grep isn't set
on(MRSK.hosts) do |host|
begin
puts_by_host host, capture_with_info(*MRSK.app.logs(since: since, lines: lines, grep: grep))
rescue SSHKit::Command::Failed
puts_by_host host, "Nothing found"
end
end
end
end
desc "remove", "Remove app containers and images from servers"
def remove
stop
remove_containers
remove_images
end
desc "remove_container [VERSION]", "Remove app container with given version from servers", hide: true
def remove_container(version)
on(MRSK.hosts) do
execute *MRSK.auditor.record("Removed app container with version #{version}"), verbosity: :debug
execute *MRSK.app.remove_container(version: version)
end
end
desc "remove_containers", "Remove all app containers from servers", hide: true
def remove_containers
on(MRSK.hosts) do
execute *MRSK.auditor.record("Removed all app containers"), verbosity: :debug
execute *MRSK.app.remove_containers
end
end
desc "remove_images", "Remove all app images from servers", hide: true
def remove_images
on(MRSK.hosts) do
execute *MRSK.auditor.record("Removed all app images"), verbosity: :debug
execute *MRSK.app.remove_images
end
end
desc "version", "Show app version currently running on servers"
def version
on(MRSK.hosts) { |host| puts_by_host host, capture_with_info(*MRSK.app.current_running_version).strip }
end
private
def using_version(new_version)
if new_version
begin
old_version = MRSK.config.version
MRSK.config.version = new_version
yield new_version
ensure
MRSK.config.version = old_version
end
else
yield MRSK.config.version
end
end
def most_recent_version_available(host: MRSK.primary_host)
version = nil
on(host) { version = capture_with_info(*MRSK.app.most_recent_version_from_available_images).strip }
if version == "<none>"
raise "Most recent image available was not tagged with a version (returned <none>)"
else
version.presence
end
end
def current_running_version(host: MRSK.primary_host)
version = nil
on(host) { version = capture_with_info(*MRSK.app.current_running_version).strip }
version.presence
end
end

View File

@@ -1,74 +0,0 @@
require "thor"
require "dotenv"
require "mrsk/sshkit_with_ext"
module Mrsk::Cli
class Base < Thor
include SSHKit::DSL
def self.exit_on_failure?() true end
class_option :verbose, type: :boolean, aliases: "-v", desc: "Detailed logging"
class_option :quiet, type: :boolean, aliases: "-q", desc: "Minimal logging"
class_option :version, desc: "Run commands against a specific app version"
class_option :primary, type: :boolean, aliases: "-p", desc: "Run commands only on primary host instead of all"
class_option :hosts, aliases: "-h", desc: "Run commands on these hosts instead of all (separate by comma)"
class_option :roles, aliases: "-r", desc: "Run commands on these roles instead of all (separate by comma)"
class_option :config_file, aliases: "-c", default: "config/deploy.yml", desc: "Path to config file"
class_option :destination, aliases: "-d", desc: "Specify destination to be used for config file (staging -> deploy.staging.yml)"
class_option :skip_broadcast, aliases: "-B", type: :boolean, default: false, desc: "Skip audit broadcasts"
def initialize(*)
super
load_envs
initialize_commander(options)
end
private
def load_envs
if destination = options[:destination]
Dotenv.load(".env.#{destination}", ".env")
else
Dotenv.load(".env")
end
end
def initialize_commander(options)
MRSK.tap do |commander|
commander.config_file = Pathname.new(File.expand_path(options[:config_file]))
commander.destination = options[:destination]
commander.version = options[:version]
commander.specific_hosts = options[:hosts]&.split(",")
commander.specific_roles = options[:roles]&.split(",")
commander.specific_primary! if options[:primary]
if options[:verbose]
ENV["VERBOSE"] = "1" # For backtraces via cli/start
commander.verbosity = :debug
end
if options[:quiet]
commander.verbosity = :error
end
end
end
def print_runtime
started_at = Time.now
yield
return Time.now - started_at
ensure
runtime = Time.now - started_at
puts " Finished all in #{sprintf("%.1f seconds", runtime)}"
end
def audit_broadcast(line)
run_locally { execute *MRSK.auditor.broadcast(line), verbosity: :debug }
end
end
end

View File

@@ -1,70 +0,0 @@
class Mrsk::Cli::Build < Mrsk::Cli::Base
desc "deliver", "Build app and push app image to registry then pull image on servers"
def deliver
push
pull
end
desc "push", "Build and push app image to registry"
def push
cli = self
run_locally do
begin
MRSK.with_verbosity(:debug) { execute *MRSK.builder.push }
rescue SSHKit::Command::Failed => e
if e.message =~ /(no builder)|(no such file or directory)/
error "Missing compatible builder, so creating a new one first"
if cli.create
MRSK.with_verbosity(:debug) { execute *MRSK.builder.push }
end
else
raise
end
end
end
end
desc "pull", "Pull app image from registry onto servers"
def pull
on(MRSK.hosts) do
execute *MRSK.auditor.record("Pulled image with version #{MRSK.version}"), verbosity: :debug
execute *MRSK.builder.clean, raise_on_non_zero_exit: false
execute *MRSK.builder.pull
end
end
desc "create", "Create a build setup"
def create
run_locally do
begin
debug "Using builder: #{MRSK.builder.name}"
execute *MRSK.builder.create
rescue SSHKit::Command::Failed => e
if e.message =~ /stderr=(.*)/
error "Couldn't create remote builder: #{$1}"
false
else
raise
end
end
end
end
desc "remove", "Remove build setup"
def remove
run_locally do
debug "Using builder: #{MRSK.builder.name}"
execute *MRSK.builder.remove
end
end
desc "details", "Show build setup"
def details
run_locally do
puts "Builder: #{MRSK.builder.name}"
puts capture(*MRSK.builder.info)
end
end
end

View File

@@ -1,50 +0,0 @@
class Mrsk::Cli::Healthcheck < Mrsk::Cli::Base
MAX_ATTEMPTS = 7
class HealthcheckError < StandardError; end
default_command :perform
desc "perform", "Health check current app version"
def perform
on(MRSK.primary_host) do
begin
execute *MRSK.healthcheck.run
target = "Health check against #{MRSK.config.healthcheck["path"]}"
attempt = 1
begin
status = capture_with_info(*MRSK.healthcheck.curl)
if status == "200"
info "#{target} succeeded with 200 OK!"
else
raise HealthcheckError, "#{target} failed with status #{status}"
end
rescue SSHKit::Command::Failed
if attempt <= MAX_ATTEMPTS
info "#{target} failed to respond, retrying in #{attempt}s..."
sleep attempt
attempt += 1
retry
else
raise
end
end
rescue SSHKit::Command::Failed, HealthcheckError => e
error capture_with_info(*MRSK.healthcheck.logs)
if e.message =~ /curl/
raise SSHKit::Command::Failed, "#{target} failed to return 200 OK!"
else
raise
end
ensure
execute *MRSK.healthcheck.stop, raise_on_non_zero_exit: false
execute *MRSK.healthcheck.remove, raise_on_non_zero_exit: false
end
end
end
end

View File

@@ -1,188 +0,0 @@
class Mrsk::Cli::Main < Mrsk::Cli::Base
desc "setup", "Setup all accessories and deploy app to servers"
def setup
print_runtime do
invoke "mrsk:cli:server:bootstrap"
invoke "mrsk:cli:accessory:boot", [ "all" ]
deploy
end
end
desc "deploy", "Deploy app to servers"
def deploy
runtime = print_runtime do
say "Ensure curl and Docker are installed...", :magenta
invoke "mrsk:cli:server:bootstrap"
say "Log into image registry...", :magenta
invoke "mrsk:cli:registry:login"
say "Build and push app image...", :magenta
invoke "mrsk:cli:build:deliver"
say "Ensure Traefik is running...", :magenta
invoke "mrsk:cli:traefik:boot"
say "Ensure app can pass healthcheck...", :magenta
invoke "mrsk:cli:healthcheck:perform"
invoke "mrsk:cli:app:boot"
say "Prune old containers and images...", :magenta
invoke "mrsk:cli:prune:all"
end
audit_broadcast "Deployed app in #{runtime.to_i} seconds" unless options[:skip_broadcast]
end
desc "redeploy", "Deploy app to servers without bootstrapping servers, starting Traefik, pruning, and registry login"
def redeploy
runtime = print_runtime do
say "Build and push app image...", :magenta
invoke "mrsk:cli:build:deliver"
say "Ensure app can pass healthcheck...", :magenta
invoke "mrsk:cli:healthcheck:perform"
invoke "mrsk:cli:app:boot"
end
audit_broadcast "Redeployed app in #{runtime.to_i} seconds" unless options[:skip_broadcast]
end
desc "rollback [VERSION]", "Rollback app to VERSION"
def rollback(version)
MRSK.version = version
if container_name_available?(MRSK.config.service_with_version)
say "Start version #{version}, then wait #{MRSK.config.readiness_delay}s for app to boot before stopping the old version...", :magenta
cli = self
on(MRSK.hosts) do |host|
old_version = capture_with_info(*MRSK.app.current_running_version).strip.presence
execute *MRSK.app.start
sleep MRSK.config.readiness_delay
execute *MRSK.app.stop(version: old_version), raise_on_non_zero_exit: false
end
audit_broadcast "Rolled back app to version #{version}" unless options[:skip_broadcast]
else
say "The app version '#{version}' is not available as a container (use 'mrsk app containers' for available versions)", :red
end
end
desc "details", "Show details about all containers"
def details
invoke "mrsk:cli:traefik:details"
invoke "mrsk:cli:app:details"
invoke "mrsk:cli:accessory:details", [ "all" ]
end
desc "audit", "Show audit log from servers"
def audit
on(MRSK.hosts) do |host|
puts_by_host host, capture_with_info(*MRSK.auditor.reveal)
end
end
desc "config", "Show combined config (including secrets!)"
def config
run_locally do
puts MRSK.config.to_h.to_yaml
end
end
desc "init", "Create config stub in config/deploy.yml and env stub in .env"
option :bundle, type: :boolean, default: false, desc: "Add MRSK to the Gemfile and create a bin/mrsk binstub"
def init
require "fileutils"
if (deploy_file = Pathname.new(File.expand_path("config/deploy.yml"))).exist?
puts "Config file already exists in config/deploy.yml (remove first to create a new one)"
else
FileUtils.mkdir_p deploy_file.dirname
FileUtils.cp_r Pathname.new(File.expand_path("templates/deploy.yml", __dir__)), deploy_file
puts "Created configuration file in config/deploy.yml"
end
unless (deploy_file = Pathname.new(File.expand_path(".env"))).exist?
FileUtils.cp_r Pathname.new(File.expand_path("templates/template.env", __dir__)), deploy_file
puts "Created .env file"
end
if options[:bundle]
if (binstub = Pathname.new(File.expand_path("bin/mrsk"))).exist?
puts "Binstub already exists in bin/mrsk (remove first to create a new one)"
else
puts "Adding MRSK to Gemfile and bundle..."
`bundle add mrsk`
`bundle binstubs mrsk`
puts "Created binstub file in bin/mrsk"
end
end
end
desc "envify", "Create .env by evaluating .env.erb (or .env.staging.erb -> .env.staging when using -d staging)"
def envify
if destination = options[:destination]
env_template_path = ".env.#{destination}.erb"
env_path = ".env.#{destination}"
else
env_template_path = ".env.erb"
env_path = ".env"
end
File.write(env_path, ERB.new(File.read(env_template_path)).result, perm: 0600)
end
desc "remove", "Remove Traefik, app, accessories, and registry session from servers"
option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
def remove
if options[:confirmed] || ask("This will remove all containers and images. Are you sure?", limited_to: %w( y N ), default: "N") == "y"
invoke "mrsk:cli:traefik:remove", [], options.without(:confirmed)
invoke "mrsk:cli:app:remove", [], options.without(:confirmed)
invoke "mrsk:cli:accessory:remove", [ "all" ], options
invoke "mrsk:cli:registry:logout", [], options.without(:confirmed)
end
end
desc "version", "Show MRSK version"
def version
puts Mrsk::VERSION
end
desc "accessory", "Manage accessories (db/redis/search)"
subcommand "accessory", Mrsk::Cli::Accessory
desc "app", "Manage application"
subcommand "app", Mrsk::Cli::App
desc "build", "Build application image"
subcommand "build", Mrsk::Cli::Build
desc "healthcheck", "Healthcheck application"
subcommand "healthcheck", Mrsk::Cli::Healthcheck
desc "prune", "Prune old application images and containers"
subcommand "prune", Mrsk::Cli::Prune
desc "registry", "Login and -out of the image registry"
subcommand "registry", Mrsk::Cli::Registry
desc "server", "Bootstrap servers with curl and Docker"
subcommand "server", Mrsk::Cli::Server
desc "traefik", "Manage Traefik load balancer"
subcommand "traefik", Mrsk::Cli::Traefik
private
def container_name_available?(container_name, host: MRSK.primary_host)
container_names = nil
on(host) { container_names = capture_with_info(*MRSK.app.list_container_names).split("\n") }
Array(container_names).include?(container_name)
end
end

View File

@@ -1,23 +0,0 @@
class Mrsk::Cli::Prune < Mrsk::Cli::Base
desc "all", "Prune unused images and stopped containers"
def all
containers
images
end
desc "images", "Prune unused images older than 7 days"
def images
on(MRSK.hosts) do
execute *MRSK.auditor.record("Pruned images"), verbosity: :debug
execute *MRSK.prune.images
end
end
desc "containers", "Prune stopped containers older than 3 days"
def containers
on(MRSK.hosts) do
execute *MRSK.auditor.record("Pruned containers"), verbosity: :debug
execute *MRSK.prune.containers
end
end
end

View File

@@ -1,15 +0,0 @@
class Mrsk::Cli::Server < Mrsk::Cli::Base
desc "bootstrap", "Ensure curl and Docker are installed on servers"
def bootstrap
on(MRSK.hosts + MRSK.accessory_hosts) do
dependencies_to_install = Array.new.tap do |dependencies|
dependencies << "curl" unless execute "which curl", raise_on_non_zero_exit: false
dependencies << "docker.io" unless execute "which docker", raise_on_non_zero_exit: false
end
if dependencies_to_install.any?
execute "apt-get update -y && apt-get install #{dependencies_to_install.join(" ")} -y)"
end
end
end
end

View File

@@ -1,2 +0,0 @@
MRSK_REGISTRY_PASSWORD=change-this
RAILS_MASTER_KEY=another-env

View File

@@ -1,87 +0,0 @@
class Mrsk::Cli::Traefik < Mrsk::Cli::Base
desc "boot", "Boot Traefik on servers"
def boot
on(MRSK.traefik_hosts) { execute *MRSK.traefik.run, raise_on_non_zero_exit: false }
end
desc "reboot", "Reboot Traefik on servers (stop container, remove container, start new container)"
def reboot
stop
remove_container
boot
end
desc "start", "Start existing Traefik container on servers"
def start
on(MRSK.traefik_hosts) do
execute *MRSK.auditor.record("Started traefik"), verbosity: :debug
execute *MRSK.traefik.start, raise_on_non_zero_exit: false
end
end
desc "stop", "Stop existing Traefik container on servers"
def stop
on(MRSK.traefik_hosts) do
execute *MRSK.auditor.record("Stopped traefik"), verbosity: :debug
execute *MRSK.traefik.stop, raise_on_non_zero_exit: false
end
end
desc "restart", "Restart existing Traefik container on servers"
def restart
stop
start
end
desc "details", "Show details about Traefik container from servers"
def details
on(MRSK.traefik_hosts) { |host| puts_by_host host, capture_with_info(*MRSK.traefik.info), type: "Traefik" }
end
desc "logs", "Show log lines from Traefik on servers"
option :since, aliases: "-s", desc: "Show logs since timestamp (e.g. 2013-01-02T13:23:37Z) or relative (e.g. 42m for 42 minutes)"
option :lines, type: :numeric, aliases: "-n", desc: "Number of log lines to pull from each server"
option :grep, aliases: "-g", desc: "Show lines with grep match only (use this to fetch specific requests by id)"
option :follow, aliases: "-f", desc: "Follow logs on primary server (or specific host set by --hosts)"
def logs
grep = options[:grep]
if options[:follow]
run_locally do
info "Following logs on #{MRSK.primary_host}..."
info MRSK.traefik.follow_logs(host: MRSK.primary_host, grep: grep)
exec MRSK.traefik.follow_logs(host: MRSK.primary_host, grep: grep)
end
else
since = options[:since]
lines = options[:lines].presence || ((since || grep) ? nil : 100) # Default to 100 lines if since or grep isn't set
on(MRSK.traefik_hosts) do |host|
puts_by_host host, capture(*MRSK.traefik.logs(since: since, lines: lines, grep: grep)), type: "Traefik"
end
end
end
desc "remove", "Remove Traefik container and image from servers"
def remove
stop
remove_container
remove_image
end
desc "remove_container", "Remove Traefik container from servers", hide: true
def remove_container
on(MRSK.traefik_hosts) do
execute *MRSK.auditor.record("Removed traefik container"), verbosity: :debug
execute *MRSK.traefik.remove_container
end
end
desc "remove_container", "Remove Traefik image from servers", hide: true
def remove_image
on(MRSK.traefik_hosts) do
execute *MRSK.auditor.record("Removed traefik image"), verbosity: :debug
execute *MRSK.traefik.remove_image
end
end
end

View File

@@ -1,119 +0,0 @@
require "active_support/core_ext/enumerable"
class Mrsk::Commander
attr_accessor :config_file, :destination, :verbosity, :version
def initialize(config_file: nil, destination: nil, verbosity: :info)
@config_file, @destination, @verbosity = config_file, destination, verbosity
end
def config
@config ||= \
Mrsk::Configuration
.create_from(config_file, destination: destination, version: cascading_version)
.tap { |config| configure_sshkit_with(config) }
end
attr_accessor :specific_hosts
def specific_primary!
self.specific_hosts = [ config.primary_web_host ]
end
def specific_roles=(role_names)
self.specific_hosts = config.roles.select { |r| role_names.include?(r.name) }.flat_map(&:hosts) if role_names.present?
end
def primary_host
specific_hosts&.first || config.primary_web_host
end
def hosts
specific_hosts || config.all_hosts
end
def traefik_hosts
specific_hosts || config.traefik_hosts
end
def accessory_hosts
specific_hosts || config.accessories.collect(&:host)
end
def accessory_names
config.accessories&.collect(&:name) || []
end
def app
@app ||= Mrsk::Commands::App.new(config)
end
def accessory(name)
Mrsk::Commands::Accessory.new(config, name: name)
end
def auditor
@auditor ||= Mrsk::Commands::Auditor.new(config)
end
def builder
@builder ||= Mrsk::Commands::Builder.new(config)
end
def healthcheck
@healthcheck ||= Mrsk::Commands::Healthcheck.new(config)
end
def prune
@prune ||= Mrsk::Commands::Prune.new(config)
end
def registry
@registry ||= Mrsk::Commands::Registry.new(config)
end
def traefik
@traefik ||= Mrsk::Commands::Traefik.new(config)
end
def with_verbosity(level)
old_level = self.verbosity
self.verbosity = level
SSHKit.config.output_verbosity = level
yield
ensure
self.verbosity = old_level
SSHKit.config.output_verbosity = old_level
end
# Test-induced damage!
def reset
@config = @config_file = @destination = @version = nil
@app = @builder = @traefik = @registry = @prune = @auditor = nil
@verbosity = :info
end
private
def cascading_version
version.presence || ENV["VERSION"] || current_commit_hash
end
def current_commit_hash
if system("git rev-parse")
`git rev-parse HEAD`.strip
else
raise "Can't use commit hash as version, no git repository found in #{Dir.pwd}"
end
end
# Lazy setup of SSHKit
def configure_sshkit_with(config)
SSHKit::Backend::Netssh.configure { |ssh| ssh.ssh_options = config.ssh_options }
SSHKit.config.command_map[:docker] = "docker" # No need to use /usr/bin/env, just clogs up the logs
SSHKit.config.output_verbosity = verbosity
end
end

Some files were not shown because too many files have changed in this diff Show More