Upgrading kamal from `v1.8.3` to `v1.9.0` broke my [kamal playground](https://labs.iximiuz.com/playgrounds/kamal):
```
laborant@dev-machine:~/svc-a$ kamal setup
INFO [34d0def6] Running /usr/bin/env mkdir -p .kamal on 172.16.0.3
INFO [c34cf833] Running /usr/bin/env mkdir -p .kamal on 172.16.0.4
INFO [34d0def6] Finished in 0.147 seconds with exit status 0 (successful).
INFO [c34cf833] Finished in 0.204 seconds with exit status 0 (successful).
Acquiring the deploy lock...
Ensure Docker is installed...
INFO [413ee426] Running docker -v on 172.16.0.4
INFO [f1acacba] Running docker -v on 172.16.0.3
INFO [413ee426] Finished in 0.036 seconds with exit status 0 (successful).
INFO [f1acacba] Finished in 0.076 seconds with exit status 0 (successful).
Log into image registry...
INFO [94cff492] Running docker login registry.iximiuz.com -u [REDACTED] -p [REDACTED] on localhost
INFO [94cff492] Finished in 0.077 seconds with exit status 0 (successful).
INFO [605c535f] Running docker login registry.iximiuz.com -u [REDACTED] -p [REDACTED] on 172.16.0.4
INFO [6002b598] Running docker login registry.iximiuz.com -u [REDACTED] -p [REDACTED] on 172.16.0.3
INFO [605c535f] Finished in 0.083 seconds with exit status 0 (successful).
INFO [6002b598] Finished in 0.083 seconds with exit status 0 (successful).
Build and push app image...
INFO [9d172b1e] Running docker --version && docker buildx version on localhost
INFO [9d172b1e] Finished in 0.059 seconds with exit status 0 (successful).
INFO Cloning repo into build directory `/tmp/kamal-clones/svc-a-2f65914456263/workdir/`...
INFO [26fb1bd3] Running /usr/bin/env git -C /tmp/kamal-clones/svc-a-2f65914456263 clone /workdir --recurse-submodules on localhost
ERROR Error preparing clone: Failed to clone repo: git exit status: 32768
git stdout: Nothing written
git stderr: Cloning into 'workdir'...
fatal: detected dubious ownership in repository at '/workdir/.git'
To add an exception for this directory, call:
git config --global --add safe.directory /workdir/.git
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
, deleting and retrying...
INFO Cloning repo into build directory `/tmp/kamal-clones/svc-a-2f65914456263/workdir/`...
INFO [fd4aac0c] Running /usr/bin/env git -C /tmp/kamal-clones/svc-a-2f65914456263 clone /workdir --recurse-submodules on localhost
Finished all in 0.3 seconds
Releasing the deploy lock...
Finished all in 0.6 seconds
ERROR (SSHKit::Command::Failed): git exit status: 32768
git stdout: Nothing written
git stderr: Cloning into 'workdir'...
fatal: detected dubious ownership in repository at '/workdir/.git'
To add an exception for this directory, call:
git config --global --add safe.directory /workdir/.git
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
laborant@dev-machine:~/svc-a$ kamal version
2.0.0
```
I checked the [v1.8.3...v1.9.0](https://github.com/basecamp/kamal/compare/v1.8.3...v1.9.0) diff, and couldn't find anything even remotely related to the above error.
Then I checked the `git` versions in kamal `v1.8.3` and `v1.9.0` images:
```
docker run -it --rm --entrypoint sh ghcr.io/basecamp/kamal:v1.8.3
/workdir # git --version
git version 2.38.5
```
vs.
```
docker run -it --rm --entrypoint sh ghcr.io/basecamp/kamal:v2.0.0
/workdir # git --version
git version 2.39.5
```
Apparently, something changed in between `2.38.5` and `2.39.5` git releases (likely yet another CVE fix), and the `git config --global --add safe.directory /workdir` stopped working.
Here is the mitigation I currently use, but it's a bit awkward to do it:
```
docker build -t ghcr.io/basecamp/kamal:v2.0.0 - <<EOF
FROM ghcr.io/basecamp/kamal:v2.0.0
RUN git config --global --add safe.directory /workdir/.git
EOF
```
Hence, this PR.
To repro, you can start a [kamal playground](https://labs.iximiuz.com/playgrounds/kamal), then `docker pull ghcr.io/basecamp/kamal:v2.0.0` to override my patched image, and `cd svc-a && kamal setup`.
Add a downgrade command, so you can reverse the upgrade process and go
back to Kamal 1.9. This replaces kamal-proxy and reboots all the
accessories.
This gives an upgrade and downgrade path:
Upgrade:
1. Upgrade config to be Kamal 2 compatible + use kamal 2.0
2. Run `kamal upgrade`
Downgrade:
1. Switch back to previous config + use kamal 1.9
2. Run `kamal downgrade`
You can set `--rolling` to downgrade one host at a time.
The integrations tests use their own registry so avoid hitting docker
hub rate limits.
This was using a self signed certificate but instead use
`--insecure-registry` to let the docker daemon use HTTP.
Allow blocks prefixed with `x-` in the configuration as a place to
declare reusable blocks with YAML anchors and aliases.
Borrowed from the Docker Compose configuration file format -
https://github.com/compose-spec/compose-spec/blob/main/spec.md#extension
Thanks to @ruyrocha for the suggestion.
Find the first registry mirror on each host. If we find any, pull the
images on one host per mirror, then do the remainder concurrently.
The initial pulls will seed the mirrors ensuring that we pull the image
from Docker Hub once each.
This works best if there is only one mirror on each host.
