Merge branch 'main' into pr/99

* main:
  Ask for access token
  Style
  Style
  config.traefik is already nil safe
  Update README.md
  Bump dev deps and consolidate platform matches
  Deploys mention the released service@version
  Accessories aren't required to publish a port
  Accessories may be pulled from authenticated registries
  Polish destination config loading
  Allow arbitrary docker options for traefik
  Fixed typos
  Fixed readme
  Rebased on main
  Added volume configuration in response to issue coments
  Modified in response to PR comments
  Added the additional_ports configuration
This commit is contained in:
David Heinemeier Hansson
2023-03-23 14:48:13 +01:00
14 changed files with 192 additions and 68 deletions

View File

@@ -13,90 +13,79 @@ PATH
GEM
remote: https://rubygems.org/
specs:
actionpack (7.0.4)
actionview (= 7.0.4)
activesupport (= 7.0.4)
actionpack (7.0.4.3)
actionview (= 7.0.4.3)
activesupport (= 7.0.4.3)
rack (~> 2.0, >= 2.2.0)
rack-test (>= 0.6.3)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.0, >= 1.2.0)
actionview (7.0.4)
activesupport (= 7.0.4)
actionview (7.0.4.3)
activesupport (= 7.0.4.3)
builder (~> 3.1)
erubi (~> 1.4)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.1, >= 1.2.0)
activesupport (7.0.4)
activesupport (7.0.4.3)
concurrent-ruby (~> 1.0, >= 1.0.2)
i18n (>= 1.6, < 2)
minitest (>= 5.1)
tzinfo (~> 2.0)
bcrypt_pbkdf (1.1.0)
builder (3.2.4)
concurrent-ruby (1.1.10)
concurrent-ruby (1.2.2)
crass (1.0.6)
debug (1.7.1)
irb (>= 1.5.0)
reline (>= 0.3.1)
dotenv (2.8.1)
ed25519 (1.3.0)
erubi (1.12.0)
i18n (1.12.0)
concurrent-ruby (~> 1.0)
io-console (0.6.0)
irb (1.6.2)
reline (>= 0.3.0)
loofah (2.19.1)
crass (~> 1.0.2)
nokogiri (>= 1.5.9)
method_source (1.0.0)
minitest (5.17.0)
minitest (5.18.0)
mocha (2.0.2)
ruby2_keywords (>= 0.0.5)
net-scp (4.0.0)
net-ssh (>= 2.6.5, < 8.0.0)
net-ssh (7.0.1)
nokogiri (1.14.0-arm64-darwin)
net-ssh (7.1.0)
nokogiri (1.14.2-arm64-darwin)
racc (~> 1.4)
nokogiri (1.14.0-x86_64-darwin)
nokogiri (1.14.2-x86_64-darwin)
racc (~> 1.4)
nokogiri (1.14.0-x86_64-linux)
nokogiri (1.14.2-x86_64-linux)
racc (~> 1.4)
racc (1.6.2)
rack (2.2.5)
rack-test (2.0.2)
rack (2.2.6.4)
rack-test (2.1.0)
rack (>= 1.3)
rails-dom-testing (2.0.3)
activesupport (>= 4.2.0)
nokogiri (>= 1.6)
rails-html-sanitizer (1.4.4)
rails-html-sanitizer (1.5.0)
loofah (~> 2.19, >= 2.19.1)
railties (7.0.4)
actionpack (= 7.0.4)
activesupport (= 7.0.4)
railties (7.0.4.3)
actionpack (= 7.0.4.3)
activesupport (= 7.0.4.3)
method_source
rake (>= 12.2)
thor (~> 1.0)
zeitwerk (~> 2.5)
rake (13.0.6)
reline (0.3.2)
io-console (~> 0.5)
ruby2_keywords (0.0.5)
sshkit (1.21.3)
sshkit (1.21.4)
net-scp (>= 1.1.2)
net-ssh (>= 2.8.0)
thor (1.2.1)
tzinfo (2.0.5)
tzinfo (2.0.6)
concurrent-ruby (~> 1.0)
zeitwerk (2.6.6)
zeitwerk (2.6.7)
PLATFORMS
arm64-darwin-20
arm64-darwin-21
arm64-darwin-22
x86_64-darwin-20
x86_64-darwin-21
x86_64-darwin-22
arm64-darwin
x86_64-darwin
x86_64-linux
DEPENDENCIES

View File

@@ -4,7 +4,7 @@ MRSK deploys web apps anywhere from bare metal to cloud VMs using Docker with ze
Watch the screencast: https://www.youtube.com/watch?v=LL1cV2FXZ5I
Join us on Discord: https://discord.gg/DQETs3Pm
Join us on Discord: https://discord.gg/YgHVT7GCXS
## Installation
@@ -426,6 +426,46 @@ traefik:
host_port: 8080
```
### Configure docker options for traefik
We allow users to pass additional docker options to the trafik container like
```yaml
traefik:
options:
publish:
- 8080:8080
volumes:
- /tmp/example.json:/tmp/example.json
memory: 512m
```
This will start the traefik container with a command like: `docker run ... --volume /tmp/example.json:/tmp/example.json --publish 8080:8080 `
### Configure alternate entrypoints for traefik
You can configure multiple entrypoints for traefik like so:
```yaml
service: myservice
labels:
traefik.tcp.routers.other.rule: 'HostSNI(`*`)'
traefik.tcp.routers.other.entrypoints: otherentrypoint
traefik.tcp.services.other.loadbalancer.server.port: 9000
traefik.http.routers.myservice.entrypoints: web
traefik.http.services.myservice.loadbalancer.server.port: 8080
traefik:
options:
publish:
- 9000:9000
args:
entrypoints.web.address: ':80'
entrypoints.otherentrypoint.address: ':9000'
```
### Configuring build args for new images
Build arguments that aren't secret can also be configured:
@@ -445,7 +485,7 @@ FROM ruby:$RUBY_VERSION-slim as base
### Using accessories for database, cache, search services
You can manage your accessory services via MRSK as well. The services will build off public images, and will not be automatically updated when you deploy:
You can manage your accessory services via MRSK as well. Accessories are long-lived services that your app depends on. They are not updated when you deploy.
```yaml
accessories:
@@ -466,10 +506,16 @@ accessories:
port: "36379:6379"
volumes:
- /var/lib/redis:/data
internal-example:
image: registry.digitalocean.com/user/otherservice:latest
host: 1.1.1.5
port: 44444
```
Now run `mrsk accessory start mysql` to start the MySQL server on 1.1.1.3. See `mrsk accessory` for all the commands possible.
Accessory images must be public or tagged in your private registry.
### Using Cron
You can use a specific container to run your Cron jobs:

View File

@@ -9,6 +9,7 @@ class Mrsk::Cli::Accessory < Mrsk::Cli::Base
upload(name)
on(accessory.host) do
execute *MRSK.registry.login
execute *MRSK.auditor.record("Booted #{name} accessory"), verbosity: :debug
execute *accessory.run
end

View File

@@ -40,7 +40,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
invoke "mrsk:cli:prune:all", [], invoke_options
end
audit_broadcast "Deployed app in #{runtime.to_i} seconds" unless options[:skip_broadcast]
audit_broadcast "Deployed #{service_version} in #{runtime.round} seconds" unless options[:skip_broadcast]
end
desc "redeploy", "Deploy app to servers without bootstrapping servers, starting Traefik, pruning, and registry login"
@@ -63,7 +63,7 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
invoke "mrsk:cli:app:boot", [], invoke_options
end
audit_broadcast "Redeployed app in #{runtime.to_i} seconds" unless options[:skip_broadcast]
audit_broadcast "Redeployed #{service_version} in #{runtime.round} seconds" unless options[:skip_broadcast]
end
desc "rollback [VERSION]", "Rollback app to VERSION"
@@ -74,18 +74,21 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
say "Start version #{version}, then wait #{MRSK.config.readiness_delay}s for app to boot before stopping the old version...", :magenta
cli = self
old_version = nil
on(MRSK.hosts) do |host|
old_version = capture_with_info(*MRSK.app.current_running_version).strip.presence
execute *MRSK.app.start
sleep MRSK.config.readiness_delay
if old_version
sleep MRSK.config.readiness_delay
execute *MRSK.app.stop(version: old_version), raise_on_non_zero_exit: false
execute *MRSK.app.stop(version: old_version), raise_on_non_zero_exit: false
end
end
audit_broadcast "Rolled back app to version #{version}" unless options[:skip_broadcast]
audit_broadcast "Rolled back #{service_version(Mrsk::Utils.abbreviate_version(old_version))} to #{service_version}" unless options[:skip_broadcast]
else
say "The app version '#{version}' is not available as a container (use 'mrsk app containers' for available versions)", :red
end
@@ -203,4 +206,8 @@ class Mrsk::Cli::Main < Mrsk::Cli::Base
on(host) { container_names = capture_with_info(*MRSK.app.list_container_names).split("\n") }
Array(container_names).include?(container_name)
end
def service_version(version = MRSK.config.abbreviated_version)
[ MRSK.config.service, version ].compact.join("@")
end
end

View File

@@ -13,6 +13,8 @@ registry:
# Specify the registry server, if you're not using Docker Hub
# server: registry.digitalocean.com / ghcr.io / ...
username: my-user
# Always use an access token rather than real password when possible.
password:
- MRSK_REGISTRY_PASSWORD

View File

@@ -1,6 +1,6 @@
class Mrsk::Commands::Accessory < Mrsk::Commands::Base
attr_reader :accessory_config
delegate :service_name, :image, :host, :port, :files, :directories, :env_args, :volume_args, :label_args, to: :accessory_config
delegate :service_name, :image, :host, :port, :files, :directories, :publish_args, :env_args, :volume_args, :label_args, to: :accessory_config
def initialize(config, name:)
super(config)
@@ -13,7 +13,7 @@ class Mrsk::Commands::Accessory < Mrsk::Commands::Base
"--detach",
"--restart", "unless-stopped",
"--log-opt", "max-size=#{MAX_LOG_SIZE}",
"--publish", port,
*publish_args,
*env_args,
*volume_args,
*label_args,

View File

@@ -10,6 +10,7 @@ class Mrsk::Commands::Traefik < Mrsk::Commands::Base
"--log-opt", "max-size=#{MAX_LOG_SIZE}",
"--publish", port,
"--volume", "/var/run/docker.sock:/var/run/docker.sock",
*docker_options_args,
"traefik",
"--providers.docker",
"--log.level=DEBUG",
@@ -54,6 +55,10 @@ class Mrsk::Commands::Traefik < Mrsk::Commands::Base
end
private
def docker_options_args
optionize(config.traefik["options"] || {})
end
def cmd_option_args
if args = config.traefik["args"]
optionize args, with: "="

View File

@@ -15,15 +15,16 @@ class Mrsk::Configuration
class << self
def create_from(base_config_file, destination: nil, version: "missing")
new(load_config_file(base_config_file).tap do |config|
if destination
config.deep_merge! \
load_config_file destination_config_file(base_config_file, destination)
end
end, destination: destination, version: version)
raw_config = load_config_files(base_config_file, *destination_config_file(base_config_file, destination))
new raw_config, destination: destination, version: version
end
private
def load_config_files(*files)
files.inject({}) { |config, file| config.deep_merge! load_config_file(file) }
end
def load_config_file(file)
if file.exist?
YAML.load(ERB.new(IO.read(file)).result).symbolize_keys
@@ -33,8 +34,7 @@ class Mrsk::Configuration
end
def destination_config_file(base_config_file, destination)
dir, basename = base_config_file.split
dir.join basename.to_s.remove(".yml") + ".#{destination}.yml"
base_config_file.sub_ext(".#{destination}.yml") if destination
end
end
@@ -46,6 +46,11 @@ class Mrsk::Configuration
end
def abbreviated_version
Mrsk::Utils.abbreviate_version(version)
end
def roles
@roles ||= role_names.collect { |role_name| Role.new(role_name, config: self) }
end

View File

@@ -20,13 +20,15 @@ class Mrsk::Configuration::Accessory
end
def port
if specifics["port"].to_s.include?(":")
specifics["port"]
else
"#{specifics["port"]}:#{specifics["port"]}"
if port = specifics["port"]&.to_s
port.include?(":") ? port : "#{port}:#{port}"
end
end
def publish_args
argumentize "--publish", port if port
end
def labels
default_labels.merge(specifics["labels"] || {})
end

View File

@@ -26,14 +26,19 @@ module Mrsk::Utils
# Returns a list of shell-dashed option arguments. If the value is true, it's treated like a value-less option.
def optionize(args, with: nil)
options = if with
args.collect { |(key, value)| value == true ? "--#{key}" : "--#{key}#{with}#{escape_shell_value(value)}" }
flatten_args(args).collect { |(key, value)| value == true ? "--#{key}" : "--#{key}#{with}#{escape_shell_value(value)}" }
else
args.collect { |(key, value)| [ "--#{key}", value == true ? nil : escape_shell_value(value) ] }
flatten_args(args).collect { |(key, value)| [ "--#{key}", value == true ? nil : escape_shell_value(value) ] }
end
options.flatten.compact
end
# Flattens a one-to-many structure into an array of two-element arrays each containing a key-value pair
def flatten_args(args)
args.flat_map { |key, value| value.try(:map) { |entry| [key, entry] } || [ [ key, value ] ] }
end
# Copied from SSHKit::Backend::Abstract#redact to be available inside Commands classes
def redact(arg) # Used in execute_command to hide redact() args a user passes in
arg.to_s.extend(SSHKit::Redaction) # to_s due to our inability to extend Integer, etc
@@ -43,4 +48,9 @@ module Mrsk::Utils
def escape_shell_value(value)
value.to_s.dump.gsub(/`/, '\\\\`')
end
# Abbreviate a git revhash for concise display
def abbreviate_version(version)
version[0...7] if version
end
end

View File

@@ -6,6 +6,7 @@ class CliAccessoryTest < CliTestCase
Mrsk::Cli::Accessory.any_instance.expects(:upload).with("mysql")
run_command("boot", "mysql").tap do |output|
assert_match /docker login.*on 1.1.1.3/, output
assert_match "docker run --name app-mysql --detach --restart unless-stopped --log-opt max-size=10m --publish 3306:3306 -e [REDACTED] -e MYSQL_ROOT_HOST=\"%\" --volume $PWD/app-mysql/etc/mysql/my.cnf:/etc/mysql/my.cnf --volume $PWD/app-mysql/data:/var/lib/mysql --label service=\"app-mysql\" mysql:5.7 on 1.1.1.3", output
end
end
@@ -17,6 +18,8 @@ class CliAccessoryTest < CliTestCase
Mrsk::Cli::Accessory.any_instance.expects(:upload).with("redis")
run_command("boot", "all").tap do |output|
assert_match /docker login.*on 1.1.1.3/, output
assert_match /docker login.*on 1.1.1.4/, output
assert_match "docker run --name app-mysql --detach --restart unless-stopped --log-opt max-size=10m --publish 3306:3306 -e [REDACTED] -e MYSQL_ROOT_HOST=\"%\" --volume $PWD/app-mysql/etc/mysql/my.cnf:/etc/mysql/my.cnf --volume $PWD/app-mysql/data:/var/lib/mysql --label service=\"app-mysql\" mysql:5.7 on 1.1.1.3", output
assert_match "docker run --name app-redis --detach --restart unless-stopped --log-opt max-size=10m --publish 6379:6379 --volume $PWD/app-redis/data:/data --label service=\"app-redis\" redis:latest on 1.1.1.4", output
end

View File

@@ -88,11 +88,23 @@ class CliMainTest < CliTestCase
test "rollback good version" do
Mrsk::Cli::Main.any_instance.stubs(:container_name_available?).returns(true)
SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info).with(:docker, :ps, "--filter", "label=service=app", "--format", "\"{{.Names}}\"", "|", "sed 's/-/\\n/g'", "|", "tail -n 1").returns("version-to-rollback\n").times(2)
run_command("rollback", "123").tap do |output|
assert_match /Start version 123/, output
assert_match /docker ps -q --filter label=service=app | xargs docker stop/, output
assert_match /docker start app-123/, output
assert_match "Start version 123", output
assert_match "docker start app-123", output
assert_match "docker container ls --all --filter name=app-version-to-rollback --quiet | xargs docker stop", output, "Should stop the container that was previously running"
end
end
test "rollback without old version" do
Mrsk::Cli::Main.any_instance.stubs(:container_name_available?).returns(true)
SSHKit::Backend::Abstract.any_instance.expects(:capture_with_info).with(:docker, :ps, "--filter", "label=service=app", "--format", "\"{{.Names}}\"", "|", "sed 's/-/\\n/g'", "|", "tail -n 1").returns("").times(2)
run_command("rollback", "123").tap do |output|
assert_match "Start version 123", output
assert_match "docker start app-123", output
assert_no_match "docker stop", output
end
end

View File

@@ -3,11 +3,11 @@ require "test_helper"
class CommandsAccessoryTest < ActiveSupport::TestCase
setup do
@config = {
service: "app", image: "dhh/app", registry: { "username" => "dhh", "password" => "secret" },
service: "app", image: "dhh/app", registry: { "server" => "private.registry", "username" => "dhh", "password" => "secret" },
servers: [ "1.1.1.1" ],
accessories: {
"mysql" => {
"image" => "mysql:8.0",
"image" => "private.registry/mysql:8.0",
"host" => "1.1.1.5",
"port" => "3306",
"env" => {
@@ -32,13 +32,18 @@ class CommandsAccessoryTest < ActiveSupport::TestCase
"volumes" => [
"/var/lib/redis:/data"
]
},
"busybox" => {
"image" => "busybox:latest",
"host" => "1.1.1.7"
}
}
}
@config = Mrsk::Configuration.new(@config)
@mysql = Mrsk::Commands::Accessory.new(@config, name: :mysql)
@redis = Mrsk::Commands::Accessory.new(@config, name: :redis)
@config = Mrsk::Configuration.new(@config)
@mysql = Mrsk::Commands::Accessory.new(@config, name: :mysql)
@redis = Mrsk::Commands::Accessory.new(@config, name: :redis)
@busybox = Mrsk::Commands::Accessory.new(@config, name: :busybox)
ENV["MYSQL_ROOT_PASSWORD"] = "secret123"
end
@@ -49,12 +54,16 @@ class CommandsAccessoryTest < ActiveSupport::TestCase
test "run" do
assert_equal \
"docker run --name app-mysql --detach --restart unless-stopped --log-opt max-size=10m --publish 3306:3306 -e MYSQL_ROOT_PASSWORD=\"secret123\" -e MYSQL_ROOT_HOST=\"%\" --label service=\"app-mysql\" mysql:8.0",
"docker run --name app-mysql --detach --restart unless-stopped --log-opt max-size=10m --publish 3306:3306 -e MYSQL_ROOT_PASSWORD=\"secret123\" -e MYSQL_ROOT_HOST=\"%\" --label service=\"app-mysql\" private.registry/mysql:8.0",
@mysql.run.join(" ")
assert_equal \
"docker run --name app-redis --detach --restart unless-stopped --log-opt max-size=10m --publish 6379:6379 -e SOMETHING=\"else\" --volume /var/lib/redis:/data --label service=\"app-redis\" --label cache=\"true\" redis:latest",
@redis.run.join(" ")
assert_equal \
"docker run --name app-busybox --detach --restart unless-stopped --log-opt max-size=10m --label service=\"app-busybox\" busybox:latest",
@busybox.run.join(" ")
end
test "start" do
@@ -78,7 +87,7 @@ class CommandsAccessoryTest < ActiveSupport::TestCase
test "execute in new container" do
assert_equal \
"docker run --rm -e MYSQL_ROOT_PASSWORD=\"secret123\" -e MYSQL_ROOT_HOST=\"%\" mysql:8.0 mysql -u root",
"docker run --rm -e MYSQL_ROOT_PASSWORD=\"secret123\" -e MYSQL_ROOT_HOST=\"%\" private.registry/mysql:8.0 mysql -u root",
@mysql.execute_in_new_container("mysql", "-u", "root").join(" ")
end
@@ -90,7 +99,7 @@ class CommandsAccessoryTest < ActiveSupport::TestCase
test "execute in new container over ssh" do
@mysql.stub(:run_over_ssh, ->(cmd) { cmd.join(" ") }) do
assert_match %r|docker run -it --rm -e MYSQL_ROOT_PASSWORD=\"secret123\" -e MYSQL_ROOT_HOST=\"%\" mysql:8.0 mysql -u root|,
assert_match %r|docker run -it --rm -e MYSQL_ROOT_PASSWORD=\"secret123\" -e MYSQL_ROOT_HOST=\"%\" private.registry/mysql:8.0 mysql -u root|,
@mysql.execute_in_new_container_over_ssh("mysql", "-u", "root")
end
end

View File

@@ -19,6 +19,39 @@ class CommandsTraefikTest < ActiveSupport::TestCase
new_command.run.join(" ")
end
test "run with ports configured" do
assert_equal \
"docker run --name traefik --detach --restart unless-stopped --log-opt max-size=10m --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock traefik --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
new_command.run.join(" ")
@config[:traefik]["options"] = {"publish" => %w[9000:9000 9001:9001]}
assert_equal \
"docker run --name traefik --detach --restart unless-stopped --log-opt max-size=10m --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --publish \"9000:9000\" --publish \"9001:9001\" traefik --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
new_command.run.join(" ")
end
test "run with volumes configured" do
assert_equal \
"docker run --name traefik --detach --restart unless-stopped --log-opt max-size=10m --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock traefik --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
new_command.run.join(" ")
@config[:traefik]["options"] = {"volume" => %w[./letsencrypt/acme.json:/letsencrypt/acme.json] }
assert_equal \
"docker run --name traefik --detach --restart unless-stopped --log-opt max-size=10m --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" traefik --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
new_command.run.join(" ")
end
test "run with several options configured" do
assert_equal \
"docker run --name traefik --detach --restart unless-stopped --log-opt max-size=10m --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock traefik --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
new_command.run.join(" ")
@config[:traefik]["options"] = {"volume" => %w[./letsencrypt/acme.json:/letsencrypt/acme.json], "publish" => %w[8080:8080], "memory" => "512m"}
assert_equal \
"docker run --name traefik --detach --restart unless-stopped --log-opt max-size=10m --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --volume \"./letsencrypt/acme.json:/letsencrypt/acme.json\" --publish \"8080:8080\" --memory \"512m\" traefik --providers.docker --log.level=DEBUG --accesslog.format=\"json\" --api.insecure --metrics.prometheus.buckets=\"0.1,0.3,1.2,5.0\"",
new_command.run.join(" ")
end
test "run without configuration" do
@config.delete(:traefik)