standard-procedure-anvil 0.1.6 → 0.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7ce4bfd700a5d2b870087b7d983902c43bc377ebcba1abbb911caea507aba0e8
-  data.tar.gz: 4e902474667b96efaf5a86435b381f6e54f28d58a1ae6a216441d9de6eb35925
+  metadata.gz: 7330f81fa4611c93e7bd6424be609a05da36a04e0372c8401fdc58a3e1b09afb
+  data.tar.gz: 7f19466d160effbd3050ae33d1273487ccfd55e87f22575b75e578b6c3798b17
 SHA512:
-  metadata.gz: 6a6c16d63f3b0c2bbbafcce079bdb7ba7ea8b44f2c5e0f4ac0f5a9beb3e1f2999852be11aeaed5c107bb88eaef1b7a09b0bf3fef4eaeabb9f5760c625f73c845
-  data.tar.gz: ca793262284e8938121d155fe18f0b6c527a50b9bf0bf437f9a3973d02d708060c317ece52f85713913ec0995937a88eb93d58af85293d01abcf50f5005d5493
+  metadata.gz: 0e0c452fab0076406770c3cff17458451cfe4ff58ae69e4fdc456b474f38cd1b7f9a5cf8cd3f619b0f6be8ac426e3ee7b618992b6f91de3a6400fa4d849363c4
+  data.tar.gz: b42162b6f694f90d4ab33b4734dbdfa465fd506998ddf687ec49b48142dd590021ad1af37d3518cc781415ff77585f7979449dc4f4a6754ca5fb9e152090163a

data/CHANGELOG.md

@@ -3,3 +3,17 @@
 ## [0.1.0] - 2023-06-19
 
 - Initial release
+
+## [0.2.0] - 2023-07-05
+
+- It works for me
+Successfully deployed a number of apps into production using this
+
+## [0.2.1] - 2023-07-06
+
+- Corrected dokku proxy SSL settings
+- Tidy up of various bits of code and configuration
+
+## [0.2.2] - 2023-08-14
+
+- Updated the redis cloudinit file to use the latest version from Redis, instead of the older one that is included with Ubuntu.

data/README.md

@@ -2,136 +2,44 @@
 
 Some simple scripts for installing [Dokku](https://dokku.com) applications on Ubuntu servers.
 
+## Why does this exist?
+
+I needed a tool to [simplify the management](/docs/why.md) of my many dokku-deployed Ruby on Rails apps.
+
 ## Installation
 
+Anvil requires Ruby 2.7 or newer, as it uses ConcurrentRuby to handle doing more than one thing at once.
+
 ```ruby
 gem install standard-procedure-anvil
 ```
 
 ## Usage
 
-### To build a new server
-
-Coming soon (plan is to use [Fog](https://github.com/fog/fog) to handle building servers)
-
-### To install an application onto a blank server
-
-Move to your application's root folder and create the anvil.yml file (see below).
-
-Then run `anvil host --user=user --use-sudo --identity=~/.ssh/my_key`.
-
-This will SSH into each server and:
-
-- Sets the server hostname and timezone
-- Installs various necessary packages, plus dokku itself
-- Sets up the firewall
-- Creates unix users for each app, adding them to the sudo and docker groups, and setting their authorized_keys files with the given public key
-- Schedule a `docker system prune` once per week to clean up any dangling images or containers
-- Configure nginx
-- Install dokku plugins and run any configuration you have defined
-- Sets the dokku deployment branch to `main`
-- Disallows root and passwordless logins over SSH
-
-For each app it will then:
-
-- Create the dokku app
-- Set the environment variables to those defined in your configuration and secrets files
-- Set the app's domain and set up a proxy from nginx to the app's port
-- Sets resource limits for the app
-- Disables checks for workers
-
-Then a git remote for each app is created on your local machine and then pushed.  This performs the initial dokku deployment. Once complete:
-
-- The app is scaled to the correct number of workers
-- Plugins are configured for the app
-
-### Configuration Files
-
-An Anvil configuration file specifies the configuration for multiple servers and multiple apps.  Each server is configured, then each app is installed onto each server.
-
-So you could have one app on two servers (and, we assume, a load-balancer set up in front of them).  Or two apps on one server.  Or even two apps on two servers (again, using a load-balancer)
-
-
-```yml
-version: 0.1
-servers:
-  hosts:
-    - server1.example.com
-  user: user
-  public_key: /home/local-user/.ssh/my-key.pub
-  timezone: Europe/London
-  ports:
-    - 22/tcp
-    - 80/tcp
-    - 443/tcp
-  nginx:
-    forward_proxy_headers: false
-    client_max_body_size: 512m
-    proxy_read_timeout: 60s
-  plugins:
-    cron-restart:
-      url: https://github.com/dokku/dokku-cron-restart.git
-    maintenance:
-      url: https://github.com/dokku/dokku-maintenance.git
-    redis:
-      url: https://github.com/dokku/dokku-redis.git
-    memcached:
-      url: https://github.com/dokku/dokku-memcached.git
-    letsencrypt:
-      url: https://github.com/dokku/dokku-letsencrypt.git
-      config:
-        - set --global email ssl-admin@mycompany.com
-        - cronjob --add
-apps:
-  first_app:
-    hostname: first_app.example.com
-    port: 3000
-    environment:
-      - ENV_VAR=value
-      - ENV_VAR2=value2
-      - RAILS_ENV=production
-    secrets: secrets.yml
-    resource_limit: 2048m
-    scale: web=2 worker=1
-    plugins:
-        cron-restart:
-          - set first_app schedule '0 3 * * *'
-        redis:
-          - create first_app_redis_db
-          - link first_app_redis_db first_app
-        memcached:
-          - create first_app_memcached
-          - link first_app_memcached first_app
-        letsencrypt:
-          - set first_app email ssl-admin@mycompany.com
-          - enable first_app
-  second_app:
-    hostname: second_app.example.com
-    port: 3000
-    environment:
-      - ENV_VAR=value
-      - ENV_VAR2=value2
-      - RAILS_ENV=production
-    secrets: secrets.yml
-    resource_limit: 2048m
-    scale: web=2 worker=1
-    plugins:
-        cron-restart:
-          - set second_app schedule '0 3 * * *'
-        letsencrypt:
-          - set second_app email ssl-admin@mycompany.com
-          - enable second_app
-```
-`secrets.yml` is an optional additional file containing environment variables that you do not want to check into your source code repository.  It is a simple KEY=VALUE format:
+### Build a server
 
-```
-DB_PASSWORD=letmein
-ENCRYPTION_KEY=secretstuff
-```
+Ultimately the plan is to use [Fog](https://github.com/fog/fog) to handle building servers.
+
+But until then, you can prepare your servers using [CloudInit](https://cloudinit.readthedocs.io/en/latest/)
 
+[Generating a cloudinit file](/docs/cloudinit.md) with `anvil cloudinit generate`
+
+### Install and deploy
+
+Use the `anvil app install` and `anvil app deploy` commands to [install and deploy](/docs/app.md) your app to your server.
+
+### Manage and reconfigure
+
+Use `anvil app scale` and `anvil app reconfigure` to manage and reconfigure your app.  (Docs coming soon)
+
+### Ruby on Rails
+
+I'm a Rails developer and I built anvil to help me with my Rails apps.  Here are [some things I learnt along the way](/docs/ruby-on-rails.md).
 
 ## Contributing
 
+Check out the [Roadmap](/docs/roadmap.md)
+
 Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/standard-procedure-anvil.
 
 ## License

data/assets/cloudinit/{dokku.ubuntu-22.yml → docker.yml}

@@ -11,11 +11,24 @@ packages:
   - ufw
   - wget
   - apt-transport-https
+  - ca-certificates
+  - curl
+  - gpg-agent
+  - software-properties-common
 package_update: true
 package_upgrade: true
 runcmd:
   # General server setup
   - timedatectl set-timezone UTC
+  - echo "${USER}:${USER}" | chpasswd
+  # Prepare for Docker
+  - sudo install -m 0755 -d /etc/apt/keyrings
+  - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+  - sudo chmod a+r /etc/apt/keyrings/docker.gpg
+  - echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+  # Install docker
+  - apt-get update
+  - apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
   # Fail2Ban setup
   - printf "[sshd]\nenabled = true\nbanaction = iptables-multiport" > /etc/fail2ban/jail.local
   - systemctl enable fail2ban

data/assets/cloudinit/{dokku.mysql.ubuntu-22.yml → dokku.yml}

@@ -11,35 +11,29 @@ packages:
   - ufw
   - wget
   - apt-transport-https
-  - mysql-client
-  - libmysqlclient-dev
+  - ca-certificates
+  - curl
+  - gpg-agent
+  - software-properties-common
 package_update: true
 package_upgrade: true
 runcmd:
   # General server setup
   - timedatectl set-timezone UTC
-  # Install MySQL
-  - echo "mysql-server mysql-server/root_password password root" | sudo debconf-set-selections
-  - echo "mysql-server mysql-server/root_password_again password root" | sudo debconf-set-selections
-  - sudo apt-get -y install mysql-server
-  - |
-    cat >> /etc/mysql/mysql.conf.d/utf8.cnf << CONF
-    [client]
-    default-character-set=utf8mb4
-
-    [mysql]
-    default-character-set=utf8mb4
-
-    [mysqld]
-    init_connect='SET collation_connection = utf8mb4_unicode_ci'
-    init_connect='SET NAMES utf8mb4'
-    character-set-server=utf8mb4
-    collation-server=utf8mb4_unicode_ci
-    skip-character-set-client-handshake
-    CONF
-  - sed -i -e '/^\(#\|\)bind-address/s/^.*$/bind-address = 0.0.0.0/' /etc/mysql/mysql.conf.d/mysqld.cnf
-  # Start MySQL
-  - systemctl start mysql.service
+  - echo "${USER}:${USER}" | chpasswd
+  # Prepare for Docker
+  - sudo install -m 0755 -d /etc/apt/keyrings
+  - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+  - sudo chmod a+r /etc/apt/keyrings/docker.gpg
+  - echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+  # Install docker
+  - apt-get update
+  - apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+  # Install dokku
+  - echo "dokku dokku/vhost_enable boolean true" | sudo debconf-set-selections
+  - wget https://dokku.com/install/v0.30.7/bootstrap.sh && sudo DOKKU_TAG=v0.30.7 bash bootstrap.sh
+  - cat /home/app/.ssh/authorized_keys | dokku ssh-keys:add admin
+  - dokku git:set --global deploy-branch main
   # Fail2Ban setup
   - printf "[sshd]\nenabled = true\nbanaction = iptables-multiport" > /etc/fail2ban/jail.local
   - systemctl enable fail2ban
@@ -59,3 +53,4 @@ runcmd:
   - sed -i '$a AllowUsers %{USER} dokku' /etc/ssh/sshd_config
   # And we're done
   - reboot
+

data/assets/cloudinit/{memcached.ubuntu-22.yml → memcached.yml}

@@ -10,13 +10,13 @@ packages:
   - fail2ban
   - ufw
   - wget
-  - memcached
   - logrotate
 package_update: true
 package_upgrade: true
 runcmd:
   # General server setup
   - timedatectl set-timezone UTC
+  - echo "${USER}:${USER}" | chpasswd
   # Fail2Ban setup
   - printf "[sshd]\nenabled = true\nbanaction = iptables-multiport" > /etc/fail2ban/jail.local
   - systemctl enable fail2ban
@@ -34,6 +34,7 @@ runcmd:
   - sed -i -e '/^\(#\|\)AuthorizedKeysFile/s/^.*$/AuthorizedKeysFile .ssh\/authorized_keys/' /etc/ssh/sshd_config
   - sed -i '$a AllowUsers %{USER}' /etc/ssh/sshd_config
   # Set up memcached
+  - apt-get -y install memcached
   - sed -i 's/-m 64/-m 512/g' /etc/memcached.conf
   - sed -i 's/-l 127.0.0.1/-l 0.0.0.0/g' /etc/memcached.conf
   - systemctl restart memcached.service

data/assets/cloudinit/{mysql.ubuntu-22.yml → mysql.yml}

@@ -16,10 +16,11 @@ package_upgrade: true
 runcmd:
   # General server setup
   - timedatectl set-timezone UTC
+  - echo "${USER}:${USER}" | chpasswd
   # Install MySQL
   - echo "mysql-server mysql-server/root_password password root" | sudo debconf-set-selections
   - echo "mysql-server mysql-server/root_password_again password root" | sudo debconf-set-selections
-  - sudo apt-get -y install mysql-server
+  - apt-get -y install mysql-server
   - |
     cat >> /etc/mysql/mysql.conf.d/utf8.cnf << CONF
     [client]

data/assets/cloudinit/{opensearch.ubuntu-22.yml → opensearch.yml}

@@ -10,14 +10,13 @@ packages:
   - fail2ban
   - ufw
   - wget
-  - docker.io
-  - docker-compose
   - apt-transport-https
 package_update: true
 package_upgrade: true
 runcmd:
   # General server setup
   - timedatectl set-timezone UTC
+  - echo "${USER}:${USER}" | chpasswd
   # Prepare for OpenSearch
   - swapoff -a
   - echo "vm.max_map_count=262144" > /etc/sysctl.d/98-opensearch.conf
@@ -39,6 +38,14 @@ runcmd:
   - sed -i -e '/^\(#\|\)AllowAgentForwarding/s/^.*$/AllowAgentForwarding no/' /etc/ssh/sshd_config
   - sed -i -e '/^\(#\|\)AuthorizedKeysFile/s/^.*$/AuthorizedKeysFile .ssh\/authorized_keys/' /etc/ssh/sshd_config
   - sed -i '$a AllowUsers %{USER}' /etc/ssh/sshd_config
+  # Install docker
+  - apt-get install -y ca-certificates curl gnupg
+  - install -m 0755 -d /etc/apt/keyrings
+  - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+  - chmod a+r /etc/apt/keyrings/docker.gpg
+  - echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
+  - apt-get update
+  - apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
   # OpenSearch setup
   - mkdir -p /etc/opensearch
   - docker pull opensearchproject/opensearch:latest
@@ -55,7 +62,7 @@ runcmd:
           - node.name=search_db
           - bootstrap.memory_lock=true
           - plugins.security.disabled=true
-          - "OPENSEARCH_JAVA_OPTS=-Xms4096m -Xmx4096m"
+          - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx3184m"
         ulimits:
           memlock:
             soft: -1
@@ -71,16 +78,18 @@ runcmd:
     volumes:
       opensearch_data:
     EOF
+  - cd /etc/opensearch && docker compose build
   - |
     cat >> /etc/systemd/system/opensearch.service << EOF
-    Description=OpenSearch container
+    [Unit]
+    Description=OpenSearch
     Requires=docker.service
     After=docker.service
     [Service]
     WorkingDirectory=/etc/opensearch
     Restart=always
-    ExecStart=/usr/bin/docker-compose up
-    ExecStop=/usr/bin/docker-compose down
+    ExecStart=/usr/bin/docker compose up
+    ExecStop=/usr/bin/docker compose down
     [Install]
     WantedBy=multi-user.target
     EOF

data/assets/cloudinit/{redis.ubuntu-22.yml → redis.yml}

@@ -10,13 +10,13 @@ packages:
   - fail2ban
   - ufw
   - wget
-  - redis-server
   - logrotate
 package_update: true
 package_upgrade: true
 runcmd:
   # General server setup
   - timedatectl set-timezone UTC
+  - echo "${USER}:${USER}" | chpasswd
   # Fail2Ban setup
   - printf "[sshd]\nenabled = true\nbanaction = iptables-multiport" > /etc/fail2ban/jail.local
   - systemctl enable fail2ban
@@ -34,10 +34,14 @@ runcmd:
   - sed -i -e '/^\(#\|\)AuthorizedKeysFile/s/^.*$/AuthorizedKeysFile .ssh\/authorized_keys/' /etc/ssh/sshd_config
   - sed -i '$a AllowUsers %{USER}' /etc/ssh/sshd_config
   # Set up Redis
+  - curl -fsSL https://packages.redis.io/gpg | gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
+  - echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/redis.list
+  - apt-get update
+  - apt-get -y install redis-server
   - sed -i 's/supervised no/supervised systemd/g' /etc/redis/redis.conf
   - sed -i 's/bind 127.0.0.1 ::1/# bind 127.0.0.1 ::1/g' /etc/redis/redis.conf
   - sed -i 's/protected-mode yes/protected-mode no/g' /etc/redis/redis.conf
-  - systemctl restart redis.service
+  - systemctl restart redis-server.service
   - |
     cat > /etc/logrotate.d/redis-server << EOF
     /var/log/redis/redis-server*.log {

data/checksums/standard-procedure-anvil-0.1.7.gem.sha512

@@ -0,0 +1 @@
+fb8d74684e40d48b82c6ad58503e52b035a76fe4f534a8fcb5ec043227af442035ef5f0d1a845cb8e8bebc3556bfcb4718fca26dcb53b4f816f76c1990f0c4d5

data/checksums/standard-procedure-anvil-0.2.0.gem.sha512

@@ -0,0 +1 @@
+5c4b010036ad155590705bf75b564c167513f342524abf34073080de20ccdf2b2c0ef4671f38dccb53781603f7373537436c8a6bbc7a43a5eb33c3fabbd1821d

data/checksums/standard-procedure-anvil-0.2.1.gem.sha512

@@ -0,0 +1 @@
+2b56322e96127aa43986eaa2de436eafd64c7d5260d61db1a13605d95bf0fc3b331f006c0b534d845c05e781659518f37d3f8aa17475e6e926675d5ea1985a50

data/docs/app.md

@@ -0,0 +1,51 @@
+# The `app` command
+
+## Installing an application
+
+Move to your application's root folder and create the deploy.yml file (see below).  Then use the `app install` command to set dokku up for your first deployment.
+
+```sh
+anvil app install
+```
+
+This will SSH into the server (or servers if you have multiple) from your config file and:
+
+- Installs any dokku plugins that you have specified
+- Tells dokku to create the app
+- Uses your config file to set the environment variables for the app
+- Sets some sensible defaults for Nginx and makes sure it proxies correctly to your app
+- Optionally forwards the correct SSL/TLS headers if your app is behind a load-balancer
+- Finally it runs the post-installation scripts from your config file, which you can use to configure your plugins
+
+## Deploying an application
+
+Next up we deploy the app.
+
+```sh
+anvil app deploy
+```
+As this is the first deployment, anvil will create git remotes for each host, then do the initial git push.  If you have multiple servers configured, these should run in parallel (coming soon).  Once each deployment has completed, anvil will SSH in, scale your app and run the post-first-deployment scripts.
+
+You can then use the same `anvil app deploy` command to deploy the app again - but as it knows this isn't the first deployment (as it does not need to create the git remotes), it will run your post-deployment scripts (not post-first-deployment) each time.
+
+To change the number of processes (as defined by your Procfile), you can set the `scale` key(s) in your config file and then call:
+
+```sh
+anvil app scale
+```
+
+(COMING SOON)
+Finally, if you need to change the values of any environment variables, update your config file and use:
+
+```sh
+anvil app configure
+```
+
+## Configuration files
+
+The [anvil configuration file](/docs/configuration.md) is the heart of the system.
+
+## Secrets
+
+You don't want to store your secrets (passwords, encryption keys) in your anvil configuration.  Instead anvil can [read your secrets](/docs/secrets.md) from a separate file or from the command line.
+

data/docs/cloudinit.md

@@ -0,0 +1,33 @@
+# Building a server
+
+## Cloudinit
+
+A [CloudInit](https://cloudinit.readthedocs.io/en/latest/) file is a YML file that you load into a virtual machine while it is being created.  As the server boots, uses the cloudinit configuration to install software and set itself up.  With most cloud hosting providers, you will find an option for "user data", or something similar, on the "create a new server" page.
+
+### Generate a configuration
+
+So firstly we ask Anvil which cloudinit configurations it has available:
+
+```sh
+anvil cloudinit list
+```
+
+This will give us a list of prewritten cloud init scripts - of which dokku is probably the one we're most interested in.
+
+Next we tell anvil to generate our configuration:
+
+```sh
+anvil cloudinit generate dokku --user app --public-key ~/.ssh/my_key.pub > ~/Desktop/my_server.yml
+```
+
+Anvil generates a dokku configuration (and places it on our desktop) that will create an Ubuntu 22.04 box with docker and dokku preinstalled.  Plus it will create a user called `app` that can log in through SSH using a public key `my_key.pub`.  The server itself is locked down so only ports 80, 443 and 22 are open, only the users `app` and `dokku` are allowed to log in and they must use public/private key encryption - no passwords allowed.
+
+### Testing your configuration
+
+To test this, it's worth taking a look at [Multipass](https://multipass.run) - a tool from Canonical that lets you create virtual machines (using cloud init files) on your local machine.  This means you can try out various configurations without spending money at a hosting company.
+
+One thing to note when using multipass - it requires SSH access for a user called "ubuntu".  So take your generated cloudinit file, locate the SSH configuration section and the "AllowUsers" line - and add the "ubuntu" user to it.  Something like: `  - sed -i '$a AllowUsers %{USER} ubuntu dokku' /etc/ssh/sshd_config`.
+
+Multipass has its own private key generated for the ubuntu user, and uses this to manage the server.  Of course, the multipass VM is only on your machine, plus its private key is hidden away, so it's not a security risk.  But in general `anvil cloudinit generate` disallows all SSH access apart from your named user (using your own key), and the `dokku` user if applicable.
+
+Once you've built a preconfigured virtual machine, we can move on to getting our dokku application installed.  However, note that it can take several minutes for the initialisation process to complete - so don't start your deployment too early, or your server won't be ready and will reboot whilst your setting things up.

data/docs/configuration.md

@@ -0,0 +1,7 @@
+# Anvil configuration files
+
+An Anvil configuration file specifies the configuration for multiple servers and multiple apps.  Each server is configured, then each app is installed onto each server.
+
+For now take a look at the two samples in the spec folder - [single-server](/spec/fixtures/single-server.config.yml) and [multi-server](/spec/fixtures/multi-server.config.yml).
+
+Also check out the [tips for Ruby on Rails](/docs/ruby-on-rails.md) which has an example configuration file.

data/docs/roadmap.md

@@ -0,0 +1,12 @@
+# Roadmap
+
+As I mentioned, this is pretty much designed for my own use.
+
+There are a few bits I still need (which will be V1) and then I want to open it up.  But if it gets too generic, you might as well just write a load of shell scripts to manage dokku yourself - it's important to keep it simple.
+
+## To do
+
+- [ ] `app reconfigure`
+- [ ] Add `--first`/`--not-first` options to `app deploy` so you can override the first-deployment behaviour (in case you get a failure and need to re-run everything)
+- [ ] Instead of relying on the ssh-agent, allow the use of your private key when connecting to servers
+- [ ] Parallel execution across multiple hosts

data/docs/ruby-on-rails.md

@@ -0,0 +1,69 @@
+# Dokku and Ruby on Rails
+
+(incomplete - coming soon)
+
+- If using the Mysql plugin, use the Mysql2 protocol
+- store your RAILS_MASTER_KEY and SECRET_KEY_BASE outside of your configuration file (see [secrets](/docs/secrets.md))
+- in your config/environments/production.rb or config/environments/staging.rb set `config.force_ssl = false` - dokku's nginx configuration will do the redirect for you if you're using the Let's Encrypt plugin, or you can set the redirect on your load-balancer.  Setting `config.force_ssl = true` causes issues with the health checks
+- Make sure your app knows its hostname; set it as an environment variable in your configuration file and then use that hostname in your environment file as follows: `config.action_mailer.default_url_options = {host: ENV["HOSTNAME"]}` and `Rails.application.routes.default_url_options[:host] = config.action_mailer.default_url_options[:host]`.  This means that when you need to generate a full URL (as opposed to a relative path), Rails knows what to use.
+- Use a CHECKS file that looks like this (again using that HOSTNAME environment variable), so dokku's zero-deployment checks can connect correctly.  My `/health_check` route just returns a `200 OK` in most apps, but in some it actually checks the database connection, as well as some other services (although this causes problems with the initial deployment, which is why checks are switched off the first time through)
+```
+WAIT=10
+ATTEMPTS=5
+http://{{ var "HOSTNAME" }}/health_check
+```
+
+A typical Rails deploy.yml for a single-server, totally self-contained app, looks like this:
+
+```yaml
+version: 0.1
+hosts:
+  - myapp.example.com:
+      user: app
+app:
+  domain: myapp.example.com
+  port: 3000
+  environment:
+    - BUNDLE_WITHOUT=test:development
+    - CABLE_CHANNEL_PREFIX=myapp
+    - EMAIL_DOMAIN=mail.myapp.example.com
+    - EMAIL_HOST=smtp.myapp.example.com
+    - EMAIL_PORT=587
+    - EMAIL_USER=postmaster@mail.myapp.example.com
+    - HOSTNAME=myapp.example.com
+    - NODE_ENV=production
+    - RACK_ENV=production
+    - RAILS_ENV=production
+    - RAILS_LOG_TO_STDOUT=true
+    - RAILS_MAX_THREADS=10
+    - RAILS_SERVE_STATIC_FILES=true
+  resource_limit: 2048m
+  scale: web=2 worker=1
+  load_balancer: false
+  nginx:
+    client_max_body_size: 512m
+    proxy_read_timeout: 60s
+  plugins:
+    - cron-restart
+    - maintenance
+    - redis
+    - memcached
+    - mysql
+    - letsencrypt
+  scripts:
+    after_install:
+      - dokku cron-restart:set app schedule '0 3 * * *'
+      - dokku memcached:create memcached
+      - dokku memcached:link memcached app
+      - dokku redis:create redis_db
+      - dokku redis:link redis_db app
+      - dokku mysql:create mysql_db
+      - dokku config:set app MYSQL_DATABASE_SCHEME=mysql2
+      - dokku mysql:link mysql_db app
+    after_first_deploy:
+      - dokku letsencrypt:set app email me@myapp.example.com
+      - dokku letsencrypt:enable app
+      - dokku letsencrypt:cron-job --add
+      - dokku run app bin/rails db:seed
+
+```

data/docs/secrets.md

@@ -0,0 +1,20 @@
+# Secrets
+
+Finally, you'll probably want to check your deploy.yml file into source control.  But you _definitely_ don't want to be storing important secrets - database passwords, encryption keys and so on - where everyone can see them.
+
+So the `anvil app` commands also allow you to specify secrets, either from another file, or via the command line.
+
+The secrets are just extra environment variables that are added to the ones defined in your config file - in the format:
+
+```
+SECRET1=VALUE1 SECRET2=VALUE2
+```
+
+You can either specify `--secrets my-secrets-file.env` to load these from a separate file.  Or you can load them from stdin.
+
+For example, I use [Bitwarden](https://bitwarden.com) as my password locker and use the Bitwarden CLI to access my secrets.  The CLI is installed through homebrew, I then authenticate and can use a command like:
+
+```sh
+bw get notes secrets@myapp.com | anvil app install deploy.myapp.yml -S
+```
+I have the environment variables for myapp.com stored in Bitwarden as a secure note with the title "secrets@myapp.com".  So `bw get notes secrets@myapp.com` loads them from my vault and pipes them to the `anvil app install` command.  The anvil command is using the `-S` (or `--secrets-stdin`) option which means it will read the information piped in by bitwarden.  So, once decrypted, the confidential data never touches a disk until it gets written into the dokku app configuration on the server.

data/docs/why.md

@@ -0,0 +1,17 @@
+# Why does this exist?
+
+[Dokku](https://dokku.com) is great at installing and configuring containers on a single host.
+
+But you do need to install dokku, generate your configuration and environment variables, install your plugins and app and then configure all those plugins.
+
+Unfortunately, there's no [single configuration file](https://github.com/dokku/dokku/issues/1558) for dokku.
+
+In addition, dokku is really designed for managing a single server.  But I'm actually using it to manage multiple servers that are hidden behind a load-balancer.
+
+So to manage this, I wanted a single configuration file that I could user for all my dokku information, that could then use that configuration across multiple servers.
+
+Currently it's extremely tailored to my needs - it's built for Ubuntu 22.04, it creates a user called "app" (although you can change that), it names your dokku app "app".
+
+I've also added in cloudinit configs for some of the other servers I have to use.  Of course, these are not related to dokku, but anvil can generate them easily so it's useful to keep them all in one place.
+
+There are several [limitations](/docs/roadmap.md) to how it works - it does what I need but does need expansion.  That will come soon.

data/lib/anvil/app/deploy.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module Anvil
+  class App
+    require_relative "host_deployer"
+    class Deploy < Struct.new(:configuration)
+      include ConfigurationReader
+      def call
+        branch = `git rev-parse --abbrev-ref HEAD`.strip
+        hosts.each do |host|
+          HostDeployer.new(configuration, host.to_s.strip, branch).call
+        end
+      end
+    end
+  end
+end

data/lib/anvil/app/host_deployer.rb

@@ -0,0 +1,90 @@
+# frozen_string_literal: true
+
+require "standard_procedure/async"
+module Anvil
+  require_relative "../logger"
+  require_relative "../ssh_executor"
+  require_relative "../configuration_reader"
+  class App
+    class HostDeployer < Struct.new(:configuration, :host, :branch)
+      include StandardProcedure::Async::Actor
+      include Anvil::ConfigurationReader
+
+      def call
+        first_deployment = !git_remote_exists?
+        if first_deployment
+          switch_off_downtime_checks
+          create_git_remote
+        end
+
+        do_git_push
+        if first_deployment
+          switch_on_downtime_checks
+          run_after_first_deployment_scripts
+          scale_processes
+        else
+          run_after_deployment_scripts
+        end
+        logger.info "Deployment for #{host} complete"
+      end
+
+      protected
+
+      def git_remote_exists?
+        `git remote show`.split("\n").include? host
+      end
+
+      def switch_off_downtime_checks
+        Anvil::SshExecutor.new(host, user_for(host), logger).call do |ssh|
+          ssh.exec! "dokku checks:disable app worker,web"
+        end
+      end
+
+      def create_git_remote
+        logger.info "git remote add #{host} dokku@#{host}:/app"
+        logger.info `git remote add #{host} dokku@#{host}:/app`
+      end
+
+      def do_git_push
+        logger.info "git push #{host} #{branch}:main"
+        logger.info `git push #{host} #{branch}:main`
+      end
+
+      def switch_on_downtime_checks
+        Anvil::SshExecutor.new(host, user_for(host), logger).call do |ssh|
+          ssh.exec! "dokku checks:enable app web"
+        end
+      end
+
+      def run_after_first_deployment_scripts
+        Anvil::SshExecutor.new(host, user_for(host), logger).call do |ssh|
+          (configuration_for(host).dig("scripts")&.dig("after_first_deploy") || []).each do |script|
+            ssh.exec! script, "run_after_install_scripts"
+          end
+          (configuration_for_app.dig("scripts")&.dig("after_first_deploy") || []).each do |script|
+            ssh.exec! script, "run_after_install_scripts"
+          end
+        end
+      end
+
+      def scale_processes
+        Anvil::App::HostScaler.new(configuration, host).call
+      end
+
+      def run_after_deployment_scripts
+        Anvil::SshExecutor.new(host, user_for(host), logger).call do |ssh|
+          (configuration_for(host).dig("scripts")&.dig("after_deploy") || []).each do |script|
+            ssh.exec! script, "run_after_install_scripts"
+          end
+          (configuration_for_app.dig("scripts")&.dig("after_deploy") || []).each do |script|
+            ssh.exec! script, "run_after_install_scripts"
+          end
+        end
+      end
+
+      def logger
+        @logger ||= Anvil::Logger.new(host)
+      end
+    end
+  end
+end

data/lib/anvil/app/host_installer.rb

@@ -13,15 +13,22 @@ module Anvil
 
       def call
         Anvil::SshExecutor.new(host, user_for(host), logger).call do |ssh|
+          install_plugins ssh
           create_app ssh
           set_environment ssh
           set_dokku_options ssh
-          run_post_installation_scripts ssh
+          run_after_install_scripts ssh
         end
       end
 
       protected
 
+      def install_plugins ssh
+        (configuration_for_app.fetch("plugins") | []).each do |plugin|
+          ssh.exec! "sudo dokku plugin:install https://github.com/dokku/dokku-#{plugin}.git #{plugin}"
+        end
+      end
+
       def create_app ssh
         ssh.exec! "dokku apps:create app", "create_app"
       end
@@ -34,27 +41,28 @@ module Anvil
         ssh.exec! "dokku docker-options:add app run \"--add-host=host.docker.internal:host-gateway\"", "set_dokku_options"
         ssh.exec! "dokku domains:set app #{configuration_for_app["domain"]}", "set_dokku_options"
         ssh.exec! "dokku proxy:ports-add app http:80:#{configuration_for_app["port"]}", "set_dokku_options"
+        ssh.exec! "dokku proxy:ports-add app https:443:#{configuration_for_app["port"]}", "set_dokku_options"
         ssh.exec! "dokku nginx:set app client-max-body-size #{configuration_for_app["nginx"]["client_max_body_size"]}", "set_dokku_options"
         ssh.exec! "dokku nginx:set app proxy-read-timeout #{configuration_for_app["nginx"]["proxy_read_timeout"]}", "set_dokku_options"
-        if configuration_for_app["nginx"]["forward_proxy_headers"]
-          ssh.exec! "dokku nginx:set $APP x-forwarded-for-value \"$http_x_forwarded_for\"", "set_dokku_options"
-          ssh.exec! "dokku nginx:set $APP x-forwarded-port-value \"$http_x_forwarded_port\"", "set_dokku_options"
-          ssh.exec! "dokku nginx:set $APP x-forwarded-proto-value \"$http_x_forwarded_proto\"", "set_dokku_options"
+        if configuration_for_app["load_balancer"]
+          ssh.exec! "dokku nginx:set app x-forwarded-for-value \"$http_x_forwarded_for\"", "set_dokku_options"
+          ssh.exec! "dokku nginx:set app x-forwarded-port-value \"$http_x_forwarded_port\"", "set_dokku_options"
+          ssh.exec! "dokku nginx:set app x-forwarded-proto-value \"$http_x_forwarded_proto\"", "set_dokku_options"
         end
         ssh.exec! "dokku proxy:build-config app", "set_dokku_options"
       end
 
-      def run_post_installation_scripts ssh
-        configuration_for_app.fetch("scripts")&.fetch("post_install")&.each do |script|
-          ssh.exec! script, "run_post_installation_scripts"
+      def run_after_install_scripts ssh
+        (configuration_for(host).dig("scripts")&.dig("after_install") || []).each do |script|
+          ssh.exec! script, "run_after_install_scripts"
         end
-        configuration_for(host).fetch("scripts")&.fetch("post_install")&.each do |script|
-          ssh.exec! script, "run_post_installation_scripts"
+        (configuration_for_app.dig("scripts")&.dig("after_install") || []).each do |script|
+          ssh.exec! script, "run_after_install_scripts"
         end
       end
 
       def logger
-        @logger ||= Anvil::Logger.new("HostInstaller - #{host}")
+        @logger ||= Anvil::Logger.new(host)
       end
     end
   end

data/lib/anvil/app/host_scaler.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+require "standard_procedure/async"
+module Anvil
+  require_relative "../logger"
+  require_relative "../ssh_executor"
+  require_relative "../configuration_reader"
+  class App
+    class HostScaler < Struct.new(:configuration, :host)
+      include StandardProcedure::Async::Actor
+      include Anvil::ConfigurationReader
+
+      def call
+        scale_processes
+      end
+
+      protected
+
+      def scale_processes
+        scale = configuration_for(host).dig("scale") || configuration_for_app.dig("scale") || "web=1"
+        Anvil::SshExecutor.new(host, user_for(host), logger).call do |ssh|
+          ssh.exec! "dokku ps:scale app #{scale}"
+        end
+      end
+
+      def logger
+        @logger ||= Anvil::Logger.new(host)
+      end
+    end
+  end
+end

data/lib/anvil/app/scale.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module Anvil
+  class App
+    require_relative "host_scaler"
+    class Scale < Struct.new(:configuration)
+      include ConfigurationReader
+      def call
+        hosts.each do |host|
+          HostScaler.new(configuration, host.to_s.strip).call
+        end
+      end
+    end
+  end
+end

data/lib/anvil/app.rb

@@ -7,6 +7,8 @@ module Anvil
   class App < Anvil::SubCommandBase
     require_relative "app/env"
     require_relative "app/install"
+    require_relative "app/deploy"
+    require_relative "app/scale"
 
     desc "env /path/to/config.yml", "Generate environment variables for an app"
     long_desc <<-DESC
@@ -49,6 +51,34 @@ module Anvil
       Anvil::App::Install.new(configuration, secrets).call
     end
 
+    desc "deploy /path/to/config.yml", "Deploy an app"
+    long_desc <<-DESC
+    Deploy an app on the hosts specified in the configuration.
+
+    First it checks to see if a git remote exists for each host.
+
+    If not, this counts as a first deployment and it creates the git remote
+
+    Then, whether first deployment or not, it does a `git push` of the current branch to main on the remote.
+
+    Finally, if this is the first deployment, it runs the "after_first_deployment" scripts, otherwise it runs the "after_deployment" scripts.
+    DESC
+    def deploy filename = "deploy.yml"
+      configuration = YAML.load_file(filename)
+      Anvil::App::Deploy.new(configuration).call
+    end
+
+    desc "scale /path/to/config.yml", "Scale an app"
+    long_desc <<-DESC
+    Scale a previously deployed app, using the scale values from the config file.
+
+    The scale can either be set per host or per app (with host settings taking priority).
+    DESC
+    def scale filename = "deploy.yml"
+      configuration = YAML.load_file(filename)
+      Anvil::App::Scale.new(configuration).call
+    end
+
     protected
 
     def read_secrets(filename: nil, stdin: false)

data/lib/anvil/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Anvil
-  VERSION = "0.1.6"
+  VERSION = "0.2.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: standard-procedure-anvil
 version: !ruby/object:Gem::Version
-  version: 0.1.6
+  version: 0.2.2
 platform: ruby
 authors:
 - Rahoul Baruah
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-07-03 00:00:00.000000000 Z
+date: 2023-08-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -110,22 +110,35 @@ files:
 - LICENSE.txt
 - README.md
 - Rakefile
-- assets/cloudinit/dokku.mysql.ubuntu-22.yml
-- assets/cloudinit/dokku.ubuntu-22.yml
-- assets/cloudinit/memcached.ubuntu-22.yml
-- assets/cloudinit/mysql.ubuntu-22.yml
-- assets/cloudinit/opensearch.ubuntu-22.yml
-- assets/cloudinit/redis.ubuntu-22.yml
-- assets/install/dokku.txt
+- assets/cloudinit/docker.yml
+- assets/cloudinit/dokku.yml
+- assets/cloudinit/memcached.yml
+- assets/cloudinit/mysql.yml
+- assets/cloudinit/opensearch.yml
+- assets/cloudinit/redis.yml
 - checksums/standard-procedure-anvil-0.1.4.gem.sha512
 - checksums/standard-procedure-anvil-0.1.5.gem.sha512
 - checksums/standard-procedure-anvil-0.1.6.gem.sha512
+- checksums/standard-procedure-anvil-0.1.7.gem.sha512
+- checksums/standard-procedure-anvil-0.2.0.gem.sha512
+- checksums/standard-procedure-anvil-0.2.1.gem.sha512
+- docs/app.md
+- docs/cloudinit.md
+- docs/configuration.md
+- docs/roadmap.md
+- docs/ruby-on-rails.md
+- docs/secrets.md
+- docs/why.md
 - exe/anvil
 - lib/anvil.rb
 - lib/anvil/app.rb
+- lib/anvil/app/deploy.rb
 - lib/anvil/app/env.rb
+- lib/anvil/app/host_deployer.rb
 - lib/anvil/app/host_installer.rb
+- lib/anvil/app/host_scaler.rb
 - lib/anvil/app/install.rb
+- lib/anvil/app/scale.rb
 - lib/anvil/cli.rb
 - lib/anvil/cloudinit.rb
 - lib/anvil/cloudinit/generator.rb

data/assets/install/dokku.txt

@@ -1,13 +0,0 @@
-# SSH into your server and paste the script
-sudo bash
-echo "dokku dokku/vhost_enable boolean true" | sudo debconf-set-selections
-wget https://dokku.com/install/v0.30.7/bootstrap.sh && sudo DOKKU_TAG=v0.30.7 bash bootstrap.sh
-cat /home/app/.ssh/authorized_keys | dokku ssh-keys:add admin
-dokku plugin:install https://github.com/dokku/dokku-cron-restart.git cron-restart
-dokku plugin:install https://github.com/dokku/dokku-maintenance.git maintenance
-dokku plugin:install https://github.com/dokku/dokku-redis.git redis
-dokku plugin:install https://github.com/dokku/dokku-memcached.git memcached
-dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git letsencrypt
-dokku config:set --global DOKKU_LETSENCRYPT_EMAIL=sysadmin@echodek.co
-dokku git:set --global deploy-branch main
-exit