standard-procedure-anvil 0.2.0 → 0.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bd79d747f6dacee61d78024e0bea3789ce4e143fb8307c5f21df6dfdac5ca357
-  data.tar.gz: d633f9eff054f41ab3e0253606f920d7e258235f25fc84b2eadf6f336125e773
+  metadata.gz: 7330f81fa4611c93e7bd6424be609a05da36a04e0372c8401fdc58a3e1b09afb
+  data.tar.gz: 7f19466d160effbd3050ae33d1273487ccfd55e87f22575b75e578b6c3798b17
 SHA512:
-  metadata.gz: 903ec97ce720953550c05f15cb15aa5d420f5e510645f688991a860c213ec2025da72049bfae1198d569390bd9cdf1b6c38bfb4e62ff05f470659ee3d5282a2d
-  data.tar.gz: ef6be87643818b9c07183d6cbc5fa4e4a6cea75b00a692313ff088cf0754a00aad33a94ba7b459daf644cb7a94d64c38f1d57bdef48434ec3d5c471d5c06043f
+  metadata.gz: 0e0c452fab0076406770c3cff17458451cfe4ff58ae69e4fdc456b474f38cd1b7f9a5cf8cd3f619b0f6be8ac426e3ee7b618992b6f91de3a6400fa4d849363c4
+  data.tar.gz: b42162b6f694f90d4ab33b4734dbdfa465fd506998ddf687ec49b48142dd590021ad1af37d3518cc781415ff77585f7979449dc4f4a6754ca5fb9e152090163a

data/CHANGELOG.md

@@ -3,3 +3,17 @@
 ## [0.1.0] - 2023-06-19
 
 - Initial release
+
+## [0.2.0] - 2023-07-05
+
+- It works for me
+Successfully deployed a number of apps into production using this
+
+## [0.2.1] - 2023-07-06
+
+- Corrected dokku proxy SSL settings
+- Tidy up of various bits of code and configuration
+
+## [0.2.2] - 2023-08-14
+
+- Updated the redis cloudinit file to use the latest version from Redis, instead of the older one that is included with Ubuntu.

data/README.md

@@ -2,110 +2,44 @@
 
 Some simple scripts for installing [Dokku](https://dokku.com) applications on Ubuntu servers.
 
+## Why does this exist?
+
+I needed a tool to [simplify the management](/docs/why.md) of my many dokku-deployed Ruby on Rails apps.
+
 ## Installation
 
+Anvil requires Ruby 2.7 or newer, as it uses ConcurrentRuby to handle doing more than one thing at once.
+
 ```ruby
 gem install standard-procedure-anvil
 ```
 
 ## Usage
 
-### To build a new server
+### Build a server
 
 Ultimately the plan is to use [Fog](https://github.com/fog/fog) to handle building servers.
 
 But until then, you can prepare your servers using [CloudInit](https://cloudinit.readthedocs.io/en/latest/)
 
-A cloudinit file is a YML file that you load into a virtual machine while it is being created.  As the server boots, uses the cloudinit configuration to install software and set itself up.  With most cloud hosting providers, you will find an option for "user data", or something similar, on the "create a new server" page.
-
-So firstly we ask Anvil which cloudinit configurations it has available:
-
-```sh
-anvil cloudinit list
-```
-
-This will give us a list of prewritten cloud init scripts - of which dokku is probably the one we're most interested in.
-
-Next we tell anvil to generate our configuration:
-
-```sh
-anvil cloudinit generate dokku --user app --public-key ~/.ssh/my_key.pub > ~/Desktop/my_server.yml
-```
-
-Anvil generates a dokku configuration (and places it on our desktop) that will create an Ubuntu 22.04 box with docker and dokku preinstalled.  Plus it will create a user called `app` that can log in through SSH using a public key `my_key.pub`.  The server itself is locked down so only ports 80, 443 and 22 are open, only the users `app` and `dokku` are allowed to log in and they must use public/private key encryption - no passwords allowed.
-
-To test this, it's worth taking a look at [Multipass](https://multipass.run) - a tool from Canonical that lets you create virtual machines (using cloud init files) on your local machine - meaning you can try out various configurations without spending money at a hosting company.
-
-Once you've built a preconfigured virtual machine, we can move on to getting our dokku application installed.
-
-### Installing an application onto the server
-
-Move to your application's root folder and create the deploy.yml file (see below).  Then use the `app install` command to set dokku up for your first deployment.
-
-```sh
-anvil app install
-```
-
-This will SSH into the server (or servers if you have multiple) from your config file and:
-
-- Installs any dokku plugins that you have specified
-- Tells dokku to create the app
-- Uses your config file to set the environment variables for the app
-- Sets some sensible defaults for Nginx and makes sure it proxies correctly to your app
-- Optionally forwards the correct SSL/TLS headers if your app is behind a load-balancer
-- Finally it runs the post-installation scripts from your config file, which you can use to configure your plugins
+[Generating a cloudinit file](/docs/cloudinit.md) with `anvil cloudinit generate`
 
-Next up we deploy the app.
+### Install and deploy
 
-```sh
-anvil app deploy
-```
-As this is the first deployment, anvil will create git remotes for each host, then do the initial git push.  If you have multiple servers configured, these should run in parallel (coming soon).  Once each deployment has completed, anvil will SSH in, scale your app and run the post-first-deployment scripts.
-
-You can then use the same `anvil app deploy` command to deploy the app again - but as it knows this isn't the first deployment (as it does not need to create the git remotes), it will run your post-deployment scripts (not post-first-deployment) each time.
-
-To change the number of processes (as defined by your Procfile), you can set the `scale` key(s) in your config file and then call:
-
-```sh
-anvil app scale
-```
-
-(COMING SOON)
-Finally, if you need to change the values of any environment variables, update your config file and use:
-
-```sh
-anvil app configure
-```
-
-### Configuration Files
-
-An Anvil configuration file specifies the configuration for multiple servers and multiple apps.  Each server is configured, then each app is installed onto each server.
+Use the `anvil app install` and `anvil app deploy` commands to [install and deploy](/docs/app.md) your app to your server.
 
-For now take a look at the two samples in the spec folder - [multi-server](/spec/fixtures/multi-server.config.yml) and [single-server](/spec/fixtures/single-server.config.yml).
+### Manage and reconfigure
 
-### Secrets
+Use `anvil app scale` and `anvil app reconfigure` to manage and reconfigure your app.  (Docs coming soon)
 
-Finally, you'll probably want to check your deploy.yml file into source control.  But you _definitely_ don't want to be storing important secrets - database passwords, encryption keys and so on - where everyone can see them.
+### Ruby on Rails
 
-So the `anvil app` commands also allow you to specify secrets, either from another file, or via the command line.
-
-The secrets are just extra environment variables that are added to the ones defined in your config file - in the format:
-
-```
-SECRET1=VALUE1 SECRET2=VALUE2
-```
-
-You can either specify `--secrets my-secrets-file.env` to load these from a separate file.  Or you can load them from stdin.
-
-For example, I use [Bitwarden](https://bitwarden.com) as my password locker and use the Bitwarden CLI to access my secrets.  The CLI is installed through homebrew, I then authenticate and can use a command like:
-
-```sh
-bw get notes secrets@myapp.com | anvil app install deploy.myapp.yml -S
-```
-I have the environment variables for myapp.com stored in Bitwarden as a secure note with the title "secrets@myapp.com".  So `bw get notes secrets@myapp.com` loads them from my vault and pipes them to the `anvil app install` command.  The anvil command is using the `-S` (or `--secrets-stdin`) option which means it will read the information piped in by bitwarden.  So, once decrypted, the confidential data never touches a disk until it gets written into the dokku app configuration on the server.
+I'm a Rails developer and I built anvil to help me with my Rails apps.  Here are [some things I learnt along the way](/docs/ruby-on-rails.md).
 
 ## Contributing
 
+Check out the [Roadmap](/docs/roadmap.md)
+
 Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/standard-procedure-anvil.
 
 ## License

data/assets/cloudinit/redis.yml

@@ -34,11 +34,14 @@ runcmd:
   - sed -i -e '/^\(#\|\)AuthorizedKeysFile/s/^.*$/AuthorizedKeysFile .ssh\/authorized_keys/' /etc/ssh/sshd_config
   - sed -i '$a AllowUsers %{USER}' /etc/ssh/sshd_config
   # Set up Redis
+  - curl -fsSL https://packages.redis.io/gpg | gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
+  - echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/redis.list
+  - apt-get update
   - apt-get -y install redis-server
   - sed -i 's/supervised no/supervised systemd/g' /etc/redis/redis.conf
   - sed -i 's/bind 127.0.0.1 ::1/# bind 127.0.0.1 ::1/g' /etc/redis/redis.conf
   - sed -i 's/protected-mode yes/protected-mode no/g' /etc/redis/redis.conf
-  - systemctl restart redis.service
+  - systemctl restart redis-server.service
   - |
     cat > /etc/logrotate.d/redis-server << EOF
     /var/log/redis/redis-server*.log {

data/checksums/standard-procedure-anvil-0.2.1.gem.sha512

@@ -0,0 +1 @@
+2b56322e96127aa43986eaa2de436eafd64c7d5260d61db1a13605d95bf0fc3b331f006c0b534d845c05e781659518f37d3f8aa17475e6e926675d5ea1985a50

data/docs/app.md

@@ -0,0 +1,51 @@
+# The `app` command
+
+## Installing an application
+
+Move to your application's root folder and create the deploy.yml file (see below).  Then use the `app install` command to set dokku up for your first deployment.
+
+```sh
+anvil app install
+```
+
+This will SSH into the server (or servers if you have multiple) from your config file and:
+
+- Installs any dokku plugins that you have specified
+- Tells dokku to create the app
+- Uses your config file to set the environment variables for the app
+- Sets some sensible defaults for Nginx and makes sure it proxies correctly to your app
+- Optionally forwards the correct SSL/TLS headers if your app is behind a load-balancer
+- Finally it runs the post-installation scripts from your config file, which you can use to configure your plugins
+
+## Deploying an application
+
+Next up we deploy the app.
+
+```sh
+anvil app deploy
+```
+As this is the first deployment, anvil will create git remotes for each host, then do the initial git push.  If you have multiple servers configured, these should run in parallel (coming soon).  Once each deployment has completed, anvil will SSH in, scale your app and run the post-first-deployment scripts.
+
+You can then use the same `anvil app deploy` command to deploy the app again - but as it knows this isn't the first deployment (as it does not need to create the git remotes), it will run your post-deployment scripts (not post-first-deployment) each time.
+
+To change the number of processes (as defined by your Procfile), you can set the `scale` key(s) in your config file and then call:
+
+```sh
+anvil app scale
+```
+
+(COMING SOON)
+Finally, if you need to change the values of any environment variables, update your config file and use:
+
+```sh
+anvil app configure
+```
+
+## Configuration files
+
+The [anvil configuration file](/docs/configuration.md) is the heart of the system.
+
+## Secrets
+
+You don't want to store your secrets (passwords, encryption keys) in your anvil configuration.  Instead anvil can [read your secrets](/docs/secrets.md) from a separate file or from the command line.
+

data/docs/cloudinit.md

@@ -0,0 +1,33 @@
+# Building a server
+
+## Cloudinit
+
+A [CloudInit](https://cloudinit.readthedocs.io/en/latest/) file is a YML file that you load into a virtual machine while it is being created.  As the server boots, uses the cloudinit configuration to install software and set itself up.  With most cloud hosting providers, you will find an option for "user data", or something similar, on the "create a new server" page.
+
+### Generate a configuration
+
+So firstly we ask Anvil which cloudinit configurations it has available:
+
+```sh
+anvil cloudinit list
+```
+
+This will give us a list of prewritten cloud init scripts - of which dokku is probably the one we're most interested in.
+
+Next we tell anvil to generate our configuration:
+
+```sh
+anvil cloudinit generate dokku --user app --public-key ~/.ssh/my_key.pub > ~/Desktop/my_server.yml
+```
+
+Anvil generates a dokku configuration (and places it on our desktop) that will create an Ubuntu 22.04 box with docker and dokku preinstalled.  Plus it will create a user called `app` that can log in through SSH using a public key `my_key.pub`.  The server itself is locked down so only ports 80, 443 and 22 are open, only the users `app` and `dokku` are allowed to log in and they must use public/private key encryption - no passwords allowed.
+
+### Testing your configuration
+
+To test this, it's worth taking a look at [Multipass](https://multipass.run) - a tool from Canonical that lets you create virtual machines (using cloud init files) on your local machine.  This means you can try out various configurations without spending money at a hosting company.
+
+One thing to note when using multipass - it requires SSH access for a user called "ubuntu".  So take your generated cloudinit file, locate the SSH configuration section and the "AllowUsers" line - and add the "ubuntu" user to it.  Something like: `  - sed -i '$a AllowUsers %{USER} ubuntu dokku' /etc/ssh/sshd_config`.
+
+Multipass has its own private key generated for the ubuntu user, and uses this to manage the server.  Of course, the multipass VM is only on your machine, plus its private key is hidden away, so it's not a security risk.  But in general `anvil cloudinit generate` disallows all SSH access apart from your named user (using your own key), and the `dokku` user if applicable.
+
+Once you've built a preconfigured virtual machine, we can move on to getting our dokku application installed.  However, note that it can take several minutes for the initialisation process to complete - so don't start your deployment too early, or your server won't be ready and will reboot whilst your setting things up.

data/docs/configuration.md

@@ -0,0 +1,7 @@
+# Anvil configuration files
+
+An Anvil configuration file specifies the configuration for multiple servers and multiple apps.  Each server is configured, then each app is installed onto each server.
+
+For now take a look at the two samples in the spec folder - [single-server](/spec/fixtures/single-server.config.yml) and [multi-server](/spec/fixtures/multi-server.config.yml).
+
+Also check out the [tips for Ruby on Rails](/docs/ruby-on-rails.md) which has an example configuration file.

data/docs/roadmap.md

@@ -0,0 +1,12 @@
+# Roadmap
+
+As I mentioned, this is pretty much designed for my own use.
+
+There are a few bits I still need (which will be V1) and then I want to open it up.  But if it gets too generic, you might as well just write a load of shell scripts to manage dokku yourself - it's important to keep it simple.
+
+## To do
+
+- [ ] `app reconfigure`
+- [ ] Add `--first`/`--not-first` options to `app deploy` so you can override the first-deployment behaviour (in case you get a failure and need to re-run everything)
+- [ ] Instead of relying on the ssh-agent, allow the use of your private key when connecting to servers
+- [ ] Parallel execution across multiple hosts

data/docs/ruby-on-rails.md

@@ -0,0 +1,69 @@
+# Dokku and Ruby on Rails
+
+(incomplete - coming soon)
+
+- If using the Mysql plugin, use the Mysql2 protocol
+- store your RAILS_MASTER_KEY and SECRET_KEY_BASE outside of your configuration file (see [secrets](/docs/secrets.md))
+- in your config/environments/production.rb or config/environments/staging.rb set `config.force_ssl = false` - dokku's nginx configuration will do the redirect for you if you're using the Let's Encrypt plugin, or you can set the redirect on your load-balancer.  Setting `config.force_ssl = true` causes issues with the health checks
+- Make sure your app knows its hostname; set it as an environment variable in your configuration file and then use that hostname in your environment file as follows: `config.action_mailer.default_url_options = {host: ENV["HOSTNAME"]}` and `Rails.application.routes.default_url_options[:host] = config.action_mailer.default_url_options[:host]`.  This means that when you need to generate a full URL (as opposed to a relative path), Rails knows what to use.
+- Use a CHECKS file that looks like this (again using that HOSTNAME environment variable), so dokku's zero-deployment checks can connect correctly.  My `/health_check` route just returns a `200 OK` in most apps, but in some it actually checks the database connection, as well as some other services (although this causes problems with the initial deployment, which is why checks are switched off the first time through)
+```
+WAIT=10
+ATTEMPTS=5
+http://{{ var "HOSTNAME" }}/health_check
+```
+
+A typical Rails deploy.yml for a single-server, totally self-contained app, looks like this:
+
+```yaml
+version: 0.1
+hosts:
+  - myapp.example.com:
+      user: app
+app:
+  domain: myapp.example.com
+  port: 3000
+  environment:
+    - BUNDLE_WITHOUT=test:development
+    - CABLE_CHANNEL_PREFIX=myapp
+    - EMAIL_DOMAIN=mail.myapp.example.com
+    - EMAIL_HOST=smtp.myapp.example.com
+    - EMAIL_PORT=587
+    - EMAIL_USER=postmaster@mail.myapp.example.com
+    - HOSTNAME=myapp.example.com
+    - NODE_ENV=production
+    - RACK_ENV=production
+    - RAILS_ENV=production
+    - RAILS_LOG_TO_STDOUT=true
+    - RAILS_MAX_THREADS=10
+    - RAILS_SERVE_STATIC_FILES=true
+  resource_limit: 2048m
+  scale: web=2 worker=1
+  load_balancer: false
+  nginx:
+    client_max_body_size: 512m
+    proxy_read_timeout: 60s
+  plugins:
+    - cron-restart
+    - maintenance
+    - redis
+    - memcached
+    - mysql
+    - letsencrypt
+  scripts:
+    after_install:
+      - dokku cron-restart:set app schedule '0 3 * * *'
+      - dokku memcached:create memcached
+      - dokku memcached:link memcached app
+      - dokku redis:create redis_db
+      - dokku redis:link redis_db app
+      - dokku mysql:create mysql_db
+      - dokku config:set app MYSQL_DATABASE_SCHEME=mysql2
+      - dokku mysql:link mysql_db app
+    after_first_deploy:
+      - dokku letsencrypt:set app email me@myapp.example.com
+      - dokku letsencrypt:enable app
+      - dokku letsencrypt:cron-job --add
+      - dokku run app bin/rails db:seed
+
+```

data/docs/secrets.md

@@ -0,0 +1,20 @@
+# Secrets
+
+Finally, you'll probably want to check your deploy.yml file into source control.  But you _definitely_ don't want to be storing important secrets - database passwords, encryption keys and so on - where everyone can see them.
+
+So the `anvil app` commands also allow you to specify secrets, either from another file, or via the command line.
+
+The secrets are just extra environment variables that are added to the ones defined in your config file - in the format:
+
+```
+SECRET1=VALUE1 SECRET2=VALUE2
+```
+
+You can either specify `--secrets my-secrets-file.env` to load these from a separate file.  Or you can load them from stdin.
+
+For example, I use [Bitwarden](https://bitwarden.com) as my password locker and use the Bitwarden CLI to access my secrets.  The CLI is installed through homebrew, I then authenticate and can use a command like:
+
+```sh
+bw get notes secrets@myapp.com | anvil app install deploy.myapp.yml -S
+```
+I have the environment variables for myapp.com stored in Bitwarden as a secure note with the title "secrets@myapp.com".  So `bw get notes secrets@myapp.com` loads them from my vault and pipes them to the `anvil app install` command.  The anvil command is using the `-S` (or `--secrets-stdin`) option which means it will read the information piped in by bitwarden.  So, once decrypted, the confidential data never touches a disk until it gets written into the dokku app configuration on the server.

data/docs/why.md

@@ -0,0 +1,17 @@
+# Why does this exist?
+
+[Dokku](https://dokku.com) is great at installing and configuring containers on a single host.
+
+But you do need to install dokku, generate your configuration and environment variables, install your plugins and app and then configure all those plugins.
+
+Unfortunately, there's no [single configuration file](https://github.com/dokku/dokku/issues/1558) for dokku.
+
+In addition, dokku is really designed for managing a single server.  But I'm actually using it to manage multiple servers that are hidden behind a load-balancer.
+
+So to manage this, I wanted a single configuration file that I could user for all my dokku information, that could then use that configuration across multiple servers.
+
+Currently it's extremely tailored to my needs - it's built for Ubuntu 22.04, it creates a user called "app" (although you can change that), it names your dokku app "app".
+
+I've also added in cloudinit configs for some of the other servers I have to use.  Of course, these are not related to dokku, but anvil can generate them easily so it's useful to keep them all in one place.
+
+There are several [limitations](/docs/roadmap.md) to how it works - it does what I need but does need expansion.  That will come soon.

data/lib/anvil/app/host_deployer.rb

@@ -41,10 +41,12 @@ module Anvil
       end
 
       def create_git_remote
+        logger.info "git remote add #{host} dokku@#{host}:/app"
         logger.info `git remote add #{host} dokku@#{host}:/app`
       end
 
       def do_git_push
+        logger.info "git push #{host} #{branch}:main"
         logger.info `git push #{host} #{branch}:main`
       end
 

data/lib/anvil/app/host_installer.rb

@@ -41,6 +41,7 @@ module Anvil
         ssh.exec! "dokku docker-options:add app run \"--add-host=host.docker.internal:host-gateway\"", "set_dokku_options"
         ssh.exec! "dokku domains:set app #{configuration_for_app["domain"]}", "set_dokku_options"
         ssh.exec! "dokku proxy:ports-add app http:80:#{configuration_for_app["port"]}", "set_dokku_options"
+        ssh.exec! "dokku proxy:ports-add app https:443:#{configuration_for_app["port"]}", "set_dokku_options"
         ssh.exec! "dokku nginx:set app client-max-body-size #{configuration_for_app["nginx"]["client_max_body_size"]}", "set_dokku_options"
         ssh.exec! "dokku nginx:set app proxy-read-timeout #{configuration_for_app["nginx"]["proxy_read_timeout"]}", "set_dokku_options"
         if configuration_for_app["load_balancer"]

data/lib/anvil/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Anvil
-  VERSION = "0.2.0"
+  VERSION = "0.2.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: standard-procedure-anvil
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.2.2
 platform: ruby
 authors:
 - Rahoul Baruah
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-07-05 00:00:00.000000000 Z
+date: 2023-08-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -121,6 +121,14 @@ files:
 - checksums/standard-procedure-anvil-0.1.6.gem.sha512
 - checksums/standard-procedure-anvil-0.1.7.gem.sha512
 - checksums/standard-procedure-anvil-0.2.0.gem.sha512
+- checksums/standard-procedure-anvil-0.2.1.gem.sha512
+- docs/app.md
+- docs/cloudinit.md
+- docs/configuration.md
+- docs/roadmap.md
+- docs/ruby-on-rails.md
+- docs/secrets.md
+- docs/why.md
 - exe/anvil
 - lib/anvil.rb
 - lib/anvil/app.rb