cpl 0.1.0

data/postgres.md

@@ -0,0 +1,436 @@
+# Migrating Postgres database from Heroku infrastructure
+
+One of the biggest problems that will appear when moving from Heroku infrastructure is migrating the database. And
+despite it being rather easy if done between Heroku-hosted databases or non-Heroku-hosted databases (as Postgres has
+tools to do that naturally) it is not easily possible between Heroku and anything outside Heroku,
+**as Heroku doesn't allow setting up WAL replication for Postgres**. Period. No... any... replication outside of
+Heroku infrastructure for Postgres.
+
+Previously, it was said to be possible to ask Heroku support to manually set up WAL log shipping, but they
+don't want to do that anymore. Which leaves only 2 options:
+
+### Option A: dump and restore way
+
+Nothing problematic here in general **if you can withstand long application maintenance time**.
+You basically need to:
+
+1. enable maintenance
+2. stop the application completely and wait for all the database writes to finish
+3. dump database on Heroku
+4. restore the database on RDS
+5. start the application
+6. disable maintenance
+
+And if the database is small or it is a hobby app, this should not be looked any further.
+However, this is not acceptable for 99% of production apps as their databases are huge and maintenance time
+should be as small as possible.
+
+Rough timing for a 1Gb database can be (but your mileage may vary):
+
+- 2.5h creating Heroku backup
+- 0.5h downloading backup to EC2
+- 13h restoring a backup on RDS (in 4 threads)
+
+**~16h total time, equals maintenance downtime**
+
+### Option B: logical replication way
+
+There are several logical replication solutions exist for Postgres - Slony, Bucardo, Londiste, Mimeo, but... when
+you try to dig deeper, the only viable and up-to-date solution for purpose of migrating from Heroku to RDS is Bucardo.
+
+The migration process with Bucardo looks as follows:
+
+1. setup Bucardo on the dedicated EC2 instance
+2. dump Heroku database schema and restore it on RDS - rather fast as there is no data
+3. start Bucardo replication - this will install triggers and special schema to your database
+4. wait for replication to catch up - this may take a long time, but the application can continue working as usual
+5. enable maintenance
+6. stop the application completely and wait for replication to finally finish
+7. switch the database connection strings
+8. start the application
+9. disable maintenance
+
+Maintenance downtime here can be minutes not hours or days like in p1, but no free lunches - the process is more complex.
+
+Rough timing for a 1Gb database can be (but your mileage may vary):
+
+- whatever setup time, no hurry
+- 1.5 days for onetimecopy (in 1 thread) - DDL changes not allowed, but no downtime
+- 1-2 min for database switch, maintenance downtime
+
+**~2 days total time, ~1-2 min maintenance downtime**
+
+### Some considerations:
+
+- DDL changes should be "frozen and postponed" while Bucardo replication. There is also a way to stop replication,
+update DDL in both databases, and restart replication, however, as well no-DDL for a day or two seems a
+reasonable restriction for production databases vs potential errors.
+
+- there is a "speed up" option to restore dump (with threads) and then run Bucardo to catch up only deltas, but it
+looks unnecessary as speed gain is minimal vs potential errors. It will not speed up things dramatically, but will just
+save a couple of hours of non-maintenance time (which most probably be spent on the command line) so not worth doing.
+
+## Before replication
+
+### Application code changes
+
+Before everything, we need to recheck the database schema and ensure **that every table has a primary key (PK)
+in place** as Bucardo is using PKs for replication.
+
+> NOTE: theoretically Bucardo can work with uniq indexes as well, but having a PK on each table is easy and avoids
+unnesessary complications
+
+So, please stop, and do whatever is needed for your application.
+
+### Choosing database location and EC2 location
+
+All Heroku Postgres databases for location US are running in AWS on `us-east-1`. Control Plane, on the other side,
+recommends `us-east-2` as a default. So we need to choose:
+
+- either simple setup - main database in `us-east-2`
+- or a bit more complex - main database in `us-east-2`, replica in `us-east-1` (which can be removed later)
+
+This makes sense if your application supports working with replicas. Then read-only `SELECT` queries will go to the
+read replica and write `INSERT/UPDATE` queries will go to the main write-enabled database.
+This way we can keep most reading latency to the minimum.
+
+Anyway, it is worth to consider developing such a mode in the application if you want to scale in more than 1 region.
+
+### Create new EC2 instance which we will use for database replication
+
+- better if it will be in the same AWS location where RDS database will be (most probably `us-east-2`)
+- choose Ubuntu as OS
+- use some bigger instance, e.g. `m6i.4xlarge` - price doesn't matter much, as such instance will not run long time
+- if you will be copying backup via this instance, choose sufficient space for both OS and backup and some free space
+- create security group `public-access` with all inbound and outbound traffic allowed. this will be handy as well for
+database setup. if you need tighter access controls, up to you
+- generate a new certificate and save it locally (e.g. `bucardo.pem`), will be used for SSH connection. Do not forget to
+update correct permissions e.g. `chown TODO ~/Dowloads/bucardo.pem`
+
+After the instance will be running on AWS, you can connect to it via SSH as follows:
+```sh
+ssh ubuntu@1.2.3.4 -i ~/Downloads/bucardo.pem
+```
+
+### Creating RDS instance
+
+- check `public` box
+- pick `public-access` security group (or whatever you need)
+- if you will be restoring from backup, it is possible to choose temporary bigger instance e.g. `db.r6i.4xlarge`, which
+can be downgraded later.
+- if you will be using bucardo onetimecopy, then it is ok to select any instance you may need, as bucardo does copying
+in a single thread
+- it is fairly easy to switch database instance type afterwards and requires only minimal downtime
+- storage space. this needs a good pick, as it is a) not possible to shrink and b) auto-expanding which AWS offers and
+which should be enabled by default can block database modifications for quite long periods (days)
+
+### Running commands in detached mode on the EC2 instance
+
+Some commands that run on EC2 may take a long time and we may want to disconnect from the SSH session while the command
+will continue running. And we want to reconnect to the session and see the progress. Possibly without installing
+special tools. This can be accomplished with `screen` command, e.g.:
+
+```sh
+# this will start a backround process and return to terminal (which can be closed)
+screen -dmL ...your command...
+
+# checking if screen is still running in the background
+ps aux | grep -i screen
+
+# see the output log
+cat screenlog.0
+```
+
+### Installing Postgres and Bucardo on EC2
+
+Now, when RDS is running and EC2 is running we can start installing local Postgres and Bucardo itself. Let's install
+Postgres 13 first. It may be possible to install the latest Postgres, but 13 seems the best choice atm.
+
+```sh
+# update all your packages
+sudo apt update
+sudo apt upgrade -y
+
+# add postgres repository key
+sudo sh -c 'echo "deb [arch=$(dpkg --print-architecture)] http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
+
+wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
+
+# update again
+sudo apt-get update
+
+# install packages
+sudo apt-get -y install make postgresql-13 postgresql-client-13 postgresql-plperl-13 libdbd-pg-perl libdbix-safe-perl
+```
+
+Postgres Perl language `plperl` as well as DBD and DBIx packages are needed for Bucardo.
+
+Now, as all dependencies are installed, we can install Bucardo from latest tarball.
+
+```sh
+# install Bucardo itself
+wget https://bucardo.org/downloads/Bucardo-5.6.0.tar.gz
+tar xzf Bucardo-5.6.0.tar.gz
+cd Bucardo-5.6.0
+perl Makefile.PL
+sudo make install
+
+# create dirs and fix permissions
+sudo mkdir /var/run/bucardo
+sudo mkdir /var/log/bucardo
+sudo chown ubuntu /var/run/bucardo/
+sudo chown ubuntu /var/log/bucardo
+```
+
+After that, Bucardo is physically installed as a package and runnable but we need to configure everything.
+Let's start with Postgres. As this is a temporary installation (only for the period of replication),
+it is rather safe to set `trust` localhost connections (or set up another way if you want this).
+
+
+For this, we need to edit `pg_hba.conf` as follows:
+```sh
+# edit pg config to make postgres trusted
+sudo nano /etc/postgresql/13/main/pg_hba.conf
+```
+
+in that file change the following lines to `trust`
+```sh
+# in pg_hba.conf
+local   all             postgres                                trust
+local   all             all                                     trust
+```
+
+and restart Postgres to pick up changes
+```sh
+# restart postgres
+sudo systemctl restart postgresql
+```
+
+And finally-finally we can install Bucardo service database on local Postgres and see if everything runs.
+
+```sh
+# this will create local bucardo "service" database
+bucardo install
+
+# for option 3 pick `postgres` as a user
+# for option 4 pick `postgres` as a database from where initial connection should be attempted
+```
+
+:tada: :tada: :tada: now we have local Postgres and Bucardo running, and can continue with external services
+configuration.
+
+### Configuring external (Heroku, RDS) database connections
+
+For this, we will use `pg_service.conf`. It will not work in all places, and sometimes we will need to provide
+connection properties manually, but for many commands, it is very useful.
+
+```sh
+# create and edit .pg_service.conf
+touch ~/.pg_service.conf
+nano ~/.pg_service.conf
+```
+
+```ini
+# ~/.pg_service.conf
+
+[heroku]
+host=ec2-xxx.compute-1.amazonaws.com
+port=5432
+dbname=xxx
+user=xxx
+password=xxx
+
+[rds]
+host=xxx.us-east-2.rds.amazonaws.com
+port=5432
+dbname=xxx
+user=postgres
+password=xxx
+```
+
+Test connectivity to databases with:
+```sh
+psql service=heroku -c '\l+'
+psql service=rds -c '\l+'
+```
+
+You will see all databases set up on each server (and can see their size):
+```console
+       Name        |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |   Size    | Tablespace |                Description
+-------------------+----------+----------+-------------+-------------+-----------------------+-----------+------------+--------------------------------------------
+ my-production-db  | xxx      | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       | 821 GB    | pg_default |
+ postgres          | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       | 8205 kB   | pg_default | default administrative connection database
+ rdsadmin          | rdsadmin | UTF8     | en_US.UTF-8 | en_US.UTF-8 | rdsadmin=CTc/rdsadmin+| No Access | pg_default |
+                   |          |          |             |             | rdstopmgr=Tc/rdsadmin |           |            |
+ template0         | rdsadmin | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/rdsadmin          +| 8033 kB   | pg_default | unmodifiable empty database
+                   |          |          |             |             | rdsadmin=CTc/rdsadmin |           |            |
+ template1         | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8205 kB   | pg_default | default template for new databases
+                   |          |          |             |             | postgres=CTc/postgres |           |            |
+(5 rows)
+```
+
+After both databases are connectable, we can proceed with replication itself.
+
+## Performing replication
+
+:fire: :fire: :fire: **IMPORTANT: from this step, DDL changes are not allowed** :fire: :fire: :fire:
+
+### Application changes
+
+- temporary freeze all DDL changes till replication will finish
+- temporary disable all background jobs or services that are possible to stop. This will help to lessen database
+load and especially, where possible, to decrease database write operations that will decrease replication pipe as well.
+
+### Dump and restore the initial schema
+
+This step doesn't take much time (as it is only a database schema without data),
+but it is definitely handy to save all output, and closely check for any errors.
+
+```sh
+# save heroku schema to `schema.sql`
+pg_dump service=heroku --schema-only --no-acl --no-owner -v > schema.sql
+
+# restore `schema.sql` on RDS
+psql service=rds < schema.sql
+```
+
+### Configure Bucardo replication
+
+After we have all databases connectable and all same schema, we can tell Bucardo what it needs to replicate:
+```sh
+# add databases
+bucardo add db from_db dbhost=xxx dbport=5432 dbuser=xxx dbpass=xxx dbname=xxx
+bucardo add db to_db dbhost=xxx dbport=5432 dbuser=postgres dbpass=xxx dbname=xxx
+
+# mark all tables and sequences for replication
+bucardo add all tables
+bucardo add all sequences
+```
+Here, Bucardo will connect to databases and collect object metadata for replication.
+After that, we can add sync as well:
+
+```sh
+# add sync itself
+bucardo add sync mysync tables=all dbs=from_db,to_db onetimecopy=1
+```
+
+The most important option here is `onetimecopy=1` which will tell Bucardo to perform initial data copying
+(when sync will start). Such a copying is done *in a single thread* by creating a pipe (via Bucardo) as follows:
+```SQL
+-- on heroku
+COPY xxx TO STDOUT
+-- on rds
+COPY xxx FROM STDIN
+```
+
+### Run sync
+
+And now, when everything is ready, we can push the button and go for a long :coffee: or maybe even a weekend.
+
+```sh
+# starts Bucardo sync daemon
+bucardo start
+```
+
+As well it is ok to disconnect from SSH as Bucardo daemon will continue working in background.
+
+### Monitor status
+
+To check the progress of sync (from Bucardo perspective):
+```sh
+# overall progress of all syncs
+bucardo status
+
+# single sync progress
+bucardo status mysync
+```
+
+To check what's going on in databases directly:
+```sh
+# Bucardo adds a comment to it's queries, so it is fairly easy to grep those
+psql service=heroku -c 'select * from pg_stat_activity' | grep -i bucardo
+psql service=rds -c 'select * from pg_stat_activity' | grep -i bucardo
+```
+
+### After replication will catch up, but before databases switch
+
+1. Please do a sanity check of data in tables. E.g. check:
+
+- table `COUNT`
+- min/max of PK ids where applicable
+- min/max of `created_at/updated_at` where applicable
+
+2. For p1 it is possible to use our checker script that will do this automatically (TODO)
+
+3. Refresh materialized views manually (as they are not synced by Bucardo).
+Just go to `psql` and `REFRESH MATERIALIZED VIEW ...`
+
+## Switch databases
+
+:fire: :fire: :fire: **This is final non reverisble step now** :fire: :fire: :fire:
+Before this point, all changes can be easily removed or reversed and database can stay on Heroku as it was before,
+afther this switch it is not possible (at least easily).
+
+So... after sync will catch up, basically it is needed to:
+
+1. start maintenance mode on heroku `heroku maintenance:on`
+2. scale down and stop all the dynos
+3. wait a bit for all queries to finish and replication catch up latest changes
+4. detach heroku postgres from DATABASE_URL
+5. set `DATABASE_URL` to RDS url (plaintext now)
+6. start dynos
+7. wait for their readiness with `heroku ps:wait`
+8. stop maintenance with `heroku maintenance:off`
+9. :fire: **Now we are fully on RDS, so DDL changes are allowed** :fire:
+10. you could gradually enable all background jobs and services which were temporary stopped
+
+## After switch
+
+As we now running on RDS, there is only single task left to do on Heroku - make a final backup of database and save it.
+
+```sh
+# to capture backup (will take lots of time), can be disconnected
+heroku pg:backups:capture -a example-app
+
+# to get url of backup
+heroku pg:backups:url bXXXX -a example-app
+```
+
+Now you can download it locally or copy to S3 via EC2 as it will take quite some time and traffic.
+
+```sh
+# download dump to EC2
+screen -dmL time curl 'your-url' -o latest.dump
+
+# install aws cli (in a way reccomended by Amazon)
+# ...TODO...
+
+# configure aws credentials
+aws configure
+
+# check S3 access
+aws s3 ls
+
+# upload to S3
+screen -dmL time aws s3 cp latest.dump s3://my-dumps-bucket/ --region us-east-1
+```
+
+# Refs
+
+https://bucardo.org
+
+https://stackoverflow.com/questions/22264753/linux-how-to-install-dbdpg-module
+
+https://gist.github.com/luizomf/1a7994cf4263e10dce416a75b9180f01
+
+https://www.waytoeasylearn.com/learn/bucardo-installation/
+
+https://gist.github.com/knasyrov/97301801733a31c60521
+
+https://www.cloudnation.nl/inspiratie/blogs/migrating-heroku-postgresql-to-aurora-rds-with-almost-minimal-downtime
+
+https://blog.porter.run/migrating-postgres-from-heroku-to-rds/
+
+https://www.endpointdev.com/blog/2017/06/amazon-aws-upgrades-to-postgres-with/
+
+https://aws.amazon.com/blogs/database/migrating-legacy-postgresql-databases-to-amazon-rds-or-aurora-postgresql-using-bucardo/

data/redis.md

@@ -0,0 +1,112 @@
+# Migrating Redis database from Heroku infrastructure
+
+**General considerations:**
+
+1. Heroku uses self-signed TLS certificates, which are not verifiable. It needs special handling by setting
+TLS verification to `none`, otherwise most apps are not able to connect.
+
+2. We are moving to private Redis that don't have a public URL, so have to do it from a Control Plane GVC container.
+
+The tool that satisfies those criteria is [Redis-RIOT](https://developer.redis.com/riot/riot-redis/index.html)
+
+**Heroku Redis:**
+
+As Redis-RIOT says, master redis should have keyspace-notifications set to `KA` to be able to do live replication.
+To do that:
+
+```sh
+heroku redis:keyspace-notifications -c KA -a my-app
+```
+
+Connect to heroku Redis CLI:
+```sh
+heroku redis:cli -a my-app
+```
+
+**Control Plane Redis:**
+
+Connect to Control Plane Redis CLI:
+
+```sh
+# open cpl interactive shell
+cpl run bash -a my-app
+
+# install redis CLI if you don't have it in Docker
+apt-get update
+apt-get install redis -y
+
+# connect to local cloud Redis
+redis-cli -u MY_CONTROL_PLANE_REDIS_URL
+```
+
+**Useful Redis CLI commands:**
+
+Quick-check keys qty:
+```
+info keyspace
+
+# Keyspace
+db0:keys=9496,expires=2941,avg_ttl=77670114535
+```
+
+**Create a Control Plane sync workload**
+
+```
+name: riot-redis
+
+suspend: true
+min/max scale: 1/1
+
+firewall: all firewalls off
+
+image: fieldengineering/riot-redis
+
+CPU: 1 Core
+RAM: 1 GB
+
+command args:
+  --info
+  -u
+  rediss://...your_heroku_redis_url...
+  --tls-verify=NONE
+  replicate
+  -h
+  ...your_control_plane_redis_host...
+  --mode
+  live
+```
+
+**Sync process**
+
+1. open 1st terminal window with heroku redis CLI, check keys qty
+2. open 2nd terminal window with controlplane redis CLI, check keys qty
+3. start sync container
+4. open logs with `cpl logs -a my-app -w riot-redis`
+4. re-check keys sync qty again
+5. stop sync container
+
+Result:
+```
+Setting commit interval to default value (1)
+Setting commit interval to default value (1)
+Job: [SimpleJob: [name=snapshot-replication]] launched with the following parameters: [{}]
+Executing step: [snapshot-replication]
+Scanning   0% ╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━    0/8891 (0:00:00 / ?)Job: [SimpleJob: [name=scan-reader]] launched with the following parameters: [{}]
+Executing step: [scan-reader]
+Scanning  61% ━━━━━━━━━━━━━━━━╸━━━━━━━━━━ 5460/8891 (0:00:07 / 0:00:04) 780.0/sStep: [scan-reader] executed in 7s918ms
+Closing with items still in queue
+Job: [SimpleJob: [name=scan-reader]] completed with the following parameters: [{}] and the following status: [COMPLETED] in 7s925ms
+Scanning 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9482/9482 (0:00:11 / 0:00:00) 862.0/s
+Step: [snapshot-replication] executed in 13s333ms
+Executing step: [verification]
+Verifying   0% ╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━    0/8942 (0:00:00 / ?)Job: [SimpleJob: [name=RedisItemReader]] launched with the following parameters: [{}]
+Executing step: [RedisItemReader]
+Verifying   2% ╺━━━━━━━━━━━━━━━━━  220/8942 (0:00:00 / 0:00:19) ?/s >0 T0 ≠Step: [RedisItemReader] executed in 7s521ms
+Closing with items still in queue
+Job: [SimpleJob: [name=RedisItemReader]] completed with the following parameters: [{}] and the following status: [COMPLETED] in 7s522ms
+Verification completed - all OK
+Step: [verification] executed in 7s776ms
+Job: [SimpleJob: [name=snapshot-replication]] completed with the following parameters: [{}] and the following status: [COMPLETED] in 21s320ms
+```
+
+Total sync time ~1min

data/script/generate_commands_docs

@@ -0,0 +1,60 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative "../lib/cpl"
+
+commands_str_arr = []
+
+commands = Command::Base.all_commands
+commands.each do |_command_key, command_class|
+  next if command_class::HIDE
+
+  name = command_class::NAME
+  usage = command_class::USAGE.empty? ? name : command_class::USAGE
+  options = command_class::OPTIONS
+  long_description = command_class::LONG_DESCRIPTION
+  examples = command_class::EXAMPLES
+
+  command_str = "### `#{name}`\n\n"
+  command_str += "#{long_description.strip}\n\n"
+
+  if examples.empty?
+    options_str_arr = []
+    options.each do |option|
+      next unless option[:params][:required]
+
+      options_str_arr.push("#{option[:params][:aliases][0]} $#{option[:params][:banner]}")
+    end
+    options_str = options_str_arr.join(" ")
+
+    command_str += "```sh\ncpl #{usage}"
+    command_str += " #{options_str}" unless options_str.empty?
+    command_str += "\n```"
+  else
+    command_str += examples.strip
+  end
+
+  commands_str_arr.push(command_str)
+end
+
+commands_str = commands_str_arr.join("\n\n")
+
+commands_path = "#{__dir__}/../docs/commands.md"
+commands_data =
+  <<~HEREDOC
+    <!-- NOTE: This file is automatically generated by running `script/generate_commands_docs`. Do NOT edit it manually. -->
+
+    ### Common Options
+
+    ```
+    -a XXX, --app XXX         app ref on Control Plane (GVC)
+    ```
+
+    This `-a` option is used in most of the commands and will pick all other app configurations from the project-specific
+    `.controlplane/controlplane.yml` file.
+
+    ### Commands
+
+    #{commands_str}
+  HEREDOC
+File.binwrite(commands_path, commands_data)

data/templates/gvc.yml

@@ -0,0 +1,13 @@
+kind: gvc
+name: APP_GVC
+spec:
+  env:
+    - name: MEMCACHE_SERVERS
+      value: memcached.APP_GVC.cpln.local
+    - name: REDIS_URL
+      value: redis://redis.APP_GVC.cpln.local:6379
+    - name: DATABASE_URL
+      value: postgres://postgres:password123@postgres.APP_GVC.cpln.local:5432/APP_GVC
+  staticPlacement:
+    locationLinks:
+      - /org/APP_ORG/location/APP_LOCATION

data/templates/identity.yml

@@ -0,0 +1,2 @@
+kind: identity
+name: APP_GVC-identity

data/templates/memcached.yml

@@ -0,0 +1,23 @@
+kind: workload
+name: memcached
+spec:
+  type: standard
+  containers:
+    - name: memcached
+      cpu: 3m
+      memory: 10Mi
+      args:
+        - '-l'
+        - 0.0.0.0
+      image: 'memcached:alpine'
+      ports:
+        - number: 11211
+          protocol: tcp
+  defaultOptions:
+    autoscaling:
+      metric: latency
+      maxScale: 1
+    capacityAI: false
+  firewallConfig:
+    internal:
+      inboundAllowType: same-gvc

data/templates/postgres.yml

@@ -0,0 +1,31 @@
+kind: workload
+name: postgres
+spec:
+  type: standard
+  containers:
+    - name: postgres
+      cpu: 50m
+      memory: 200Mi
+      env:
+        - name: PGUSER
+          value: postgres
+        - name: POSTGRES_PASSWORD
+          value: password123
+        - name: POSTGRES_USER
+          value: postgres
+      image: 'postgres:13.8-alpine'
+      ports:
+        - number: 5432
+          protocol: tcp
+      volumes:
+        - path: /var/lib/postgresql/data
+          recoveryPolicy: retain
+          uri: 'scratch://postgres-vol'
+  defaultOptions:
+    autoscaling:
+      metric: latency
+      maxScale: 1
+    capacityAI: false
+  firewallConfig:
+    internal:
+      inboundAllowType: same-gvc