@anyakichi/cdk-rsync-backup 0.5.4 → 0.5.6

package/CLAUDE.md

@@ -0,0 +1,42 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Commands
+
+- `npm run build` — compile TypeScript (`lib/*.ts` → `lib/*.js` + `.d.ts`). `prebuild` regenerates `lib/version.ts` from `package.json` version.
+- `npm test` — run Jest tests (`ts-jest`, matches `test/**/*.test.ts`).
+- `npm run watch` — `tsc -w` for incremental compilation.
+- `npm run format` — run `sort-package-json` + Prettier over `package.json` and `lib`. Prettier sorts imports (`prettier.config.js`).
+
+Compiled JS/`.d.ts` output is committed alongside the sources in `lib/` (this is the published npm package, `main: lib/index.js`). Run `npm run build` after editing `lib/index.ts` so the shipped `.js` stays in sync.
+
+There is no lint step and no CDK app here — this is a library (L3 construct) consumed by other CDK stacks. Publishing is automated: `.github/workflows/publish.yml` runs `npm publish` (npm trusted publishing / OIDC) on GitHub release creation.
+
+## Architecture
+
+This package exports one L3 construct, `RsyncBackup` (`lib/index.ts`). The design splits responsibilities between **synth time** (TypeScript, generates CloudFormation) and **backup runtime** (bash on the EC2 instance).
+
+### Synth time — `lib/index.ts`
+
+The constructor provisions: an S3 logs bucket, a dual-stack VPC (public subnet, no NAT), a security group (SSH open to the world), an IAM role granting the EC2/S3/SSM actions the backup script needs, an EC2 key pair, and a single long-lived EC2 instance (Ubuntu 24.04, `t4g.nano` default). Most props have sensible defaults and can be overridden (`vpc`, `securityGroup`, `instanceType`, `logsBucket`, etc.).
+
+Nearly all behavior is wired through **EC2 UserData** built up incrementally in the constructor:
+- Installs `aws-cli`, downloads the `assets/` directory (packaged as an `s3assets.Asset`, unzipped to `/srv/rsync-backup`), installs `rsync-backup.sh` as `/usr/local/bin/rsync-backup`.
+- Writes `MAX_SNAPSHOTS` and `S3_LOGS_BUCKET` into `/etc/environment`.
+- For each `RsyncBackupModule`, renders a per-module `rsyncd.<name>.conf` (via `sed` on `@host@`) and appends a **forced-command** line to `/root/.ssh/authorized_keys`. That forced command is what pins an incoming SSH/rsync session to `rsync-backup <name> <size> /dev/sd<X>` — the client-supplied rsync command is ignored. Device letters are assigned sequentially from `f` per module index.
+- With no `modules`, a single `backup` module is configured as a fallback.
+
+Two notable optional flags:
+- `swapSize` — appends `fallocate`/`mkswap`/`swapon` + `/etc/fstab` commands (helps small instance types survive heavy rsync).
+- `useEIP` — allocates an EIP **and** injects a UserData block that persists SSH host keys to SSM Parameter Store (`/ec2/hostkeys/rsync-backup/*`), so a re-created fixed-IP instance keeps the same host keys.
+
+The instance's CDK logical ID embeds the library version (`Instance-<major.minor>`, optionally `-<instanceVersion>`); bumping it forces instance replacement.
+
+### Runtime — `assets/rsync-backup.sh`
+
+This is the actual backup engine, invoked once per rsync connection via the authorized_keys forced command. Per run it: finds/creates an encrypted gp3 EBS volume (from the latest matching snapshot if one exists, else fresh + `mkfs.ext4`), attaches and mounts it, runs `rsync --server --daemon` with the module's config to receive data, unmounts/detaches, creates a snapshot tagged `Name=backup-<host>` + `rsync-backup`, uploads a gzipped log to S3, then **backgrounds** (via `disown`) the snapshot-wait → volume-delete → old-snapshot pruning (keeping `MAX_SNAPSHOTS`, default 15). Size changes are handled at runtime: growing creates the volume larger from the snapshot + `resize2fs`; shrinking receives the transfer on the old-size volume, then migrates data to a fresh smaller volume in the disowned background block (interim tag `Name=backup-<host>-migrating`, module device letter mapped f–p→q–z) and snapshots the new volume. Runs are serialized per host with `flock` on `/srv/rsync-backup/rsync-backup.<host>.lock`, held by the background block until cleanup/migration completes. Volumes and snapshots are correlated by the `Name=backup-<host>` tag. `assets/rsyncd.conf` + `rsyncd.inc` are the rsyncd templates (`@host@` is substituted at synth time).
+
+### Tests
+
+`test/cdk-rsync-backup.test.ts` is currently a commented-out placeholder — there is no meaningful test coverage yet. New tests would use `aws-cdk-lib/assertions` `Template.fromStack` against a stack containing `RsyncBackup`.

package/README.md

@@ -50,9 +50,22 @@ export class CdkDemoStack extends cdk.Stack {
 If no EBS snapshot for `hostname` does not exist, a new EBS volume is
 created on rsync execution with specified `size`.
 
-If an EBS snapshot exist, a new EBS volume is created from the latest
-EBS snapshot. The volume size is depends on the snapshot and `size`
-parameter is ignored.
+If an EBS snapshot exists, a new EBS volume is created from the latest
+EBS snapshot. When the `size` parameter differs from the snapshot
+size:
+
+- Growing: the volume is created with the new size and the filesystem
+  is expanded automatically before receiving data.
+- Shrinking: the backup is received on a volume of the previous size
+  first, so the client never has to wait. After the transfer
+  completes, the data is copied to a new smaller volume in the
+  background and the snapshot is taken from the new volume. If the
+  data does not fit into the new size, shrinking is skipped and the
+  previous size is kept.
+
+Note that `size` is embedded in the instance's authorized_keys at
+first boot. To apply a size change to an existing deployment, replace
+the instance by setting or bumping the `instanceVersion` property.
 
 When you create a new backup, simply execute rsync.
 

package/assets/rsync-backup.sh

@@ -11,6 +11,23 @@ AWS_DEFAULT_REGION=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.25
 INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
 INSTANCE_AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/availability-zone)
 
+prune_snapshots() {
+    local name="$1"
+    local i
+    local -a snapshot_ids
+
+    if ((MAX_SNAPSHOTS > 0)); then
+        read -r -a snapshot_ids <<<"$(aws ec2 describe-snapshots \
+            --owner-ids self \
+            --filter "Name=tag:Name,Values=$name" \
+            --query "reverse(sort_by(Snapshots,&StartTime))[].SnapshotId" \
+            --output text)"
+        for ((i = MAX_SNAPSHOTS; i < ${#snapshot_ids[@]}; i++)); do
+            aws ec2 delete-snapshot --snapshot-id "${snapshot_ids[$i]}"
+        done
+    fi
+}
+
 main() {
     local host="$1"
     local size="$2"
@@ -18,8 +35,18 @@ main() {
     local name="backup-$1"
     local mnt="/srv/rsync-backup/mnt/$host"
     local logfile="/srv/rsync-backup/rsync-backup.$host.log"
-    local args dev logfile_gz s3base snapshot_id timestamp volume_id
-    local -a snapshot_ids
+    local args dev logfile_gz migrate new_dev new_device new_mnt \
+        new_volume_id s3base snapshot_id snapshot_size stale_volume_ids \
+        timestamp used v volume_id volume_size
+
+    # Serialize runs per host. The lock is inherited by the disowned
+    # background jobs, so it is held until the snapshot cleanup or the
+    # shrink migration is finished.
+    exec 9>"/srv/rsync-backup/rsync-backup.$host.lock"
+    if ! flock -n 9; then
+        echo "backup or migration for $host is in progress" >&2
+        exit 1
+    fi
 
     timestamp="$(date -u "+%Y%m%dT%H%MZ")"
     s3base=s3://$S3_LOGS_BUCKET/RsyncBackupLogs/$(date -u "+%Y/%m/%d")/
@@ -39,15 +66,26 @@ main() {
             exit 1
         fi
 
-        # Get the latest snapshot ID.
-        snapshot_id=$(aws ec2 describe-snapshots --owner-ids self \
+        # Get the latest snapshot ID and its volume size.
+        read -r snapshot_id snapshot_size <<<"$(aws ec2 describe-snapshots \
+            --owner-ids self \
             --filter "Name=tag:Name,Values=$name" \
-            --query "reverse(sort_by(Snapshots,&StartTime))[0].SnapshotId" \
-            --output text)
+            --query "reverse(sort_by(Snapshots,&StartTime))[0].[SnapshotId,VolumeSize]" \
+            --output text)"
 
         args=()
         if [[ $snapshot_id != None ]]; then
             args+=(--snapshot-id "$snapshot_id")
+            if ((size > snapshot_size)); then
+                # Grow the volume. The filesystem is expanded with
+                # resize2fs after mount.
+                args+=(--size "$size")
+            elif ((size < snapshot_size)); then
+                # The shrink migration reads the whole volume, which is
+                # painfully slow against lazily-loaded snapshot blocks.
+                # Pay for provisioned-rate initialization instead.
+                args+=(--volume-initialization-rate 100)
+            fi
         else
             args+=(--size "$size")
         fi
@@ -78,8 +116,17 @@ main() {
             mkfs.ext4 -q "$dev"
         fi
         mount "$dev" "$mnt"
+    else
+        dev=$(findmnt -no SOURCE "$mnt")
     fi
 
+    # Expand the filesystem to the volume size. This is a no-op unless
+    # the volume was created larger than the snapshot.
+    resize2fs "$dev" >>"$logfile" 2>&1
+
+    volume_size=$(aws ec2 describe-volumes --volume-ids "$volume_id" \
+        --query "Volumes[0].Size" --output text)
+
     mkdir -p "$mnt/$host"
 
     df -h "$dev" >>"$logfile"
@@ -88,39 +135,127 @@ main() {
 
     df -h "$dev" >>"$logfile"
 
-    umount "$mnt"
+    # If the requested size is smaller than the current volume, migrate
+    # the data to a new smaller volume after the transfer so that the
+    # client does not have to wait.
+    migrate=""
+    if ((volume_size > size)); then
+        used=$(df -BG --output=used "$mnt" | tail -n 1 | tr -dc '0-9')
+        if ((used * 100 < size * 90)); then
+            migrate=1
+        else
+            echo "data (${used}G used) does not fit into ${size}G;" \
+                "keeping ${volume_size}G volume" >>"$logfile"
+        fi
+    fi
+
+    if [[ -z $migrate ]]; then
+        umount "$mnt"
+
+        aws ec2 detach-volume --volume-id "$volume_id" &>/dev/null
+
+        snapshot_id=$(aws ec2 create-snapshot --volume-id "$volume_id" \
+            --description "$name-$timestamp" \
+            --tag-specifications "ResourceType=snapshot,Tags=[{Key=Name,Value=$name},{Key=rsync-backup,Value=''}]" \
+            --query SnapshotId --output text)
 
-    aws ec2 detach-volume --volume-id "$volume_id" &>/dev/null
+        gzip -c "$logfile" >"$logfile_gz"
 
-    snapshot_id=$(aws ec2 create-snapshot --volume-id "$volume_id" \
-        --description "$name-$timestamp" \
-        --tag-specifications "ResourceType=snapshot,Tags=[{Key=Name,Value=$name},{Key=rsync-backup,Value=''}]" \
-        --query SnapshotId --output text)
+        aws s3 cp "$logfile_gz" "$s3base"
 
-    gzip -c "$logfile" >"$logfile_gz"
+        rm -f "$logfile" "$logfile_gz"
 
-    aws s3 cp "$logfile_gz" "$s3base"
+        (
+            while true; do
+                aws ec2 wait snapshot-completed --snapshot-ids "$snapshot_id" &&
+                    break
+            done
+            aws ec2 wait volume-available --volume-ids "$volume_id"
+            aws ec2 delete-volume --volume-id "$volume_id"
+
+            prune_snapshots "$name"
+        ) &>/dev/null &
+        disown
+        return
+    fi
 
-    rm -f "$logfile" "$logfile_gz"
+    new_mnt="$mnt.new"
+    # Attach in the q-z range so we never collide with module devices
+    # (f-p). The real device node is resolved by volume ID anyway.
+    new_device="${device%?}$(tr 'f-p' 'q-z' <<<"${device: -1}")"
 
     (
+        # Clean up a stale volume from a previously failed migration.
+        umount "$new_mnt" 2>/dev/null || true
+        stale_volume_ids=$(aws ec2 describe-volumes \
+            --filter "Name=tag:Name,Values=$name-migrating" \
+            --query "Volumes[].VolumeId" --output text)
+        for v in $stale_volume_ids; do
+            aws ec2 detach-volume --volume-id "$v" &>/dev/null || true
+            aws ec2 wait volume-available --volume-ids "$v"
+            aws ec2 delete-volume --volume-id "$v"
+        done
+
+        new_volume_id=$(aws ec2 create-volume \
+            --availability-zone "$INSTANCE_AZ" \
+            --encrypted --volume-type gp3 --size "$size" \
+            --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=$name-migrating},{Key=rsync-backup,Value=''}]" \
+            --query VolumeId --output text)
+
+        aws ec2 wait volume-available --volume-ids "$new_volume_id"
+        aws ec2 attach-volume --volume-id "$new_volume_id" \
+            --instance-id "$INSTANCE_ID" --device "$new_device" &>/dev/null
+
+        if [[ -e /dev/disk/by-id ]]; then
+            new_dev="/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_${new_volume_id/-/}_1"
+        else
+            new_dev="${new_device/sd/xvd}"
+        fi
+
+        while true; do
+            if [[ -e $new_dev ]]; then
+                break
+            fi
+            sleep 1
+        done
+
+        mkfs.ext4 -q "$new_dev"
+        mkdir -p "$new_mnt"
+        mount "$new_dev" "$new_mnt"
+
+        rsync -aAXHS --numeric-ids --exclude=/lost+found \
+            "$mnt/" "$new_mnt/" >>"$logfile" 2>&1
+
+        df -h "$new_dev" >>"$logfile"
+
+        umount "$new_mnt" "$mnt"
+
+        aws ec2 detach-volume --volume-id "$volume_id" &>/dev/null
+        aws ec2 detach-volume --volume-id "$new_volume_id" &>/dev/null
+
+        aws ec2 create-tags --resources "$new_volume_id" \
+            --tags "Key=Name,Value=$name"
+
+        snapshot_id=$(aws ec2 create-snapshot --volume-id "$new_volume_id" \
+            --description "$name-$timestamp" \
+            --tag-specifications "ResourceType=snapshot,Tags=[{Key=Name,Value=$name},{Key=rsync-backup,Value=''}]" \
+            --query SnapshotId --output text)
+
+        gzip -c "$logfile" >"$logfile_gz"
+
+        aws s3 cp "$logfile_gz" "$s3base"
+
+        rm -f "$logfile" "$logfile_gz"
+
         while true; do
             aws ec2 wait snapshot-completed --snapshot-ids "$snapshot_id" &&
                 break
         done
-        aws ec2 wait volume-available --volume-ids "$volume_id"
+        aws ec2 wait volume-available --volume-ids "$volume_id" "$new_volume_id"
         aws ec2 delete-volume --volume-id "$volume_id"
+        aws ec2 delete-volume --volume-id "$new_volume_id"
 
-        if ((MAX_SNAPSHOTS > 0)); then
-            read -r -a snapshot_ids <<<"$(aws ec2 describe-snapshots \
-                --owner-ids self \
-                --filter "Name=tag:Name,Values=$name" \
-                --query "reverse(sort_by(Snapshots,&StartTime))[].SnapshotId" \
-                --output text)"
-            for ((i = MAX_SNAPSHOTS; i < ${#snapshot_ids}; i++)); do
-                aws ec2 delete-snapshot --snapshot-id "${snapshot_ids[$i]}"
-            done
-        fi
+        prune_snapshots "$name"
     ) &>/dev/null &
     disown
 }

package/lib/version.d.ts

@@ -1 +1 @@
-export declare const LIB_VERSION = "0.5.4";
+export declare const LIB_VERSION = "0.5.6";

package/lib/version.js

@@ -1,5 +1,5 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.LIB_VERSION = void 0;
-exports.LIB_VERSION = "0.5.4";
-//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoidmVyc2lvbi5qcyIsInNvdXJjZVJvb3QiOiIiLCJzb3VyY2VzIjpbInZlcnNpb24udHMiXSwibmFtZXMiOltdLCJtYXBwaW5ncyI6Ijs7O0FBQWEsUUFBQSxXQUFXLEdBQUcsT0FBTyxDQUFDIiwic291cmNlc0NvbnRlbnQiOlsiZXhwb3J0IGNvbnN0IExJQl9WRVJTSU9OID0gXCIwLjUuNFwiO1xuIl19
+exports.LIB_VERSION = "0.5.6";
+//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoidmVyc2lvbi5qcyIsInNvdXJjZVJvb3QiOiIiLCJzb3VyY2VzIjpbInZlcnNpb24udHMiXSwibmFtZXMiOltdLCJtYXBwaW5ncyI6Ijs7O0FBQWEsUUFBQSxXQUFXLEdBQUcsT0FBTyxDQUFDIiwic291cmNlc0NvbnRlbnQiOlsiZXhwb3J0IGNvbnN0IExJQl9WRVJTSU9OID0gXCIwLjUuNlwiO1xuIl19

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@anyakichi/cdk-rsync-backup",
-  "version": "0.5.4",
+  "version": "0.5.6",
   "description": "AWS CDK L3 construct for cloud backup system with rsync",
   "keywords": [
     "awscdk",