db-cutover 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- db_cutover-0.2.0/.gitignore +22 -0
- db_cutover-0.2.0/AGENTS.md +49 -0
- db_cutover-0.2.0/CHANGELOG.md +25 -0
- db_cutover-0.2.0/LICENSE +21 -0
- db_cutover-0.2.0/PKG-INFO +231 -0
- db_cutover-0.2.0/README.md +203 -0
- db_cutover-0.2.0/SECURITY.md +41 -0
- db_cutover-0.2.0/docs/design.md +115 -0
- db_cutover-0.2.0/docs/procedure.md +226 -0
- db_cutover-0.2.0/examples/config.toml +91 -0
- db_cutover-0.2.0/pyproject.toml +63 -0
- db_cutover-0.2.0/skills/db-cutover/SKILL.md +140 -0
- db_cutover-0.2.0/skills/db-cutover/references/commands.md +104 -0
- db_cutover-0.2.0/skills/db-cutover/references/recovery.md +64 -0
- db_cutover-0.2.0/skills/db-cutover/references/rehearsal.md +69 -0
- db_cutover-0.2.0/src/db_cutover/__init__.py +3 -0
- db_cutover-0.2.0/src/db_cutover/__main__.py +3 -0
- db_cutover-0.2.0/src/db_cutover/cli.py +230 -0
- db_cutover-0.2.0/src/db_cutover/commands/__init__.py +0 -0
- db_cutover-0.2.0/src/db_cutover/commands/blue.py +189 -0
- db_cutover-0.2.0/src/db_cutover/commands/inspect.py +590 -0
- db_cutover-0.2.0/src/db_cutover/commands/replication.py +535 -0
- db_cutover-0.2.0/src/db_cutover/commands/switch.py +429 -0
- db_cutover-0.2.0/src/db_cutover/commands/terraform.py +346 -0
- db_cutover-0.2.0/src/db_cutover/config.py +286 -0
- db_cutover-0.2.0/src/db_cutover/db.py +489 -0
- db_cutover-0.2.0/src/db_cutover/output.py +77 -0
- db_cutover-0.2.0/src/db_cutover/pgbouncer.py +220 -0
- db_cutover-0.2.0/src/db_cutover/shell.py +88 -0
- db_cutover-0.2.0/tests/integration/README.md +16 -0
- db_cutover-0.2.0/tests/integration/compose.yml +24 -0
- db_cutover-0.2.0/tests/integration/test_replication.py +98 -0
- db_cutover-0.2.0/tests/test_config.py +182 -0
- db_cutover-0.2.0/tests/test_db.py +121 -0
- db_cutover-0.2.0/tests/test_health.py +39 -0
- db_cutover-0.2.0/tests/test_pgbouncer.py +32 -0
- db_cutover-0.2.0/tests/test_reboot.py +50 -0
- db_cutover-0.2.0/tests/test_safety.py +109 -0
- db_cutover-0.2.0/tests/test_skill.py +73 -0
- db_cutover-0.2.0/tests/test_switch.py +186 -0
- db_cutover-0.2.0/tests/test_terraform.py +187 -0
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
__pycache__/
|
|
2
|
+
*.pyc
|
|
3
|
+
.venv/
|
|
4
|
+
dist/
|
|
5
|
+
.pytest_cache/
|
|
6
|
+
.ruff_cache/
|
|
7
|
+
.DS_Store
|
|
8
|
+
|
|
9
|
+
# Run state (schema dumps, cutover reports) when a config lives in this repo, e.g. while testing
|
|
10
|
+
.db_cutover/
|
|
11
|
+
|
|
12
|
+
# Local operational inputs and outputs; committed examples use synthetic values.
|
|
13
|
+
/cutover.toml
|
|
14
|
+
*.local.toml
|
|
15
|
+
.env
|
|
16
|
+
.env.*
|
|
17
|
+
*.tfstate
|
|
18
|
+
*.tfstate.*
|
|
19
|
+
*.tfplan
|
|
20
|
+
*.log
|
|
21
|
+
*.pem
|
|
22
|
+
*.key
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# db-cutover
|
|
2
|
+
|
|
3
|
+
`db-cutover`: a Python CLI for zero-downtime blue/green PostgreSQL cutovers on RDS (logical replication plus a
|
|
4
|
+
PgBouncer switch). `README.md` covers use, `docs/procedure.md` the steps of a cutover, `docs/design.md` the
|
|
5
|
+
reasoning. Read `SECURITY.md` before handling operational configuration or logs.
|
|
6
|
+
|
|
7
|
+
## Layout
|
|
8
|
+
|
|
9
|
+
- `src/db_cutover/cli.py`: argument parsing; one subcommand per step.
|
|
10
|
+
- `config.py`: the TOML config (`examples/config.toml` documents every key; keep it in sync).
|
|
11
|
+
- `db.py`: connections and catalog queries. `pgbouncer.py`: the admin console of every pod through
|
|
12
|
+
`kubectl port-forward`. `shell.py`: aws, kubectl, pg_dump, psql. `output.py`: log lines, checks, JSON.
|
|
13
|
+
- `commands/`: `inspect.py` (read-only and long-running), `blue.py`, `replication.py`, `switch.py` (cutover,
|
|
14
|
+
finish), `terraform.py` (green-tf, promote-tf).
|
|
15
|
+
- `skills/db-cutover/`: the [Agent Skill](https://agentskills.io/specification) for agents that operate the tool
|
|
16
|
+
(`SKILL.md` plus `references/`). The wheel includes it as `db_cutover/skills/db-cutover`, and `db-cutover skill`
|
|
17
|
+
prints or installs it. It is written for an agent running a cutover; this file is for an agent changing the code.
|
|
18
|
+
|
|
19
|
+
## Rules
|
|
20
|
+
|
|
21
|
+
- Every command that changes anything is a dry run unless `--execute` is passed. Keep it that way for new commands.
|
|
22
|
+
- Never print a secret or a connection string that holds one. Secrets go to subprocesses through the environment
|
|
23
|
+
(`PGPASSWORD`), never the command line.
|
|
24
|
+
- The switch (`commands/switch.py`) must resume on BLUE for any failure before RELOAD, including Ctrl-C. Don't add
|
|
25
|
+
steps between PAUSE and RESUME without measuring the pause on a staging run.
|
|
26
|
+
- After RELOAD starts, never resume mixed or unknown targets. Report failed resumes as failures. A sequence gap
|
|
27
|
+
and one-way replication cannot make concurrent writers on BLUE and GREEN safe.
|
|
28
|
+
- Probe writes, replication markers and manual RESUME require `--execute`, including retry paths.
|
|
29
|
+
- Inspect all client roles when draining BLUE. A failed deployment lookup must not be interpreted as zero replicas.
|
|
30
|
+
- Never print query text, server row diagnostics or a secret helper's stderr. Keep fixtures synthetic and review
|
|
31
|
+
all history before public release; removing a value from the current tree does not remove it from history.
|
|
32
|
+
- Logic that can be pure (sequence planning, HCL edits, ini parsing) stays pure and has tests in `tests/`.
|
|
33
|
+
- Long-running commands print one line per change, not per poll, so they work as background monitors.
|
|
34
|
+
- Before committing: `uv run pytest`, `uv run ruff check .`, `uv run ruff format --check .` (CI runs the same).
|
|
35
|
+
- When behaviour changes, update `README.md`, `docs/procedure.md`, `docs/design.md` and the skill in the same
|
|
36
|
+
commit. A new command needs a line in `skills/db-cutover/references/commands.md` (a test checks).
|
|
37
|
+
- The skill's frontmatter has only `name`, `description`, `license` and `metadata`. Keep `SKILL.md` under 500
|
|
38
|
+
lines with the rules first, and write standing instructions for "you" (the agent) and "the operator".
|
|
39
|
+
|
|
40
|
+
## Releasing
|
|
41
|
+
|
|
42
|
+
1. Set the version in `pyproject.toml`, `src/db_cutover/__init__.py`, the skill's `metadata.version` and the
|
|
43
|
+
README's install command, run `uv lock`, and give `CHANGELOG.md` a `## <version> (<date>)` section. A test
|
|
44
|
+
checks that they agree.
|
|
45
|
+
2. Merge to `main`, then tag that commit `v<version>` and push the tag.
|
|
46
|
+
3. The `Release` workflow builds the package, checks the tag against the version, publishes to PyPI with trusted
|
|
47
|
+
publishing (no token: PyPI trusts this repository's `release.yml` in the `pypi` environment) and creates the
|
|
48
|
+
GitHub release from the changelog section. PyPI never accepts the same version twice, so a mistake needs a new
|
|
49
|
+
version.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.2.0 (2026-11-16)
|
|
4
|
+
|
|
5
|
+
The first release on PyPI.
|
|
6
|
+
|
|
7
|
+
- Install with `uv tool install db-cutover` or `pipx install db-cutover`. `db-cutover --version` prints the version.
|
|
8
|
+
- Add an agent skill (`skills/db-cutover`) for planning, rehearsing and running a cutover with an operator: each
|
|
9
|
+
command and its dry run, the stop points, the rehearsal checklist and recovery. It ships inside the package:
|
|
10
|
+
`db-cutover skill` prints its directory and `db-cutover skill --install DIR` copies it into an agent's skills
|
|
11
|
+
directory.
|
|
12
|
+
- README: who the tool is for, a rehearsal walkthrough, and its guarantees and limits.
|
|
13
|
+
- Stop and require recovery after a partial or unknown PgBouncer reload. Never resume mixed BLUE/GREEN targets.
|
|
14
|
+
Report failed resumes and post-switch verification failures with unsuccessful exit status.
|
|
15
|
+
- Require `--execute` for `probe`, `wait-sync`, and `pgbouncer resume`. `verify` is read-only by default;
|
|
16
|
+
`verify --execute` adds a replication marker. Cutover and finish dry runs no longer write markers.
|
|
17
|
+
- Remove `cutover --allow-direct`: every client role must drain before switching. Failed deployment lookups block
|
|
18
|
+
the operation. Reboot's separate `--allow-direct` option still explicitly permits direct-client errors.
|
|
19
|
+
- Default to text replication. Validate publication coverage, subscription identity, sequence limits and marker
|
|
20
|
+
tokens. Reject unsupported data objects before switching.
|
|
21
|
+
- Withhold PostgreSQL/subprocess/secret-helper diagnostics that may expose credentials or row values. Remove query
|
|
22
|
+
text and health response bodies from inspection output; keep state files in a private directory.
|
|
23
|
+
- Bound port-forward startup and settle pending admin commands before confirming recovery from a late PAUSE.
|
|
24
|
+
- Add synthetic fault tests, PostgreSQL 16-to-17 integration tests, an MIT license, standalone examples and an
|
|
25
|
+
expanded operator/agent runbook. Infrastructure and application rehearsals remain required before live use.
|
db_cutover-0.2.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 DocSpring
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,231 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: db-cutover
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Agent-friendly tools for zero-downtime blue/green PostgreSQL cutovers on RDS with logical replication and a PgBouncer switch
|
|
5
|
+
Project-URL: Repository, https://github.com/DocSpring/db-cutover
|
|
6
|
+
Project-URL: Changelog, https://github.com/DocSpring/db-cutover/blob/main/CHANGELOG.md
|
|
7
|
+
Project-URL: Issues, https://github.com/DocSpring/db-cutover/issues
|
|
8
|
+
Project-URL: Documentation, https://github.com/DocSpring/db-cutover/blob/main/docs/procedure.md
|
|
9
|
+
Author-email: "DocSpring, Inc." <support@docspring.com>
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: blue-green,logical-replication,pgbouncer,postgresql,rds,upgrade,zero-downtime
|
|
13
|
+
Classifier: Development Status :: 4 - Beta
|
|
14
|
+
Classifier: Environment :: Console
|
|
15
|
+
Classifier: Intended Audience :: System Administrators
|
|
16
|
+
Classifier: Operating System :: MacOS
|
|
17
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
18
|
+
Classifier: Programming Language :: Python :: 3
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
23
|
+
Classifier: Topic :: Database
|
|
24
|
+
Classifier: Topic :: System :: Systems Administration
|
|
25
|
+
Requires-Python: >=3.11
|
|
26
|
+
Requires-Dist: psycopg[binary]<4,>=3.2
|
|
27
|
+
Description-Content-Type: text/markdown
|
|
28
|
+
|
|
29
|
+
# db-cutover
|
|
30
|
+
|
|
31
|
+
Upgrade a PostgreSQL database by copying it to a new instance, keeping that copy current with logical replication,
|
|
32
|
+
and switching application traffic through PgBouncer. The application keeps using the old database during the copy.
|
|
33
|
+
At the switch, PgBouncer holds new queries while in-flight transactions finish, then sends queued work to the new
|
|
34
|
+
instance.
|
|
35
|
+
|
|
36
|
+
`db-cutover` is the Python CLI that runs each step. It comes with an [example configuration](examples/config.toml),
|
|
37
|
+
a [step-by-step procedure](docs/procedure.md), [design notes](docs/design.md) and an
|
|
38
|
+
[agent skill](skills/db-cutover/SKILL.md). It is made for a human operator working with a coding agent: each command
|
|
39
|
+
has a narrow purpose, and every command that changes database, infrastructure or PgBouncer state is a dry run until
|
|
40
|
+
it gets `--execute`.
|
|
41
|
+
|
|
42
|
+
**Release candidate:** failure-path tests and PostgreSQL replication tests do not replace a rehearsal on your
|
|
43
|
+
application's RDS and PgBouncer deployment. Read the requirements below before planning a cutover.
|
|
44
|
+
|
|
45
|
+
## Who it's for
|
|
46
|
+
|
|
47
|
+
Teams that run PostgreSQL on RDS behind PgBouncer in Kubernetes and need a major version upgrade (or a move to a new
|
|
48
|
+
instance) without a maintenance window. It suits you if:
|
|
49
|
+
|
|
50
|
+
- you want to choose the moment of the switch and see every check before it, instead of handing the upgrade to a
|
|
51
|
+
managed service;
|
|
52
|
+
- your application already connects through PgBouncer with transaction pooling, or can be made to;
|
|
53
|
+
- someone on the team can provision an RDS instance and change a Kubernetes ConfigMap.
|
|
54
|
+
|
|
55
|
+
It is not for databases older than PostgreSQL 16, setups without PgBouncer, or schemas that rely on large objects,
|
|
56
|
+
materialized views, unlogged tables or foreign tables. For those, look at
|
|
57
|
+
[RDS Blue/Green Deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html) or
|
|
58
|
+
`pg_upgrade` with a maintenance window.
|
|
59
|
+
|
|
60
|
+
## Requirements
|
|
61
|
+
|
|
62
|
+
- PostgreSQL **16 or newer**, running on RDS. BLUE is the existing primary; GREEN is a separate instance at the
|
|
63
|
+
target version. One database is migrated at a time. Test the specific source/target versions and extensions.
|
|
64
|
+
- PgBouncer in Kubernetes, with transaction pooling. Every foreground database client must use the selected pods.
|
|
65
|
+
These commands pause/resume **all databases on those pods**, and RELOAD applies the entire configuration.
|
|
66
|
+
Use dedicated pods for this workflow.
|
|
67
|
+
- A directory-mounted ConfigMap containing `pgbouncer.ini`, with no automatic reload. `subPath` mounts do not receive
|
|
68
|
+
ConfigMap updates. Prevent rollouts, scaling and automatic restarts during the prepared-config window; see the
|
|
69
|
+
[procedure](docs/procedure.md#7-prepare-and-switch).
|
|
70
|
+
- All connections bypassing PgBouncer must be stopped and prevented from reconnecting during the switch. Inventory
|
|
71
|
+
consoles, schedulers, maintenance jobs, migrations and every application role, not just your web process.
|
|
72
|
+
- A schema freeze from `copy-schema` through `finish`. Roles, grants, extensions, settings and other databases need
|
|
73
|
+
separate preparation. Large objects, unlogged tables, materialized views and foreign tables are blocked.
|
|
74
|
+
- Sequences with `INCREMENT BY 1`, no cycling, sufficient capacity, and application code that tolerates gaps in IDs.
|
|
75
|
+
- Request, proxy, job and PgBouncer timeouts longer than the measured pause. Queries wait during the switch;
|
|
76
|
+
**zero failed requests depends on those timeouts and successful recovery**. An RDS reboot can take much longer.
|
|
77
|
+
|
|
78
|
+
## Install
|
|
79
|
+
|
|
80
|
+
`db-cutover` needs Python 3.11 or newer. Install it as a standalone tool with [uv](https://docs.astral.sh/uv/) or
|
|
81
|
+
[pipx](https://pipx.pypa.io/):
|
|
82
|
+
|
|
83
|
+
```sh
|
|
84
|
+
uv tool install db-cutover==0.2.0 # or: pipx install db-cutover==0.2.0
|
|
85
|
+
db-cutover --version
|
|
86
|
+
db-cutover --help
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Pin the version and keep it fixed from the first rehearsal to the end of the migration. To run from source instead,
|
|
90
|
+
clone the repository at a release tag and use its locked dependencies (`uv sync --frozen`, then `uv run db-cutover`).
|
|
91
|
+
|
|
92
|
+
The machine running the tool also needs `aws`, `kubectl`, and matching, current `pg_dump`/`psql` clients at least as
|
|
93
|
+
new as **both** database servers. `op` is optional for 1Password references; Terraform is optional for the two HCL
|
|
94
|
+
helpers. It needs network access to both databases and the Kubernetes API. GREEN must reach BLUE for replication.
|
|
95
|
+
|
|
96
|
+
Copy the [example](examples/config.toml) to a private working directory, fill in your identifiers and contexts,
|
|
97
|
+
and supply credentials through environment variables or a secret manager. Never commit the operational config,
|
|
98
|
+
schema dumps, reports, Terraform state or logs. `cmd:` secret references execute shell commands: treat configs as
|
|
99
|
+
trusted executable input.
|
|
100
|
+
|
|
101
|
+
## Working with an agent
|
|
102
|
+
|
|
103
|
+
The package includes an [Agent Skill](https://agentskills.io/specification) that teaches a coding agent (Claude
|
|
104
|
+
Code, Codex, Cursor and others) to plan, rehearse and run a cutover with you: what each command does, which ones
|
|
105
|
+
need `--execute`, the rehearsal checklist, the points where it must stop for your decision, and recovery.
|
|
106
|
+
|
|
107
|
+
```sh
|
|
108
|
+
db-cutover skill --install .claude/skills # or .agents/skills, or your agent's skills directory
|
|
109
|
+
npx skills add DocSpring/db-cutover # the same skill, straight from this repository
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Give the agent the config path, the approved versions, the traffic controls and the limits of what it may execute.
|
|
113
|
+
Have it run each changing command without `--execute` first, explain the output, and follow your agreed execution
|
|
114
|
+
policy. A failed gate is a stop condition.
|
|
115
|
+
|
|
116
|
+
Exit status 0 means the command completed its requested mode, 1 means failure, 2 means an argument/config error,
|
|
117
|
+
and 130 means interruption. A dry-run success is not a completed migration. Long-running monitors continue until
|
|
118
|
+
stopped; quiet output alone is not proof that a monitor is still alive.
|
|
119
|
+
|
|
120
|
+
## Workflow
|
|
121
|
+
|
|
122
|
+
Read [the procedure](docs/procedure.md) for the provisioning, permissions, traffic controls and recovery steps
|
|
123
|
+
between these commands. In the examples below, `C` is a private config path and `TARGET_VERSION` is the RDS engine
|
|
124
|
+
version you have selected and tested. Prefix commands with `uv run` if you run from a source checkout.
|
|
125
|
+
|
|
126
|
+
```sh
|
|
127
|
+
db-cutover "$C" preflight
|
|
128
|
+
# Enable logical replication; reboot-blue --execute only if the static parameter needs it.
|
|
129
|
+
db-cutover "$C" prepare-blue --execute
|
|
130
|
+
db-cutover "$C" green-tf --engine-version "$TARGET_VERSION" --execute
|
|
131
|
+
# Review and provision GREEN, including version-compatible parameters and network access.
|
|
132
|
+
db-cutover "$C" copy-schema --execute
|
|
133
|
+
# Create application roles, grants and required extensions on GREEN.
|
|
134
|
+
db-cutover "$C" start-replication --execute
|
|
135
|
+
db-cutover "$C" wait-sync --execute
|
|
136
|
+
db-cutover "$C" verify --execute
|
|
137
|
+
db-cutover "$C" sequences --execute
|
|
138
|
+
# Stop and fence direct writers before the rehearsal.
|
|
139
|
+
db-cutover "$C" cutover --execute --rehearse
|
|
140
|
+
# Prepare PgBouncer's config as described in the procedure, without reloading it.
|
|
141
|
+
db-cutover "$C" cutover
|
|
142
|
+
db-cutover "$C" cutover --execute
|
|
143
|
+
# Move direct writers to GREEN and prevent any application writes to BLUE.
|
|
144
|
+
db-cutover "$C" finish --execute
|
|
145
|
+
db-cutover "$C" promote-tf --execute
|
|
146
|
+
# Review and run the printed state moves, then verify the Terraform plan.
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
If health endpoints require authentication, set `[secrets] health_check_token`; `watch` sends it as
|
|
150
|
+
`X-Health-Check-Token` to every configured health URL. Only configure trusted HTTPS endpoints. Redirects are reported
|
|
151
|
+
as failures rather than followed, so the header cannot be forwarded to a different endpoint.
|
|
152
|
+
|
|
153
|
+
Use `watch` to monitor health/replication and `probe --execute` to write synthetic probe rows through PgBouncer.
|
|
154
|
+
Start the probe after `prepare-blue`; stop it before `finish` drops its table. `verify` without `--execute` is
|
|
155
|
+
read-only; `--execute` adds a marker write to test replication. Dry runs can connect, read data, create local state
|
|
156
|
+
files, and consume database resources. They do not pause clients or modify remote state.
|
|
157
|
+
|
|
158
|
+
## Rehearse first
|
|
159
|
+
|
|
160
|
+
Rehearse twice: the whole procedure on staging, then the pause on production.
|
|
161
|
+
|
|
162
|
+
1. **Build a staging setup that matches production** where it matters: the same PgBouncer image, pool mode and
|
|
163
|
+
ConfigMap mount, the same PostgreSQL source and target versions and extensions, and the application with its
|
|
164
|
+
real timeouts. Put traffic on it.
|
|
165
|
+
2. **Run every step there, including a real switch**, with `watch` and `probe --execute` running. Write down how
|
|
166
|
+
long the copy took, how much WAL BLUE kept, and the pause the switch reports.
|
|
167
|
+
3. **Cause the failures on purpose.** Leave a direct writer connected, hold a transaction open on BLUE, press
|
|
168
|
+
Ctrl-C during a rehearsal's pause, restart a PgBouncer pod after the ConfigMap names GREEN. Each one should stop
|
|
169
|
+
the switch and leave traffic on BLUE. Practise the [recovery steps](docs/procedure.md#recovery) for the last one.
|
|
170
|
+
4. **On production, rehearse the pause** once replication has caught up and the direct writers are stopped:
|
|
171
|
+
|
|
172
|
+
```sh
|
|
173
|
+
db-cutover "$C" cutover --execute --rehearse
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
This pauses live traffic, runs every check the real switch runs, and resumes on BLUE. Its report gives the pause
|
|
177
|
+
in milliseconds. Compare it with every timeout in the request path before you go on.
|
|
178
|
+
|
|
179
|
+
`--rehearse` never reloads PgBouncer, so only the staging switch tests RELOAD, GREEN under the application and
|
|
180
|
+
recovery from a partial switch. The skill's
|
|
181
|
+
[rehearsal checklist](skills/db-cutover/references/rehearsal.md) lists each thing to check and record.
|
|
182
|
+
|
|
183
|
+
## Guarantees and limits
|
|
184
|
+
|
|
185
|
+
What the tool does:
|
|
186
|
+
|
|
187
|
+
- **Nothing changes without `--execute`.** A dry run reads, checks and reports. It can connect, read data, create
|
|
188
|
+
local state files and use database resources, but it never pauses clients or changes remote state.
|
|
189
|
+
- **No switch without proof that GREEN is current.** With every PgBouncer pod paused, the tool checks that no
|
|
190
|
+
client session is left on BLUE, writes a marker on BLUE and waits for it on GREEN, and compares the highest ID of
|
|
191
|
+
every table before any pod reloads.
|
|
192
|
+
- **A failure before RELOAD leaves BLUE as primary.** The tool resumes traffic on BLUE, including on Ctrl-C, and
|
|
193
|
+
reports a resume it could not confirm.
|
|
194
|
+
- **It never resumes pods that point at different databases.** After RELOAD starts, mixed or unknown targets
|
|
195
|
+
**remain paused** and the command fails. An operator aligns every pod to GREEN before resuming.
|
|
196
|
+
- **It keeps secrets and row data out of its output.** PostgreSQL diagnostics, query text and health response
|
|
197
|
+
bodies are withheld. State files go in a private directory.
|
|
198
|
+
|
|
199
|
+
What it can't do:
|
|
200
|
+
|
|
201
|
+
- **Promise zero failed requests.** Queries wait during the pause. Requests succeed only if every timeout in their
|
|
202
|
+
path is longer than the pause and recovery works. An RDS reboot can take much longer than a switch.
|
|
203
|
+
- **Fence clients it doesn't know about.** Anything that connects without PgBouncer has to be stopped and kept
|
|
204
|
+
away by you. The tool checks for sessions; it can't prevent new ones.
|
|
205
|
+
- **Close the window between the ConfigMap change and the switch.** A PgBouncer pod restarted in that window
|
|
206
|
+
serves GREEN early. The tool detects it before switching but can't stop it.
|
|
207
|
+
- **Go back to BLUE once GREEN has taken writes.** Replication runs one way. Never read replication or a sequence
|
|
208
|
+
gap as permission to write to both databases. After the switch, fix forward on GREEN.
|
|
209
|
+
- **Survive its own process being killed.** After SIGKILL or a lost connection during a switch, an operator has to
|
|
210
|
+
read the state from the pods.
|
|
211
|
+
- **Prove the data is identical.** `verify` compares schemas, counts and sampled rows. It is not a full comparison.
|
|
212
|
+
- **Copy everything.** Roles, grants, extensions, database settings and other databases need separate preparation.
|
|
213
|
+
Sequence values are advanced, not replicated.
|
|
214
|
+
|
|
215
|
+
The [design notes](docs/design.md) give the reasoning, and [SECURITY.md](SECURITY.md) the limits of the tests.
|
|
216
|
+
|
|
217
|
+
## Development
|
|
218
|
+
|
|
219
|
+
```sh
|
|
220
|
+
uv sync --frozen --group dev
|
|
221
|
+
uv run pytest
|
|
222
|
+
uv run ruff check .
|
|
223
|
+
uv run ruff format --check .
|
|
224
|
+
uv build
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
The default suite uses no cloud credentials. [Integration tests](tests/integration/README.md) use disposable
|
|
228
|
+
PostgreSQL 16 and 17 containers with synthetic data. CI runs both suites. See [SECURITY.md](SECURITY.md) for data
|
|
229
|
+
handling and release checks, and [AGENTS.md](AGENTS.md) for the code's rules and how to release.
|
|
230
|
+
|
|
231
|
+
Built by [DocSpring](https://docspring.com). Licensed under [MIT](LICENSE).
|
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# db-cutover
|
|
2
|
+
|
|
3
|
+
Upgrade a PostgreSQL database by copying it to a new instance, keeping that copy current with logical replication,
|
|
4
|
+
and switching application traffic through PgBouncer. The application keeps using the old database during the copy.
|
|
5
|
+
At the switch, PgBouncer holds new queries while in-flight transactions finish, then sends queued work to the new
|
|
6
|
+
instance.
|
|
7
|
+
|
|
8
|
+
`db-cutover` is the Python CLI that runs each step. It comes with an [example configuration](examples/config.toml),
|
|
9
|
+
a [step-by-step procedure](docs/procedure.md), [design notes](docs/design.md) and an
|
|
10
|
+
[agent skill](skills/db-cutover/SKILL.md). It is made for a human operator working with a coding agent: each command
|
|
11
|
+
has a narrow purpose, and every command that changes database, infrastructure or PgBouncer state is a dry run until
|
|
12
|
+
it gets `--execute`.
|
|
13
|
+
|
|
14
|
+
**Release candidate:** failure-path tests and PostgreSQL replication tests do not replace a rehearsal on your
|
|
15
|
+
application's RDS and PgBouncer deployment. Read the requirements below before planning a cutover.
|
|
16
|
+
|
|
17
|
+
## Who it's for
|
|
18
|
+
|
|
19
|
+
Teams that run PostgreSQL on RDS behind PgBouncer in Kubernetes and need a major version upgrade (or a move to a new
|
|
20
|
+
instance) without a maintenance window. It suits you if:
|
|
21
|
+
|
|
22
|
+
- you want to choose the moment of the switch and see every check before it, instead of handing the upgrade to a
|
|
23
|
+
managed service;
|
|
24
|
+
- your application already connects through PgBouncer with transaction pooling, or can be made to;
|
|
25
|
+
- someone on the team can provision an RDS instance and change a Kubernetes ConfigMap.
|
|
26
|
+
|
|
27
|
+
It is not for databases older than PostgreSQL 16, setups without PgBouncer, or schemas that rely on large objects,
|
|
28
|
+
materialized views, unlogged tables or foreign tables. For those, look at
|
|
29
|
+
[RDS Blue/Green Deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html) or
|
|
30
|
+
`pg_upgrade` with a maintenance window.
|
|
31
|
+
|
|
32
|
+
## Requirements
|
|
33
|
+
|
|
34
|
+
- PostgreSQL **16 or newer**, running on RDS. BLUE is the existing primary; GREEN is a separate instance at the
|
|
35
|
+
target version. One database is migrated at a time. Test the specific source/target versions and extensions.
|
|
36
|
+
- PgBouncer in Kubernetes, with transaction pooling. Every foreground database client must use the selected pods.
|
|
37
|
+
These commands pause/resume **all databases on those pods**, and RELOAD applies the entire configuration.
|
|
38
|
+
Use dedicated pods for this workflow.
|
|
39
|
+
- A directory-mounted ConfigMap containing `pgbouncer.ini`, with no automatic reload. `subPath` mounts do not receive
|
|
40
|
+
ConfigMap updates. Prevent rollouts, scaling and automatic restarts during the prepared-config window; see the
|
|
41
|
+
[procedure](docs/procedure.md#7-prepare-and-switch).
|
|
42
|
+
- All connections bypassing PgBouncer must be stopped and prevented from reconnecting during the switch. Inventory
|
|
43
|
+
consoles, schedulers, maintenance jobs, migrations and every application role, not just your web process.
|
|
44
|
+
- A schema freeze from `copy-schema` through `finish`. Roles, grants, extensions, settings and other databases need
|
|
45
|
+
separate preparation. Large objects, unlogged tables, materialized views and foreign tables are blocked.
|
|
46
|
+
- Sequences with `INCREMENT BY 1`, no cycling, sufficient capacity, and application code that tolerates gaps in IDs.
|
|
47
|
+
- Request, proxy, job and PgBouncer timeouts longer than the measured pause. Queries wait during the switch;
|
|
48
|
+
**zero failed requests depends on those timeouts and successful recovery**. An RDS reboot can take much longer.
|
|
49
|
+
|
|
50
|
+
## Install
|
|
51
|
+
|
|
52
|
+
`db-cutover` needs Python 3.11 or newer. Install it as a standalone tool with [uv](https://docs.astral.sh/uv/) or
|
|
53
|
+
[pipx](https://pipx.pypa.io/):
|
|
54
|
+
|
|
55
|
+
```sh
|
|
56
|
+
uv tool install db-cutover==0.2.0 # or: pipx install db-cutover==0.2.0
|
|
57
|
+
db-cutover --version
|
|
58
|
+
db-cutover --help
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Pin the version and keep it fixed from the first rehearsal to the end of the migration. To run from source instead,
|
|
62
|
+
clone the repository at a release tag and use its locked dependencies (`uv sync --frozen`, then `uv run db-cutover`).
|
|
63
|
+
|
|
64
|
+
The machine running the tool also needs `aws`, `kubectl`, and matching, current `pg_dump`/`psql` clients at least as
|
|
65
|
+
new as **both** database servers. `op` is optional for 1Password references; Terraform is optional for the two HCL
|
|
66
|
+
helpers. It needs network access to both databases and the Kubernetes API. GREEN must reach BLUE for replication.
|
|
67
|
+
|
|
68
|
+
Copy the [example](examples/config.toml) to a private working directory, fill in your identifiers and contexts,
|
|
69
|
+
and supply credentials through environment variables or a secret manager. Never commit the operational config,
|
|
70
|
+
schema dumps, reports, Terraform state or logs. `cmd:` secret references execute shell commands: treat configs as
|
|
71
|
+
trusted executable input.
|
|
72
|
+
|
|
73
|
+
## Working with an agent
|
|
74
|
+
|
|
75
|
+
The package includes an [Agent Skill](https://agentskills.io/specification) that teaches a coding agent (Claude
|
|
76
|
+
Code, Codex, Cursor and others) to plan, rehearse and run a cutover with you: what each command does, which ones
|
|
77
|
+
need `--execute`, the rehearsal checklist, the points where it must stop for your decision, and recovery.
|
|
78
|
+
|
|
79
|
+
```sh
|
|
80
|
+
db-cutover skill --install .claude/skills # or .agents/skills, or your agent's skills directory
|
|
81
|
+
npx skills add DocSpring/db-cutover # the same skill, straight from this repository
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Give the agent the config path, the approved versions, the traffic controls and the limits of what it may execute.
|
|
85
|
+
Have it run each changing command without `--execute` first, explain the output, and follow your agreed execution
|
|
86
|
+
policy. A failed gate is a stop condition.
|
|
87
|
+
|
|
88
|
+
Exit status 0 means the command completed its requested mode, 1 means failure, 2 means an argument/config error,
|
|
89
|
+
and 130 means interruption. A dry-run success is not a completed migration. Long-running monitors continue until
|
|
90
|
+
stopped; quiet output alone is not proof that a monitor is still alive.
|
|
91
|
+
|
|
92
|
+
## Workflow
|
|
93
|
+
|
|
94
|
+
Read [the procedure](docs/procedure.md) for the provisioning, permissions, traffic controls and recovery steps
|
|
95
|
+
between these commands. In the examples below, `C` is a private config path and `TARGET_VERSION` is the RDS engine
|
|
96
|
+
version you have selected and tested. Prefix commands with `uv run` if you run from a source checkout.
|
|
97
|
+
|
|
98
|
+
```sh
|
|
99
|
+
db-cutover "$C" preflight
|
|
100
|
+
# Enable logical replication; reboot-blue --execute only if the static parameter needs it.
|
|
101
|
+
db-cutover "$C" prepare-blue --execute
|
|
102
|
+
db-cutover "$C" green-tf --engine-version "$TARGET_VERSION" --execute
|
|
103
|
+
# Review and provision GREEN, including version-compatible parameters and network access.
|
|
104
|
+
db-cutover "$C" copy-schema --execute
|
|
105
|
+
# Create application roles, grants and required extensions on GREEN.
|
|
106
|
+
db-cutover "$C" start-replication --execute
|
|
107
|
+
db-cutover "$C" wait-sync --execute
|
|
108
|
+
db-cutover "$C" verify --execute
|
|
109
|
+
db-cutover "$C" sequences --execute
|
|
110
|
+
# Stop and fence direct writers before the rehearsal.
|
|
111
|
+
db-cutover "$C" cutover --execute --rehearse
|
|
112
|
+
# Prepare PgBouncer's config as described in the procedure, without reloading it.
|
|
113
|
+
db-cutover "$C" cutover
|
|
114
|
+
db-cutover "$C" cutover --execute
|
|
115
|
+
# Move direct writers to GREEN and prevent any application writes to BLUE.
|
|
116
|
+
db-cutover "$C" finish --execute
|
|
117
|
+
db-cutover "$C" promote-tf --execute
|
|
118
|
+
# Review and run the printed state moves, then verify the Terraform plan.
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
If health endpoints require authentication, set `[secrets] health_check_token`; `watch` sends it as
|
|
122
|
+
`X-Health-Check-Token` to every configured health URL. Only configure trusted HTTPS endpoints. Redirects are reported
|
|
123
|
+
as failures rather than followed, so the header cannot be forwarded to a different endpoint.
|
|
124
|
+
|
|
125
|
+
Use `watch` to monitor health/replication and `probe --execute` to write synthetic probe rows through PgBouncer.
|
|
126
|
+
Start the probe after `prepare-blue`; stop it before `finish` drops its table. `verify` without `--execute` is
|
|
127
|
+
read-only; `--execute` adds a marker write to test replication. Dry runs can connect, read data, create local state
|
|
128
|
+
files, and consume database resources. They do not pause clients or modify remote state.
|
|
129
|
+
|
|
130
|
+
## Rehearse first
|
|
131
|
+
|
|
132
|
+
Rehearse twice: the whole procedure on staging, then the pause on production.
|
|
133
|
+
|
|
134
|
+
1. **Build a staging setup that matches production** where it matters: the same PgBouncer image, pool mode and
|
|
135
|
+
ConfigMap mount, the same PostgreSQL source and target versions and extensions, and the application with its
|
|
136
|
+
real timeouts. Put traffic on it.
|
|
137
|
+
2. **Run every step there, including a real switch**, with `watch` and `probe --execute` running. Write down how
|
|
138
|
+
long the copy took, how much WAL BLUE kept, and the pause the switch reports.
|
|
139
|
+
3. **Cause the failures on purpose.** Leave a direct writer connected, hold a transaction open on BLUE, press
|
|
140
|
+
Ctrl-C during a rehearsal's pause, restart a PgBouncer pod after the ConfigMap names GREEN. Each one should stop
|
|
141
|
+
the switch and leave traffic on BLUE. Practise the [recovery steps](docs/procedure.md#recovery) for the last one.
|
|
142
|
+
4. **On production, rehearse the pause** once replication has caught up and the direct writers are stopped:
|
|
143
|
+
|
|
144
|
+
```sh
|
|
145
|
+
db-cutover "$C" cutover --execute --rehearse
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
This pauses live traffic, runs every check the real switch runs, and resumes on BLUE. Its report gives the pause
|
|
149
|
+
in milliseconds. Compare it with every timeout in the request path before you go on.
|
|
150
|
+
|
|
151
|
+
`--rehearse` never reloads PgBouncer, so only the staging switch tests RELOAD, GREEN under the application and
|
|
152
|
+
recovery from a partial switch. The skill's
|
|
153
|
+
[rehearsal checklist](skills/db-cutover/references/rehearsal.md) lists each thing to check and record.
|
|
154
|
+
|
|
155
|
+
## Guarantees and limits
|
|
156
|
+
|
|
157
|
+
What the tool does:
|
|
158
|
+
|
|
159
|
+
- **Nothing changes without `--execute`.** A dry run reads, checks and reports. It can connect, read data, create
|
|
160
|
+
local state files and use database resources, but it never pauses clients or changes remote state.
|
|
161
|
+
- **No switch without proof that GREEN is current.** With every PgBouncer pod paused, the tool checks that no
|
|
162
|
+
client session is left on BLUE, writes a marker on BLUE and waits for it on GREEN, and compares the highest ID of
|
|
163
|
+
every table before any pod reloads.
|
|
164
|
+
- **A failure before RELOAD leaves BLUE as primary.** The tool resumes traffic on BLUE, including on Ctrl-C, and
|
|
165
|
+
reports a resume it could not confirm.
|
|
166
|
+
- **It never resumes pods that point at different databases.** After RELOAD starts, mixed or unknown targets
|
|
167
|
+
**remain paused** and the command fails. An operator aligns every pod to GREEN before resuming.
|
|
168
|
+
- **It keeps secrets and row data out of its output.** PostgreSQL diagnostics, query text and health response
|
|
169
|
+
bodies are withheld. State files go in a private directory.
|
|
170
|
+
|
|
171
|
+
What it can't do:
|
|
172
|
+
|
|
173
|
+
- **Promise zero failed requests.** Queries wait during the pause. Requests succeed only if every timeout in their
|
|
174
|
+
path is longer than the pause and recovery works. An RDS reboot can take much longer than a switch.
|
|
175
|
+
- **Fence clients it doesn't know about.** Anything that connects without PgBouncer has to be stopped and kept
|
|
176
|
+
away by you. The tool checks for sessions; it can't prevent new ones.
|
|
177
|
+
- **Close the window between the ConfigMap change and the switch.** A PgBouncer pod restarted in that window
|
|
178
|
+
serves GREEN early. The tool detects it before switching but can't stop it.
|
|
179
|
+
- **Go back to BLUE once GREEN has taken writes.** Replication runs one way. Never read replication or a sequence
|
|
180
|
+
gap as permission to write to both databases. After the switch, fix forward on GREEN.
|
|
181
|
+
- **Survive its own process being killed.** After SIGKILL or a lost connection during a switch, an operator has to
|
|
182
|
+
read the state from the pods.
|
|
183
|
+
- **Prove the data is identical.** `verify` compares schemas, counts and sampled rows. It is not a full comparison.
|
|
184
|
+
- **Copy everything.** Roles, grants, extensions, database settings and other databases need separate preparation.
|
|
185
|
+
Sequence values are advanced, not replicated.
|
|
186
|
+
|
|
187
|
+
The [design notes](docs/design.md) give the reasoning, and [SECURITY.md](SECURITY.md) the limits of the tests.
|
|
188
|
+
|
|
189
|
+
## Development
|
|
190
|
+
|
|
191
|
+
```sh
|
|
192
|
+
uv sync --frozen --group dev
|
|
193
|
+
uv run pytest
|
|
194
|
+
uv run ruff check .
|
|
195
|
+
uv run ruff format --check .
|
|
196
|
+
uv build
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
The default suite uses no cloud credentials. [Integration tests](tests/integration/README.md) use disposable
|
|
200
|
+
PostgreSQL 16 and 17 containers with synthetic data. CI runs both suites. See [SECURITY.md](SECURITY.md) for data
|
|
201
|
+
handling and release checks, and [AGENTS.md](AGENTS.md) for the code's rules and how to release.
|
|
202
|
+
|
|
203
|
+
Built by [DocSpring](https://docspring.com). Licensed under [MIT](LICENSE).
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Security and operational data
|
|
2
|
+
|
|
3
|
+
This tool operates with administrative database and infrastructure access. Follow your organization's change
|
|
4
|
+
authorization policy. A successful dry run is not authorization to execute a production change.
|
|
5
|
+
|
|
6
|
+
## Private inputs and output
|
|
7
|
+
|
|
8
|
+
- Keep operational TOML files outside the source repository. Use environment variables or a secret manager,
|
|
9
|
+
never literal credentials in config or on a command line. `cmd:` references execute shell commands; configs,
|
|
10
|
+
Terraform settings and schema dumps are trusted executable inputs, not safe inputs from strangers.
|
|
11
|
+
- Keep schema dumps, reports, Terraform state/plans and logs private. Schema functions and comments can contain
|
|
12
|
+
secrets even without table rows. The state directory uses mode 0700.
|
|
13
|
+
- PostgreSQL diagnostics and subprocess error output are withheld because they can contain SQL, credentials and
|
|
14
|
+
row values. Inspect failures privately. Session inspection excludes query text; health monitoring omits response
|
|
15
|
+
bodies. Identifiers, URLs, Terraform dry-run output and reports may still contain operator-supplied private data.
|
|
16
|
+
- PostgreSQL stores the subscription's connection information. Administrators and some server logging settings can
|
|
17
|
+
expose it. Review logging before creating a subscription with a password.
|
|
18
|
+
- Database connections use `sslmode=require`, which encrypts without verifying hostname identity. Restrict access
|
|
19
|
+
and use trusted networks. PgBouncer admin connections use local Kubernetes port-forwards and `sslmode=prefer`.
|
|
20
|
+
Certificate verification is not configured by this CLI.
|
|
21
|
+
|
|
22
|
+
Never attach live config, Terraform state, schema dumps, connection strings, customer rows, filled documents or raw
|
|
23
|
+
logs to public issues. Use synthetic fixtures and error categories. Revoke committed credentials first: deleting
|
|
24
|
+
their current file does not remove Git history, cached artifacts or copies.
|
|
25
|
+
|
|
26
|
+
## Source release review
|
|
27
|
+
|
|
28
|
+
Review every published ref and its history. Run a redacted secret scan, then inspect for internal endpoints,
|
|
29
|
+
account identifiers, infrastructure names, private incident notes and copied data that a credential detector will
|
|
30
|
+
not recognize. Check GitHub metadata, release archives and build artifacts separately.
|
|
31
|
+
|
|
32
|
+
Package only tracked source and synthetic tests. Exclude operational configs, `.db_cutover`, `.env`, credentials,
|
|
33
|
+
Terraform state, logs, virtual environments and Git history from source archives. A clean scan records the checks
|
|
34
|
+
run; it does not prove that every possible secret is absent.
|
|
35
|
+
|
|
36
|
+
## Validation scope
|
|
37
|
+
|
|
38
|
+
Unit tests cover cutover failures, interrupts, resume failures, dry runs, output handling and pure logic. Disposable
|
|
39
|
+
PostgreSQL tests cover 16-to-17 text replication, publication coverage, markers and sequences. These do not validate
|
|
40
|
+
RDS restart timing, Kubernetes pod churn, real PgBouncer network failures or application timeout budgets. Rehearse
|
|
41
|
+
those in staging and keep an operator available during a live switch.
|