db-cutover 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. db_cutover-0.2.0/.gitignore +22 -0
  2. db_cutover-0.2.0/AGENTS.md +49 -0
  3. db_cutover-0.2.0/CHANGELOG.md +25 -0
  4. db_cutover-0.2.0/LICENSE +21 -0
  5. db_cutover-0.2.0/PKG-INFO +231 -0
  6. db_cutover-0.2.0/README.md +203 -0
  7. db_cutover-0.2.0/SECURITY.md +41 -0
  8. db_cutover-0.2.0/docs/design.md +115 -0
  9. db_cutover-0.2.0/docs/procedure.md +226 -0
  10. db_cutover-0.2.0/examples/config.toml +91 -0
  11. db_cutover-0.2.0/pyproject.toml +63 -0
  12. db_cutover-0.2.0/skills/db-cutover/SKILL.md +140 -0
  13. db_cutover-0.2.0/skills/db-cutover/references/commands.md +104 -0
  14. db_cutover-0.2.0/skills/db-cutover/references/recovery.md +64 -0
  15. db_cutover-0.2.0/skills/db-cutover/references/rehearsal.md +69 -0
  16. db_cutover-0.2.0/src/db_cutover/__init__.py +3 -0
  17. db_cutover-0.2.0/src/db_cutover/__main__.py +3 -0
  18. db_cutover-0.2.0/src/db_cutover/cli.py +230 -0
  19. db_cutover-0.2.0/src/db_cutover/commands/__init__.py +0 -0
  20. db_cutover-0.2.0/src/db_cutover/commands/blue.py +189 -0
  21. db_cutover-0.2.0/src/db_cutover/commands/inspect.py +590 -0
  22. db_cutover-0.2.0/src/db_cutover/commands/replication.py +535 -0
  23. db_cutover-0.2.0/src/db_cutover/commands/switch.py +429 -0
  24. db_cutover-0.2.0/src/db_cutover/commands/terraform.py +346 -0
  25. db_cutover-0.2.0/src/db_cutover/config.py +286 -0
  26. db_cutover-0.2.0/src/db_cutover/db.py +489 -0
  27. db_cutover-0.2.0/src/db_cutover/output.py +77 -0
  28. db_cutover-0.2.0/src/db_cutover/pgbouncer.py +220 -0
  29. db_cutover-0.2.0/src/db_cutover/shell.py +88 -0
  30. db_cutover-0.2.0/tests/integration/README.md +16 -0
  31. db_cutover-0.2.0/tests/integration/compose.yml +24 -0
  32. db_cutover-0.2.0/tests/integration/test_replication.py +98 -0
  33. db_cutover-0.2.0/tests/test_config.py +182 -0
  34. db_cutover-0.2.0/tests/test_db.py +121 -0
  35. db_cutover-0.2.0/tests/test_health.py +39 -0
  36. db_cutover-0.2.0/tests/test_pgbouncer.py +32 -0
  37. db_cutover-0.2.0/tests/test_reboot.py +50 -0
  38. db_cutover-0.2.0/tests/test_safety.py +109 -0
  39. db_cutover-0.2.0/tests/test_skill.py +73 -0
  40. db_cutover-0.2.0/tests/test_switch.py +186 -0
  41. db_cutover-0.2.0/tests/test_terraform.py +187 -0
@@ -0,0 +1,22 @@
1
+ __pycache__/
2
+ *.pyc
3
+ .venv/
4
+ dist/
5
+ .pytest_cache/
6
+ .ruff_cache/
7
+ .DS_Store
8
+
9
+ # Run state (schema dumps, cutover reports) when a config lives in this repo, e.g. while testing
10
+ .db_cutover/
11
+
12
+ # Local operational inputs and outputs; committed examples use synthetic values.
13
+ /cutover.toml
14
+ *.local.toml
15
+ .env
16
+ .env.*
17
+ *.tfstate
18
+ *.tfstate.*
19
+ *.tfplan
20
+ *.log
21
+ *.pem
22
+ *.key
@@ -0,0 +1,49 @@
1
+ # db-cutover
2
+
3
+ `db-cutover`: a Python CLI for zero-downtime blue/green PostgreSQL cutovers on RDS (logical replication plus a
4
+ PgBouncer switch). `README.md` covers use, `docs/procedure.md` the steps of a cutover, `docs/design.md` the
5
+ reasoning. Read `SECURITY.md` before handling operational configuration or logs.
6
+
7
+ ## Layout
8
+
9
+ - `src/db_cutover/cli.py`: argument parsing; one subcommand per step.
10
+ - `config.py`: the TOML config (`examples/config.toml` documents every key; keep it in sync).
11
+ - `db.py`: connections and catalog queries. `pgbouncer.py`: the admin console of every pod through
12
+ `kubectl port-forward`. `shell.py`: aws, kubectl, pg_dump, psql. `output.py`: log lines, checks, JSON.
13
+ - `commands/`: `inspect.py` (read-only and long-running), `blue.py`, `replication.py`, `switch.py` (cutover,
14
+ finish), `terraform.py` (green-tf, promote-tf).
15
+ - `skills/db-cutover/`: the [Agent Skill](https://agentskills.io/specification) for agents that operate the tool
16
+ (`SKILL.md` plus `references/`). The wheel includes it as `db_cutover/skills/db-cutover`, and `db-cutover skill`
17
+ prints or installs it. It is written for an agent running a cutover; this file is for an agent changing the code.
18
+
19
+ ## Rules
20
+
21
+ - Every command that changes anything is a dry run unless `--execute` is passed. Keep it that way for new commands.
22
+ - Never print a secret or a connection string that holds one. Secrets go to subprocesses through the environment
23
+ (`PGPASSWORD`), never the command line.
24
+ - The switch (`commands/switch.py`) must resume on BLUE for any failure before RELOAD, including Ctrl-C. Don't add
25
+ steps between PAUSE and RESUME without measuring the pause on a staging run.
26
+ - After RELOAD starts, never resume mixed or unknown targets. Report failed resumes as failures. A sequence gap
27
+ and one-way replication cannot make concurrent writers on BLUE and GREEN safe.
28
+ - Probe writes, replication markers and manual RESUME require `--execute`, including retry paths.
29
+ - Inspect all client roles when draining BLUE. A failed deployment lookup must not be interpreted as zero replicas.
30
+ - Never print query text, server row diagnostics or a secret helper's stderr. Keep fixtures synthetic and review
31
+ all history before public release; removing a value from the current tree does not remove it from history.
32
+ - Logic that can be pure (sequence planning, HCL edits, ini parsing) stays pure and has tests in `tests/`.
33
+ - Long-running commands print one line per change, not per poll, so they work as background monitors.
34
+ - Before committing: `uv run pytest`, `uv run ruff check .`, `uv run ruff format --check .` (CI runs the same).
35
+ - When behaviour changes, update `README.md`, `docs/procedure.md`, `docs/design.md` and the skill in the same
36
+ commit. A new command needs a line in `skills/db-cutover/references/commands.md` (a test checks).
37
+ - The skill's frontmatter has only `name`, `description`, `license` and `metadata`. Keep `SKILL.md` under 500
38
+ lines with the rules first, and write standing instructions for "you" (the agent) and "the operator".
39
+
40
+ ## Releasing
41
+
42
+ 1. Set the version in `pyproject.toml`, `src/db_cutover/__init__.py`, the skill's `metadata.version` and the
43
+ README's install command, run `uv lock`, and give `CHANGELOG.md` a `## <version> (<date>)` section. A test
44
+ checks that they agree.
45
+ 2. Merge to `main`, then tag that commit `v<version>` and push the tag.
46
+ 3. The `Release` workflow builds the package, checks the tag against the version, publishes to PyPI with trusted
47
+ publishing (no token: PyPI trusts this repository's `release.yml` in the `pypi` environment) and creates the
48
+ GitHub release from the changelog section. PyPI never accepts the same version twice, so a mistake needs a new
49
+ version.
@@ -0,0 +1,25 @@
1
+ # Changelog
2
+
3
+ ## 0.2.0 (2026-11-16)
4
+
5
+ The first release on PyPI.
6
+
7
+ - Install with `uv tool install db-cutover` or `pipx install db-cutover`. `db-cutover --version` prints the version.
8
+ - Add an agent skill (`skills/db-cutover`) for planning, rehearsing and running a cutover with an operator: each
9
+ command and its dry run, the stop points, the rehearsal checklist and recovery. It ships inside the package:
10
+ `db-cutover skill` prints its directory and `db-cutover skill --install DIR` copies it into an agent's skills
11
+ directory.
12
+ - README: who the tool is for, a rehearsal walkthrough, and its guarantees and limits.
13
+ - Stop and require recovery after a partial or unknown PgBouncer reload. Never resume mixed BLUE/GREEN targets.
14
+ Report failed resumes and post-switch verification failures with unsuccessful exit status.
15
+ - Require `--execute` for `probe`, `wait-sync`, and `pgbouncer resume`. `verify` is read-only by default;
16
+ `verify --execute` adds a replication marker. Cutover and finish dry runs no longer write markers.
17
+ - Remove `cutover --allow-direct`: every client role must drain before switching. Failed deployment lookups block
18
+ the operation. Reboot's separate `--allow-direct` option still explicitly permits direct-client errors.
19
+ - Default to text replication. Validate publication coverage, subscription identity, sequence limits and marker
20
+ tokens. Reject unsupported data objects before switching.
21
+ - Withhold PostgreSQL/subprocess/secret-helper diagnostics that may expose credentials or row values. Remove query
22
+ text and health response bodies from inspection output; keep state files in a private directory.
23
+ - Bound port-forward startup and settle pending admin commands before confirming recovery from a late PAUSE.
24
+ - Add synthetic fault tests, PostgreSQL 16-to-17 integration tests, an MIT license, standalone examples and an
25
+ expanded operator/agent runbook. Infrastructure and application rehearsals remain required before live use.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 DocSpring
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,231 @@
1
+ Metadata-Version: 2.5
2
+ Name: db-cutover
3
+ Version: 0.2.0
4
+ Summary: Agent-friendly tools for zero-downtime blue/green PostgreSQL cutovers on RDS with logical replication and a PgBouncer switch
5
+ Project-URL: Repository, https://github.com/DocSpring/db-cutover
6
+ Project-URL: Changelog, https://github.com/DocSpring/db-cutover/blob/main/CHANGELOG.md
7
+ Project-URL: Issues, https://github.com/DocSpring/db-cutover/issues
8
+ Project-URL: Documentation, https://github.com/DocSpring/db-cutover/blob/main/docs/procedure.md
9
+ Author-email: "DocSpring, Inc." <support@docspring.com>
10
+ License-Expression: MIT
11
+ License-File: LICENSE
12
+ Keywords: blue-green,logical-replication,pgbouncer,postgresql,rds,upgrade,zero-downtime
13
+ Classifier: Development Status :: 4 - Beta
14
+ Classifier: Environment :: Console
15
+ Classifier: Intended Audience :: System Administrators
16
+ Classifier: Operating System :: MacOS
17
+ Classifier: Operating System :: POSIX :: Linux
18
+ Classifier: Programming Language :: Python :: 3
19
+ Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Programming Language :: Python :: 3.13
22
+ Classifier: Programming Language :: Python :: 3.14
23
+ Classifier: Topic :: Database
24
+ Classifier: Topic :: System :: Systems Administration
25
+ Requires-Python: >=3.11
26
+ Requires-Dist: psycopg[binary]<4,>=3.2
27
+ Description-Content-Type: text/markdown
28
+
29
+ # db-cutover
30
+
31
+ Upgrade a PostgreSQL database by copying it to a new instance, keeping that copy current with logical replication,
32
+ and switching application traffic through PgBouncer. The application keeps using the old database during the copy.
33
+ At the switch, PgBouncer holds new queries while in-flight transactions finish, then sends queued work to the new
34
+ instance.
35
+
36
+ `db-cutover` is the Python CLI that runs each step. It comes with an [example configuration](examples/config.toml),
37
+ a [step-by-step procedure](docs/procedure.md), [design notes](docs/design.md) and an
38
+ [agent skill](skills/db-cutover/SKILL.md). It is made for a human operator working with a coding agent: each command
39
+ has a narrow purpose, and every command that changes database, infrastructure or PgBouncer state is a dry run until
40
+ it gets `--execute`.
41
+
42
+ **Release candidate:** failure-path tests and PostgreSQL replication tests do not replace a rehearsal on your
43
+ application's RDS and PgBouncer deployment. Read the requirements below before planning a cutover.
44
+
45
+ ## Who it's for
46
+
47
+ Teams that run PostgreSQL on RDS behind PgBouncer in Kubernetes and need a major version upgrade (or a move to a new
48
+ instance) without a maintenance window. It suits you if:
49
+
50
+ - you want to choose the moment of the switch and see every check before it, instead of handing the upgrade to a
51
+ managed service;
52
+ - your application already connects through PgBouncer with transaction pooling, or can be made to;
53
+ - someone on the team can provision an RDS instance and change a Kubernetes ConfigMap.
54
+
55
+ It is not for databases older than PostgreSQL 16, setups without PgBouncer, or schemas that rely on large objects,
56
+ materialized views, unlogged tables or foreign tables. For those, look at
57
+ [RDS Blue/Green Deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html) or
58
+ `pg_upgrade` with a maintenance window.
59
+
60
+ ## Requirements
61
+
62
+ - PostgreSQL **16 or newer**, running on RDS. BLUE is the existing primary; GREEN is a separate instance at the
63
+ target version. One database is migrated at a time. Test the specific source/target versions and extensions.
64
+ - PgBouncer in Kubernetes, with transaction pooling. Every foreground database client must use the selected pods.
65
+ These commands pause/resume **all databases on those pods**, and RELOAD applies the entire configuration.
66
+ Use dedicated pods for this workflow.
67
+ - A directory-mounted ConfigMap containing `pgbouncer.ini`, with no automatic reload. `subPath` mounts do not receive
68
+ ConfigMap updates. Prevent rollouts, scaling and automatic restarts during the prepared-config window; see the
69
+ [procedure](docs/procedure.md#7-prepare-and-switch).
70
+ - All connections bypassing PgBouncer must be stopped and prevented from reconnecting during the switch. Inventory
71
+ consoles, schedulers, maintenance jobs, migrations and every application role, not just your web process.
72
+ - A schema freeze from `copy-schema` through `finish`. Roles, grants, extensions, settings and other databases need
73
+ separate preparation. Large objects, unlogged tables, materialized views and foreign tables are blocked.
74
+ - Sequences with `INCREMENT BY 1`, no cycling, sufficient capacity, and application code that tolerates gaps in IDs.
75
+ - Request, proxy, job and PgBouncer timeouts longer than the measured pause. Queries wait during the switch;
76
+ **zero failed requests depends on those timeouts and successful recovery**. An RDS reboot can take much longer.
77
+
78
+ ## Install
79
+
80
+ `db-cutover` needs Python 3.11 or newer. Install it as a standalone tool with [uv](https://docs.astral.sh/uv/) or
81
+ [pipx](https://pipx.pypa.io/):
82
+
83
+ ```sh
84
+ uv tool install db-cutover==0.2.0 # or: pipx install db-cutover==0.2.0
85
+ db-cutover --version
86
+ db-cutover --help
87
+ ```
88
+
89
+ Pin the version and keep it fixed from the first rehearsal to the end of the migration. To run from source instead,
90
+ clone the repository at a release tag and use its locked dependencies (`uv sync --frozen`, then `uv run db-cutover`).
91
+
92
+ The machine running the tool also needs `aws`, `kubectl`, and matching, current `pg_dump`/`psql` clients at least as
93
+ new as **both** database servers. `op` is optional for 1Password references; Terraform is optional for the two HCL
94
+ helpers. It needs network access to both databases and the Kubernetes API. GREEN must reach BLUE for replication.
95
+
96
+ Copy the [example](examples/config.toml) to a private working directory, fill in your identifiers and contexts,
97
+ and supply credentials through environment variables or a secret manager. Never commit the operational config,
98
+ schema dumps, reports, Terraform state or logs. `cmd:` secret references execute shell commands: treat configs as
99
+ trusted executable input.
100
+
101
+ ## Working with an agent
102
+
103
+ The package includes an [Agent Skill](https://agentskills.io/specification) that teaches a coding agent (Claude
104
+ Code, Codex, Cursor and others) to plan, rehearse and run a cutover with you: what each command does, which ones
105
+ need `--execute`, the rehearsal checklist, the points where it must stop for your decision, and recovery.
106
+
107
+ ```sh
108
+ db-cutover skill --install .claude/skills # or .agents/skills, or your agent's skills directory
109
+ npx skills add DocSpring/db-cutover # the same skill, straight from this repository
110
+ ```
111
+
112
+ Give the agent the config path, the approved versions, the traffic controls and the limits of what it may execute.
113
+ Have it run each changing command without `--execute` first, explain the output, and follow your agreed execution
114
+ policy. A failed gate is a stop condition.
115
+
116
+ Exit status 0 means the command completed its requested mode, 1 means failure, 2 means an argument/config error,
117
+ and 130 means interruption. A dry-run success is not a completed migration. Long-running monitors continue until
118
+ stopped; quiet output alone is not proof that a monitor is still alive.
119
+
120
+ ## Workflow
121
+
122
+ Read [the procedure](docs/procedure.md) for the provisioning, permissions, traffic controls and recovery steps
123
+ between these commands. In the examples below, `C` is a private config path and `TARGET_VERSION` is the RDS engine
124
+ version you have selected and tested. Prefix commands with `uv run` if you run from a source checkout.
125
+
126
+ ```sh
127
+ db-cutover "$C" preflight
128
+ # Enable logical replication; reboot-blue --execute only if the static parameter needs it.
129
+ db-cutover "$C" prepare-blue --execute
130
+ db-cutover "$C" green-tf --engine-version "$TARGET_VERSION" --execute
131
+ # Review and provision GREEN, including version-compatible parameters and network access.
132
+ db-cutover "$C" copy-schema --execute
133
+ # Create application roles, grants and required extensions on GREEN.
134
+ db-cutover "$C" start-replication --execute
135
+ db-cutover "$C" wait-sync --execute
136
+ db-cutover "$C" verify --execute
137
+ db-cutover "$C" sequences --execute
138
+ # Stop and fence direct writers before the rehearsal.
139
+ db-cutover "$C" cutover --execute --rehearse
140
+ # Prepare PgBouncer's config as described in the procedure, without reloading it.
141
+ db-cutover "$C" cutover
142
+ db-cutover "$C" cutover --execute
143
+ # Move direct writers to GREEN and prevent any application writes to BLUE.
144
+ db-cutover "$C" finish --execute
145
+ db-cutover "$C" promote-tf --execute
146
+ # Review and run the printed state moves, then verify the Terraform plan.
147
+ ```
148
+
149
+ If health endpoints require authentication, set `[secrets] health_check_token`; `watch` sends it as
150
+ `X-Health-Check-Token` to every configured health URL. Only configure trusted HTTPS endpoints. Redirects are reported
151
+ as failures rather than followed, so the header cannot be forwarded to a different endpoint.
152
+
153
+ Use `watch` to monitor health/replication and `probe --execute` to write synthetic probe rows through PgBouncer.
154
+ Start the probe after `prepare-blue`; stop it before `finish` drops its table. `verify` without `--execute` is
155
+ read-only; `--execute` adds a marker write to test replication. Dry runs can connect, read data, create local state
156
+ files, and consume database resources. They do not pause clients or modify remote state.
157
+
158
+ ## Rehearse first
159
+
160
+ Rehearse twice: the whole procedure on staging, then the pause on production.
161
+
162
+ 1. **Build a staging setup that matches production** where it matters: the same PgBouncer image, pool mode and
163
+ ConfigMap mount, the same PostgreSQL source and target versions and extensions, and the application with its
164
+ real timeouts. Put traffic on it.
165
+ 2. **Run every step there, including a real switch**, with `watch` and `probe --execute` running. Write down how
166
+ long the copy took, how much WAL BLUE kept, and the pause the switch reports.
167
+ 3. **Cause the failures on purpose.** Leave a direct writer connected, hold a transaction open on BLUE, press
168
+ Ctrl-C during a rehearsal's pause, restart a PgBouncer pod after the ConfigMap names GREEN. Each one should stop
169
+ the switch and leave traffic on BLUE. Practise the [recovery steps](docs/procedure.md#recovery) for the last one.
170
+ 4. **On production, rehearse the pause** once replication has caught up and the direct writers are stopped:
171
+
172
+ ```sh
173
+ db-cutover "$C" cutover --execute --rehearse
174
+ ```
175
+
176
+ This pauses live traffic, runs every check the real switch runs, and resumes on BLUE. Its report gives the pause
177
+ in milliseconds. Compare it with every timeout in the request path before you go on.
178
+
179
+ `--rehearse` never reloads PgBouncer, so only the staging switch tests RELOAD, GREEN under the application and
180
+ recovery from a partial switch. The skill's
181
+ [rehearsal checklist](skills/db-cutover/references/rehearsal.md) lists each thing to check and record.
182
+
183
+ ## Guarantees and limits
184
+
185
+ What the tool does:
186
+
187
+ - **Nothing changes without `--execute`.** A dry run reads, checks and reports. It can connect, read data, create
188
+ local state files and use database resources, but it never pauses clients or changes remote state.
189
+ - **No switch without proof that GREEN is current.** With every PgBouncer pod paused, the tool checks that no
190
+ client session is left on BLUE, writes a marker on BLUE and waits for it on GREEN, and compares the highest ID of
191
+ every table before any pod reloads.
192
+ - **A failure before RELOAD leaves BLUE as primary.** The tool resumes traffic on BLUE, including on Ctrl-C, and
193
+ reports a resume it could not confirm.
194
+ - **It never resumes pods that point at different databases.** After RELOAD starts, mixed or unknown targets
195
+ **remain paused** and the command fails. An operator aligns every pod to GREEN before resuming.
196
+ - **It keeps secrets and row data out of its output.** PostgreSQL diagnostics, query text and health response
197
+ bodies are withheld. State files go in a private directory.
198
+
199
+ What it can't do:
200
+
201
+ - **Promise zero failed requests.** Queries wait during the pause. Requests succeed only if every timeout in their
202
+ path is longer than the pause and recovery works. An RDS reboot can take much longer than a switch.
203
+ - **Fence clients it doesn't know about.** Anything that connects without PgBouncer has to be stopped and kept
204
+ away by you. The tool checks for sessions; it can't prevent new ones.
205
+ - **Close the window between the ConfigMap change and the switch.** A PgBouncer pod restarted in that window
206
+ serves GREEN early. The tool detects it before switching but can't stop it.
207
+ - **Go back to BLUE once GREEN has taken writes.** Replication runs one way. Never read replication or a sequence
208
+ gap as permission to write to both databases. After the switch, fix forward on GREEN.
209
+ - **Survive its own process being killed.** After SIGKILL or a lost connection during a switch, an operator has to
210
+ read the state from the pods.
211
+ - **Prove the data is identical.** `verify` compares schemas, counts and sampled rows. It is not a full comparison.
212
+ - **Copy everything.** Roles, grants, extensions, database settings and other databases need separate preparation.
213
+ Sequence values are advanced, not replicated.
214
+
215
+ The [design notes](docs/design.md) give the reasoning, and [SECURITY.md](SECURITY.md) the limits of the tests.
216
+
217
+ ## Development
218
+
219
+ ```sh
220
+ uv sync --frozen --group dev
221
+ uv run pytest
222
+ uv run ruff check .
223
+ uv run ruff format --check .
224
+ uv build
225
+ ```
226
+
227
+ The default suite uses no cloud credentials. [Integration tests](tests/integration/README.md) use disposable
228
+ PostgreSQL 16 and 17 containers with synthetic data. CI runs both suites. See [SECURITY.md](SECURITY.md) for data
229
+ handling and release checks, and [AGENTS.md](AGENTS.md) for the code's rules and how to release.
230
+
231
+ Built by [DocSpring](https://docspring.com). Licensed under [MIT](LICENSE).
@@ -0,0 +1,203 @@
1
+ # db-cutover
2
+
3
+ Upgrade a PostgreSQL database by copying it to a new instance, keeping that copy current with logical replication,
4
+ and switching application traffic through PgBouncer. The application keeps using the old database during the copy.
5
+ At the switch, PgBouncer holds new queries while in-flight transactions finish, then sends queued work to the new
6
+ instance.
7
+
8
+ `db-cutover` is the Python CLI that runs each step. It comes with an [example configuration](examples/config.toml),
9
+ a [step-by-step procedure](docs/procedure.md), [design notes](docs/design.md) and an
10
+ [agent skill](skills/db-cutover/SKILL.md). It is made for a human operator working with a coding agent: each command
11
+ has a narrow purpose, and every command that changes database, infrastructure or PgBouncer state is a dry run until
12
+ it gets `--execute`.
13
+
14
+ **Release candidate:** failure-path tests and PostgreSQL replication tests do not replace a rehearsal on your
15
+ application's RDS and PgBouncer deployment. Read the requirements below before planning a cutover.
16
+
17
+ ## Who it's for
18
+
19
+ Teams that run PostgreSQL on RDS behind PgBouncer in Kubernetes and need a major version upgrade (or a move to a new
20
+ instance) without a maintenance window. It suits you if:
21
+
22
+ - you want to choose the moment of the switch and see every check before it, instead of handing the upgrade to a
23
+ managed service;
24
+ - your application already connects through PgBouncer with transaction pooling, or can be made to;
25
+ - someone on the team can provision an RDS instance and change a Kubernetes ConfigMap.
26
+
27
+ It is not for databases older than PostgreSQL 16, setups without PgBouncer, or schemas that rely on large objects,
28
+ materialized views, unlogged tables or foreign tables. For those, look at
29
+ [RDS Blue/Green Deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html) or
30
+ `pg_upgrade` with a maintenance window.
31
+
32
+ ## Requirements
33
+
34
+ - PostgreSQL **16 or newer**, running on RDS. BLUE is the existing primary; GREEN is a separate instance at the
35
+ target version. One database is migrated at a time. Test the specific source/target versions and extensions.
36
+ - PgBouncer in Kubernetes, with transaction pooling. Every foreground database client must use the selected pods.
37
+ These commands pause/resume **all databases on those pods**, and RELOAD applies the entire configuration.
38
+ Use dedicated pods for this workflow.
39
+ - A directory-mounted ConfigMap containing `pgbouncer.ini`, with no automatic reload. `subPath` mounts do not receive
40
+ ConfigMap updates. Prevent rollouts, scaling and automatic restarts during the prepared-config window; see the
41
+ [procedure](docs/procedure.md#7-prepare-and-switch).
42
+ - All connections bypassing PgBouncer must be stopped and prevented from reconnecting during the switch. Inventory
43
+ consoles, schedulers, maintenance jobs, migrations and every application role, not just your web process.
44
+ - A schema freeze from `copy-schema` through `finish`. Roles, grants, extensions, settings and other databases need
45
+ separate preparation. Large objects, unlogged tables, materialized views and foreign tables are blocked.
46
+ - Sequences with `INCREMENT BY 1`, no cycling, sufficient capacity, and application code that tolerates gaps in IDs.
47
+ - Request, proxy, job and PgBouncer timeouts longer than the measured pause. Queries wait during the switch;
48
+ **zero failed requests depends on those timeouts and successful recovery**. An RDS reboot can take much longer.
49
+
50
+ ## Install
51
+
52
+ `db-cutover` needs Python 3.11 or newer. Install it as a standalone tool with [uv](https://docs.astral.sh/uv/) or
53
+ [pipx](https://pipx.pypa.io/):
54
+
55
+ ```sh
56
+ uv tool install db-cutover==0.2.0 # or: pipx install db-cutover==0.2.0
57
+ db-cutover --version
58
+ db-cutover --help
59
+ ```
60
+
61
+ Pin the version and keep it fixed from the first rehearsal to the end of the migration. To run from source instead,
62
+ clone the repository at a release tag and use its locked dependencies (`uv sync --frozen`, then `uv run db-cutover`).
63
+
64
+ The machine running the tool also needs `aws`, `kubectl`, and matching, current `pg_dump`/`psql` clients at least as
65
+ new as **both** database servers. `op` is optional for 1Password references; Terraform is optional for the two HCL
66
+ helpers. It needs network access to both databases and the Kubernetes API. GREEN must reach BLUE for replication.
67
+
68
+ Copy the [example](examples/config.toml) to a private working directory, fill in your identifiers and contexts,
69
+ and supply credentials through environment variables or a secret manager. Never commit the operational config,
70
+ schema dumps, reports, Terraform state or logs. `cmd:` secret references execute shell commands: treat configs as
71
+ trusted executable input.
72
+
73
+ ## Working with an agent
74
+
75
+ The package includes an [Agent Skill](https://agentskills.io/specification) that teaches a coding agent (Claude
76
+ Code, Codex, Cursor and others) to plan, rehearse and run a cutover with you: what each command does, which ones
77
+ need `--execute`, the rehearsal checklist, the points where it must stop for your decision, and recovery.
78
+
79
+ ```sh
80
+ db-cutover skill --install .claude/skills # or .agents/skills, or your agent's skills directory
81
+ npx skills add DocSpring/db-cutover # the same skill, straight from this repository
82
+ ```
83
+
84
+ Give the agent the config path, the approved versions, the traffic controls and the limits of what it may execute.
85
+ Have it run each changing command without `--execute` first, explain the output, and follow your agreed execution
86
+ policy. A failed gate is a stop condition.
87
+
88
+ Exit status 0 means the command completed its requested mode, 1 means failure, 2 means an argument/config error,
89
+ and 130 means interruption. A dry-run success is not a completed migration. Long-running monitors continue until
90
+ stopped; quiet output alone is not proof that a monitor is still alive.
91
+
92
+ ## Workflow
93
+
94
+ Read [the procedure](docs/procedure.md) for the provisioning, permissions, traffic controls and recovery steps
95
+ between these commands. In the examples below, `C` is a private config path and `TARGET_VERSION` is the RDS engine
96
+ version you have selected and tested. Prefix commands with `uv run` if you run from a source checkout.
97
+
98
+ ```sh
99
+ db-cutover "$C" preflight
100
+ # Enable logical replication; reboot-blue --execute only if the static parameter needs it.
101
+ db-cutover "$C" prepare-blue --execute
102
+ db-cutover "$C" green-tf --engine-version "$TARGET_VERSION" --execute
103
+ # Review and provision GREEN, including version-compatible parameters and network access.
104
+ db-cutover "$C" copy-schema --execute
105
+ # Create application roles, grants and required extensions on GREEN.
106
+ db-cutover "$C" start-replication --execute
107
+ db-cutover "$C" wait-sync --execute
108
+ db-cutover "$C" verify --execute
109
+ db-cutover "$C" sequences --execute
110
+ # Stop and fence direct writers before the rehearsal.
111
+ db-cutover "$C" cutover --execute --rehearse
112
+ # Prepare PgBouncer's config as described in the procedure, without reloading it.
113
+ db-cutover "$C" cutover
114
+ db-cutover "$C" cutover --execute
115
+ # Move direct writers to GREEN and prevent any application writes to BLUE.
116
+ db-cutover "$C" finish --execute
117
+ db-cutover "$C" promote-tf --execute
118
+ # Review and run the printed state moves, then verify the Terraform plan.
119
+ ```
120
+
121
+ If health endpoints require authentication, set `[secrets] health_check_token`; `watch` sends it as
122
+ `X-Health-Check-Token` to every configured health URL. Only configure trusted HTTPS endpoints. Redirects are reported
123
+ as failures rather than followed, so the header cannot be forwarded to a different endpoint.
124
+
125
+ Use `watch` to monitor health/replication and `probe --execute` to write synthetic probe rows through PgBouncer.
126
+ Start the probe after `prepare-blue`; stop it before `finish` drops its table. `verify` without `--execute` is
127
+ read-only; `--execute` adds a marker write to test replication. Dry runs can connect, read data, create local state
128
+ files, and consume database resources. They do not pause clients or modify remote state.
129
+
130
+ ## Rehearse first
131
+
132
+ Rehearse twice: the whole procedure on staging, then the pause on production.
133
+
134
+ 1. **Build a staging setup that matches production** where it matters: the same PgBouncer image, pool mode and
135
+ ConfigMap mount, the same PostgreSQL source and target versions and extensions, and the application with its
136
+ real timeouts. Put traffic on it.
137
+ 2. **Run every step there, including a real switch**, with `watch` and `probe --execute` running. Write down how
138
+ long the copy took, how much WAL BLUE kept, and the pause the switch reports.
139
+ 3. **Cause the failures on purpose.** Leave a direct writer connected, hold a transaction open on BLUE, press
140
+ Ctrl-C during a rehearsal's pause, restart a PgBouncer pod after the ConfigMap names GREEN. Each one should stop
141
+ the switch and leave traffic on BLUE. Practise the [recovery steps](docs/procedure.md#recovery) for the last one.
142
+ 4. **On production, rehearse the pause** once replication has caught up and the direct writers are stopped:
143
+
144
+ ```sh
145
+ db-cutover "$C" cutover --execute --rehearse
146
+ ```
147
+
148
+ This pauses live traffic, runs every check the real switch runs, and resumes on BLUE. Its report gives the pause
149
+ in milliseconds. Compare it with every timeout in the request path before you go on.
150
+
151
+ `--rehearse` never reloads PgBouncer, so only the staging switch tests RELOAD, GREEN under the application and
152
+ recovery from a partial switch. The skill's
153
+ [rehearsal checklist](skills/db-cutover/references/rehearsal.md) lists each thing to check and record.
154
+
155
+ ## Guarantees and limits
156
+
157
+ What the tool does:
158
+
159
+ - **Nothing changes without `--execute`.** A dry run reads, checks and reports. It can connect, read data, create
160
+ local state files and use database resources, but it never pauses clients or changes remote state.
161
+ - **No switch without proof that GREEN is current.** With every PgBouncer pod paused, the tool checks that no
162
+ client session is left on BLUE, writes a marker on BLUE and waits for it on GREEN, and compares the highest ID of
163
+ every table before any pod reloads.
164
+ - **A failure before RELOAD leaves BLUE as primary.** The tool resumes traffic on BLUE, including on Ctrl-C, and
165
+ reports a resume it could not confirm.
166
+ - **It never resumes pods that point at different databases.** After RELOAD starts, mixed or unknown targets
167
+ **remain paused** and the command fails. An operator aligns every pod to GREEN before resuming.
168
+ - **It keeps secrets and row data out of its output.** PostgreSQL diagnostics, query text and health response
169
+ bodies are withheld. State files go in a private directory.
170
+
171
+ What it can't do:
172
+
173
+ - **Promise zero failed requests.** Queries wait during the pause. Requests succeed only if every timeout in their
174
+ path is longer than the pause and recovery works. An RDS reboot can take much longer than a switch.
175
+ - **Fence clients it doesn't know about.** Anything that connects without PgBouncer has to be stopped and kept
176
+ away by you. The tool checks for sessions; it can't prevent new ones.
177
+ - **Close the window between the ConfigMap change and the switch.** A PgBouncer pod restarted in that window
178
+ serves GREEN early. The tool detects it before switching but can't stop it.
179
+ - **Go back to BLUE once GREEN has taken writes.** Replication runs one way. Never read replication or a sequence
180
+ gap as permission to write to both databases. After the switch, fix forward on GREEN.
181
+ - **Survive its own process being killed.** After SIGKILL or a lost connection during a switch, an operator has to
182
+ read the state from the pods.
183
+ - **Prove the data is identical.** `verify` compares schemas, counts and sampled rows. It is not a full comparison.
184
+ - **Copy everything.** Roles, grants, extensions, database settings and other databases need separate preparation.
185
+ Sequence values are advanced, not replicated.
186
+
187
+ The [design notes](docs/design.md) give the reasoning, and [SECURITY.md](SECURITY.md) the limits of the tests.
188
+
189
+ ## Development
190
+
191
+ ```sh
192
+ uv sync --frozen --group dev
193
+ uv run pytest
194
+ uv run ruff check .
195
+ uv run ruff format --check .
196
+ uv build
197
+ ```
198
+
199
+ The default suite uses no cloud credentials. [Integration tests](tests/integration/README.md) use disposable
200
+ PostgreSQL 16 and 17 containers with synthetic data. CI runs both suites. See [SECURITY.md](SECURITY.md) for data
201
+ handling and release checks, and [AGENTS.md](AGENTS.md) for the code's rules and how to release.
202
+
203
+ Built by [DocSpring](https://docspring.com). Licensed under [MIT](LICENSE).
@@ -0,0 +1,41 @@
1
+ # Security and operational data
2
+
3
+ This tool operates with administrative database and infrastructure access. Follow your organization's change
4
+ authorization policy. A successful dry run is not authorization to execute a production change.
5
+
6
+ ## Private inputs and output
7
+
8
+ - Keep operational TOML files outside the source repository. Use environment variables or a secret manager,
9
+ never literal credentials in config or on a command line. `cmd:` references execute shell commands; configs,
10
+ Terraform settings and schema dumps are trusted executable inputs, not safe inputs from strangers.
11
+ - Keep schema dumps, reports, Terraform state/plans and logs private. Schema functions and comments can contain
12
+ secrets even without table rows. The state directory uses mode 0700.
13
+ - PostgreSQL diagnostics and subprocess error output are withheld because they can contain SQL, credentials and
14
+ row values. Inspect failures privately. Session inspection excludes query text; health monitoring omits response
15
+ bodies. Identifiers, URLs, Terraform dry-run output and reports may still contain operator-supplied private data.
16
+ - PostgreSQL stores the subscription's connection information. Administrators and some server logging settings can
17
+ expose it. Review logging before creating a subscription with a password.
18
+ - Database connections use `sslmode=require`, which encrypts without verifying hostname identity. Restrict access
19
+ and use trusted networks. PgBouncer admin connections use local Kubernetes port-forwards and `sslmode=prefer`.
20
+ Certificate verification is not configured by this CLI.
21
+
22
+ Never attach live config, Terraform state, schema dumps, connection strings, customer rows, filled documents or raw
23
+ logs to public issues. Use synthetic fixtures and error categories. Revoke committed credentials first: deleting
24
+ their current file does not remove Git history, cached artifacts or copies.
25
+
26
+ ## Source release review
27
+
28
+ Review every published ref and its history. Run a redacted secret scan, then inspect for internal endpoints,
29
+ account identifiers, infrastructure names, private incident notes and copied data that a credential detector will
30
+ not recognize. Check GitHub metadata, release archives and build artifacts separately.
31
+
32
+ Package only tracked source and synthetic tests. Exclude operational configs, `.db_cutover`, `.env`, credentials,
33
+ Terraform state, logs, virtual environments and Git history from source archives. A clean scan records the checks
34
+ run; it does not prove that every possible secret is absent.
35
+
36
+ ## Validation scope
37
+
38
+ Unit tests cover cutover failures, interrupts, resume failures, dry runs, output handling and pure logic. Disposable
39
+ PostgreSQL tests cover 16-to-17 text replication, publication coverage, markers and sequences. These do not validate
40
+ RDS restart timing, Kubernetes pod churn, real PgBouncer network failures or application timeout budgets. Rehearse
41
+ those in staging and keep an operator available during a live switch.