pgnudge 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pgnudge-1.0.0/.github/workflows/ci.yml +56 -0
- pgnudge-1.0.0/.github/workflows/release.yml +19 -0
- pgnudge-1.0.0/.gitignore +29 -0
- pgnudge-1.0.0/.python-version +1 -0
- pgnudge-1.0.0/AGENTS.md +465 -0
- pgnudge-1.0.0/CLAUDE.md +5 -0
- pgnudge-1.0.0/LICENSE +21 -0
- pgnudge-1.0.0/PKG-INFO +219 -0
- pgnudge-1.0.0/README.md +198 -0
- pgnudge-1.0.0/docs/temporary-slots.md +203 -0
- pgnudge-1.0.0/examples/minimal.py +17 -0
- pgnudge-1.0.0/pyproject.toml +74 -0
- pgnudge-1.0.0/src/pgnudge/__init__.py +17 -0
- pgnudge-1.0.0/src/pgnudge/core.py +39 -0
- pgnudge-1.0.0/src/pgnudge/engine.py +286 -0
- pgnudge-1.0.0/src/pgnudge/proto.py +248 -0
- pgnudge-1.0.0/src/pgnudge/py.typed +0 -0
- pgnudge-1.0.0/src/pgnudge/wal.py +229 -0
- pgnudge-1.0.0/tests/conftest.py +103 -0
- pgnudge-1.0.0/tests/test_engine.py +266 -0
- pgnudge-1.0.0/tests/test_proto.py +413 -0
- pgnudge-1.0.0/tests/test_wal.py +172 -0
- pgnudge-1.0.0/tests/test_wal_unit.py +365 -0
- pgnudge-1.0.0/tests/wire.py +117 -0
- pgnudge-1.0.0/uv.lock +840 -0
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
lint:
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
steps:
|
|
12
|
+
- uses: actions/checkout@v4
|
|
13
|
+
- uses: astral-sh/setup-uv@v5
|
|
14
|
+
- run: uv sync
|
|
15
|
+
- run: uv run ruff check .
|
|
16
|
+
- run: uv run mypy
|
|
17
|
+
|
|
18
|
+
test:
|
|
19
|
+
runs-on: ubuntu-latest
|
|
20
|
+
strategy:
|
|
21
|
+
fail-fast: false
|
|
22
|
+
matrix:
|
|
23
|
+
python: ["3.11", "3.12", "3.13", "3.14"]
|
|
24
|
+
postgres: ["16", "17", "18"]
|
|
25
|
+
env:
|
|
26
|
+
POSTGRES_IMAGE: postgres:${{ matrix.postgres }}
|
|
27
|
+
steps:
|
|
28
|
+
- uses: actions/checkout@v4
|
|
29
|
+
- uses: astral-sh/setup-uv@v5
|
|
30
|
+
with:
|
|
31
|
+
python-version: ${{ matrix.python }}
|
|
32
|
+
- run: uv sync
|
|
33
|
+
# 100% line+branch locally; 98 leaves headroom for the TLS-dependent
|
|
34
|
+
# lines when the container's best-effort TLS setup fails
|
|
35
|
+
- run: uv run pytest -q --cov=pgnudge --cov-report=term-missing --cov-fail-under=98
|
|
36
|
+
|
|
37
|
+
# keeps the wal2json path honest (default matrix runs test_decoding)
|
|
38
|
+
test-wal2json:
|
|
39
|
+
runs-on: ubuntu-latest
|
|
40
|
+
env:
|
|
41
|
+
POSTGRES_IMAGE: pgnudge-wal2json:16
|
|
42
|
+
PGNUDGE_PLUGIN: wal2json
|
|
43
|
+
steps:
|
|
44
|
+
- uses: actions/checkout@v4
|
|
45
|
+
# no public image ships wal2json for PG 16 (debezium/postgres dropped it),
|
|
46
|
+
# so build one: official postgres + the PGDG package
|
|
47
|
+
- run: |
|
|
48
|
+
docker build -t pgnudge-wal2json:16 - <<'EOF'
|
|
49
|
+
FROM postgres:16
|
|
50
|
+
RUN apt-get update -o Dir::Etc::sourcelist="sources.list.d/pgdg.list" -o Dir::Etc::sourceparts="-" \
|
|
51
|
+
&& apt-get install -y --no-install-recommends postgresql-16-wal2json \
|
|
52
|
+
&& rm -rf /var/lib/apt/lists/*
|
|
53
|
+
EOF
|
|
54
|
+
- uses: astral-sh/setup-uv@v5
|
|
55
|
+
- run: uv sync
|
|
56
|
+
- run: uv run pytest -q --cov=pgnudge --cov-report=term-missing --cov-fail-under=98
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
name: Release
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
tags: ["v*"]
|
|
6
|
+
|
|
7
|
+
jobs:
|
|
8
|
+
release:
|
|
9
|
+
runs-on: ubuntu-latest
|
|
10
|
+
environment: release
|
|
11
|
+
permissions:
|
|
12
|
+
id-token: write
|
|
13
|
+
steps:
|
|
14
|
+
- uses: actions/checkout@v4
|
|
15
|
+
- uses: astral-sh/setup-uv@v5
|
|
16
|
+
# a tag/version mismatch would burn the PyPI version for good
|
|
17
|
+
- run: test "v$(uv version --short)" = "$GITHUB_REF_NAME"
|
|
18
|
+
- run: uv build
|
|
19
|
+
- run: uv publish --trusted-publishing always
|
pgnudge-1.0.0/.gitignore
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# python
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
|
|
5
|
+
# uv / packaging
|
|
6
|
+
.venv/
|
|
7
|
+
dist/
|
|
8
|
+
build/
|
|
9
|
+
*.egg-info/
|
|
10
|
+
|
|
11
|
+
# tooling caches
|
|
12
|
+
.mypy_cache/
|
|
13
|
+
.ruff_cache/
|
|
14
|
+
.pytest_cache/
|
|
15
|
+
.coverage
|
|
16
|
+
htmlcov/
|
|
17
|
+
|
|
18
|
+
# environment / secrets — never commit
|
|
19
|
+
.env
|
|
20
|
+
.env.*
|
|
21
|
+
|
|
22
|
+
# editors & OS
|
|
23
|
+
.DS_Store
|
|
24
|
+
.idea/
|
|
25
|
+
.vscode/
|
|
26
|
+
*.swp
|
|
27
|
+
|
|
28
|
+
# claude session artifacts — never commit
|
|
29
|
+
.claude/
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
3.13
|
pgnudge-1.0.0/AGENTS.md
ADDED
|
@@ -0,0 +1,465 @@
|
|
|
1
|
+
# AGENTS.md — Guidance for AI Code-Generation Agents
|
|
2
|
+
|
|
3
|
+
pgnudge is a Python framework for **push-only change nudges from PostgreSQL
|
|
4
|
+
with zero server footprint**. Consumers get an async iterator of two item
|
|
5
|
+
types — `Resync` (reload everything) and `Batch` (coalesced wakeups saying
|
|
6
|
+
*which tables moved*) — and refetch their own data. The framework never
|
|
7
|
+
carries application data. Python >= 3.11, async-first, MIT-licensed,
|
|
8
|
+
runtime dependency: scramp only.
|
|
9
|
+
|
|
10
|
+
Named `pgnudge` (v1.0.0): *pgqueuer moves work, pgnudge moves wakefulness.*
|
|
11
|
+
|
|
12
|
+
Read this file fully before changing anything. The full design history is
|
|
13
|
+
deliberately not in this repo (`docs/` holds only the public
|
|
14
|
+
temporary-slots reference); settled decisions are summarized in the five
|
|
15
|
+
laws below — ask the owner before re-opening any of them.
|
|
16
|
+
|
|
17
|
+
## The five laws (non-negotiable, owner-imposed)
|
|
18
|
+
|
|
19
|
+
These were each fought for across a long design process. Do not trade any of
|
|
20
|
+
them away, even for apparently good reasons. If a task seems to require
|
|
21
|
+
violating one, stop and ask the owner.
|
|
22
|
+
|
|
23
|
+
1. **Push only.** No polling of any kind — no `pg_logical_slot_get_changes`,
|
|
24
|
+
no periodic SELECTs, no sentinel queries. The only timers permitted:
|
|
25
|
+
reconnect backoff, standby-status feedback (a protocol keepalive on an
|
|
26
|
+
open stream, not a data poll), and the *user-opt-in* `failsafe` knob
|
|
27
|
+
(default off).
|
|
28
|
+
2. **Native Postgres only.** No external CDC systems, brokers, or sidecars
|
|
29
|
+
beyond an optional bridge daemon that is itself just this library.
|
|
30
|
+
3. **Zero server persistence.** Nothing created on the server may outlive
|
|
31
|
+
the connection. The only primitive in all of PostgreSQL that satisfies
|
|
32
|
+
this for a change feed is the **temporary replication slot** ("not saved
|
|
33
|
+
to disk, automatically dropped on error or when the session has
|
|
34
|
+
finished"). Triggers were explicitly vetoed by the owner: persistent
|
|
35
|
+
catalog objects fail this law by definition. `ddl.py` (a trigger
|
|
36
|
+
generator) was the boxed exception; the owner had it deleted 2026-07-03.
|
|
37
|
+
Do not reintroduce a trigger path.
|
|
38
|
+
4. **Driver-free.** The replication transport is a hand-rolled walsender
|
|
39
|
+
client (`proto.py`, stdlib + scramp), because no Python library outside
|
|
40
|
+
psycopg2 speaks the replication protocol (asyncpg issue #91 has been
|
|
41
|
+
open since 2017) and psycopg2 is vetoed. Runtime dependency: scramp
|
|
42
|
+
only. asyncpg exists solely in the dev group for the live test's admin
|
|
43
|
+
connection.
|
|
44
|
+
5. **From-connect-only.** No history, no backfill, no replay. Slots are
|
|
45
|
+
created fresh at every (re)connect with `SNAPSHOT 'nothing'`; reconnect
|
|
46
|
+
*resyncs*, never resumes. Missing changes while disconnected is a
|
|
47
|
+
requirement, not a bug — the `Resync` item brackets every gap and the
|
|
48
|
+
consumer's refetch is the source of truth.
|
|
49
|
+
|
|
50
|
+
## Things you will be tempted to do — don't
|
|
51
|
+
|
|
52
|
+
- **Add a graceful `DROP_REPLICATION_SLOT` or protocol goodbye on close.**
|
|
53
|
+
`WalFeed._extra_close()` hard-aborts the socket *on purpose*: crash and
|
|
54
|
+
clean exit must exercise the identical server-side cleanup path, and the
|
|
55
|
+
test suite depends on proving that. Cleanup is the server's job.
|
|
56
|
+
- **Persist the slot / store an LSN cursor to "not miss events".** Violates
|
|
57
|
+
laws 3 and 5. Resync-plus-refetch *is* the reliability model.
|
|
58
|
+
- **Wrap protocol reads in `asyncio.wait_for`.** Cancelling a read between
|
|
59
|
+
frame header and body desyncs the stream. Feedback runs on its own timer
|
|
60
|
+
task precisely so reads can block freely. Keep it that way.
|
|
61
|
+
- **Put row data in payloads by default.** The v1 payload contract is
|
|
62
|
+
`schema.table`, and coalescing depends on payload identity. A `:pk`
|
|
63
|
+
payload is a planned *opt-in* (see Backlog).
|
|
64
|
+
- **Switch to pgoutput to drop the wal2json dependency.** Careful: pgoutput
|
|
65
|
+
requires `publication_names`, and a **publication is a persistent catalog
|
|
66
|
+
object** — it likely violates law 3. This roadmap item is conditional at
|
|
67
|
+
best; raise it with the owner before starting (a pre-existing
|
|
68
|
+
app-owned publication *might* be acceptable as "config", might not).
|
|
69
|
+
- **Reintroduce a trigger-based emitter.** `ddl.py` existed once and was
|
|
70
|
+
deleted on owner instruction (2026-07-03). Persistent catalog objects
|
|
71
|
+
violate law 3; fan-out goes through the bridge daemon.
|
|
72
|
+
- **Reintroduce a NOTIFY consumer.** There was a second transport,
|
|
73
|
+
`ChangeFeed` (NOTIFY consumer on asyncpg); the owner had it removed
|
|
74
|
+
2026-07-03. pgnudge is WalFeed-only. Fan-out is a bridge daemon
|
|
75
|
+
republishing via `pg_notify`; consumers LISTEN with whatever driver they
|
|
76
|
+
already have.
|
|
77
|
+
|
|
78
|
+
## Project Structure
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
src/pgnudge/
|
|
82
|
+
proto.py WalsenderConnection: the protocol client (~250 lines, stdlib
|
|
83
|
+
asyncio + scramp only). Startup with replication=database,
|
|
84
|
+
optional TLS via StreamWriter.start_tls, auth = trust /
|
|
85
|
+
cleartext (refused unless the connection is TLS) /
|
|
86
|
+
SCRAM-SHA-256 (-PLUS mechanisms filtered out; no channel
|
|
87
|
+
binding), simple_query (walsender mode speaks ONLY the
|
|
88
|
+
simple query subprotocol), start_replication -> CopyBoth,
|
|
89
|
+
read_stream() -> XLogData | Keepalive (frozen slots
|
|
90
|
+
dataclasses), send_standby_status(lsn, reply=...), abort() =
|
|
91
|
+
deliberate hard close.
|
|
92
|
+
core.py The contract only: Event/Batch/Resync dataclasses + FeedItem.
|
|
93
|
+
Event is payload/first_seen/count — `channel` and
|
|
94
|
+
`payload_filter` were dropped 2026-07-04 (breaking, owner
|
|
95
|
+
call); do not reintroduce.
|
|
96
|
+
engine.py The machinery, one dataclass per concern, pure stdlib:
|
|
97
|
+
Wakeup (one raw arrival, pre-coalescing),
|
|
98
|
+
Intake (bounded wakeup buffer, overflow flag),
|
|
99
|
+
Coalescer (dedup buffer, count per payload),
|
|
100
|
+
Debouncer (rolling window, hard max_batch_wait cap; overflow
|
|
101
|
+
-> Resync("overflow")), Backoff (jittered exponential),
|
|
102
|
+
FeedService (wires intake -> debouncer -> out queue, owns
|
|
103
|
+
tasks/failsafe/shutdown), BaseFeed (thin async-iterator
|
|
104
|
+
facade; subclass hooks _supervisor/_extra_close). Unit-tested
|
|
105
|
+
without PostgreSQL in tests/test_engine.py.
|
|
106
|
+
wal.py WalFeed(BaseFeed): the transport. Supervisor creates a fresh
|
|
107
|
+
TEMPORARY slot per (re)connect (name: pgnudge_<pid>_<hex>),
|
|
108
|
+
SNAPSHOT 'nothing', starts replication at 0/0, emits Resync
|
|
109
|
+
only once the stream is live (this ordering is the gap-free
|
|
110
|
+
handshake argument — see README), parses wal2json
|
|
111
|
+
format-version 2 (default) or test_decoding (zero-install
|
|
112
|
+
fallback) — I/U/D/TRUNCATE nudge, logical messages ("M")
|
|
113
|
+
don't. Feedback task every status_interval (default 10s,
|
|
114
|
+
must stay under wal_sender_timeout, default 60s) plus
|
|
115
|
+
immediate reply on keepalive reply-requested; with
|
|
116
|
+
liveness_timeout set (default 30s) every status requests a
|
|
117
|
+
keepalive back, and inbound silence past the timeout aborts
|
|
118
|
+
the socket to force a reconnect (never wait_for on reads).
|
|
119
|
+
Lifecycle logging on the "pgnudge.wal" logger.
|
|
120
|
+
tests/
|
|
121
|
+
conftest.py session-scoped PostgreSQL via testcontainers (pgqueuer
|
|
122
|
+
pattern), scratch database per test, best-effort TLS enable
|
|
123
|
+
(self-signed cert) inside the container. Env knobs:
|
|
124
|
+
EXTERNAL_POSTGRES_DSN, POSTGRES_IMAGE, PGNUDGE_PLUGIN,
|
|
125
|
+
PGNUDGE_TLS.
|
|
126
|
+
test_engine.py unit tests for the engine classes (Intake, Coalescer,
|
|
127
|
+
Debouncer, Backoff, FeedService, BaseFeed via a FakeFeed) —
|
|
128
|
+
pure asyncio, no PostgreSQL.
|
|
129
|
+
wire.py shared wire-protocol helpers for the fake-walsender tests:
|
|
130
|
+
backend frame builders, frontend frame readers, a
|
|
131
|
+
scripted_server() harness on an ephemeral localhost port.
|
|
132
|
+
test_proto.py WalsenderConnection unit tests against scripted handlers —
|
|
133
|
+
no PostgreSQL. Auth edge cases (cleartext, missing password,
|
|
134
|
+
MD5 rejected, SASL -PLUS-only rejected), SSL-refused, silent-
|
|
135
|
+
server timeout, error/notice handling, read_stream frame
|
|
136
|
+
parsing, standby-status wire format, abort idempotence.
|
|
137
|
+
test_wal_unit.py WalFeed unit tests — no PostgreSQL. wal2json/
|
|
138
|
+
test_decoding payload parsers, plugin option assembly +
|
|
139
|
+
SQL quoting, feedback loop, connect/slot-failure retry, and
|
|
140
|
+
a full lifecycle against a scripted FakeWalsender (connect ->
|
|
141
|
+
Resync -> Batch -> keepalive ack -> stream end -> reconnect),
|
|
142
|
+
incl. asserting TEMPORARY + SNAPSHOT 'nothing' on the wire.
|
|
143
|
+
test_wal.py 7 tests incl. the two proofs: no backfill of pre-connect
|
|
144
|
+
writes, and hard abort -> pg_replication_slots EMPTY.
|
|
145
|
+
docs/
|
|
146
|
+
temporary-slots.md public reference (added 2026-07-04): logical decoding
|
|
147
|
+
+ temporary-slot mechanics in PostgreSQL's own terms, the
|
|
148
|
+
gap-free handshake, polling vs pgnudge, when NOT to use
|
|
149
|
+
pgnudge, operational limits. Reference material only — not
|
|
150
|
+
a design-history document.
|
|
151
|
+
examples/
|
|
152
|
+
minimal.py smallest end-to-end consumer: WalFeed + match on Resync/Batch.
|
|
153
|
+
.github/workflows/
|
|
154
|
+
ci.yml lint (ruff + mypy), test matrix Python 3.11-3.14 × PG 16-18
|
|
155
|
+
(test_decoding), one wal2json job on a CI-built image.
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
The consumer contract, coalescing semantics, and ops notes are in
|
|
159
|
+
`README.md` — treat it as the spec. The full decision history (why not
|
|
160
|
+
sentinel polling, why not triggers, why not Go/Rust/psycopg2, the Zig
|
|
161
|
+
track) lived in `docs/design-history.md` and `docs/research-notifications.md`,
|
|
162
|
+
removed from the repo 2026-07-03 (owner wanted no origin-story/internal
|
|
163
|
+
context public). `docs/temporary-slots.md` (2026-07-04) is NOT that history
|
|
164
|
+
returning — it is a public reference on the temp-slot mechanism itself.
|
|
165
|
+
Settled decisions are summarized in the five laws above; ask the owner
|
|
166
|
+
before re-litigating any of them.
|
|
167
|
+
|
|
168
|
+
## Build, Lint, and Test Commands
|
|
169
|
+
|
|
170
|
+
All commands use `uv` as the package manager.
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
uv sync # .venv + uv.lock, editable + dev group
|
|
174
|
+
|
|
175
|
+
# Run all tests (spins postgres:17 via testcontainers, runs the proofs)
|
|
176
|
+
uv run pytest
|
|
177
|
+
|
|
178
|
+
# Run a single test file
|
|
179
|
+
uv run pytest tests/test_engine.py
|
|
180
|
+
|
|
181
|
+
# Run a single test function
|
|
182
|
+
uv run pytest tests/test_wal.py::test_hard_abort_leaves_no_slots
|
|
183
|
+
|
|
184
|
+
# Coverage (100% line+branch as of 2026-07-04; keep it there.
|
|
185
|
+
# CI gate is 98 — headroom for best-effort TLS lines)
|
|
186
|
+
uv run pytest --cov=pgnudge --cov-report=term-missing
|
|
187
|
+
|
|
188
|
+
# Lint + typecheck (both must stay clean)
|
|
189
|
+
uv run ruff check . && uv run mypy
|
|
190
|
+
|
|
191
|
+
uv build
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
### Database for Tests
|
|
195
|
+
|
|
196
|
+
The test suite owns its own PostgreSQL via testcontainers — nothing to
|
|
197
|
+
install beyond Docker. Against an external server instead:
|
|
198
|
+
`EXTERNAL_POSTGRES_DSN=postgresql://... uv run pytest` (role needs
|
|
199
|
+
CREATEDB + REPLICATION; add `PGNUDGE_TLS=1` if it has TLS). Default plugin
|
|
200
|
+
is `test_decoding` (zero-install); `PGNUDGE_PLUGIN=wal2json` needs an image
|
|
201
|
+
that ships it — **no public image does for PG 16** (debezium/postgres
|
|
202
|
+
dropped wal2json; proven missing 2026-07-04, error 58P01). Build one:
|
|
203
|
+
`postgres:16` + apt `postgresql-16-wal2json` from the PGDG repo the
|
|
204
|
+
official image already has configured — see the docker build step in
|
|
205
|
+
`ci.yml`, then `POSTGRES_IMAGE=pgnudge-wal2json:16`.
|
|
206
|
+
|
|
207
|
+
Debugging trick from the PG docs: a replication connection can be tested
|
|
208
|
+
with nothing but psql —
|
|
209
|
+
`psql "dbname=x replication=database user=y" -c "IDENTIFY_SYSTEM;"`
|
|
210
|
+
|
|
211
|
+
## Code Style
|
|
212
|
+
|
|
213
|
+
### Formatting and Linting
|
|
214
|
+
|
|
215
|
+
- **Formatter/linter**: ruff (line-length=110, `src = ["src"]`)
|
|
216
|
+
- **Mypy**: `strict` + `disallow_any_explicit` + `warn_unreachable`, extra
|
|
217
|
+
error codes `redundant-expr`, `possibly-undefined`, `truthy-bool`,
|
|
218
|
+
`ignore-without-code`; covers `src` and `tests`.
|
|
219
|
+
|
|
220
|
+
### Imports
|
|
221
|
+
|
|
222
|
+
1. Standard library
|
|
223
|
+
2. Third-party packages (scramp; asyncpg/pytest/testcontainers in tests)
|
|
224
|
+
3. Internal imports using absolute paths (`from pgnudge.core import Batch`)
|
|
225
|
+
|
|
226
|
+
- **No `from __future__ import annotations`.** pgnudge targets Python >= 3.11
|
|
227
|
+
(lowered from 3.13 on 2026-07-04 after external review — 3.13 was too
|
|
228
|
+
narrow for production fleets); everything used is runtime-valid on 3.11
|
|
229
|
+
(`Self`, PEP 604 unions, `collections.abc` imports). **No PEP 695** `type`
|
|
230
|
+
aliases or generic syntax — those need 3.12+; use `typing.TypeAlias` and
|
|
231
|
+
`typing.TypeVar` instead.
|
|
232
|
+
- **No local imports.** All imports at module top level; the only exception
|
|
233
|
+
is `if TYPE_CHECKING:` blocks for breaking circular imports at runtime.
|
|
234
|
+
|
|
235
|
+
### Type Annotations
|
|
236
|
+
|
|
237
|
+
- **Always annotate** all function/method signatures, tests included.
|
|
238
|
+
- Use native types: `list[str]`, `int | None` (never `Optional`),
|
|
239
|
+
`TypeAlias` aliases (`FeedItem: TypeAlias = Resync | Batch`), `ClassVar`
|
|
240
|
+
for class constants, `Self` for fluent/classmethod returns.
|
|
241
|
+
- **No `Any`, anywhere.** `disallow_any_explicit` is on and there are no
|
|
242
|
+
exceptions — not even at protocol boundaries. Use proper types, generics,
|
|
243
|
+
protocols, or `object` instead. (Stricter than pgqueuer's
|
|
244
|
+
driver-boundary carve-out; the stricter rule wins.)
|
|
245
|
+
- **No `# type: ignore` in production code** (`src/pgnudge/`). Fix the
|
|
246
|
+
underlying type issue instead. In tests it is acceptable only with an
|
|
247
|
+
error code (`ignore-without-code` is enforced).
|
|
248
|
+
- **Generic constructors over type-annotated assignments**:
|
|
249
|
+
`fut = asyncio.Future[MyType]()` not
|
|
250
|
+
`fut: asyncio.Future[MyType] = asyncio.Future()`.
|
|
251
|
+
|
|
252
|
+
### Naming Conventions
|
|
253
|
+
|
|
254
|
+
| Element | Convention | Example |
|
|
255
|
+
|---------------------|-------------------|-------------------------------------|
|
|
256
|
+
| Classes | CamelCase | `WalFeed`, `PgServerError` |
|
|
257
|
+
| Functions/methods | snake_case | `read_stream`, `send_standby_status`|
|
|
258
|
+
| Class constants | UPPER_SNAKE `ClassVar` | `_PG_EPOCH_UNIX` |
|
|
259
|
+
| Test functions | `test_` prefix | `test_hard_abort_leaves_no_slots` |
|
|
260
|
+
| Fixtures | snake_case | `pg`, `admin`, `postgres` |
|
|
261
|
+
|
|
262
|
+
- **No module-level globals.** Constants live as class attributes
|
|
263
|
+
(`ClassVar`); test config lives inside functions. (Owner rule 2026-07-03;
|
|
264
|
+
stricter than pgqueuer's module-level constants — the stricter rule wins.)
|
|
265
|
+
- **No leading-underscore prefixes on new names.** Python has no real
|
|
266
|
+
public/private distinction; pick a descriptive name instead of hiding it
|
|
267
|
+
behind a prefix. (Adopted from pgqueuer 2026-07-04 as the stricter rule —
|
|
268
|
+
it supersedes the earlier pgnudge carve-out that kept single underscores
|
|
269
|
+
on plain-class instance attributes and helpers.) Existing `_`-prefixed
|
|
270
|
+
internals in `proto.py`/`engine.py`/`wal.py` predate this; renaming them
|
|
271
|
+
is a pending refactor — coordinate with the owner before a mass rename,
|
|
272
|
+
and never underscore-prefix anything new. Module and class names and
|
|
273
|
+
dataclass fields were already banned from underscore prefixes
|
|
274
|
+
(internal dataclass state uses `field(init=False)` with a plain name).
|
|
275
|
+
Python-mandated protocol dunders (`__init__.py`, `__aenter__`,
|
|
276
|
+
`__version__`, …) are exempt.
|
|
277
|
+
|
|
278
|
+
### Error Handling
|
|
279
|
+
|
|
280
|
+
- The protocol layer raises `PgServerError` (ErrorResponse field map
|
|
281
|
+
preserved) and stdlib connection errors; the supervisor catches, aborts,
|
|
282
|
+
and reconnects with a fresh slot — errors are a normal part of the
|
|
283
|
+
lifecycle, not exceptional control flow to hide.
|
|
284
|
+
- Use `contextlib.suppress(...)` for non-critical teardown errors.
|
|
285
|
+
- Use `pytest.raises` in tests for expected exceptions.
|
|
286
|
+
|
|
287
|
+
### Docstrings
|
|
288
|
+
|
|
289
|
+
Follow [PEP 257](https://peps.python.org/pep-0257/). Keep them tight;
|
|
290
|
+
design prose lives in README/docs, not code.
|
|
291
|
+
|
|
292
|
+
- **One-liner**: triple-quoted on the same line, period at the end.
|
|
293
|
+
- **Multi-line**: summary line, blank line, body. Closing `"""` on its own
|
|
294
|
+
line.
|
|
295
|
+
- **Don't restate the signature.** Only mention a parameter when something
|
|
296
|
+
non-obvious matters (units, constraints, ownership, side effects).
|
|
297
|
+
- **Skip docstrings on trivial helpers.** A descriptive name beats a
|
|
298
|
+
docstring that repeats it.
|
|
299
|
+
- **Test docstrings**: one line describing the behavior under test. Skip
|
|
300
|
+
when the test name is self-evident.
|
|
301
|
+
- **Module docstrings**: short; say what the module is, point to README for
|
|
302
|
+
semantics.
|
|
303
|
+
|
|
304
|
+
### Comments
|
|
305
|
+
|
|
306
|
+
Default to no comments. Code with descriptive names is more durable than
|
|
307
|
+
comments that rot.
|
|
308
|
+
|
|
309
|
+
Only add a comment when the **why** is non-obvious:
|
|
310
|
+
|
|
311
|
+
- A hidden invariant or constraint that the code relies on (e.g. "hard-close
|
|
312
|
+
on purpose, no DROP: crash and clean exit must exercise the same
|
|
313
|
+
server-side cleanup path")
|
|
314
|
+
- A workaround for a specific bug, library quirk, or platform behavior
|
|
315
|
+
- A subtle ordering or concurrency requirement
|
|
316
|
+
- Something that would surprise a reader who understood the code
|
|
317
|
+
|
|
318
|
+
**Do not write:** comments that restate the code, caller references, task
|
|
319
|
+
or issue refs in inline comments (regression tests may reference an issue
|
|
320
|
+
in their docstring), banner section headers beyond the existing light
|
|
321
|
+
`# -- section --` dividers, or TODOs without an owner and a tracking link.
|
|
322
|
+
|
|
323
|
+
### Guiding Principles
|
|
324
|
+
|
|
325
|
+
- **Follow existing patterns.** Before writing new code, read surrounding
|
|
326
|
+
modules to match conventions. Do not invent new patterns when an
|
|
327
|
+
established one exists.
|
|
328
|
+
- **Readability and correctness above speed.** Never sacrifice clarity or
|
|
329
|
+
correctness for performance unless there is a measured, proven need.
|
|
330
|
+
- **Every change must be proven correct by a test.** Never accept a code
|
|
331
|
+
change without an accompanying test. Tests must be narrow and precise —
|
|
332
|
+
test exactly the behavior being changed. Coverage is 100% line+branch;
|
|
333
|
+
keep it there.
|
|
334
|
+
- **Every user-facing change must be documented.** `README.md` is the spec —
|
|
335
|
+
new knobs, contract changes, and behavior changes update it in the same
|
|
336
|
+
change. `docs/temporary-slots.md` covers the mechanism; extend it when
|
|
337
|
+
the machinery changes.
|
|
338
|
+
|
|
339
|
+
## Testing Conventions
|
|
340
|
+
|
|
341
|
+
- **pytest config**: `asyncio_mode = "auto"`, function-scoped event loops.
|
|
342
|
+
- **2-second budget per test** (pytest-timeout, `timeout = 2` in
|
|
343
|
+
pyproject): a hung protocol read must fail fast, never wedge the suite.
|
|
344
|
+
Tests that legitimately need longer opt out with
|
|
345
|
+
`@pytest.mark.timeout(...)` — `test_wal.py` does so module-wide
|
|
346
|
+
(`pytestmark`) because the first live test pays for the container start.
|
|
347
|
+
- Declare async tests as `async def test_...` — no `@pytest.mark.asyncio`
|
|
348
|
+
decorator needed.
|
|
349
|
+
- Annotate all test function parameters and returns
|
|
350
|
+
(`async def test_foo(pg: PgParams) -> None:`).
|
|
351
|
+
- Fixtures create a **fresh scratch database per test**; the PostgreSQL
|
|
352
|
+
container is session-scoped.
|
|
353
|
+
- Key fixtures in `tests/conftest.py`: `postgres` (session server),
|
|
354
|
+
`pg` (per-test `PgParams`), `admin` (asyncpg connection into the scratch
|
|
355
|
+
DB).
|
|
356
|
+
- Engine, proto, and WalFeed-unit tests need no PostgreSQL at all — the
|
|
357
|
+
wire-level tests run against `tests/wire.py`'s scripted fake walsender.
|
|
358
|
+
Prefer that harness for new protocol behavior; reserve `test_wal.py` for
|
|
359
|
+
properties only a real server can prove.
|
|
360
|
+
|
|
361
|
+
## Versioning
|
|
362
|
+
|
|
363
|
+
pgnudge follows **strict semantic versioning** (SemVer) from v1.0.0 onward:
|
|
364
|
+
|
|
365
|
+
- **Patch** (1.0.x): bug fixes only, no API changes.
|
|
366
|
+
- **Minor** (1.x.0): new features, fully backward-compatible.
|
|
367
|
+
- **Major** (x.0.0): breaking changes — reserved for when there is no
|
|
368
|
+
alternative.
|
|
369
|
+
|
|
370
|
+
The stable API is the feed contract (`Resync | Batch` iteration) and the
|
|
371
|
+
payload contract (`schema.table`) — transports are pluggable behind them
|
|
372
|
+
(this keeps the someday Zig track possible). Never break either in a patch
|
|
373
|
+
or minor release.
|
|
374
|
+
|
|
375
|
+
## Commit Conventions
|
|
376
|
+
|
|
377
|
+
This project follows
|
|
378
|
+
[Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/).
|
|
379
|
+
|
|
380
|
+
```
|
|
381
|
+
<type>[optional scope]: <description>
|
|
382
|
+
```
|
|
383
|
+
|
|
384
|
+
- **type**: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`,
|
|
385
|
+
`build`, `ci`, `chore`
|
|
386
|
+
- **scope** (optional): area of the codebase, e.g. `proto`, `engine`,
|
|
387
|
+
`wal`, `core`
|
|
388
|
+
- **description**: imperative mood, lowercase, no period at the end
|
|
389
|
+
- **`!` after type/scope** marks a breaking change (e.g.
|
|
390
|
+
`refactor!: drop Event.channel ...`)
|
|
391
|
+
- Body: present tense, wrap at 72 characters. Explain **why**, not just
|
|
392
|
+
what. Footer `BREAKING CHANGE: <description>` for breaking changes.
|
|
393
|
+
- No PR-number suffix — pgnudge has no PR-based flow today; if one appears,
|
|
394
|
+
adopt pgqueuer's `(#{PR_ID})` suffix.
|
|
395
|
+
|
|
396
|
+
## CI Matrix
|
|
397
|
+
|
|
398
|
+
Python 3.11–3.14 × PostgreSQL 16–18 on Ubuntu (test_decoding), plus one
|
|
399
|
+
wal2json job on a CI-built `postgres:16` + PGDG `postgresql-16-wal2json`
|
|
400
|
+
image (the debezium/postgres:16 image does NOT ship wal2json — first CI
|
|
401
|
+
run failed on it 2026-07-04, fixed same day). Lint job runs ruff + mypy.
|
|
402
|
+
Coverage gate `--cov-fail-under=98`.
|
|
403
|
+
|
|
404
|
+
## Proven vs. untested (be honest in docs and commits)
|
|
405
|
+
|
|
406
|
+
Proven live (PG 16 with wal2json, PG 17 with test_decoding via the pytest
|
|
407
|
+
suite): SCRAM-SHA-256 auth; TLS handshake + SCRAM over TLS (self-signed
|
|
408
|
+
cert, CERT_NONE context); temporary slot is the only server object while
|
|
409
|
+
connected; no backfill; client-side coalescing (50-row txn -> one Event,
|
|
410
|
+
count=50); kill -> fresh slot, old auto-dropped; hard abort ->
|
|
411
|
+
`pg_replication_slots` empty; wheel installs and passes from site-packages
|
|
412
|
+
(pre-1.0 layout — re-verify from wheel before release).
|
|
413
|
+
|
|
414
|
+
Untested / absent: `ssl=True` verify-full against a real managed endpoint
|
|
415
|
+
(Azure Flexible Server is the owner's target — replication bypasses the
|
|
416
|
+
built-in PgBouncer there, use direct 5432); MD5 auth (deliberately absent);
|
|
417
|
+
pgoutput (see the publications warning above); behavior under a
|
|
418
|
+
long-running write transaction at connect (slot creation waits — connect
|
|
419
|
+
latency, never history); PG 18 (in the CI matrix, not yet run); the CI
|
|
420
|
+
workflow itself.
|
|
421
|
+
|
|
422
|
+
## Backlog, priority order
|
|
423
|
+
|
|
424
|
+
0. **Rename — DONE** (`pgnudge` → `pgnudge`, 2026-07-03, owner's choice).
|
|
425
|
+
Runners-up preserved in case of a future collision, PyPI-verified free
|
|
426
|
+
at the time: `waltail`, `pgripple`, `pgnudge`, `pgwisp`, `pgghost`,
|
|
427
|
+
`pgfollow`, `pgblip`, `pgwhisper`, `pgpulse`, `pgbeacon`, `pgfeed`
|
|
428
|
+
(`pgtail`, `pgflux`, `pglive` were taken). The PyPI name `pgnudge` is
|
|
429
|
+
**not yet registered** — claiming it with the first release is the real
|
|
430
|
+
Task 0 remainder.
|
|
431
|
+
1. ~~pytest-ify~~ — DONE 2026-07-03: testcontainers session fixture,
|
|
432
|
+
scratch DB per test, EXTERNAL_POSTGRES_DSN escape hatch.
|
|
433
|
+
2. ~~CI~~ — DONE 2026-07-03: `.github/workflows/ci.yml`, Python 3.13/3.14 ×
|
|
434
|
+
PG 16–18 (test_decoding; floor is PG 16+, owner call 2026-07-03) + one
|
|
435
|
+
wal2json job on a CI-built image.
|
|
436
|
+
Repo pushed to github.com/janbjorge/pgnudge 2026-07-03 — check Actions
|
|
437
|
+
for the first real runs.
|
|
438
|
+
3. **Bridge daemon** as a first-class example or subpackage: one WalFeed ->
|
|
439
|
+
`pg_notify` on a channel -> plain LISTEN consumers. Zero persistent
|
|
440
|
+
objects end to end (the bridge's slot dies with the bridge).
|
|
441
|
+
4. **Opt-in payload v2** (`schema.table:pk`) — wal2json carries pk columns;
|
|
442
|
+
keep v1 the default, document the coalescing trade (dedup granularity
|
|
443
|
+
changes).
|
|
444
|
+
5. Azure end-to-end validation (verify-full TLS, direct 5432, wal2json
|
|
445
|
+
preinstalled) — coordinate with owner, needs real infra.
|
|
446
|
+
6. ~~Typing/lint hardening~~ — DONE 2026-07-03: mypy --strict and ruff
|
|
447
|
+
configured in pyproject, both clean. Keep them clean. Docs polish remains.
|
|
448
|
+
|
|
449
|
+
## Release checklist
|
|
450
|
+
|
|
451
|
+
Bump `__version__` and `pyproject.toml` together; `uv build`; run the full
|
|
452
|
+
suite from the installed wheel (not the source tree) against a live server;
|
|
453
|
+
tag; `uv publish`. Never release with the live suite skipped — the
|
|
454
|
+
zero-footprint proof (`test_hard_abort_leaves_no_slots`) is the product.
|
|
455
|
+
|
|
456
|
+
## Owner context
|
|
457
|
+
|
|
458
|
+
Python/asyncio shop; the owner maintains pgqueuer (job queue on
|
|
459
|
+
LISTEN/NOTIFY) — this library is its broadcast-shaped sibling: *pgqueuer
|
|
460
|
+
moves work, this moves wakefulness*. Target deployment: Azure Database for
|
|
461
|
+
PostgreSQL Flexible Server, PG 16+, schema `versaai`, read-only consumers.
|
|
462
|
+
A native Zig implementation of the walsender core is a someday-ambition
|
|
463
|
+
("the Zig track") — design decisions here should not foreclose it: the
|
|
464
|
+
feed contract and payload contract are the stable API, transports are
|
|
465
|
+
pluggable.
|
pgnudge-1.0.0/CLAUDE.md
ADDED
pgnudge-1.0.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 pgnudge contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|