postgresql-testing 0.0.1__tar.gz → 0.0.3__tar.gz

{postgresql_testing-0.0.1 → postgresql_testing-0.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: postgresql-testing
-Version: 0.0.1
+Version: 0.0.3
 Summary: PostgreSQL testing helpers
 Author: Oli Russell
 Author-email: Oli Russell <mrxoliver@gmail.com>
@@ -12,7 +12,7 @@ Description-Content-Type: text/markdown
 
 # Postgresql testing
 
-Simple Postgres helpers for testing with Python - no docker, brew, apt, etc - uses [postgresql-binaries](https://github.com/leontrolski/postgresql-binaries).
+Simple Postgres helpers for testing with Python - no docker, brew, apt, etc - uses [postgresql-binaries](https://github.com/leontrolski/postgresql-binaries). The interface is simple, but close enough to the metal that you can use it to eg. have a new database per testing thread/use a fresh database from a `TEMPLATE` or archive.
 
 ```shell
 pip install postgresql-testing 'postgresql-binaries==18.*'
@@ -37,4 +37,20 @@ There are various useful flags and things - the source code is short enough to j
 
 <hr>
 
-There are a couple of helpers for creating/using template dbs and tar files. I have some vague long term plan for some kind of "Docker layers for migrated databases" with clever caching, but I'm not quite sure what it looks like yet.
+There are a couple of helpers for creating/using template dbs and archives. I have some vague long term plan for some kind of "Docker layers for migrated databases" with clever caching (or not), but I'm not quite sure what it looks like yet.
+
+Some benchmarking:
+
+| number of tables | create from template | dump to archive | create from archive |
+|---|---|---|---|
+| 100 | 80ms | 80ms (0.2MB) | 150ms |
+| 1000 | 500ms | 300ms (2MB) | 1100ms |
+
+On macos using a ramdisk, it is slightly quicker:
+
+| number of tables | create from template | dump to archive | create from archive |
+|---|---|---|---|
+| 100 | 70ms | 70ms | 120ms |
+| 1000 | 350ms | 300ms | 1000ms |
+
+See claims/advice from planet [Go](https://github.com/peterldowns/pgtestdb).

postgresql_testing-0.0.3/README.md

@@ -0,0 +1,44 @@
+# Postgresql testing
+
+Simple Postgres helpers for testing with Python - no docker, brew, apt, etc - uses [postgresql-binaries](https://github.com/leontrolski/postgresql-binaries). The interface is simple, but close enough to the metal that you can use it to eg. have a new database per testing thread/use a fresh database from a `TEMPLATE` or archive.
+
+```shell
+pip install postgresql-testing 'postgresql-binaries==18.*'
+```
+
+Then to use, eg:
+
+```python
+import postgresql_testing
+
+@pytest.fixture(scope="session")
+def db() -> Iterator[str]:
+    config = postgresql_testing.DatabaseConfig.default("testing-db")
+    postgresql_testing.initdb(config.directory, on_existing="use")
+    with postgresql_testing.serve(config):
+        postgresql_testing.ensure_user(config)
+        postgresql_testing.create_database(config, on_existing="replace")
+        yield config.dsn
+```
+
+There are various useful flags and things - the source code is short enough to just dive in.
+
+<hr>
+
+There are a couple of helpers for creating/using template dbs and archives. I have some vague long term plan for some kind of "Docker layers for migrated databases" with clever caching (or not), but I'm not quite sure what it looks like yet.
+
+Some benchmarking:
+
+| number of tables | create from template | dump to archive | create from archive |
+|---|---|---|---|
+| 100 | 80ms | 80ms (0.2MB) | 150ms |
+| 1000 | 500ms | 300ms (2MB) | 1100ms |
+
+On macos using a ramdisk, it is slightly quicker:
+
+| number of tables | create from template | dump to archive | create from archive |
+|---|---|---|---|
+| 100 | 70ms | 70ms | 120ms |
+| 1000 | 350ms | 300ms | 1000ms |
+
+See claims/advice from planet [Go](https://github.com/peterldowns/pgtestdb).

{postgresql_testing-0.0.1 → postgresql_testing-0.0.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "postgresql-testing"
-version = "0.0.1"
+version = "0.0.3"
 description = "PostgreSQL testing helpers"
 readme = "README.md"
 authors = [

{postgresql_testing-0.0.1 → postgresql_testing-0.0.3}/src/postgresql_testing/__init__.py

@@ -8,9 +8,8 @@ import subprocess
 from typing import Iterator, Literal, Self
 import postgresql_binaries
 import psycopg
-import tarfile
 
-DEFAULT_DIR = Path("/tmp/.postgresql_testing")
+DEFAULT_DIR = Path("/tmp/postgresql-testing")
 SUPERUSER = "postgres"
 ROOT_DATABASE = "postgres"
 
@@ -72,7 +71,6 @@ def initdb(
     *,
     on_existing: Literal["raise", "use", "replace"] = "raise",
 ) -> None:
-    """Call `initdb` with some sensible arguments for testing."""
     if directory.exists():
         if on_existing == "raise":
             raise RuntimeError(f"Directory {directory} already exists")
@@ -100,7 +98,6 @@ def initdb(
 
 @contextmanager
 def serve(c: ClusterConfig) -> Iterator[None]:
-    """Call `postgres` with some sensible arguments."""
     c.stderr.parent.mkdir(parents=True, exist_ok=True)
     with open(c.stderr, "wb") as stderr_f:
         cmd = [
@@ -119,30 +116,16 @@ def serve(c: ClusterConfig) -> Iterator[None]:
             _try_connect(c.superuser().dsn)
             yield
         finally:
+            # Stop any ongoing queries
+            with psycopg.connect(c.superuser().dsn) as conn, conn.cursor() as cur:
+                cur.execute("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid <> pg_backend_pid()")
+            # Shutdown
             process.terminate()
-            process.wait()
-
-
-@contextmanager
-def create_template(
-    c: DatabaseConfig,
-    *,
-    template: str,
-    on_existing: Literal["raise", "replace"] = "replace",
-) -> Iterator[DatabaseConfig]:
-    with psycopg.connect(c.superuser().dsn) as conn, conn.cursor() as cur:
-        conn.autocommit = True
-        template_exists = bool(cur.execute(f"SELECT 1 FROM pg_database WHERE datname = '{template}'").fetchall())
-        if template_exists:
-            if on_existing == "raise":
-                raise RuntimeError(f"Template database {template} already exists")
-            if on_existing == "replace":
-                cur.execute(f"UPDATE pg_database SET datistemplate = false WHERE datname='{template}'")
-                cur.execute(f'DROP DATABASE "{template}"')
-
-        cur.execute(f'CREATE DATABASE "{template}" OWNER "{c.user}"')
-        yield replace(c, database=template)
-        cur.execute(f"UPDATE pg_database SET datistemplate = true WHERE datname='{template}'")
+            try:
+                process.wait(timeout=1)
+            except subprocess.TimeoutExpired:
+                process.kill()
+                process.wait(timeout=5)
 
 
 def ensure_user(c: DatabaseConfig) -> None:
@@ -159,11 +142,6 @@ def create_database(
     template: str | None = None,
     on_existing: Literal["raise", "replace"] = "replace",
 ) -> None:
-    """Create a Postgres database.
-
-    - Optionally pass in a template.
-    - Takes in the tens of ms for a small db.
-    """
     template_str = "" if template is None else f'WITH TEMPLATE "{template}"'
     sql_create_database = f'CREATE DATABASE "{c.database}" {template_str} OWNER "{c.user}" STRATEGY=FILE_COPY'
 
@@ -179,41 +157,71 @@ def create_database(
                 cur.execute(sql_create_database)
 
 
+# More experimental functions
+
+
+@contextmanager
+def create_template(
+    c: DatabaseConfig,
+    *,
+    template: str,
+    on_existing: Literal["raise", "replace"] = "replace",
+) -> Iterator[DatabaseConfig]:
+    with psycopg.connect(c.superuser().dsn) as conn, conn.cursor() as cur:
+        conn.autocommit = True
+        template_exists = bool(cur.execute(f"SELECT 1 FROM pg_database WHERE datname = '{template}'").fetchall())
+        if template_exists:
+            if on_existing == "raise":
+                raise RuntimeError(f"Template database {template} already exists")
+            if on_existing == "replace":
+                cur.execute(f"UPDATE pg_database SET datistemplate = false WHERE datname='{template}'")
+                cur.execute(f'DROP DATABASE "{template}"')
+
+        cur.execute(f'CREATE DATABASE "{template}" OWNER "{c.user}"')
+        yield replace(c, database=template)
+        cur.execute(f"UPDATE pg_database SET datistemplate = true WHERE datname='{template}'")
+
+
 def dump_archive(
-    directory: Path,
+    c: DatabaseConfig,
     archive: Path,
     *,
     on_existing: Literal["raise", "replace"] = "replace",
 ) -> None:
-    """Dump the entirety of a Postgres cluster to a tar archive.
-
-    - Archives are not very portable (between system architectures or Postgres
-      versions).
-    - Takes in the hundreds of ms for a small db.
-    """
     if archive.exists():
         if on_existing == "raise":
             raise RuntimeError(f"Archive {archive} already exists")
-        archive.unlink()
+        if on_existing == "replace":
+            archive.unlink()
 
-    with tarfile.open(archive, "w") as tar:
-        tar.add(directory, arcname=directory.name)
+    cmd: list[str] = [
+        str(postgresql_binaries.bin() / "pg_dump"),
+        c.dsn,
+        *("--file", str(archive)),
+        *("--format", "t"),
+        *("--compress", "0"),
+        "--no-sync",
+    ]
+    subprocess.check_call(cmd)
 
 
 def load_archive(
     archive: Path,
-    directory: Path,
+    c: DatabaseConfig,
     *,
     on_existing: Literal["raise", "replace"] = "replace",
 ) -> None:
-    """Dump the entirety of a Postgres cluster from a tar archive."""
-    if directory.exists():
-        if on_existing == "raise":
-            raise RuntimeError(f"Directory {directory} already exists")
-        shutil.rmtree(directory)
-
-    with tarfile.open(archive) as tar:
-        tar.extractall(directory.parent)
+    cmd: list[str] = [
+        str(postgresql_binaries.bin() / "pg_restore"),
+        *("--host", c.host),
+        *("--port", str(c.port)),
+        *("--username", c.user),
+        *(("--password", c.password) if c.password else ()),
+        *("--dbname", c.database),
+        *(("--clean",) if on_existing == "replace" else ()),
+        str(archive),
+    ]
+    subprocess.check_call(cmd)
 
 
 def _try_connect(dsn: str) -> None:

postgresql_testing-0.0.1/README.md

@@ -1,28 +0,0 @@
-# Postgresql testing
-
-Simple Postgres helpers for testing with Python - no docker, brew, apt, etc - uses [postgresql-binaries](https://github.com/leontrolski/postgresql-binaries).
-
-```shell
-pip install postgresql-testing 'postgresql-binaries==18.*'
-```
-
-Then to use, eg:
-
-```python
-import postgresql_testing
-
-@pytest.fixture(scope="session")
-def db() -> Iterator[str]:
-    config = postgresql_testing.DatabaseConfig.default("testing-db")
-    postgresql_testing.initdb(config.directory, on_existing="use")
-    with postgresql_testing.serve(config):
-        postgresql_testing.ensure_user(config)
-        postgresql_testing.create_database(config, on_existing="replace")
-        yield config.dsn
-```
-
-There are various useful flags and things - the source code is short enough to just dive in.
-
-<hr>
-
-There are a couple of helpers for creating/using template dbs and tar files. I have some vague long term plan for some kind of "Docker layers for migrated databases" with clever caching, but I'm not quite sure what it looks like yet.