postgresledger 0.0.1__tar.gz → 0.0.3__tar.gz

postgresledger-0.0.1/README.md → postgresledger-0.0.3/PKG-INFO

@@ -1,5 +1,42 @@
+Metadata-Version: 2.4
+Name: postgresledger
+Version: 0.0.3
+Summary: Versioned PostgreSQL migrations from plain SQL, without an ORM: generated from schema.sql, tracked per database, with drift detection
+Keywords: postgresql,postgres,migrations,database-migrations,schema,sql,drift,pgschema
+Author: Nebojša Mrkić
+Author-email: Nebojša Mrkić <mrkic.nebojsa@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: SQL
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Version Control
+Requires-Dist: python-dotenv>=1.2.4
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/nmrkic/postgresledger
+Project-URL: Repository, https://github.com/nmrkic/postgresledger
+Project-URL: Issues, https://github.com/nmrkic/postgresledger/issues
+Project-URL: Changelog, https://github.com/nmrkic/postgresledger/releases
+Description-Content-Type: text/markdown
+
 # postgresledger
 
+[![PyPI](https://img.shields.io/pypi/v/postgresledger)](https://pypi.org/project/postgresledger/)
+[![Python](https://img.shields.io/pypi/pyversions/postgresledger)](https://pypi.org/project/postgresledger/)
+[![Tests](https://github.com/nmrkic/postgresledger/actions/workflows/tests.yml/badge.svg)](https://github.com/nmrkic/postgresledger/actions/workflows/tests.yml)
+[![License: MIT](https://img.shields.io/pypi/l/postgresledger)](https://github.com/nmrkic/postgresledger/blob/main/LICENSE)
+
 Versioned migrations for PostgreSQL from plain SQL, without an ORM (tables,
 views, materialized views, functions, triggers), built on
 [pgschema](https://www.pgschema.com). The command is `pgl`.
@@ -245,6 +282,81 @@ environment; a variable set there wins over `.env`. `pgl --env-file .env.uat …
 (or `PGL_ENV_FILE=.env.uat`) picks another file for one run.
 Run `pgl` in the folder with `postgresledger.toml`, or point to it with
 `--config PATH`. `pgl <command> -h` describes each command.
+`baseline`, `new`, `capture` and `drift` print only what changed;
+`--verbose` (`-v`) prints pgschema's whole plan, SQL included. On every
+command that dumps a database (also `snapshot` and `compare`), `--verbose`
+shows pgschema's log as it connects: when a run hangs, it tells whether the
+connection or the dump is slow.
+
+## Several schemas
+
+By default `pgl` manages one schema, `public`, with its desired state in
+`schema.sql`. To manage several schemas of the same database, list them in
+`postgresledger.toml` instead of `schema` and `schema_file`:
+
+```toml
+[project]
+schemas = ["public", "billing"]
+schema_dir = "database/schema"     # public.sql, billing.sql (the default)
+```
+
+Each schema has its own file of desired state, and every command works on
+all of them: one `pgl new`, one `pgl apply`, one `pgl drift`. A migration
+covers every schema that changed, one part per schema in the order of
+`schemas`:
+
+```sql
+-- schema: public
+SET search_path TO public;
+...
+-- schema: billing
+CREATE SCHEMA IF NOT EXISTS billing;   -- only in the first migration that has billing
+SET search_path TO billing;
+...
+```
+
+pgschema writes a schema's own objects without the schema name and other
+schemas' objects with it (`REFERENCES public.customer (id)`), which is why
+each part sets `search_path`. Snapshots hold every schema too, and `compare`
+names the objects `<schema>.<name>`.
+
+List a schema after the schemas it refers to: a view in `billing` on
+`public.customer` needs `public` first. References in both directions
+between two schemas are not supported.
+
+A schema added to `schemas` that already exists in the databases with
+objects in it: `pgl capture <name>` on one of them takes it into the schema
+files and records the migration as captured there. The other databases that
+already have the schema cannot run that migration, and `pgl` has no command
+yet to record it there without running it.
+
+## Roles and grants
+
+Roles (database users and groups) belong to the whole Postgres server and
+usually differ between databases: other users on dev and prod, the roles a
+managed database adds. pgschema never creates them. If grants were part of
+the migrations, `pgl apply` would fail on a database without one of the
+roles, and grants that differ on purpose would show up in `drift` and
+`compare`.
+
+So `pgl init` writes a `.pgschemaignore` next to `postgresledger.toml` that
+leaves every grant and default privilege out of `schema.sql`, the migrations
+and the snapshots:
+
+```toml
+[privileges]
+patterns = ["*"]
+
+[default_privileges]
+patterns = ["*"]
+```
+
+The patterns are the names of the roles that receive the grants:
+`["*", "!app_reader"]` keeps only the grants to `app_reader`, `["cloudsql*"]`
+leaves out only the grants to `cloudsql*` roles. Delete the file to keep
+every grant in the migrations. pgschema reads it from the folder `pgl` runs in,
+on every command; more in the
+[pgschema docs](https://www.pgschema.com/cli/ignore).
 
 ## Developing postgresledger
 

postgresledger-0.0.1/PKG-INFO → postgresledger-0.0.3/README.md

@@ -1,24 +1,10 @@
-Metadata-Version: 2.4
-Name: postgresledger
-Version: 0.0.1
-Summary: Versioned PostgreSQL migrations from plain SQL, without an ORM: generated from schema.sql, tracked per database, with drift detection
-Author: Nebojša Mrkić
-Author-email: Nebojša Mrkić <mrkic.nebojsa@gmail.com>
-License-Expression: MIT
-License-File: LICENSE
-Classifier: Programming Language :: Python :: 3
-Classifier: Topic :: Database
-Classifier: Environment :: Console
-Requires-Dist: python-dotenv>=1.2.4
-Requires-Python: >=3.11
-Project-URL: Homepage, https://github.com/nmrkic/postgresledger
-Project-URL: Repository, https://github.com/nmrkic/postgresledger
-Project-URL: Issues, https://github.com/nmrkic/postgresledger/issues
-Project-URL: Changelog, https://github.com/nmrkic/postgresledger/releases
-Description-Content-Type: text/markdown
-
 # postgresledger
 
+[![PyPI](https://img.shields.io/pypi/v/postgresledger)](https://pypi.org/project/postgresledger/)
+[![Python](https://img.shields.io/pypi/pyversions/postgresledger)](https://pypi.org/project/postgresledger/)
+[![Tests](https://github.com/nmrkic/postgresledger/actions/workflows/tests.yml/badge.svg)](https://github.com/nmrkic/postgresledger/actions/workflows/tests.yml)
+[![License: MIT](https://img.shields.io/pypi/l/postgresledger)](https://github.com/nmrkic/postgresledger/blob/main/LICENSE)
+
 Versioned migrations for PostgreSQL from plain SQL, without an ORM (tables,
 views, materialized views, functions, triggers), built on
 [pgschema](https://www.pgschema.com). The command is `pgl`.
@@ -264,6 +250,81 @@ environment; a variable set there wins over `.env`. `pgl --env-file .env.uat …
 (or `PGL_ENV_FILE=.env.uat`) picks another file for one run.
 Run `pgl` in the folder with `postgresledger.toml`, or point to it with
 `--config PATH`. `pgl <command> -h` describes each command.
+`baseline`, `new`, `capture` and `drift` print only what changed;
+`--verbose` (`-v`) prints pgschema's whole plan, SQL included. On every
+command that dumps a database (also `snapshot` and `compare`), `--verbose`
+shows pgschema's log as it connects: when a run hangs, it tells whether the
+connection or the dump is slow.
+
+## Several schemas
+
+By default `pgl` manages one schema, `public`, with its desired state in
+`schema.sql`. To manage several schemas of the same database, list them in
+`postgresledger.toml` instead of `schema` and `schema_file`:
+
+```toml
+[project]
+schemas = ["public", "billing"]
+schema_dir = "database/schema"     # public.sql, billing.sql (the default)
+```
+
+Each schema has its own file of desired state, and every command works on
+all of them: one `pgl new`, one `pgl apply`, one `pgl drift`. A migration
+covers every schema that changed, one part per schema in the order of
+`schemas`:
+
+```sql
+-- schema: public
+SET search_path TO public;
+...
+-- schema: billing
+CREATE SCHEMA IF NOT EXISTS billing;   -- only in the first migration that has billing
+SET search_path TO billing;
+...
+```
+
+pgschema writes a schema's own objects without the schema name and other
+schemas' objects with it (`REFERENCES public.customer (id)`), which is why
+each part sets `search_path`. Snapshots hold every schema too, and `compare`
+names the objects `<schema>.<name>`.
+
+List a schema after the schemas it refers to: a view in `billing` on
+`public.customer` needs `public` first. References in both directions
+between two schemas are not supported.
+
+A schema added to `schemas` that already exists in the databases with
+objects in it: `pgl capture <name>` on one of them takes it into the schema
+files and records the migration as captured there. The other databases that
+already have the schema cannot run that migration, and `pgl` has no command
+yet to record it there without running it.
+
+## Roles and grants
+
+Roles (database users and groups) belong to the whole Postgres server and
+usually differ between databases: other users on dev and prod, the roles a
+managed database adds. pgschema never creates them. If grants were part of
+the migrations, `pgl apply` would fail on a database without one of the
+roles, and grants that differ on purpose would show up in `drift` and
+`compare`.
+
+So `pgl init` writes a `.pgschemaignore` next to `postgresledger.toml` that
+leaves every grant and default privilege out of `schema.sql`, the migrations
+and the snapshots:
+
+```toml
+[privileges]
+patterns = ["*"]
+
+[default_privileges]
+patterns = ["*"]
+```
+
+The patterns are the names of the roles that receive the grants:
+`["*", "!app_reader"]` keeps only the grants to `app_reader`, `["cloudsql*"]`
+leaves out only the grants to `cloudsql*` roles. Delete the file to keep
+every grant in the migrations. pgschema reads it from the folder `pgl` runs in,
+on every command; more in the
+[pgschema docs](https://www.pgschema.com/cli/ignore).
 
 ## Developing postgresledger
 

{postgresledger-0.0.1 → postgresledger-0.0.3}/pyproject.toml

@@ -1,16 +1,38 @@
 [project]
 name = "postgresledger"
-version = "0.0.1"
+version = "0.0.3"
 description = "Versioned PostgreSQL migrations from plain SQL, without an ORM: generated from schema.sql, tracked per database, with drift detection"
 readme = "README.md"
 license = "MIT"
 license-files = ["LICENSE"]
 requires-python = ">=3.11"
 dependencies = ["python-dotenv>=1.2.4"]
+keywords = [
+    "postgresql",
+    "postgres",
+    "migrations",
+    "database-migrations",
+    "schema",
+    "sql",
+    "drift",
+    "pgschema",
+]
 classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Intended Audience :: System Administrators",
+    "Operating System :: POSIX :: Linux",
+    "Operating System :: MacOS",
     "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Programming Language :: SQL",
     "Topic :: Database",
-    "Environment :: Console",
+    "Topic :: Software Development :: Version Control",
 ]
 
 [[project.authors]]

{postgresledger-0.0.1 → postgresledger-0.0.3}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "postgresledger"
-version = "0.0.1"  # set from the release tag by .github/workflows/publish.yml, not here
+version = "0.0.3"  # set from the release tag by .github/workflows/publish.yml, not here
 description = "Versioned PostgreSQL migrations from plain SQL, without an ORM: generated from schema.sql, tracked per database, with drift detection"
 readme = "README.md"
 license = "MIT"
@@ -10,10 +10,23 @@ authors = [{ name = "Nebojša Mrkić", email = "mrkic.nebojsa@gmail.com" }]
 dependencies = [
     "python-dotenv>=1.2.4",
 ]
+keywords = ["postgresql", "postgres", "migrations", "database-migrations", "schema", "sql", "drift", "pgschema"]
 classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Intended Audience :: System Administrators",
+    "Operating System :: POSIX :: Linux",
+    "Operating System :: MacOS",
     "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Programming Language :: SQL",
     "Topic :: Database",
-    "Environment :: Console",
+    "Topic :: Software Development :: Version Control",
 ]
 
 [project.urls]

{postgresledger-0.0.1 → postgresledger-0.0.3}/src/postgresledger/cli.py

@@ -17,6 +17,9 @@ NO_CONFIG = {"init", "setup"}
 # commands that read the project but never touch a database
 # (compare with two snapshots; compare with one checks DATABASE_NAME itself)
 NO_DATABASE = {"new", "compare"}
+# commands that dump a database or make a pgschema plan; --verbose shows pgschema's
+# debug log while it dumps and the whole plan, SQL included
+VERBOSE = {"baseline", "new", "drift", "capture", "snapshot", "compare"}
 
 INTRO = """\
 Versioned PostgreSQL migrations from plain SQL, without an ORM.
@@ -60,6 +63,12 @@ def add_apply_arguments(command_parser: argparse.ArgumentParser) -> None:
     command_parser.add_argument("--dry-run", action="store_true", help="only list what would be applied")
 
 
+def add_verbose_argument(command_parser: argparse.ArgumentParser) -> None:
+    command_parser.add_argument("-v", "--verbose", action="store_true",
+                                help="show pgschema's debug log while it dumps the database, "
+                                     "and its whole plan including the SQL")
+
+
 def add_snapshot_arguments(command_parser: argparse.ArgumentParser) -> None:
     command_parser.add_argument("name", help="label for the database, e.g. prod")
 
@@ -85,7 +94,8 @@ COMMANDS = [
     Command(
         "setup", "init", "[--dir D]", "create postgresledger.toml, .env and folders",
         "Creates postgresledger.toml (project settings, committed) and .env with DATABASE_* (never committed, "
-        "added to .gitignore) in the current folder, plus <dir>/schema/ and <dir>/migrations/, then runs setup.",
+        "added to .gitignore) in the current folder, plus <dir>/schema/ and <dir>/migrations/, then runs setup. "
+        "Also .pgschemaignore, which keeps grants out of the migrations (delete it to keep them).",
         add_init_arguments),
     Command(
         "setup", "setup", "", "download pgschema + scratch Postgres, check psql",
@@ -95,7 +105,8 @@ COMMANDS = [
 
     Command(
         "migrations", "baseline", "", "snapshot the database as 0001_baseline",
-        "No migrations yet: dumps the database into schema.sql and creates 0001_baseline.sql from it, "
+        "No migrations yet: dumps the database into schema.sql (one file per schema with `schemas`) and "
+        "creates 0001_baseline.sql from it, "
         "recorded as 'baseline' (not executed). Migrations exist: an existing database that already has "
         "exactly their schema gets them recorded as 'baseline' instead of applying them; if it differs, "
         "the differences are listed and nothing is recorded."),
@@ -180,10 +191,22 @@ def parser() -> argparse.ArgumentParser:
         command_parser.set_defaults(fn=command_function)
         if command.add_arguments is not None:
             command.add_arguments(command_parser)
+        if command.name in VERBOSE:
+            add_verbose_argument(command_parser)
     return main_parser
 
 
 def main(argv: list[str] | None = None) -> None:
+    try:
+        run_command(argv)
+    except KeyboardInterrupt:
+        # Ctrl+C: psql / pgschema got it as well and stopped, the scratch Postgres is cleaned up;
+        # a Python traceback would only hide that
+        print("\ninterrupted", file=sys.stderr)
+        sys.exit(130)
+
+
+def run_command(argv: list[str] | None) -> None:
     args = parser().parse_args(argv)
     if args.cmd in NO_CONFIG:
         sys.exit(args.fn(args) or 0)

{postgresledger-0.0.1 → postgresledger-0.0.3}/src/postgresledger/commands.py

@@ -4,24 +4,32 @@ from __future__ import annotations
 
 import sys
 import tempfile
+import time
 from pathlib import Path
 
 from . import pgschema, scratch
 from .compare import compare
 from .config import CONFIG_NAME, Config, Connection, relative
 from .core import (
+    Migration,
+    Plan,
     applied,
     apply_file,
     build_shadow,
     check_checksums,
+    create_missing_schemas,
+    dump_schemas,
+    dump_text,
     ensure_meta,
     generate_migration,
+    has_changes,
     list_migrations,
-    pgschema_dump,
-    pgschema_plan,
+    plan_schemas,
+    print_plans,
     psql,
     record_migration_sql,
     require_up_to_date,
+    schema_files_in,
 )
 
 
@@ -35,6 +43,9 @@ def config_file_text(folder: str) -> str:
 [project]
 schema = "public"                                # schema managed by postgresledger
 schema_file = "{folder}/schema/schema.sql"          # desired state, the source of truth
+# several schemas: instead of schema and schema_file
+# schemas = ["public", "billing"]                  # in this order inside every migration
+# schema_dir = "{folder}/schema"                      # one <schema>.sql per schema
 migrations_dir = "{folder}/migrations"
 snapshots_dir = "{folder}/snapshots"                # pgl snapshot <name> writes here
 meta_schema = "postgresledger"                         # tracking table: <meta_schema>.schema_migrations
@@ -53,6 +64,30 @@ DATABASE_NAME={database}
 """
 
 
+PGSCHEMA_IGNORE_NAME = ".pgschemaignore"
+
+# what `pgl init` writes to .pgschemaignore
+PGSCHEMA_IGNORE_TEXT = """\
+# Read by pgschema on every pgl command run from this folder. Committed to git.
+#
+# Grants and default privileges are left out of schema.sql, the migrations and
+# the snapshots: roles usually differ between databases (and pgschema never
+# creates them), so a GRANT to a missing role would make `pgl apply` fail, and
+# per-database grants would show up as drift. Manage them outside migrations.
+#
+# The patterns are names of the roles that receive the grants:
+#   ["*", "!app_reader"]   keep only the grants to app_reader
+#   ["cloudsql*"]          leave out only the grants to cloudsql* roles
+# Delete this file to keep every grant in the migrations.
+
+[privileges]
+patterns = ["*"]
+
+[default_privileges]
+patterns = ["*"]
+"""
+
+
 def database_name_from_folder(folder_name: str) -> str:
     """The project folder name as a database name: lowercase letters, digits and _ ("My-App" -> "my_app")."""
     allowed_characters = "abcdefghijklmnopqrstuvwxyz0123456789_"
@@ -82,6 +117,12 @@ def cmd_init(args) -> int:
         dotenv_file.write_text(dotenv_file_text(database_name_from_folder(root.name)))
         created.append(".env")
 
+    # an existing file is the project's own choice: keep it
+    pgschema_ignore = root / PGSCHEMA_IGNORE_NAME
+    if not pgschema_ignore.exists():
+        pgschema_ignore.write_text(PGSCHEMA_IGNORE_TEXT)
+        created.append(f"{PGSCHEMA_IGNORE_NAME} (no grants)")
+
     gitignore = root / ".gitignore"
     ignored = gitignore.read_text().splitlines() if gitignore.exists() else []
     if ".env" not in ignored:
@@ -105,14 +146,30 @@ def cmd_setup(args) -> int:
     return pgschema.setup()
 
 
+def schema_files_label(cfg: Config) -> str:
+    """The schema files as the user would type them, e.g. for "dump ... -> <files>"."""
+    return ", ".join(relative(schema_file) for schema_file in cfg.schema_files.values())
+
+
+def database_plans(cfg: Config, connection: Connection, scratch_server: Connection, migrations: list[Migration],
+                   verbose: bool) -> dict[str, Plan]:
+    """What turns the state of `migrations` into the database as it is, per schema."""
+    shadow = build_shadow(cfg, scratch_server, migrations)
+    create_missing_schemas(cfg, shadow)
+    with tempfile.TemporaryDirectory() as temporary_dir:
+        actual_files = schema_files_in(cfg, Path(temporary_dir))
+        dump_schemas(cfg, connection, actual_files, verbose)
+        return plan_schemas(cfg, shadow, actual_files, scratch_server)
+
+
 def cmd_baseline(cfg: Config, connection: Connection, args) -> int:
     migrations = list_migrations(cfg)
     with scratch.server() as scratch_server:
         if not migrations:
-            cfg.schema_file.parent.mkdir(parents=True, exist_ok=True)
-            print(f"- dump {connection.label} -> {relative(cfg.schema_file)}")
-            pgschema_dump(cfg, connection, cfg.schema_file)
-            baseline = generate_migration(cfg, scratch_server, "baseline", f"baseline taken from {connection.database}")
+            print(f"- dump {connection.label} -> {schema_files_label(cfg)}")
+            dump_schemas(cfg, connection, cfg.schema_files, args.verbose)
+            baseline = generate_migration(cfg, scratch_server, "baseline", f"baseline taken from {connection.database}",
+                                          args.verbose)
             if baseline is None:
                 sys.exit("the database is empty, nothing to snapshot")
             ensure_meta(cfg, connection)
@@ -123,14 +180,10 @@ def cmd_baseline(cfg: Config, connection: Connection, args) -> int:
         if applied(cfg, connection):
             sys.exit(f"{connection.label} already tracks migrations - use `pgl apply`")
         print(f"- building the state of {len(migrations)} migrations (scratch Postgres)")
-        shadow = build_shadow(cfg, scratch_server, migrations)
-        with tempfile.TemporaryDirectory() as temporary_dir:
-            actual = Path(temporary_dir) / "actual.sql"
-            pgschema_dump(cfg, connection, actual)
-            plan = pgschema_plan(cfg, shadow, actual, scratch_server)
-    if plan.sql:
+        plans = database_plans(cfg, connection, scratch_server, migrations, args.verbose)
+    if has_changes(plans):
         print(f"✘ {connection.label} does not match the migrations, so they cannot be recorded as already there:\n")
-        print(plan.human)
+        print_plans(plans, args.verbose)
         print("\nempty database: `pgl apply` instead; otherwise align it first (see `pgl compare`)")
         return 1
     # every record in one transaction: all of them or none
@@ -142,7 +195,7 @@ def cmd_baseline(cfg: Config, connection: Connection, args) -> int:
 
 def cmd_new(cfg: Config, args) -> None:
     with scratch.server() as scratch_server:
-        generate_migration(cfg, scratch_server, args.name, args.note or "change in schema.sql")
+        generate_migration(cfg, scratch_server, args.name, args.note or "change in schema.sql", args.verbose)
 
 
 def cmd_apply(cfg: Config, connection: Connection, args) -> None:
@@ -207,17 +260,13 @@ def cmd_drift(cfg: Config, connection: Connection, args) -> int:
 
     with scratch.server() as scratch_server:
         print(f"- building the state of the {len(has)} migrations {connection.database} has (scratch Postgres)")
-        shadow = build_shadow(cfg, scratch_server, has)
-        with tempfile.TemporaryDirectory() as tmp_dir:
-            actual = Path(tmp_dir) / "actual.sql"
-            pgschema_dump(cfg, connection, actual)
-            # "what turns the expected state into the actual database" = the manual changes
-            plan = pgschema_plan(cfg, shadow, actual, scratch_server)
-    if not plan.sql:
+        # "what turns the expected state into the actual database" = the manual changes
+        plans = database_plans(cfg, connection, scratch_server, has, args.verbose)
+    if not has_changes(plans):
         print(f"✔ {connection.label}: no drift, the database matches its migrations")
         return 0
     print(f"✘ {connection.label}: DRIFT - the database was changed outside of migrations:\n")
-    print(plan.human)
+    print_plans(plans, args.verbose)
     if pending:
         print("\nrun `pgl apply` first, then keep or drop the changes:")
     print("\nkeep the changes:   pgl capture <name>")
@@ -227,15 +276,21 @@ def cmd_drift(cfg: Config, connection: Connection, args) -> int:
 
 def cmd_capture(cfg: Config, connection: Connection, args) -> None:
     require_up_to_date(cfg, connection)
-    print(f"- dump {connection.label} -> {relative(cfg.schema_file)} (manual changes become the desired state)")
-    # put back when there is nothing to capture
-    schema_before = cfg.schema_file.read_text()
-    pgschema_dump(cfg, connection, cfg.schema_file)
+    print(f"- dump {connection.label} -> {schema_files_label(cfg)} (manual changes become the desired state)")
+    # put back when there is nothing to capture; None for a schema file that isn't there yet
+    files_before = {}
+    for schema, schema_file in cfg.schema_files.items():
+        files_before[schema] = schema_file.read_text() if schema_file.exists() else None
+    dump_schemas(cfg, connection, cfg.schema_files, args.verbose)
     note = f"manual changes captured from {connection.database} (recorded as captured there)"
     with scratch.server() as scratch_server:
-        migration = generate_migration(cfg, scratch_server, args.name, note)
+        migration = generate_migration(cfg, scratch_server, args.name, note, args.verbose)
     if migration is None:
-        cfg.schema_file.write_text(schema_before)
+        for schema, schema_file in cfg.schema_files.items():
+            if files_before[schema] is None:
+                schema_file.unlink()
+            else:
+                schema_file.write_text(files_before[schema])
         print(f"{connection.label}: no drift, nothing captured")
         return
     psql(connection, record_migration_sql(cfg, migration, "captured"))
@@ -253,8 +308,11 @@ def cmd_snapshot(cfg: Config, connection: Connection, args) -> None:
     if not is_valid_snapshot_name(args.name):
         sys.exit("snapshot name: letters, digits, _ . - only")
     snapshot = cfg.snapshots_dir / f"{args.name}.sql"
-    pgschema_dump(cfg, connection, snapshot)
-    print(f"✔ {connection.label} -> {relative(snapshot)}")
+    print(f"- dump {connection.label} -> {relative(snapshot)}")
+    started = time.monotonic()
+    snapshot.parent.mkdir(parents=True, exist_ok=True)
+    snapshot.write_text(dump_text(cfg, connection, args.verbose))
+    print(f"✔ {relative(snapshot)} written in {time.monotonic() - started:.1f} s")
 
 
 def snapshot_file(cfg: Config, name_or_path: str) -> Path:
@@ -275,10 +333,8 @@ def cmd_compare(cfg: Config, args) -> int:
         # no second snapshot: compare with the database itself, dumped right now
         if cfg.target is None:
             sys.exit("comparing with the database needs DATABASE_NAME in .env - or give two snapshots")
-        with tempfile.TemporaryDirectory() as temporary_dir:
-            live_dump_file = Path(temporary_dir) / "live.sql"
-            pgschema_dump(cfg, cfg.target, live_dump_file)
-            second_dump = live_dump_file.read_text()
+        print(f"- dump {cfg.target.label}")
+        second_dump = dump_text(cfg, cfg.target, args.verbose)
         second_label = cfg.target.label
     comparison = compare(first_dump, second_dump, args.a, second_label)
     print(comparison.report)

{postgresledger-0.0.1 → postgresledger-0.0.3}/src/postgresledger/compare.py

@@ -3,6 +3,9 @@
 pgschema writes every object as its own block under a header
 `-- Name: <name>; Type: <type>; Schema: ...`, in a stable order, so two dumps
 can be compared block by block.
+
+With several schemas, a snapshot is one dump per schema, each under a
+`-- schema: <name>` line, and objects are named <schema>.<name>.
 """
 
 from __future__ import annotations
@@ -11,6 +14,8 @@ import difflib
 from dataclasses import dataclass
 
 HEADER_START = "-- Name: "
+# starts each schema's part in a snapshot or a migration of several schemas
+SCHEMA_MARKER = "-- schema: "
 # lines around an object's SQL that are not part of it
 SEPARATOR_LINES = ("", "--")
 
@@ -60,11 +65,22 @@ def objects(dump: str) -> dict[SchemaObject, str]:
     sql_by_object: dict[SchemaObject, str] = {}
     current_object = None
     current_lines: list[str] = []
+    # set by a `-- schema: <name>` line; a dump of one schema has none
+    current_schema = ""
     for line in dump.splitlines():
+        if line.startswith(SCHEMA_MARKER):
+            # the previous object ends where the next schema starts
+            if current_object is not None:
+                add_object(sql_by_object, current_object, current_lines)
+            current_object = None
+            current_schema = line.removeprefix(SCHEMA_MARKER).strip()
+            continue
         header_object = parse_header(line)
         if header_object is not None:
             if current_object is not None:
                 add_object(sql_by_object, current_object, current_lines)
+            if current_schema:
+                header_object = SchemaObject(header_object.type, f"{current_schema}.{header_object.name}")
             current_object = header_object
             current_lines = []
         elif current_object is not None:

{postgresledger-0.0.1 → postgresledger-0.0.3}/src/postgresledger/config.py

@@ -53,12 +53,16 @@ class Config:
         self.path = path.resolve()
         self.root = self.path.parent  # paths in the config file are relative to it
         project = tomllib.loads(self.path.read_text()).get("project", {})
-        self.schema = project.get("schema", "public")  # the schema pgschema manages
-        self.schema_file = self.root / project.get("schema_file", "database/schema/schema.sql")
+        # every schema pgschema manages, with the file of its desired state, in the order of the config
+        self.schema_files = schema_files(self.root, project)
+        self.schemas = list(self.schema_files)
         self.migrations_dir = self.root / project.get("migrations_dir", "database/migrations")
         self.snapshots_dir = self.root / project.get("snapshots_dir", "database/snapshots")
         self.meta_schema = project.get("meta_schema", DEFAULT_META_SCHEMA)
         self.meta_table = f"{self.meta_schema}.schema_migrations"
+        if self.meta_schema in self.schemas:
+            sys.exit(f"{CONFIG_NAME}: meta_schema '{self.meta_schema}' is for the tracking table only, "
+                     "it cannot be a managed schema")
         self.env_file = env_file or Path(os.environ.get("PGL_ENV_FILE") or self.root / ".env")
 
         # the one database this run works on; None when DATABASE_NAME is not set.
@@ -76,6 +80,49 @@ class Config:
                 sslmode=os.environ.get("DATABASE_SSLMODE", "prefer"),
             )
 
+    @property
+    def names_schemas(self) -> bool:
+        """Whether migrations and snapshots name the schema of each part.
+
+        A project with only `public` keeps the plain pgschema output; any other schema needs its
+        name, because pgschema writes the objects of a schema without it (CREATE TABLE invoice).
+        """
+        return self.schemas != ["public"]
+
+
+def is_valid_schema_name(name: str) -> bool:
+    """Lowercase letters, digits and _, not starting with a digit: safe unquoted in SQL and as a file name."""
+    allowed_characters = set("abcdefghijklmnopqrstuvwxyz0123456789_")
+    return isinstance(name, str) and name != "" and not name[0].isdigit() and set(name) <= allowed_characters
+
+
+def schema_files(root: Path, project: dict) -> dict[str, Path]:
+    """Every managed schema with the file of its desired state, in the order of the config.
+
+    `schemas = [...]` keeps one <schema_dir>/<schema>.sql per schema. The older `schema` and
+    `schema_file` are one schema with its own file.
+    """
+    if "schemas" not in project:
+        schema = project.get("schema", "public")
+        if not is_valid_schema_name(schema):
+            sys.exit(f"{CONFIG_NAME}: schema '{schema}' - lowercase letters, digits and _ only")
+        return {schema: root / project.get("schema_file", "database/schema/schema.sql")}
+
+    if "schema" in project or "schema_file" in project:
+        sys.exit(f"{CONFIG_NAME}: use either `schemas` and `schema_dir`, or `schema` and `schema_file`, not both")
+    schemas = project["schemas"]
+    if not isinstance(schemas, list) or not schemas:
+        sys.exit(f'{CONFIG_NAME}: schemas must be a list of names, e.g. ["public", "billing"]')
+    schema_dir = root / project.get("schema_dir", "database/schema")
+    files = {}
+    for schema in schemas:
+        if not is_valid_schema_name(schema):
+            sys.exit(f"{CONFIG_NAME}: schema '{schema}' - lowercase letters, digits and _ only")
+        if schema in files:
+            sys.exit(f"{CONFIG_NAME}: schema '{schema}' is listed twice")
+        files[schema] = schema_dir / f"{schema}.sql"
+    return files
+
 
 def relative(path: Path) -> str:
     """The path as the user would type it from the current folder."""

{postgresledger-0.0.1 → postgresledger-0.0.3}/src/postgresledger/core.py

@@ -14,6 +14,7 @@ from dataclasses import dataclass, replace
 from pathlib import Path
 
 from . import pgschema
+from .compare import SCHEMA_MARKER
 from .config import Config, Connection, relative
 
 MIGRATION_RE = re.compile(r"^(\d{4})_([a-z0-9_]+)\.sql$")
@@ -23,16 +24,26 @@ SHADOW_DB = "shadow"
 PLAN_DB = "plan"
 
 
-def run(command: list[str], connection: Connection, *, stdin: str | None = None) -> subprocess.CompletedProcess:
-    """Run psql or pgschema against `connection`; if it fails, show its output and stop pgl."""
+def run(command: list[str], connection: Connection, *, stdin: str | None = None,
+        show_stderr: bool = False) -> subprocess.CompletedProcess:
+    """Run psql or pgschema against `connection`; if it fails, show its output and stop pgl.
+
+    With `show_stderr` the program's log goes straight to the terminal while it runs
+    (only stdout is captured), e.g. pgschema --debug.
+    """
     # the shell environment (PATH, HOME, ...) plus the PG* variables of this connection
     environment = {**os.environ, **connection.environment()}
     program = Path(command[0]).name
+    stderr = None if show_stderr else subprocess.PIPE
+    if show_stderr:
+        # pgl's own lines first: when stdout is a pipe (CI, | less) Python would print them later
+        sys.stdout.flush()
     try:
-        return subprocess.run(command, input=stdin, env=environment, text=True, capture_output=True, check=True)
+        return subprocess.run(command, input=stdin, env=environment, text=True,
+                              stdout=subprocess.PIPE, stderr=stderr, check=True)
     except subprocess.CalledProcessError as error:
-        # the program ran and failed: its own output says why
-        program_output = (error.stdout + error.stderr).strip()
+        # the program ran and failed: its own output says why (stderr is None when already shown)
+        program_output = (error.stdout + (error.stderr or "")).strip()
         sys.exit(f"{program} failed on {connection.label}:\n{program_output}")
     except OSError as error:
         # the program could not run at all, e.g. psql is not installed
@@ -193,10 +204,55 @@ def build_shadow(cfg: Config, scratch: Connection, migrations: list[Migration])
     return shadow
 
 
+def existing_schemas(cfg: Config, connection: Connection) -> list[str]:
+    """The managed schemas `connection` has, in the order of the config."""
+    # schema names are checked in config.py: lowercase letters, digits and _ only
+    quoted_names = ", ".join(f"'{schema}'" for schema in cfg.schemas)
+    found = psql(connection, f"SELECT nspname FROM pg_namespace WHERE nspname IN ({quoted_names})").splitlines()
+    return [schema for schema in cfg.schemas if schema in found]
+
+
+def create_missing_schemas(cfg: Config, scratch_database: Connection) -> list[str]:
+    """Create the managed schemas a scratch database doesn't have yet and return them.
+
+    pgschema plans only inside a schema that exists; the schemas created here are the ones
+    no migration has created yet.
+    """
+    present = existing_schemas(cfg, scratch_database)
+    missing = [schema for schema in cfg.schemas if schema not in present]
+    for schema in missing:
+        psql(scratch_database, f"CREATE SCHEMA {schema}")
+    return missing
+
+
+def schema_files_in(cfg: Config, folder: Path) -> dict[str, Path]:
+    """One <schema>.sql in `folder` for every managed schema, e.g. for dumps that are thrown away."""
+    files = {}
+    for schema in cfg.schemas:
+        files[schema] = folder / f"{schema}.sql"
+    return files
+
+
+def load_plan_database(cfg: Config, scratch: Connection, desired_files: dict[str, Path]) -> None:
+    """Recreate the plan database; with several schemas, load the desired state of all of them into it.
+
+    pgschema loads the desired state of one schema into a temporary schema of the plan database.
+    References to another schema (REFERENCES public.customer) must find that object there.
+    """
+    plan_database = reset_database(scratch, PLAN_DB)
+    if len(cfg.schemas) == 1:
+        return
+    # a function body may use a schema that is loaded after it
+    script = ["SET check_function_bodies = off;"]
+    for schema, desired in desired_files.items():
+        script.append(f"CREATE SCHEMA IF NOT EXISTS {schema};")
+        script.append(f"SET search_path TO {schema};")
+        script.append(f"\\i {desired}")
+    psql(plan_database, "\n".join(script))
+
+
 def plan_arguments(scratch: Connection) -> list[str]:
     """pgschema validates the desired state in a database on the scratch server (not its own embedded one)."""
-    if not psql(scratch, f"SELECT 1 FROM pg_database WHERE datname = '{PLAN_DB}'"):
-        psql(scratch, f'CREATE DATABASE "{PLAN_DB}"')
     return [
         "--plan-host", scratch.host,
         "--plan-port", str(scratch.port),
@@ -206,28 +262,109 @@ def plan_arguments(scratch: Connection) -> list[str]:
     ]
 
 
-def pgschema_dump(cfg: Config, connection: Connection, output_file: Path) -> None:
+def pgschema_dump(schema: str, connection: Connection, output_file: Path, verbose: bool = False) -> None:
+    """One schema of `connection` into `output_file`.
+
+    With `verbose`, pgschema's debug log is shown as it happens: it says when the connection
+    is being made and when it is up, so a slow connection and a slow dump can be told apart.
+    """
     output_file.parent.mkdir(parents=True, exist_ok=True)
-    dump = run([pgschema.locate(), "dump", "--schema", cfg.schema], connection).stdout
+    command = [pgschema.locate(), "dump", "--schema", schema]
+    if verbose:
+        command.append("--debug")
+    dump = run(command, connection, show_stderr=verbose).stdout
     output_file.write_text(dump)
 
 
+def dump_schemas(cfg: Config, connection: Connection, output_files: dict[str, Path], verbose: bool = False) -> None:
+    """Every managed schema of `connection` into its file in `output_files`.
+
+    A schema the database doesn't have gets an empty file: an empty schema, as pgschema
+    cannot dump a schema that doesn't exist.
+    """
+    present = existing_schemas(cfg, connection)
+    for schema, output_file in output_files.items():
+        if schema in present:
+            pgschema_dump(schema, connection, output_file, verbose)
+        else:
+            output_file.parent.mkdir(parents=True, exist_ok=True)
+            output_file.write_text("")
+
+
+def dump_text(cfg: Config, connection: Connection, verbose: bool = False) -> str:
+    """The managed schemas of `connection` as one text, for snapshots and compare.
+
+    Only `public`: the plain pgschema dump. Otherwise every schema's dump under `-- schema: <name>`.
+    """
+    with tempfile.TemporaryDirectory() as temporary_dir:
+        dump_files = schema_files_in(cfg, Path(temporary_dir))
+        dump_schemas(cfg, connection, dump_files, verbose)
+        if not cfg.names_schemas:
+            return dump_files["public"].read_text()
+        parts = []
+        for schema, dump_file in dump_files.items():
+            parts.append(f"{SCHEMA_MARKER}{schema}\n{dump_file.read_text()}")
+        return "\n".join(parts)
+
+
+# where pgschema's readable plan starts repeating the whole SQL
+DDL_HEADER = "DDL to be executed:"
+
+
 @dataclass(frozen=True)
 class Plan:
     """What pgschema would change, in two forms. Both are empty when there is nothing to change."""
 
     sql: str  # the statements, e.g. for a migration file
-    human: str  # the same changes as a readable list for the terminal
+    human: str  # the same changes as a readable list for the terminal, followed by the SQL
+
+    @property
+    def changes(self) -> str:
+        """`human` without the SQL: the summary and the list of changed objects."""
+        return self.human.split(DDL_HEADER)[0].strip()
+
+    @property
+    def headline(self) -> str:
+        """The size of the plan, e.g. "52 to add" from pgschema's first line "Plan: 52 to add."."""
+        first_line = self.human.splitlines()[0] if self.human else ""
+        return first_line.removeprefix("Plan:").strip().removesuffix(".")
+
+
+def print_plans(plans: dict[str, Plan], verbose: bool) -> None:
+    """The changed objects of every schema that changes, and with --verbose also the SQL."""
+    for schema, plan in plans.items():
+        if not plan.sql:
+            continue
+        if len(plans) > 1:
+            print(f"schema {schema}:")
+        print(plan.human if verbose else plan.changes)
+
+
+def has_changes(plans: dict[str, Plan]) -> bool:
+    """Whether any schema's plan changes something."""
+    return any(plan.sql for plan in plans.values())
 
 
-def pgschema_plan(cfg: Config, connection: Connection, desired: Path, scratch: Connection) -> Plan:
-    """The plan that turns `connection` into the state from `desired`."""
+def plans_headline(plans: dict[str, Plan]) -> str:
+    """The size of the plans, e.g. "1 to add", or with several schemas "public: 1 to add, billing: 2 to add"."""
+    if len(plans) == 1:
+        only_plan = next(iter(plans.values()))
+        return only_plan.headline
+    headlines = []
+    for schema, plan in plans.items():
+        if plan.sql:
+            headlines.append(f"{schema}: {plan.headline}")
+    return ", ".join(headlines)
+
+
+def pgschema_plan(schema: str, connection: Connection, desired: Path, scratch: Connection) -> Plan:
+    """The plan that turns `schema` on `connection` into the state from `desired`."""
     with tempfile.TemporaryDirectory() as temporary_dir:
         sql_file = Path(temporary_dir) / "plan.sql"
         human_file = Path(temporary_dir) / "plan.txt"
         command = [
             pgschema.locate(), "plan",
-            "--schema", cfg.schema,
+            "--schema", schema,
             "--file", str(desired),
             "--output-sql", str(sql_file),
             "--output-human", str(human_file),
@@ -241,6 +378,37 @@ def pgschema_plan(cfg: Config, connection: Connection, desired: Path, scratch: C
     return Plan(sql=sql.strip(), human=human.strip())
 
 
+def plan_schemas(cfg: Config, scratch_database: Connection, desired_files: dict[str, Path],
+                 scratch: Connection) -> dict[str, Plan]:
+    """For every managed schema, the plan that turns it on `scratch_database` into its desired file."""
+    load_plan_database(cfg, scratch, desired_files)
+    plans = {}
+    for schema, desired in desired_files.items():
+        plans[schema] = pgschema_plan(schema, scratch_database, desired, scratch)
+    return plans
+
+
+def migration_sql(cfg: Config, plans: dict[str, Plan], new_schemas: list[str]) -> str:
+    """The plans of every schema as the SQL of one migration, in the order of the config.
+
+    pgschema writes a schema's own objects without the schema name, so each part starts with
+    SET search_path; a schema no earlier migration has is created first.
+    """
+    if not cfg.names_schemas:
+        return plans["public"].sql
+    parts = []
+    for schema, plan in plans.items():
+        if not plan.sql:
+            continue
+        lines = [f"{SCHEMA_MARKER}{schema}"]
+        if schema in new_schemas:
+            lines.append(f"CREATE SCHEMA IF NOT EXISTS {schema};")
+        lines.append(f"SET search_path TO {schema};\n")
+        lines.append(plan.sql)
+        parts.append("\n".join(lines))
+    return "\n\n".join(parts)
+
+
 # Generating migrations
 
 
@@ -265,24 +433,35 @@ def write_migration(cfg: Config, name: str, sql: str, note: str) -> Migration:
     return Migration(version, name, path)
 
 
-def generate_migration(cfg: Config, scratch: Connection, name: str, note: str) -> Migration | None:
-    """schema.sql vs (every migration applied) -> new migration, then check that it reproduces schema.sql."""
+def generate_migration(cfg: Config, scratch: Connection, name: str, note: str,
+                       verbose: bool = False) -> Migration | None:
+    """schema files vs (every migration applied) -> new migration, then check that it reproduces them.
+
+    The plan is only printed with `verbose`: it is the content of the new file anyway.
+    """
+    for schema, desired in cfg.schema_files.items():
+        if not desired.is_file():
+            sys.exit(f"no {relative(desired)} - the desired state of schema {schema} "
+                     "(an empty file is an empty schema)")
     existing = list_migrations(cfg)
     print(f"- building the state of {len(existing)} existing migrations (scratch Postgres)")
     shadow = build_shadow(cfg, scratch, existing)
-    plan = pgschema_plan(cfg, shadow, cfg.schema_file, scratch)
-    if not plan.sql:
+    new_schemas = create_missing_schemas(cfg, shadow)
+    plans = plan_schemas(cfg, shadow, cfg.schema_files, scratch)
+    if not has_changes(plans):
         print("no changes compared to the existing migrations")
         return None
-    migration = write_migration(cfg, name, plan.sql, note)
-    print(plan.human)
+    migration = write_migration(cfg, name, migration_sql(cfg, plans, new_schemas), note)
+    if verbose:
+        print_plans(plans, verbose=True)
     print(f"- verifying {migration.path.name}")
     apply_file(cfg, shadow, migration)
     # with the new migration applied, nothing should be left to change
-    leftover = pgschema_plan(cfg, shadow, cfg.schema_file, scratch)
-    if leftover.sql:
-        print(leftover.human)
+    leftover = plan_schemas(cfg, shadow, cfg.schema_files, scratch)
+    if has_changes(leftover):
+        # always in full: this is the case to debug
+        print_plans(leftover, verbose=True)
         migration.path.unlink()
-        sys.exit("after applying the migration the result still differs from schema.sql - migration deleted")
-    print(f"✔ created {relative(migration.path)}")
+        sys.exit("after applying the migration the result still differs from the schema files - migration deleted")
+    print(f"✔ created {relative(migration.path)} ({plans_headline(plans)})")
     return migration