postgres-dbman 0.1.0__tar.gz

postgres_dbman-0.1.0/PKG-INFO

@@ -0,0 +1,155 @@
+Metadata-Version: 2.4
+Name: postgres-dbman
+Version: 0.1.0
+Summary: Add your description here
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: docker>=7.1.0
+Requires-Dist: pre-commit>=4.2.0
+Requires-Dist: python-decouple>=3.8
+Requires-Dist: ruff>=0.11.4
+Requires-Dist: zstandard>=0.22.0
+
+# DBMan
+
+DBMan is an CLI tool to manage PostgreSQL databases running in separate Docker containers.
+The content of the databases is obtained from pg_dumps of production databases.
+
+It will assign each database to a unique ZFS dataset inside a predefined ZFS dataset.
+It will allow snapshotting the dataset and restoring the dataset from a snapshot.
+
+## Configuration
+
+DBMan uses environment variables for configuration. Copy `.env.example` to `.env` and modify the values as needed:
+
+```bash
+# ZFS configuration
+DBMAN_ZFS_DATASET=big/docker-postgres
+DBMAN_ZFS_MOUNT_POINT=/mnt/big/docker-postgres
+DBMAN_ZFS_COMPRESSION=lz4  # Default compression algorithm for new databases
+
+# Docker configuration
+DBMAN_DOCKER_IMAGE=postgres:latest
+
+# PostgreSQL configuration
+DBMAN_POSTGRES_USER=postgres
+DBMAN_POSTGRES_PASSWORD=postgres
+
+# Database configuration
+DBMAN_SQLITE_DB=~/.dbman/dbman.db
+```
+
+You can also override the ZFS compression algorithm using the `-c` or `--compression` command line option:
+
+```bash
+uv run dbman.py -c zstd create mydb dump.sql
+```
+
+## Installation
+
+1. You need `uv` to run dbman. More info about installing it can be found here
+   https://docs.astral.sh/uv/getting-started/installation/
+
+2. Copy `.env.example` to `.env` and configure your environment variables.
+
+3. Test it:
+
+```bash
+uv run dbman.py
+```
+
+## Usage
+
+The tool can be used to:
+
+- create new databases from pg_dumps - it will automatically create a new Docker container
+  with the data from the dump. It will allow naming the container and the database. It will
+  automatically assign a port to the container.
+
+  ```bash
+  # Create a database with default compression (inherited from parent ZFS dataset)
+  uv run dbman.py create mydb dump.sql
+  # or using the short command
+  uv run dbman.py c mydb dump.sql
+
+  # Create a database with specific compression
+  uv run dbman.py create -c zstd mydb dump.sql
+  # or using the short command
+  uv run dbman.py c -c zstd mydb dump.sql
+  ```
+
+- list all databases and their statuses
+
+  ```bash
+  uv run dbman.py list
+  # or using the short command
+  uv run dbman.py ls
+  ```
+
+- create a snapshot of a database
+
+  ```bash
+  uv run dbman.py snapshot mydb mysnapshot
+  # or using the short command
+  uv run dbman.py snap mydb mysnapshot
+  ```
+
+- restore a database from a snapshot
+
+  ```bash
+  uv run dbman.py restore mydb mysnapshot
+  # or using the short command
+  uv run dbman.py rest mydb mysnapshot
+  ```
+
+- list all snapshots of a database
+
+  ```bash
+  uv run dbman.py list-snapshots mydb
+  # or using the short command
+  uv run dbman.py lss mydb
+  ```
+
+- start a subshell with database environment variables
+
+  ```bash
+  uv run dbman.py shell mydb
+  # or using the short command
+  uv run dbman.py sh mydb
+  ```
+
+- print database environment variables
+
+  ```bash
+  # Print variables
+  uv run dbman.py env mydb
+  # or using the short command
+  uv run dbman.py e mydb
+
+  # Export variables to the current shell
+  eval $(uv run dbman.py env --export mydb)
+
+  # Unset variables from the current shell
+  eval $(uv run dbman.py env --clean mydb)
+  ```
+
+- destroy a database
+  ```bash
+  uv run dbman.py destroy mydb
+  # or using the short command
+  uv run dbman.py del mydb
+  ```
+
+For detailed usage information, run:
+
+```bash
+uv run dbman.py --help
+```
+
+## Note
+
+This tool was also used as a proof-of-concept in creating code by using AI. I started
+with a docstring in dbman.py and asked `Cursor` to create an implementation for me.
+It created a functional ~250 lines of code which mostly did what I wanted. When I tweaked
+and improved it, I also extensively used AI to do the modifications, rather than coding it
+myself.

postgres_dbman-0.1.0/README.md

@@ -0,0 +1,143 @@
+# DBMan
+
+DBMan is an CLI tool to manage PostgreSQL databases running in separate Docker containers.
+The content of the databases is obtained from pg_dumps of production databases.
+
+It will assign each database to a unique ZFS dataset inside a predefined ZFS dataset.
+It will allow snapshotting the dataset and restoring the dataset from a snapshot.
+
+## Configuration
+
+DBMan uses environment variables for configuration. Copy `.env.example` to `.env` and modify the values as needed:
+
+```bash
+# ZFS configuration
+DBMAN_ZFS_DATASET=big/docker-postgres
+DBMAN_ZFS_MOUNT_POINT=/mnt/big/docker-postgres
+DBMAN_ZFS_COMPRESSION=lz4  # Default compression algorithm for new databases
+
+# Docker configuration
+DBMAN_DOCKER_IMAGE=postgres:latest
+
+# PostgreSQL configuration
+DBMAN_POSTGRES_USER=postgres
+DBMAN_POSTGRES_PASSWORD=postgres
+
+# Database configuration
+DBMAN_SQLITE_DB=~/.dbman/dbman.db
+```
+
+You can also override the ZFS compression algorithm using the `-c` or `--compression` command line option:
+
+```bash
+uv run dbman.py -c zstd create mydb dump.sql
+```
+
+## Installation
+
+1. You need `uv` to run dbman. More info about installing it can be found here
+   https://docs.astral.sh/uv/getting-started/installation/
+
+2. Copy `.env.example` to `.env` and configure your environment variables.
+
+3. Test it:
+
+```bash
+uv run dbman.py
+```
+
+## Usage
+
+The tool can be used to:
+
+- create new databases from pg_dumps - it will automatically create a new Docker container
+  with the data from the dump. It will allow naming the container and the database. It will
+  automatically assign a port to the container.
+
+  ```bash
+  # Create a database with default compression (inherited from parent ZFS dataset)
+  uv run dbman.py create mydb dump.sql
+  # or using the short command
+  uv run dbman.py c mydb dump.sql
+
+  # Create a database with specific compression
+  uv run dbman.py create -c zstd mydb dump.sql
+  # or using the short command
+  uv run dbman.py c -c zstd mydb dump.sql
+  ```
+
+- list all databases and their statuses
+
+  ```bash
+  uv run dbman.py list
+  # or using the short command
+  uv run dbman.py ls
+  ```
+
+- create a snapshot of a database
+
+  ```bash
+  uv run dbman.py snapshot mydb mysnapshot
+  # or using the short command
+  uv run dbman.py snap mydb mysnapshot
+  ```
+
+- restore a database from a snapshot
+
+  ```bash
+  uv run dbman.py restore mydb mysnapshot
+  # or using the short command
+  uv run dbman.py rest mydb mysnapshot
+  ```
+
+- list all snapshots of a database
+
+  ```bash
+  uv run dbman.py list-snapshots mydb
+  # or using the short command
+  uv run dbman.py lss mydb
+  ```
+
+- start a subshell with database environment variables
+
+  ```bash
+  uv run dbman.py shell mydb
+  # or using the short command
+  uv run dbman.py sh mydb
+  ```
+
+- print database environment variables
+
+  ```bash
+  # Print variables
+  uv run dbman.py env mydb
+  # or using the short command
+  uv run dbman.py e mydb
+
+  # Export variables to the current shell
+  eval $(uv run dbman.py env --export mydb)
+
+  # Unset variables from the current shell
+  eval $(uv run dbman.py env --clean mydb)
+  ```
+
+- destroy a database
+  ```bash
+  uv run dbman.py destroy mydb
+  # or using the short command
+  uv run dbman.py del mydb
+  ```
+
+For detailed usage information, run:
+
+```bash
+uv run dbman.py --help
+```
+
+## Note
+
+This tool was also used as a proof-of-concept in creating code by using AI. I started
+with a docstring in dbman.py and asked `Cursor` to create an implementation for me.
+It created a functional ~250 lines of code which mostly did what I wanted. When I tweaked
+and improved it, I also extensively used AI to do the modifications, rather than coding it
+myself.

postgres_dbman-0.1.0/pyproject.toml

@@ -0,0 +1,46 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "postgres-dbman"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+    "docker>=7.1.0",
+    "pre-commit>=4.2.0",
+    "python-decouple>=3.8",
+    "ruff>=0.11.4",
+    "zstandard>=0.22.0",
+]
+
+[project.scripts]
+dbman = "dbman.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = [
+    "E",   # pycodestyle errors
+    "W",   # pycodestyle warnings
+    "F",   # pyflakes
+    "I",   # isort
+    "B",   # flake8-bugbear
+    "C4",  # flake8-comprehensions
+    "UP",  # pyupgrade
+    "PL",  # pylint
+    "RUF", # ruff-specific rules
+]
+ignore = []
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+line-ending = "auto"

postgres_dbman-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

postgres_dbman-0.1.0/src/dbman/__init__.py

@@ -0,0 +1,5 @@
+"""DBMan - PostgreSQL database management tool for Docker and ZFS."""
+
+from .core import DEFAULT_COMPRESSION, DBMan, format_output
+
+__all__ = ["DEFAULT_COMPRESSION", "DBMan", "format_output"]

postgres_dbman-0.1.0/src/dbman/__main__.py

@@ -0,0 +1,6 @@
+"""Allow running the package as python -m dbman."""
+
+from .cli import main
+
+if __name__ == "__main__":
+    main()

postgres_dbman-0.1.0/src/dbman/cli.py

@@ -0,0 +1,292 @@
+"""CLI entry point for DBMan."""
+
+import argparse
+import logging
+import os
+import re
+import sys
+import traceback
+
+from .core import DEFAULT_COMPRESSION, DBMan, format_output
+
+logger = logging.getLogger(__name__)
+
+
+def main() -> None:  # noqa: PLR0915, PLR0912
+    parser = argparse.ArgumentParser(description="DBMan - PostgreSQL database management tool")
+    parser.add_argument(
+        "-f", "--force", action="store_true", help="Force recreation of ZFS datasets"
+    )
+    parser.add_argument("-v", "--verbose", action="store_true", help="Enable verbose logging")
+    parser.add_argument("--json", action="store_true", help="Output in JSON format")
+
+    subparsers = parser.add_subparsers(dest="command", help="Commands", required=True)
+
+    # Create database command
+    create_parser = subparsers.add_parser("create", aliases=["c"], help="Create a new database")
+    create_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+    create_parser.add_argument("dump_path", help="Path to pg_dump file")
+    create_parser.add_argument("--port", type=int, help="Port to expose (optional)")
+    create_parser.add_argument(
+        "-c",
+        "--compression",
+        default=DEFAULT_COMPRESSION,
+        help=f"ZFS compression algorithm (default: {DEFAULT_COMPRESSION})",
+    )
+    create_parser.add_argument(
+        "--postgres-config",
+        action="append",
+        help="PostgreSQL configuration option in format key=value (can be used multiple times)",
+    )
+
+    # List databases command
+    subparsers.add_parser("list", aliases=["ls"], help="List all databases")
+
+    # Snapshot commands
+    snapshot_parser = subparsers.add_parser("snapshot", aliases=["snap"], help="Create a snapshot")
+    snapshot_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+    snapshot_parser.add_argument("snapshot_name", help="Snapshot name")
+    snapshot_parser.add_argument(
+        "--add",
+        action="store_true",
+        help="Add an existing ZFS snapshot to the database",
+    )
+
+    restore_parser = subparsers.add_parser(
+        "restore", aliases=["rest"], help="Restore from snapshot"
+    )
+    restore_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+    restore_parser.add_argument("snapshot_name", help="Snapshot name or number")
+
+    list_snapshots_parser = subparsers.add_parser(
+        "list-snapshots", aliases=["lss"], help="List snapshots"
+    )
+    list_snapshots_parser.add_argument("name", nargs="?", help="Database name (optional)")
+
+    # Shell command
+    shell_parser = subparsers.add_parser(
+        "shell", aliases=["sh"], help="Start a subshell with database environment variables"
+    )
+    shell_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+
+    # Env command
+    env_parser = subparsers.add_parser(
+        "env", aliases=["e"], help="Print database environment variables"
+    )
+    env_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+    env_parser.add_argument(
+        "--export",
+        "-e",
+        action="store_true",
+        help="Export environment variables to the current shell",
+    )
+    env_parser.add_argument(
+        "--clean",
+        action="store_true",
+        help="Unset environment variables from the current shell",
+    )
+
+    # Destroy command
+    destroy_parser = subparsers.add_parser(
+        "destroy", aliases=["del", "rm"], help="Destroy a database"
+    )
+    destroy_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+    destroy_parser.add_argument(
+        "--keep-zfs-dataset",
+        action="store_true",
+        help="Keep the ZFS dataset (by default, the ZFS dataset is removed)",
+    )
+
+    # Start command
+    start_parser = subparsers.add_parser("start", help="Start a stopped database")
+    start_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+
+    # Stop command
+    stop_parser = subparsers.add_parser("stop", help="Stop a running database")
+    stop_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+
+    # Restart command
+    restart_parser = subparsers.add_parser("restart", help="Restart a database")
+    restart_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+
+    # Cleanup command
+    cleanup_parser = subparsers.add_parser(
+        "cleanup", help="List orphaned mountpoints (directories without corresponding databases)"
+    )
+    cleanup_parser.add_argument(
+        "--one-line",
+        "-1",
+        action="store_true",
+        help="Output only mountpoint names on a single line separated by spaces",
+    )
+
+    # Clone command
+    clone_parser = subparsers.add_parser("clone", help="Clone a database from a snapshot")
+    clone_parser.add_argument(
+        "source_name",
+        nargs="?",
+        help="Source database name (optional if DBMAN_DB_NAME is set)",
+    )
+    clone_parser.add_argument("snapshot_name", help="Snapshot name to clone from")
+    clone_parser.add_argument("new_name", help="Name for the new cloned database")
+    clone_parser.add_argument("--port", type=int, help="Port to expose (optional)")
+
+    # Export command
+    export_parser = subparsers.add_parser(
+        "export", aliases=["exp"], help="Export a database to a ZFS send file"
+    )
+    export_parser.add_argument(
+        "name", nargs="?", help="Database name (optional if DBMAN_DB_NAME is set)"
+    )
+    export_parser.add_argument(
+        "output_file",
+        nargs="?",
+        help="Output file path (optional; auto-generated from name and snapshot if omitted)",
+    )
+    export_parser.add_argument(
+        "--snapshot",
+        "-s",
+        dest="snapshot_name",
+        help="Snapshot name to export (optional; auto-created as export-<timestamp> if omitted)",
+    )
+
+    # Import command
+    import_parser = subparsers.add_parser(
+        "import", aliases=["imp"], help="Import a database from a ZFS send file"
+    )
+    import_parser.add_argument("input_file", help="Path to the ZFS send file")
+    import_parser.add_argument(
+        "new_name",
+        nargs="?",
+        help="Name for the imported database (optional; detected from stream if omitted)",
+    )
+    import_parser.add_argument("--port", type=int, help="Port to expose (optional)")
+    import_parser.add_argument(
+        "--compression",
+        "-c",
+        help="ZFS compression algorithm for the received dataset (e.g. lz4, zstd)",
+    )
+
+    # Delete older than command
+    delete_older_parser = subparsers.add_parser(
+        "delete_older_than",
+        aliases=["dot"],
+        help="List or delete databases older than specified days",
+    )
+    delete_older_parser.add_argument(
+        "days", type=int, help="Number of days - databases older than this will be listed/deleted"
+    )
+    delete_older_parser.add_argument(
+        "--do-it",
+        action="store_true",
+        help="Actually delete the databases (without this flag, only lists them)",
+    )
+
+    args = parser.parse_args()
+
+    # Set logging level based on verbose flag
+    if args.verbose:
+        logging.getLogger("dbman").setLevel(logging.DEBUG)
+
+    # Get database name from environment variable if not provided
+    def get_db_name(name: str | None, allow_none: bool = False) -> str:
+        if name is not None:
+            return name
+        env_name = os.environ.get("DBMAN_DB_NAME")
+        if env_name is None and not allow_none:
+            raise ValueError(
+                "Database name must be provided either as an argument or via DBMAN_DB_NAME "
+                "environment variable"
+            )
+        return env_name
+
+    try:
+        dbman = DBMan(args.force)
+
+        if args.command in ["create", "c"]:
+            # Parse PostgreSQL configuration options
+            postgres_config = {}
+            if args.postgres_config:
+                for cfg in args.postgres_config:
+                    try:
+                        key, value = cfg.split("=", 1)
+                        postgres_config[key] = value
+                    except ValueError:
+                        logger.error(f"Invalid configuration format: {cfg}")
+                        sys.exit(1)
+            dbman.create_database(
+                get_db_name(args.name), args.dump_path, args.port, args.compression, postgres_config
+            )
+        elif args.command in ["list", "ls"]:
+            databases = dbman.list_databases()
+            format_output(databases, args.json)
+        elif args.command in ["snapshot", "snap"]:
+            if args.add:
+                dbman.add_snapshot(get_db_name(args.name), args.snapshot_name)
+            else:
+                dbman.create_snapshot(get_db_name(args.name), args.snapshot_name)
+        elif args.command in ["restore", "rest"]:
+            # only allow negative numbers for snapshot numbers
+            if re.match(r"^-\d+$", args.snapshot_name):
+                dbman.restore_snapshot(get_db_name(args.name), int(args.snapshot_name))
+            else:
+                dbman.restore_snapshot(get_db_name(args.name), args.snapshot_name)
+        elif args.command in ["list-snapshots", "lss"]:
+            snapshots = dbman.list_snapshots(get_db_name(args.name, allow_none=True))
+            format_output(snapshots, args.json)
+        elif args.command in ["shell", "sh"]:
+            dbman.shell(get_db_name(args.name))
+        elif args.command in ["env", "e"]:
+            dbman.env(get_db_name(args.name), args.export, args.clean)
+        elif args.command in ["destroy", "del", "rm"]:
+            dbman._destroy_database(get_db_name(args.name), keep_zfs=args.keep_zfs_dataset)
+        elif args.command == "start":
+            dbman.start(get_db_name(args.name))
+        elif args.command == "stop":
+            dbman.stop(get_db_name(args.name))
+        elif args.command == "restart":
+            dbman.restart(get_db_name(args.name))
+        elif args.command == "cleanup":
+            orphaned_mountpoints = dbman.cleanup()
+            if args.one_line:
+                # Output only mountpoint names on a single line
+                names = [mp["name"] for mp in orphaned_mountpoints]
+                print(" ".join(names))
+            else:
+                format_output(orphaned_mountpoints, args.json)
+        elif args.command == "clone":
+            dbman.clone_database(
+                get_db_name(args.source_name), args.snapshot_name, args.new_name, args.port
+            )
+        elif args.command in ["delete_older_than", "dot"]:
+            databases = dbman.delete_older_than(args.days, args.do_it)
+            format_output(databases, args.json)
+        elif args.command in ["export", "exp"]:
+            dbman.export_database(get_db_name(args.name), args.output_file, args.snapshot_name)
+        elif args.command in ["import", "imp"]:
+            dbman.import_database(args.input_file, args.new_name, args.port, args.compression)
+        else:
+            parser.print_help()
+            sys.exit(1)
+    except Exception as e:
+        logger.error(f"Error: {e!s}")
+        logger.debug(f"Stack trace: {traceback.format_exc()}")
+        sys.exit(1)