fgsmdb 0.1.0__tar.gz

fgsmdb-0.1.0/PKG-INFO

@@ -0,0 +1,168 @@
+Metadata-Version: 2.4
+Name: fgsmdb
+Version: 0.1.0
+Summary: FlightGear Scenemodels Database Tools
+Author-email: Torsten Dreyer <torsten@t3r.de>
+License-Expression: GPL-2.0-only
+Project-URL: Homepage, https://github.com/FlightGear/scenemodels
+Project-URL: Repository, https://github.com/FlightGear/scenemodels
+Project-URL: Issues, https://github.com/FlightGear/scenemodels/issues
+Keywords: flightgear,scenery,terrasync,export
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Games/Entertainment :: Simulation
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: psycopg2-binary>=2.9
+Requires-Dist: python-dotenv>=1.0
+
+# fgsmdb — FlightGear Scenemodels Database Tools
+
+Command-line tools for the FlightGear scenemodels database.
+
+## Installation
+
+```bash
+pip install fgsmdb
+```
+
+Or install from source:
+
+```bash
+cd export-legacy/fgsmdb
+pip install .
+```
+
+For development:
+
+```bash
+pip install -e .
+```
+
+## Configuration
+
+Copy `.env.example` to `.env` and fill in your values:
+
+```bash
+cp .env.example .env
+```
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `PGURI` | PostgreSQL connection URI (required) | — |
+| `FGELEV_PATH` | Path to the fgelev binary | `/usr/local/bin/fgelev` |
+| `FG_ROOT` | FlightGear data root (needed by fgelev) | `/app/data` |
+| `FG_SCENERY` | Local scenery path (Terrain, Airports, etc.) | `./fgscenery` |
+| `FG_SCENERY_EXPORT` | Working directory for exported files | `./export` |
+| `GROUNDNETS_PATH` | Path to groundnets git repository | `./groundnets` |
+| `MIN_UPDATES` | Minimum pending updates to trigger export | `1` |
+| `SYNC_TARGETS` | Comma-separated rsync target URIs | — |
+| `CDN_INVALIDATE_CMD` | Shell command to run after distribution | — |
+
+For remote distribution, rsync uses your `~/.ssh/config` for host aliases,
+usernames, and identity files.
+
+## Usage
+
+```bash
+# Run the export pipeline
+python3 -m fgsmdb export
+
+# Or use the installed command
+fgsmdb export
+```
+
+### Export Command
+
+Exports the scenemodels database to a TerraSync-compatible file tree.
+
+```bash
+# Full pipeline (all steps except distribute)
+python3 -m fgsmdb export
+
+# With a specific .env file
+python3 -m fgsmdb export --env-file /path/to/.env
+
+# Run specific steps only
+python3 -m fgsmdb export --steps sanitize,export,mkdiridx
+
+# Include distribution to mirrors
+python3 -m fgsmdb export --steps sanitize,gndelev,export,mkdiridx,upload,distribute
+
+# Skip the pending-updates threshold check
+python3 -m fgsmdb export --skip-check
+
+# Override the minimum updates threshold
+python3 -m fgsmdb export --min-updates 10
+
+# Enable verbose (debug) logging
+python3 -m fgsmdb export -v
+```
+
+### Pipeline Steps
+
+| Step | Description |
+|------|-------------|
+| `sanitize` | Fix data integrity: update tile numbers, nullify zero elevation offsets |
+| `gndelev` | Compute ground elevation for new objects using fgelev |
+| `export` | Export shared/per-hemisphere models and generate .stg files from the database |
+| `mkdiridx` | Create `.dirindex` files for TerraSync and the Airports archive |
+| `upload` | Sync exported files to the local scenery tree (hash-based, copies only changes) |
+| `distribute` | Rsync scenery to remote mirror servers (parallel) |
+
+By default, all steps except `distribute` are run. Distribution must be explicitly
+requested via `--steps`.
+
+Before any steps execute, the pipeline calls `fn_update_statistics()` and checks
+whether the number of objects pending ground elevation meets the configured threshold.
+Use `--skip-check` to bypass this gate.
+
+### Command Line Options
+
+```
+python3 -m fgsmdb export [-h] [--env-file ENV_FILE] [--steps STEPS] [--all]
+                         [--skip-check] [--min-updates N] [-v]
+
+Options:
+  --env-file ENV_FILE   Path to .env file (default: auto-detect)
+  --steps STEPS         Comma-separated steps: sanitize, gndelev, export,
+                        mkdiridx, upload, distribute
+  --all                 Run all steps (same as default, without distribute)
+  --skip-check          Skip the pending-updates threshold check
+  --min-updates N       Override the MIN_UPDATES threshold from .env
+  -v, --verbose         Enable debug logging
+  -h, --help            Show help message
+```
+
+## Output Structure
+
+```
+fgscenery/
+├── .dirindex
+├── Airports_archive.tgz
+├── SharedModels.txz
+├── Airports/
+│   ├── .dirindex
+│   └── ...
+├── Models/
+│   ├── .dirindex
+│   └── <group_path>/
+├── Objects/
+│   ├── .dirindex
+│   └── <e|w>NNN<n|s>NN/
+│       └── <e|w>NNN<n|s>NN/
+│           ├── <tile>.stg
+│           └── <model_files>
+└── Terrain/
+    └── .dirindex
+```
+
+## License
+
+GNU General Public License v2.0 — see [LICENCE](../LICENCE).

fgsmdb-0.1.0/README.md

@@ -0,0 +1,144 @@
+# fgsmdb — FlightGear Scenemodels Database Tools
+
+Command-line tools for the FlightGear scenemodels database.
+
+## Installation
+
+```bash
+pip install fgsmdb
+```
+
+Or install from source:
+
+```bash
+cd export-legacy/fgsmdb
+pip install .
+```
+
+For development:
+
+```bash
+pip install -e .
+```
+
+## Configuration
+
+Copy `.env.example` to `.env` and fill in your values:
+
+```bash
+cp .env.example .env
+```
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `PGURI` | PostgreSQL connection URI (required) | — |
+| `FGELEV_PATH` | Path to the fgelev binary | `/usr/local/bin/fgelev` |
+| `FG_ROOT` | FlightGear data root (needed by fgelev) | `/app/data` |
+| `FG_SCENERY` | Local scenery path (Terrain, Airports, etc.) | `./fgscenery` |
+| `FG_SCENERY_EXPORT` | Working directory for exported files | `./export` |
+| `GROUNDNETS_PATH` | Path to groundnets git repository | `./groundnets` |
+| `MIN_UPDATES` | Minimum pending updates to trigger export | `1` |
+| `SYNC_TARGETS` | Comma-separated rsync target URIs | — |
+| `CDN_INVALIDATE_CMD` | Shell command to run after distribution | — |
+
+For remote distribution, rsync uses your `~/.ssh/config` for host aliases,
+usernames, and identity files.
+
+## Usage
+
+```bash
+# Run the export pipeline
+python3 -m fgsmdb export
+
+# Or use the installed command
+fgsmdb export
+```
+
+### Export Command
+
+Exports the scenemodels database to a TerraSync-compatible file tree.
+
+```bash
+# Full pipeline (all steps except distribute)
+python3 -m fgsmdb export
+
+# With a specific .env file
+python3 -m fgsmdb export --env-file /path/to/.env
+
+# Run specific steps only
+python3 -m fgsmdb export --steps sanitize,export,mkdiridx
+
+# Include distribution to mirrors
+python3 -m fgsmdb export --steps sanitize,gndelev,export,mkdiridx,upload,distribute
+
+# Skip the pending-updates threshold check
+python3 -m fgsmdb export --skip-check
+
+# Override the minimum updates threshold
+python3 -m fgsmdb export --min-updates 10
+
+# Enable verbose (debug) logging
+python3 -m fgsmdb export -v
+```
+
+### Pipeline Steps
+
+| Step | Description |
+|------|-------------|
+| `sanitize` | Fix data integrity: update tile numbers, nullify zero elevation offsets |
+| `gndelev` | Compute ground elevation for new objects using fgelev |
+| `export` | Export shared/per-hemisphere models and generate .stg files from the database |
+| `mkdiridx` | Create `.dirindex` files for TerraSync and the Airports archive |
+| `upload` | Sync exported files to the local scenery tree (hash-based, copies only changes) |
+| `distribute` | Rsync scenery to remote mirror servers (parallel) |
+
+By default, all steps except `distribute` are run. Distribution must be explicitly
+requested via `--steps`.
+
+Before any steps execute, the pipeline calls `fn_update_statistics()` and checks
+whether the number of objects pending ground elevation meets the configured threshold.
+Use `--skip-check` to bypass this gate.
+
+### Command Line Options
+
+```
+python3 -m fgsmdb export [-h] [--env-file ENV_FILE] [--steps STEPS] [--all]
+                         [--skip-check] [--min-updates N] [-v]
+
+Options:
+  --env-file ENV_FILE   Path to .env file (default: auto-detect)
+  --steps STEPS         Comma-separated steps: sanitize, gndelev, export,
+                        mkdiridx, upload, distribute
+  --all                 Run all steps (same as default, without distribute)
+  --skip-check          Skip the pending-updates threshold check
+  --min-updates N       Override the MIN_UPDATES threshold from .env
+  -v, --verbose         Enable debug logging
+  -h, --help            Show help message
+```
+
+## Output Structure
+
+```
+fgscenery/
+├── .dirindex
+├── Airports_archive.tgz
+├── SharedModels.txz
+├── Airports/
+│   ├── .dirindex
+│   └── ...
+├── Models/
+│   ├── .dirindex
+│   └── <group_path>/
+├── Objects/
+│   ├── .dirindex
+│   └── <e|w>NNN<n|s>NN/
+│       └── <e|w>NNN<n|s>NN/
+│           ├── <tile>.stg
+│           └── <model_files>
+└── Terrain/
+    └── .dirindex
+```
+
+## License
+
+GNU General Public License v2.0 — see [LICENCE](../LICENCE).

fgsmdb-0.1.0/fgsmdb/__init__.py

@@ -0,0 +1,3 @@
+"""fgsmdb — FlightGear Scenemodels Database Tools."""
+
+__version__ = "0.1.0"

fgsmdb-0.1.0/fgsmdb/__main__.py

@@ -0,0 +1,34 @@
+"""CLI entry point for fgsmdb.
+
+Usage:
+    python3 -m fgsmdb export [options]
+"""
+
+import argparse
+import sys
+
+from .commands import export
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="fgsmdb",
+        description="FlightGear Scenemodels Database Tools",
+    )
+    subparsers = parser.add_subparsers(dest="command", help="Available commands")
+
+    # Register the export subcommand
+    export.register(subparsers)
+
+    args = parser.parse_args()
+
+    if args.command is None:
+        parser.print_help()
+        sys.exit(1)
+
+    # Dispatch to the command handler
+    args.func(args)
+
+
+if __name__ == "__main__":
+    main()

fgsmdb-0.1.0/fgsmdb/commands/__init__.py

@@ -0,0 +1 @@
+# fgsmdb command modules

fgsmdb-0.1.0/fgsmdb/commands/export.py

@@ -0,0 +1,180 @@
+"""Export command — orchestrates the full scenemodels export pipeline.
+
+Usage:
+    python3 -m fgsmdb export [--env-file .env] [--steps sanitize,gndelev,export,mkdiridx,upload,distribute]
+    python3 -m fgsmdb export --all
+    python3 -m fgsmdb export --steps export,mkdiridx
+
+Available steps:
+    sanitize   - Fix data integrity (tile numbers, elevation offsets)
+    gndelev    - Compute ground elevations using fgelev
+    export     - Export models and STG files from the database
+    mkdiridx   - Create .dirindex files and Airports archive
+    upload     - Sync export directory to local scenery tree
+    distribute - Rsync scenery to mirror servers
+"""
+
+import logging
+import os
+import tarfile
+
+from ..config import load_config
+from ..db import Database
+from ..steps import sanitize, gndelev, export_models, export_stg, dirindex, sync, rsync_distribute
+
+ALL_STEPS = ["sanitize", "gndelev", "export", "mkdiridx", "upload"]
+VALID_STEPS = ALL_STEPS + ["distribute"]
+
+logger = logging.getLogger(__name__)
+
+
+def register(subparsers):
+    """Register the export subcommand with the argument parser."""
+    parser = subparsers.add_parser(
+        "export",
+        help="Export scenemodels database to TerraSync file tree",
+        description="Export the FlightGear scenemodels database into a "
+                    "TerraSync-compatible file tree and optionally distribute to mirrors.",
+    )
+    parser.add_argument(
+        "--env-file",
+        default=None,
+        help="Path to .env file (default: auto-detect)",
+    )
+    parser.add_argument(
+        "--steps",
+        default=None,
+        help="Comma-separated list of steps to run (default: all except distribute)",
+    )
+    parser.add_argument(
+        "--all",
+        action="store_true",
+        help="Run all steps (equivalent to: %s)" % ",".join(ALL_STEPS),
+    )
+    parser.add_argument(
+        "--skip-check",
+        action="store_true",
+        help="Skip the pending-updates threshold check",
+    )
+    parser.add_argument(
+        "--min-updates",
+        type=int,
+        default=None,
+        help="Override MIN_UPDATES threshold from .env",
+    )
+    parser.add_argument(
+        "--verbose", "-v",
+        action="store_true",
+        help="Enable debug logging",
+    )
+    parser.set_defaults(func=run)
+
+
+def run(args):
+    """Execute the export pipeline."""
+    # Setup logging
+    log_level = logging.DEBUG if args.verbose else logging.INFO
+    logging.basicConfig(
+        level=log_level,
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+        datefmt="%Y-%m-%d %H:%M:%S",
+    )
+
+    # Load configuration
+    config = load_config(args.env_file)
+
+    # Determine which steps to run
+    if args.steps:
+        steps = [s.strip() for s in args.steps.split(",")]
+        for s in steps:
+            if s not in VALID_STEPS:
+                logger.error("Unknown step: %s. Valid steps: %s", s, ", ".join(VALID_STEPS))
+                raise SystemExit(1)
+    else:
+        steps = ALL_STEPS
+
+    # Override min_updates if specified
+    min_updates = args.min_updates if args.min_updates is not None else config["min_updates"]
+
+    # Connect to the database
+    db = Database(config["pguri"])
+
+    try:
+        # Always update statistics first
+        logger.info("Updating statistics...")
+        db.execute("SELECT fn_update_statistics()")
+
+        # Check if there are enough pending updates
+        if not args.skip_check:
+            rows = db.execute(
+                "SELECT count(*) AS cnt FROM fgs_objects WHERE ob_gndelev = -9999"
+            )
+            count = rows[0]["cnt"] if rows else 0
+            logger.info("Pending model updates: %d (minimum: %d)", count, min_updates)
+            if count < min_updates:
+                logger.info("Not enough pending updates. Exiting.")
+                return
+
+        # Run pipeline steps
+        if "sanitize" in steps:
+            sanitize.run(db)
+
+        if "gndelev" in steps:
+            gndelev.run(
+                db,
+                fgelev_path=config["fgelev_path"],
+                fg_root=config["fg_root"],
+                fg_scenery=config["fg_scenery"],
+            )
+
+        if "export" in steps:
+            export_models.run(db, target=config["fg_scenery_export"])
+            export_stg.run(db, target=config["fg_scenery_export"])
+
+        if "mkdiridx" in steps:
+            dirindex.create_root_dirindex(
+                fg_scenery=config["fg_scenery"],
+                fg_scenery_export=config["fg_scenery_export"],
+            )
+
+        if "upload" in steps:
+            sync.run(
+                fg_scenery_export=config["fg_scenery_export"],
+                fg_scenery=config["fg_scenery"],
+            )
+
+        # Update statistics after export
+        if any(s in steps for s in ("export", "upload")):
+            db.execute("SELECT fn_update_statistics()")
+
+        # Create SharedModels tarball
+        if "upload" in steps:
+            _create_shared_models_tarball(config["fg_scenery"])
+
+        # Distribution (rsync to mirrors)
+        if "distribute" in steps:
+            rsync_distribute.run(
+                fg_scenery=config["fg_scenery"],
+                sync_targets=config["sync_targets"],
+                cdn_invalidate_cmd=config["cdn_invalidate_cmd"],
+            )
+
+        logger.info("Export pipeline complete.")
+
+    finally:
+        db.close()
+
+
+def _create_shared_models_tarball(fg_scenery: str):
+    """Create SharedModels.txz from the Models directory."""
+    models_path = os.path.join(fg_scenery, "Models")
+    output_path = os.path.join(fg_scenery, "SharedModels.txz")
+
+    if not os.path.isdir(models_path):
+        logger.warning("No Models directory at %s, skipping tarball.", models_path)
+        return
+
+    logger.info("Creating SharedModels.txz...")
+    with tarfile.open(output_path, "w:xz") as tar:
+        tar.add(models_path, arcname="Models")
+    logger.info("SharedModels.txz created at %s", output_path)

fgsmdb-0.1.0/fgsmdb/config.py

@@ -0,0 +1,37 @@
+"""Configuration loader — reads .env and provides settings to all modules."""
+
+import os
+from dotenv import load_dotenv
+
+
+def load_config(env_file: str | None = None) -> dict:
+    """Load configuration from .env file and environment variables.
+
+    Args:
+        env_file: Path to .env file. If None, searches in current directory
+                  and parent directories.
+
+    Returns:
+        Dictionary with all configuration values.
+    """
+    if env_file:
+        load_dotenv(env_file)
+    else:
+        load_dotenv()
+
+    config = {
+        "pguri": os.environ["PGURI"],
+        "fgelev_path": os.environ.get("FGELEV_PATH", "/usr/local/bin/fgelev"),
+        "fg_root": os.environ.get("FG_ROOT", "/app/data"),
+        "fg_scenery": os.environ.get("FG_SCENERY", "./fgscenery"),
+        "fg_scenery_export": os.environ.get("FG_SCENERY_EXPORT", "./export"),
+        "groundnets_path": os.environ.get("GROUNDNETS_PATH", "./groundnets"),
+        "min_updates": int(os.environ.get("MIN_UPDATES", "1")),
+        "sync_targets": [
+            t.strip()
+            for t in os.environ.get("SYNC_TARGETS", "").split(",")
+            if t.strip()
+        ],
+        "cdn_invalidate_cmd": os.environ.get("CDN_INVALIDATE_CMD", ""),
+    }
+    return config

fgsmdb-0.1.0/fgsmdb/db.py

@@ -0,0 +1,66 @@
+"""Database connection helper with automatic reconnect."""
+
+import logging
+import psycopg2
+import psycopg2.extras
+
+logger = logging.getLogger(__name__)
+
+
+class Database:
+    """Thin wrapper around psycopg2 with reconnect logic."""
+
+    def __init__(self, pguri: str):
+        self.pguri = pguri
+        self._conn = None
+
+    def connect(self):
+        """Establish database connection."""
+        if self._conn is None or self._conn.closed:
+            logger.info("Connecting to database...")
+            self._conn = psycopg2.connect(self.pguri)
+            self._conn.autocommit = True
+        return self._conn
+
+    @property
+    def conn(self):
+        return self.connect()
+
+    def cursor(self):
+        """Return a DictCursor, reconnecting if needed."""
+        return self.conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
+
+    def execute(self, sql: str, params=None, retries: int = 3):
+        """Execute a query with automatic reconnect on OperationalError.
+
+        Returns:
+            List of result rows, or empty list for non-SELECT statements.
+        """
+        for attempt in range(retries):
+            try:
+                cur = self.cursor()
+                cur.execute(sql, params)
+                if cur.description:
+                    return cur.fetchall()
+                return []
+            except psycopg2.OperationalError:
+                logger.warning(
+                    "Database connection lost (attempt %d/%d), reconnecting...",
+                    attempt + 1,
+                    retries,
+                )
+                self._conn = None
+                if attempt == retries - 1:
+                    raise
+
+    def execute_batch(self, statements: list[str]):
+        """Execute multiple SQL statements in sequence."""
+        cur = self.cursor()
+        for stmt in statements:
+            cur.execute(stmt)
+
+    def close(self):
+        """Close the database connection."""
+        if self._conn and not self._conn.closed:
+            self._conn.close()
+            self._conn = None

fgsmdb-0.1.0/fgsmdb/steps/__init__.py

@@ -0,0 +1 @@
+# fgsmdb pipeline step modules