thistle-db 0.6.0__tar.gz

thistle_db-0.6.0/.gitignore

@@ -0,0 +1,133 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+__about__.py
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# hatch-vcs
+_version.py

thistle_db-0.6.0/CLAUDE.md

@@ -0,0 +1,148 @@
+# CLAUDE.md
+
+This file is the **target design spec** for thistle-db. Where the code and this
+document disagree, this document wins — update the code to match, not the other
+way around. Sections marked **[NOT YET IMPLEMENTED]** are known gaps.
+
+## What thistle-db is
+
+thistle-db is a database manager for orbital element sets. It ingests TLE
+(Two-Line Element) and OMM (Orbit Mean-Elements Message) files from configured
+directories into a SQL database with uniqueness guarantees, and generates
+organized output files (by date and by satellite) from that database.
+
+## Core workflow
+
+1. TLE/OMM files are delivered into directories defined in `config.toml`
+   (`[[ingest.sources]]` entries: a `path` plus a glob `pattern`).
+2. `thistle-db ingest` scans those directories, auto-detects each file's format,
+   parses the element sets, and inserts them into the database. Duplicates are
+   silently skipped — ingest is **idempotent**: running it twice over the same
+   files must never create duplicate rows or fail.
+3. `thistle-db generate` writes output files from the database.
+
+### File formats
+
+Format is auto-detected per file (`reader.detect_format`):
+
+| Format | Extensions | Reader |
+|---|---|---|
+| TLE (2-line or 3-line) | `.tle`, `.txt`, `.3le` | `read_tle` |
+| OMM JSON (Space-Track) | `.json` | `read_omm_json` |
+| OMM CSV | `.csv` | `read_omm_csv` |
+| OMM XML | `.xml` | `read_omm_xml` |
+
+### Delivery patterns to handle
+
+- **New file per delivery** (e.g. daily `YYYYMMDD.txt`): straightforward scan-and-ingest.
+- **In-place updates**: a provider may append to or rewrite an existing file
+  rather than delivering a delta. This must be handled elegantly:
+  - Correctness comes from DB-level dedup — re-reading a whole updated file and
+    inserting only the new element sets is always safe.
+  - Efficiency comes from **file-state tracking**: each successfully ingested
+    file's path, size, mtime, and content hash is recorded in the
+    `ingest_files` table (`IngestFile` model). On scan
+    (`ingest.ingest_source_file`), unchanged files (size + mtime match) are
+    skipped without being opened; a changed file whose content hash still
+    matches just refreshes its state; otherwise the file is re-ingested (dedup
+    absorbs the already-seen records). A file that fails to parse gets no
+    state recorded, so it is retried next scan. `--force` bypasses skipping.
+
+## Data model (`model.py`)
+
+Single canonical element-set table plus an OMM metadata sidecar. Do **not**
+split OMM into a fully separate element-set table.
+
+- **`tle`** — one row per unique element set. Stores the raw `line1`/`line2`
+  text plus parsed fields (norad_cat_id, epoch, Keplerian elements, drag terms)
+  and derived values (semimajor axis, period, apoapsis/periapsis altitude).
+  Parsing goes through `sgp4.Satrec`.
+- **`omm_metadata`** — OMM-only fields (object name/type, country code, RCS
+  size, launch/decay date, site, originator, GP_ID), one-to-one with a `tle`
+  row via `tle_id`.
+- **`ingest_files`** — per-file ingest state (path, size, mtime_ns, sha256)
+  used to skip unchanged source files on scan; keyed by a sha256 of the
+  resolved path (bounded length for cross-dialect unique indexes).
+
+### Uniqueness
+
+The dedup key is **exact text of `(line1, line2)`** (a `UniqueConstraint`).
+This is deliberate: it is lossless and simple; textual near-duplicates from
+different providers coexist as separate rows. Do not change the key to
+`(norad_cat_id, epoch)` or similar without an explicit decision.
+
+If a TLE is delivered first and the OMM version of the same element set arrives
+later (with identical lines), the TLE row is **not** duplicated — the OMM
+delivery attaches an `omm_metadata` row to the existing `tle` row. Both
+representations are thereby preserved: the TLE lines in `tle`, the OMM extras
+in `omm_metadata`.
+
+Dedup is enforced at the database with dialect-aware upserts
+(`ingest._bulk_insert_tles`): `ON CONFLICT DO NOTHING` for SQLite/PostgreSQL,
+`INSERT IGNORE` for MySQL/MariaDB. Inserts are chunked (`CHUNK_SIZE = 5000`).
+Any new bulk-write path must follow this pattern and work on all three dialects.
+
+## Configuration (`config.py`)
+
+- `config.toml` (path via `-c`, default `./config.toml`), parsed into
+  pydantic-settings models: `[database]`, `[[ingest.sources]]`, `[output]`,
+  `[logging]`.
+- Database credentials are **never** stored in `config.toml`. Resolution order
+  (highest priority first):
+  1. Env vars `THISTLE_DB_DATABASE__USERNAME` / `THISTLE_DB_DATABASE__PASSWORD`
+  2. User secrets file `~/.config/thistle-db.toml`
+  3. System secrets file (`database.secrets_file` in config)
+- Supported databases: SQLite (default/dev), MariaDB/MySQL (operational,
+  `thistle-db[mysql]` extra), PostgreSQL (tested in CI).
+
+## CLI (`cli.py`)
+
+`thistle-db [-c CONFIG] <command>` — the config path defaults to the
+`THISTLE_DB_CONFIG` env var when set (shared with thistle's db fallback),
+else `./config.toml`:
+
+- `init` — scaffold `config.toml` and `~/.config/thistle-db.toml`.
+- `ingest [FILES...] [--force]` — ingest specific files (always parsed, even
+  if recorded as unchanged), or scan all configured sources when no files are
+  given; `--force` re-ingests scanned files regardless of recorded state.
+- `get-tle TARGET [--days N]` — print TLEs to stdout. `TARGET` is either an
+  8-digit date `YYYYMMDD` (→ the nearest TLE per object to 12:00 UTC on that
+  date, within ±N days, default 7) or an alpha-5-compatible NORAD ID
+  (e.g. `25544`, `00022`, `E5693` → every TLE for that object, epoch order).
+  Exits 1 if nothing matches.
+- `generate` — write output files per `[output]` config:
+  - `date_files`: `YYYYMMDD.{tle,omm}` — latest element set per satellite for
+    that date.
+  - `object_files`: `NORAD_ID.{tle,omm}` — all element sets for a satellite,
+    ordered by epoch.
+  - Formats toggleable via `[output.formats]` (tle = two-line text, omm = CSV).
+
+## Module map
+
+| Module | Responsibility |
+|---|---|
+| `cli.py` | typer CLI, subcommand dispatch, logging setup |
+| `config.py` | pydantic-settings config + layered secrets resolution |
+| `model.py` | SQLAlchemy ORM models (`TLE`, `OmmMetadata`), TLE parsing via sgp4 |
+| `reader.py` | format detection + file readers (TLE, OMM JSON/CSV/XML) |
+| `ingest.py` | bulk insert with dedup, source-directory scanning |
+| `generator.py` | output file generation from the database |
+
+## Error handling philosophy
+
+Ingest is tolerant: a malformed TLE or OMM record is logged (loguru, WARNING)
+and skipped — one bad record must never abort a file, and one bad file must
+never abort a scan. A missing source directory is a warning, not an error.
+
+## Development
+
+- Python ≥ 3.11. Package manager: `uv`. Build: hatchling + hatch-vcs
+  (version comes from git tags — never hardcode it).
+- Run tests: `uv run pytest tests/thistle_db` from the workspace root. Lint:
+  `uv run ruff check`. Types: `uv run pyright`.
+- Tests cover SQLite by default; set `THISTLE_DB_TEST_MARIADB=1` /
+  `THISTLE_DB_TEST_POSTGRES=1` (requires Docker) to run the MariaDB and
+  PostgreSQL backends via testcontainers — CI runs all three. Dialect-specific
+  behavior (upserts) must be tested against all three.
+- Test fixtures live in `tests/thistle_db/data/` (real TLE text files and
+  Space-Track OMM JSON).

thistle_db-0.6.0/LICENSE

@@ -0,0 +1,7 @@
+Copyright (c) 2026 Jonathan Olsten
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

thistle_db-0.6.0/PKG-INFO

@@ -0,0 +1,269 @@
+Metadata-Version: 2.4
+Name: thistle-db
+Version: 0.6.0
+Summary: Orbital element database management tool
+Author-email: Jonathan Olsten <jolsten@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: pydantic-settings>=2.9.1
+Requires-Dist: sgp4>=2.24
+Requires-Dist: sqlalchemy>=2.0.40
+Requires-Dist: typer>=0.27.0
+Provides-Extra: mysql
+Requires-Dist: pymysql>=1.1; extra == 'mysql'
+Description-Content-Type: text/markdown
+
+# thistle-db
+
+Orbital element database manager. Ingests TLE (Two-Line Element) and OMM (Orbit Mean-Elements Message) files into a database and generates organized output files by date and satellite.
+
+## Installation
+
+```bash
+pip install thistle-db
+
+# For MariaDB/MySQL support:
+pip install thistle-db[mysql]
+```
+
+Or with [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv add thistle-db
+```
+
+## Quick Start
+
+### 1. Scaffold configuration
+
+```bash
+thistle-db init
+```
+
+This creates two files:
+
+- `./config.toml` -- main configuration (database, ingest sources, output settings)
+- `~/.config/thistle-db.toml` -- user-local database credentials
+
+Use `-c` to specify a different config path:
+
+```bash
+thistle-db -c /etc/thistle-db/config.toml init
+```
+
+### 2. Configure
+
+Edit `config.toml` to set your database and ingest sources. The generated file is fully commented -- see below for a summary.
+
+**SQLite (default):**
+
+```toml
+[database]
+drivername = "sqlite"
+name = "thistle-db.db"
+```
+
+**MariaDB/MySQL:**
+
+```toml
+[database]
+drivername = "mysql+pymysql"
+host = "localhost"
+port = 3306
+name = "thistle-db"
+secrets_file = "/etc/thistle-db/secrets.toml"
+```
+
+Then add your credentials to `~/.config/thistle-db.toml`:
+
+```toml
+username = "myuser"
+password = "mypassword"
+```
+
+### 3. Ingest TLE/OMM files
+
+Scan configured source directories:
+
+```bash
+thistle-db ingest
+```
+
+Or ingest specific files:
+
+```bash
+thistle-db ingest /path/to/20260327.tle /path/to/20260327.json
+```
+
+File format is auto-detected by extension:
+
+| Extension          | Format              |
+| ------------------ | ------------------- |
+| `.tle`, `.txt`, `.3le` | Two-Line Element |
+| `.json`            | Space-Track OMM JSON |
+| `.csv`             | OMM CSV              |
+| `.xml`             | OMM XML              |
+
+Ingestion is idempotent -- duplicate records are silently skipped.
+
+### 4. Generate output files
+
+```bash
+thistle-db generate
+```
+
+This produces files in the configured output directory:
+
+- **Date files** (`YYYYMMDD.tle` / `YYYYMMDD.omm`) -- one TLE per satellite for each date (latest epoch that day)
+- **Object files** (`25544.tle` / `25544.omm`) -- all TLEs for a single satellite, ordered by epoch
+
+## Automating with Cron
+
+thistle-db is designed to run via cron rather than as a long-running service. Both `ingest` and `generate` are idempotent and safe to re-run.
+
+**Ingest and generate every 4 hours:**
+
+```cron
+0 */4 * * * thistle-db -c /etc/thistle-db/config.toml ingest && thistle-db -c /etc/thistle-db/config.toml generate
+```
+
+**Ingest hourly, generate once daily at 03:00 UTC:**
+
+```cron
+0 * * * * thistle-db -c /etc/thistle-db/config.toml ingest
+0 3 * * * thistle-db -c /etc/thistle-db/config.toml generate
+```
+
+**With logging to a file:**
+
+```cron
+0 */4 * * * thistle-db -c /etc/thistle-db/config.toml ingest >> /var/log/thistle-db.log 2>&1 && thistle-db -c /etc/thistle-db/config.toml generate >> /var/log/thistle-db.log 2>&1
+```
+
+## Credential Resolution
+
+Database credentials are resolved in priority order:
+
+1. **Environment variables** -- `THISTLE_DB_DATABASE__USERNAME` / `THISTLE_DB_DATABASE__PASSWORD`
+2. **User secrets file** -- `~/.config/thistle-db.toml`
+3. **System secrets file** -- path set via `secrets_file` in `config.toml`
+4. **config.toml values** -- not recommended for credentials
+
+For cron jobs, either use the user secrets file or export environment variables in the crontab:
+
+```cron
+THISTLE_DB_DATABASE__USERNAME=myuser
+THISTLE_DB_DATABASE__PASSWORD=mypassword
+0 */4 * * * thistle-db -c /etc/thistle-db/config.toml ingest && thistle-db -c /etc/thistle-db/config.toml generate
+```
+
+## CLI Reference
+
+```
+thistle-db [-c CONFIG] COMMAND
+
+Commands:
+  init       Scaffold config.toml and ~/.config/thistle-db.toml
+  ingest     Ingest TLE/OMM files into the database
+  generate   Generate output TLE/OMM files from the database
+  get-tle    Print TLEs from the database to stdout
+
+Options:
+  -c, --config PATH   Path to config.toml
+                      (default: $THISTLE_DB_CONFIG if set, else ./config.toml)
+```
+
+### `get-tle`
+
+Query the database directly and print TLEs to stdout. The positional argument
+is either a NORAD ID (alpha-5 compatible, e.g. `25544`, `00022`, `E5693`) or
+an 8-digit date (`YYYYMMDD`):
+
+```bash
+# All TLEs for one satellite, ordered by epoch
+thistle-db get-tle 25544
+thistle-db get-tle E5693   # alpha-5 IDs work too (= 145693)
+
+# Nearest TLE per satellite to 12:00 UTC on a date, within +/- 7 days
+thistle-db get-tle 20260717
+
+# Widen (or narrow) the search window
+thistle-db get-tle 20260717 --days 3
+```
+
+Exits with status 1 if no TLEs match.
+
+## Configuration Reference
+
+### `[database]`
+
+| Field          | Default      | Description                                |
+| -------------- | ------------ | ------------------------------------------ |
+| `drivername`   | `"sqlite"`   | SQLAlchemy driver (`sqlite`, `mysql+pymysql`) |
+| `name`         | `":memory:"` | Database name or file path                 |
+| `host`         |              | Database host                              |
+| `port`         |              | Database port                              |
+| `username`     |              | Database username (prefer secrets file)    |
+| `password`     |              | Database password (prefer secrets file)    |
+| `secrets_file` |              | Path to a TOML file with username/password |
+
+### `[[ingest.sources]]`
+
+| Field     | Default   | Description                        |
+| --------- | --------- | ---------------------------------- |
+| `path`    |           | Directory to scan for files        |
+| `pattern` | `"*.tle"` | Glob pattern for matching files    |
+
+### `[output]`
+
+| Field   | Default      | Description                     |
+| ------- | ------------ | ------------------------------- |
+| `dir`   | `"./output"` | Output directory                |
+
+### `[output.formats]`
+
+| Field | Default | Description                 |
+| ----- | ------- | --------------------------- |
+| `tle` | `true`  | Generate .tle output files  |
+| `omm` | `true`  | Generate .omm (CSV) output  |
+
+### `[output.types]`
+
+| Field          | Default | Description                                        |
+| -------------- | ------- | -------------------------------------------------- |
+| `date_files`   | `true`  | YYYYMMDD files with latest TLE per satellite       |
+| `object_files` | `true`  | Per-satellite files with all TLEs ordered by epoch  |
+
+### `[logging]`
+
+| Field   | Default  | Description                                    |
+| ------- | -------- | ---------------------------------------------- |
+| `level` | `"INFO"` | Log level: DEBUG, INFO, WARNING, ERROR, CRITICAL |
+
+## Development
+
+### Running tests
+
+Tests live at the workspace root under `tests/thistle_db/` and are
+parametrized to run against SQLite, MariaDB, and PostgreSQL. SQLite runs
+unconditionally; the MariaDB and PostgreSQL backends are opt-in and managed
+automatically by [testcontainers](https://testcontainers-python.readthedocs.io/)
+— one container per test session, one throwaway database per test. No manual
+`docker run` needed, just a running Docker daemon.
+
+SQLite only:
+
+```bash
+uv run pytest tests/thistle_db
+```
+
+All backends (requires Docker):
+
+```bash
+THISTLE_DB_TEST_MARIADB=1 THISTLE_DB_TEST_POSTGRES=1 uv run pytest tests/thistle_db
+```
+
+The images default to `mariadb:11` and `postgres:16`; override with
+`THISTLE_DB_MARIADB_IMAGE` / `THISTLE_DB_POSTGRES_IMAGE`.