bottomtime 0.1.0__tar.gz

bottomtime-0.1.0/.github/workflows/test.yml

@@ -0,0 +1,20 @@
+name: test
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.14"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e .[dev]
+      - run: pytest -q

bottomtime-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+.venv/
+__pycache__/
+*.egg-info/
+dist/
+.pytest_cache/
+
+# Personal dive data lives outside version control
+data/
+archive/
+.DS_Store

bottomtime-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Snehan Kekre
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

bottomtime-0.1.0/PKG-INFO

@@ -0,0 +1,108 @@
+Metadata-Version: 2.4
+Name: bottomtime
+Version: 0.1.0
+Summary: Lossless dive-log store: ingest, decode, and reconcile dives from Garmin (FIT) and Shearwater (Petrel Native Format) dive computers
+Project-URL: Homepage, https://github.com/snehankekre/bottomtime
+Project-URL: Issues, https://github.com/snehankekre/bottomtime/issues
+Author: Snehan Kekre
+License-Expression: MIT
+License-File: LICENSE
+Keywords: decompression,dive-log,diving,fit,garmin,scuba,shearwater,uddf
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.10
+Requires-Dist: garmin-fit-sdk>=21.0
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# bottomtime
+
+A lossless dive-log store for divers who log with more than one computer.
+
+Tec divers often wear two dive computers (say, a Garmin Descent on one wrist
+and a Shearwater on the other). Each records the same dive differently: the
+Garmin logs at 1 Hz with heart rate, GPS, CNS and tissue loading; the
+Shearwater logs its native record with GF99, deco ceiling, ppO2 sensors and
+per-sample battery. No existing interchange format (UDDF, DL7, Subsurface)
+preserves the union, so exporting to any of them throws data away.
+
+bottomtime keeps everything:
+
+- **Ingests Garmin FIT files** (official garmin-fit-sdk). Known dive channels
+  become typed columns; unknown fields and Garmin's undocumented message
+  types are preserved raw, per sample and per message.
+- **Decodes Shearwater's Petrel Native Format** directly from a Shearwater
+  Cloud database (`dive_data.db`, the "Export Database" output) with a pure
+  Python decoder — including per-sample **GF99, deco ceiling, CNS and battery
+  voltage** that Shearwater's own XML/CSV exports omit. Unknown record types
+  are preserved raw.
+- **Archives every source file verbatim**, content-addressed by SHA-256. The
+  database is a decoded view; the archive is the source of truth.
+- **Reconciles dual-computer dives**: interval overlap plus depth-profile
+  cross-correlation links the two logs of the same physical dive (storing
+  clock offset and residual skew) without ever merging or resampling the
+  original series.
+- **Verifies itself**: the decoder is cross-checked per-sample against
+  Shearwater XML exports, against Shearwater Cloud's own computed values
+  (EndGF99), and matched dives are checked for depth agreement.
+
+Everything lands in a single SQLite file with a stable schema, ready for SQL,
+pandas, or whatever you analyze with.
+
+## Install
+
+```sh
+pip install bottomtime
+```
+
+## Usage
+
+```sh
+# create a store
+bottomtime init
+
+# ingest a directory of Garmin FIT files
+# (optionally with a Garmin Connect index JSON for true UTC offsets and metadata)
+bottomtime ingest garmin ~/dives/garmin-fits
+
+# ingest a Shearwater Cloud database export
+bottomtime ingest shearwater ~/dives/dive_data.db
+
+# link dual-computer dives and build the canonical dive list
+bottomtime match
+
+# run the verification suite (XML dir optional but recommended)
+bottomtime verify --xml-dir ~/dives/shearwater-xml
+
+bottomtime status
+```
+
+All commands are idempotent: re-running an ingest skips already-stored dives,
+so syncing after a dive trip only adds what's new.
+
+## Schema in one breath
+
+`raw_artifacts` (archived files) → `source_dives` (one row per computer log,
+with the verbatim header and, for Shearwater, the raw PNF blob) →
+`garmin_samples` / `shearwater_samples` (wide, per-source, never resampled) +
+`gases`, `gas_segments`, `events`, `undecoded_payloads` (raw bytes of
+anything not yet understood) → `matches` (dual-computer links) → `dives` +
+`dive_members` (canonical dives). Views `v_dive_summary` and
+`v_samples_unified` cover the common queries.
+
+## Notes on the PNF decoder
+
+The decoder follows libdivecomputer's `shearwater_predator_parser.c` for the
+documented fields and adds empirically verified mappings for GF99 (byte 25),
+deco ceiling (byte 24), battery voltage (byte 18) and @+5 TTS (bytes 26-27),
+validated against Shearwater Cloud's displayed values across hundreds of
+dives. Bytes without a known meaning are preserved per sample. If your dives
+disagree, `bottomtime verify` will say so loudly — issue reports with a
+failing blob are very welcome.
+
+## License
+
+MIT

bottomtime-0.1.0/README.md

@@ -0,0 +1,88 @@
+# bottomtime
+
+A lossless dive-log store for divers who log with more than one computer.
+
+Tec divers often wear two dive computers (say, a Garmin Descent on one wrist
+and a Shearwater on the other). Each records the same dive differently: the
+Garmin logs at 1 Hz with heart rate, GPS, CNS and tissue loading; the
+Shearwater logs its native record with GF99, deco ceiling, ppO2 sensors and
+per-sample battery. No existing interchange format (UDDF, DL7, Subsurface)
+preserves the union, so exporting to any of them throws data away.
+
+bottomtime keeps everything:
+
+- **Ingests Garmin FIT files** (official garmin-fit-sdk). Known dive channels
+  become typed columns; unknown fields and Garmin's undocumented message
+  types are preserved raw, per sample and per message.
+- **Decodes Shearwater's Petrel Native Format** directly from a Shearwater
+  Cloud database (`dive_data.db`, the "Export Database" output) with a pure
+  Python decoder — including per-sample **GF99, deco ceiling, CNS and battery
+  voltage** that Shearwater's own XML/CSV exports omit. Unknown record types
+  are preserved raw.
+- **Archives every source file verbatim**, content-addressed by SHA-256. The
+  database is a decoded view; the archive is the source of truth.
+- **Reconciles dual-computer dives**: interval overlap plus depth-profile
+  cross-correlation links the two logs of the same physical dive (storing
+  clock offset and residual skew) without ever merging or resampling the
+  original series.
+- **Verifies itself**: the decoder is cross-checked per-sample against
+  Shearwater XML exports, against Shearwater Cloud's own computed values
+  (EndGF99), and matched dives are checked for depth agreement.
+
+Everything lands in a single SQLite file with a stable schema, ready for SQL,
+pandas, or whatever you analyze with.
+
+## Install
+
+```sh
+pip install bottomtime
+```
+
+## Usage
+
+```sh
+# create a store
+bottomtime init
+
+# ingest a directory of Garmin FIT files
+# (optionally with a Garmin Connect index JSON for true UTC offsets and metadata)
+bottomtime ingest garmin ~/dives/garmin-fits
+
+# ingest a Shearwater Cloud database export
+bottomtime ingest shearwater ~/dives/dive_data.db
+
+# link dual-computer dives and build the canonical dive list
+bottomtime match
+
+# run the verification suite (XML dir optional but recommended)
+bottomtime verify --xml-dir ~/dives/shearwater-xml
+
+bottomtime status
+```
+
+All commands are idempotent: re-running an ingest skips already-stored dives,
+so syncing after a dive trip only adds what's new.
+
+## Schema in one breath
+
+`raw_artifacts` (archived files) → `source_dives` (one row per computer log,
+with the verbatim header and, for Shearwater, the raw PNF blob) →
+`garmin_samples` / `shearwater_samples` (wide, per-source, never resampled) +
+`gases`, `gas_segments`, `events`, `undecoded_payloads` (raw bytes of
+anything not yet understood) → `matches` (dual-computer links) → `dives` +
+`dive_members` (canonical dives). Views `v_dive_summary` and
+`v_samples_unified` cover the common queries.
+
+## Notes on the PNF decoder
+
+The decoder follows libdivecomputer's `shearwater_predator_parser.c` for the
+documented fields and adds empirically verified mappings for GF99 (byte 25),
+deco ceiling (byte 24), battery voltage (byte 18) and @+5 TTS (bytes 26-27),
+validated against Shearwater Cloud's displayed values across hundreds of
+dives. Bytes without a known meaning are preserved per sample. If your dives
+disagree, `bottomtime verify` will say so loudly — issue reports with a
+failing blob are very welcome.
+
+## License
+
+MIT

bottomtime-0.1.0/pyproject.toml

@@ -0,0 +1,34 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "bottomtime"
+version = "0.1.0"
+description = "Lossless dive-log store: ingest, decode, and reconcile dives from Garmin (FIT) and Shearwater (Petrel Native Format) dive computers"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.10"
+authors = [{ name = "Snehan Kekre" }]
+keywords = ["diving", "scuba", "dive-log", "garmin", "shearwater", "fit", "uddf", "decompression"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: End Users/Desktop",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering",
+]
+dependencies = ["garmin-fit-sdk>=21.0"]
+
+[project.optional-dependencies]
+dev = ["pytest"]
+
+[project.scripts]
+bottomtime = "bottomtime.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/snehankekre/bottomtime"
+Issues = "https://github.com/snehankekre/bottomtime/issues"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/bottomtime"]

bottomtime-0.1.0/src/bottomtime/__init__.py

@@ -0,0 +1,7 @@
+"""bottomtime: lossless dive-log store for multi-computer divers."""
+
+__version__ = "0.1.0"
+
+from .api import list_dives, load_dive
+
+__all__ = ["load_dive", "list_dives", "__version__"]

bottomtime-0.1.0/src/bottomtime/api.py

@@ -0,0 +1,88 @@
+"""Small read API for analysis code.
+
+    import bottomtime
+    dive = bottomtime.load_dive("data/dives.db", 291)
+    dive["sources"]["garmin"]["samples"]["depth_m"]
+"""
+
+from __future__ import annotations
+
+import json
+import sqlite3
+from pathlib import Path
+
+_SAMPLE_TABLES = {"garmin": "garmin_samples", "shearwater": "shearwater_samples"}
+
+
+def _columns_of(con: sqlite3.Connection, table: str) -> list[str]:
+    return [r[1] for r in con.execute(f"PRAGMA table_info({table})")]
+
+
+def load_dive(db_path: str | Path, dive_number: int) -> dict:
+    """Load one canonical dive with every member source log's samples as
+    column-oriented lists (ready for plotting / DataFrame construction)."""
+    con = sqlite3.connect(db_path)
+    con.row_factory = sqlite3.Row
+    try:
+        dive = con.execute(
+            "SELECT * FROM dives WHERE dive_number=?", (dive_number,)
+        ).fetchone()
+        if dive is None:
+            raise KeyError(f"no dive with number {dive_number}")
+        out = {k: dive[k] for k in dive.keys()}
+        out["sources"] = {}
+        members = con.execute(
+            "SELECT sd.* FROM source_dives sd JOIN dive_members dm"
+            " ON dm.source_dive_id = sd.id WHERE dm.dive_id=?",
+            (dive["id"],),
+        ).fetchall()
+        for sd in members:
+            table = _SAMPLE_TABLES[sd["source"]]
+            cols = [c for c in _columns_of(con, table) if c != "source_dive_id"]
+            rows = con.execute(
+                f"SELECT {', '.join(cols)} FROM {table}"
+                " WHERE source_dive_id=? ORDER BY t_s",
+                (sd["id"],),
+            ).fetchall()
+            samples = {c: [r[i] for r in rows] for i, c in enumerate(cols)}
+            entry = {
+                "source_dive_id": sd["id"],
+                "source_key": sd["source_key"],
+                "start_time_utc": sd["start_time_utc"],
+                "start_time_local": sd["start_time_local"],
+                "duration_s": sd["duration_s"],
+                "max_depth_m": sd["max_depth_m"],
+                "mode": sd["mode"],
+                "header": json.loads(sd["header_json"]),
+                "samples": samples,
+            }
+            key = sd["source"]
+            if key in out["sources"]:  # several logs from one source (splits)
+                existing = out["sources"][key]
+                if isinstance(existing, list):
+                    existing.append(entry)
+                else:
+                    out["sources"][key] = [existing, entry]
+            else:
+                out["sources"][key] = entry
+        return out
+    finally:
+        con.close()
+
+
+def list_dives(db_path: str | Path, include_tests: bool = False) -> list[dict]:
+    con = sqlite3.connect(db_path)
+    con.row_factory = sqlite3.Row
+    try:
+        where = "" if include_tests else "WHERE d.is_test = 0"
+        return [
+            {k: r[k] for k in r.keys()}
+            for r in con.execute(
+                "SELECT d.*, group_concat(sd.source) AS sources FROM dives d"
+                " JOIN dive_members dm ON dm.dive_id = d.id"
+                " JOIN source_dives sd ON sd.id = dm.source_dive_id"
+                f" {where} GROUP BY d.id ORDER BY d.start_time_utc"
+            )
+        ]
+    finally:
+        con.close()

bottomtime-0.1.0/src/bottomtime/cli.py

@@ -0,0 +1,107 @@
+"""bottomtime command line interface.
+
+    bottomtime init
+    bottomtime ingest garmin <fit-dir>
+    bottomtime ingest shearwater <dive_data.db>
+    bottomtime match
+    bottomtime verify [--xml-dir DIR]
+    bottomtime status
+
+All commands are idempotent: re-running an ingest skips anything already
+stored (content-hash keyed), match recomputes only 'auto' links.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from .store import db as store_db
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="bottomtime")
+    parser.add_argument("--db", default="data/dives.db", help="path to the dive store")
+    parser.add_argument(
+        "--archive", default="archive", help="directory for verbatim raw source copies"
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    sub.add_parser("init", help="create an empty dive store")
+
+    p_ingest = sub.add_parser("ingest", help="ingest a data source")
+    ingest_sub = p_ingest.add_subparsers(dest="source", required=True)
+    p_garmin = ingest_sub.add_parser("garmin", help="a directory of FIT files")
+    p_garmin.add_argument("fit_dir", type=Path)
+    p_garmin.add_argument(
+        "--index",
+        default="dives_index.json",
+        help="Garmin Connect index JSON filename inside fit-dir",
+    )
+    p_sw = ingest_sub.add_parser("shearwater", help="a Shearwater Cloud dive_data.db")
+    p_sw.add_argument("db_path", type=Path)
+
+    sub.add_parser("match", help="match dual-computer dives, build canonical dives")
+
+    p_verify = sub.add_parser("verify", help="run the verification suite")
+    p_verify.add_argument("--xml-dir", type=Path, default=None,
+                          help="directory of Shearwater XML exports to cross-check")
+
+    sub.add_parser("status", help="summarize the store")
+
+    args = parser.parse_args(argv)
+    con = store_db.connect(args.db)
+    archive_dir = Path(args.archive)
+
+    if args.command == "init":
+        print(f"initialized {args.db}")
+        return 0
+
+    if args.command == "ingest" and args.source == "garmin":
+        from .ingest.garmin_fit import ingest_garmin_dir
+
+        stats = ingest_garmin_dir(con, archive_dir, args.fit_dir, args.index)
+        print(f"garmin: {stats}")
+        return 1 if stats["failed"] else 0
+
+    if args.command == "ingest" and args.source == "shearwater":
+        from .ingest.shearwater_db import ingest_shearwater_db
+
+        stats = ingest_shearwater_db(con, archive_dir, args.db_path)
+        print(f"shearwater: {stats}")
+        return 1 if stats["failed"] else 0
+
+    if args.command == "match":
+        from .match.matcher import run_match
+
+        stats = run_match(con)
+        print(f"match: {stats}")
+        return 0
+
+    if args.command == "verify":
+        from .verify.crosscheck import run_verify
+
+        return run_verify(con, args.xml_dir)
+
+    if args.command == "status":
+        for line in con.execute(
+            "SELECT source || ' source dives: ' || count(*) FROM source_dives GROUP BY source"
+        ):
+            print(line[0])
+        for line in con.execute(
+            "SELECT 'canonical dives: ' || count(*) || ' (' ||"
+            " sum(CASE WHEN is_test THEN 1 ELSE 0 END) || ' test)' FROM dives"
+        ):
+            print(line[0])
+        for line in con.execute(
+            "SELECT 'matched pairs: ' || count(*) FROM matches WHERE status IN ('auto','confirmed')"
+        ):
+            print(line[0])
+        return 0
+
+    return 2
+
+
+if __name__ == "__main__":
+    sys.exit(main())

bottomtime-0.1.0/src/bottomtime/decode/__init__.py

@@ -0,0 +1 @@
+