pyroparse 0.3.6__tar.gz

pyroparse-0.3.6/.agents/skills/pyroparse/SKILL.md

@@ -0,0 +1,180 @@
+---
+name: pyroparse
+description: >
+  Parse FIT files (activities, courses) into typed PyArrow tables with structured
+  metadata. Covers read_fit, Activity/Session/Course classes, all_messages(),
+  batch operations, Parquet round-trips, column selection, and the pyroparse CLI.
+  Use when writing Python code that reads FIT files, processes activity/workout/
+  route data, or builds fitness data pipelines — even if the user just says
+  "parse FIT" or "activity data" without naming the library.
+---
+
+# Pyroparse
+
+Rust-backed FIT file parser with Python bindings. Reads FIT files into typed
+PyArrow tables with structured metadata. Normalizes manufacturer-specific
+field names into a consistent schema. Round-trips to Parquet with metadata
+preserved. Zero-copy into Polars, DuckDB, and pandas.
+
+**Install:** `uv add pyroparse` or `pip install pyroparse`
+
+## Core patterns
+
+### Read a FIT file
+
+```python
+import pyroparse as pp
+
+# Table only (no metadata access)
+table = pp.read_fit("ride.fit")                    # -> pa.Table
+
+# With metadata
+activity = pp.Activity.load_fit("ride.fit")
+activity.data                                      # -> pa.Table
+activity.metadata                                  # -> ActivityMetadata
+activity.metadata.sport                            # "cycling.road"
+activity.metadata.devices                          # [Device(...), ...]
+```
+
+### Column selection
+
+```python
+# Default: 11 standard columns
+table = pp.read_fit("ride.fit")
+
+# All columns (standard + extras like core_temperature, smo2, form_power)
+table = pp.read_fit("ride.fit", columns="all")
+
+# Explicit list
+table = pp.read_fit("ride.fit", columns=["timestamp", "power", "heart_rate"])
+
+# Standard + specific extras
+table = pp.read_fit("ride.fit", extra_columns=["core_temperature"])
+```
+
+### Parquet round-trip
+
+```python
+activity = pp.Activity.load_fit("ride.fit")
+activity.to_parquet("ride.parquet")                # ZSTD, metadata preserved
+
+loaded = pp.Activity.load_parquet("ride.parquet")
+loaded.metadata.sport                              # "cycling.road" — survived
+```
+
+### Multi-activity FIT files
+
+```python
+# Activity.load_fit raises MultipleActivitiesError for multi-session files
+session = pp.Session.load_fit("triathlon.fit")
+session.activities[0].metadata.sport               # "swimming"
+session.activities[1].metadata.sport               # "cycling"
+```
+
+### Course files (planned routes)
+
+```python
+course = pp.Course.load_fit("stage3.fit")
+course.track                                       # -> pa.Table (lat, lon, alt, distance)
+course.metadata.name                               # "Volta Ciclista Stage 3"
+course.metadata.waypoints                          # -> list[Waypoint] (turns, climbs, sprints)
+course.metadata.waypoints[0].name                  # "km 0"
+course.metadata.waypoints[0].type                  # "generic"
+course.to_parquet("stage3.parquet")                # single file, waypoints in metadata
+```
+
+### Raw FIT messages (escape hatch)
+
+```python
+msgs = pp.all_messages("ride.fit")
+# Returns list[dict] — every message, no normalization, fitparser-native format.
+# Each dict: {"kind": "record", "fields": [{"name": ..., "value": ..., "units": ...}, ...]}
+
+events = [m["fields"] for m in msgs if m["kind"] == "event"]
+zones = [m["fields"] for m in msgs if m["kind"] == "hr_zone"]
+```
+
+### Batch operations
+
+```python
+# Scan metadata only (fast)
+catalog = pp.scan_fit("~/activities/")             # -> pa.Table (one row per file)
+
+# Load timeseries from multiple files
+data = pp.load_fit_batch(paths, columns=["timestamp", "power"])
+
+# Batch FIT -> Parquet conversion
+result = pp.convert_fit_tree("~/fit/", "~/parquet/", workers=-1, progress=True)
+```
+
+### CLI
+
+```bash
+pyroparse convert ride.fit                         # -> ride.parquet
+pyroparse convert ./activities/ -w -1              # batch, all cores
+pyroparse dump ride.fit                            # raw JSON to stdout
+pyroparse dump ride.fit --kind event,hr_zone       # filter by message type
+```
+
+## API surface
+
+| Function / Class | Description |
+|---|---|
+| `pp.read_fit(source, ...)` | FIT -> `pa.Table` (convenience, no metadata) |
+| `pp.read_parquet(source, ...)` | Parquet -> `pa.Table` |
+| `pp.read_csv(source, ...)` | CSV -> `pa.Table` |
+| `pp.all_messages(source)` | FIT -> `list[dict]` (raw, no normalization) |
+| `pp.Activity.load_fit(source, ...)` | FIT -> `Activity` (data + metadata) |
+| `pp.Activity.load_parquet(source, ...)` | Parquet -> `Activity` |
+| `pp.Activity.load_csv(source, ...)` | CSV -> `Activity` |
+| `pp.Activity.open_fit(path, ...)` | Lazy: metadata now, data on `.data` access |
+| `pp.Activity.open_parquet(path, ...)` | Lazy Parquet loader |
+| `pp.Course.load_fit(source)` | Course FIT -> `Course` (track + waypoints) |
+| `pp.Course.load_parquet(path)` | Parquet -> `Course` |
+| `pp.Session.load_fit(source, ...)` | Multi-activity FIT -> `Session` |
+| `pp.Session.open_fit(path, ...)` | Lazy multi-activity loader |
+| `pp.scan_fit(path, ...)` | Directory -> catalog `pa.Table` (metadata only) |
+| `pp.scan_parquet(path, ...)` | Same for Parquet directories |
+| `pp.load_fit_batch(paths, ...)` | Multiple FIT files -> concatenated `pa.Table` |
+| `pp.convert_fit_file(src, dst)` | Single FIT -> Parquet |
+| `pp.convert_fit_tree(src, dst, ...)` | Batch FIT -> Parquet with directory mirroring |
+| `pp.classify_sport(sport, sub_sport, has_gps)` | -> `Sport` enum value |
+| `pp.STANDARD_COLUMNS` | The 11 default column names |
+
+## Reference
+
+**Schema & columns** — standard columns, types, extras, column selection. Read [schema.md](schema.md)
+
+**Metadata & devices** — ActivityMetadata, Device, Sport enum. Read [metadata.md](metadata.md)
+
+**Batch & convert** — scan, batch load, conversion, CLI commands. Read [batch-and-convert.md](batch-and-convert.md)
+
+**Raw messages** — all_messages() format and dump CLI. Read [raw-messages.md](raw-messages.md)
+
+**Integrations** — Polars, DuckDB, CSV, Parquet metadata queries. Read [integrations.md](integrations.md)
+
+## Gotchas
+
+- `Activity.load_fit()` raises `FileTypeMismatchError` for non-activity FIT
+  files (e.g. course files). Use `Course.load_fit()` for course/route files.
+- `Activity.load_fit()` raises `MultipleActivitiesError` for multi-session
+  FIT files (triathlon, multisport). Use `Session.load_fit()` instead.
+- `read_fit()` returns a bare `pa.Table` with no metadata access. Use
+  `Activity.load_fit()` when you need sport, duration, devices, etc.
+- Default columns are 11 standard columns only. Use `columns="all"` to get
+  extras like `core_temperature`, `smo2`, `form_power`, `stance_time`.
+- `extra_columns` cannot be combined with `columns="all"` or an explicit
+  column list. It only works with the default (standard) column set.
+- GPS coordinates (`latitude`, `longitude`) are degrees (already converted
+  from FIT semicircles). Do NOT convert them again.
+- All timestamps are `Timestamp(us, UTC)` — microsecond precision, UTC.
+  `start_time_local` on metadata is naive (no timezone).
+- `all_messages()` returns raw FIT profile names with no normalization.
+  Field names will differ from the standard pyroparse schema (e.g.
+  `enhanced_speed` instead of `speed`, `position_lat` instead of `latitude`).
+- `open_fit()` uses an experimental binary scanner for metadata. Values
+  should be validated against `load_fit()` for critical workflows.
+- `Source` type accepts `str`, `PathLike`, `bytes`, or `BinaryIO` (file-like
+  object opened in binary mode).
+- `missing="ignore"` fills absent columns with typed nulls instead of raising.
+  Useful for batch loading files with different sensor configurations.

pyroparse-0.3.6/.agents/skills/pyroparse/batch-and-convert.md

@@ -0,0 +1,161 @@
+# Batch Operations & Conversion
+
+## scan_fit
+
+Scan a directory for FIT files. Returns metadata only — no timeseries parsed.
+
+```python
+catalog = pp.scan_fit("~/activities/", recursive=True, errors="warn")
+```
+
+Returns a `pa.Table` with one row per file:
+
+| Column | Type | Description |
+|---|---|---|
+| `file_path` | `Utf8` | Absolute path |
+| `sport` | `Utf8` | Classified sport string |
+| `name` | `Utf8` | Activity name (if set) |
+| `start_time` | `Timestamp(us, UTC)` | UTC start |
+| `start_time_local` | `Timestamp(us)` | Naive local time |
+| `duration` | `Float64` | Seconds |
+| `distance` | `Float64` | Meters |
+| `metrics` | `List(Utf8)` | Available metrics |
+| `device_name` | `Utf8` | Creator device name |
+| `device_type` | `Utf8` | Creator device type |
+
+Parameters:
+- `path: str` — directory to scan
+- `recursive: bool = True` — search subdirectories
+- `errors: str = "warn"` — `"warn"` skips corrupt files, `"raise"` fails immediately
+
+Multi-activity files are skipped with a warning (use `Session.load_fit()`).
+
+## scan_parquet
+
+Same interface and schema as `scan_fit`, but reads Parquet schema footers.
+
+```python
+catalog = pp.scan_parquet("~/parquet/", recursive=True, errors="warn")
+```
+
+## load_fit_batch
+
+Parse multiple FIT files into a single concatenated table.
+
+```python
+data = pp.load_fit_batch(
+    paths,
+    columns=["timestamp", "power"],
+    errors="warn",
+)
+```
+
+Returns a `pa.Table` with a `file_path` column prepended. Supports the same
+`columns`, `extra_columns`, `missing` parameters as `read_fit()`.
+
+Parameters:
+- `paths: list[str]` — file paths to load
+- `columns`, `extra_columns`, `missing` — same as `read_fit()`
+- `errors: str = "warn"` — `"warn"` skips corrupt files, `"raise"` fails
+
+Uses `ThreadPoolExecutor` for concurrent parsing.
+
+## convert_fit_file
+
+Convert a single FIT file to Parquet.
+
+```python
+result = pp.convert_fit_file("ride.fit", "ride.parquet")
+# -> Path("ride.parquet")
+
+# Multi-activity files produce indexed outputs:
+result = pp.convert_fit_file("triathlon.fit", "triathlon.parquet")
+# -> [Path("triathlon_0.parquet"), Path("triathlon_1.parquet"), ...]
+```
+
+Returns `Path` for single-activity files, `list[Path]` for multi-activity.
+
+## convert_fit_tree
+
+Batch-convert a directory of FIT files to Parquet.
+
+```python
+result = pp.convert_fit_tree(
+    "~/fit/",
+    "~/parquet/",          # None = in-place (next to source)
+    glob="**/*.[fF][iI][tT]",  # default, case-insensitive
+    overwrite=False,       # skip existing (idempotent re-runs)
+    workers=-1,            # -1 = all CPU cores
+    progress=True,         # tqdm bar
+)
+result.converted           # list[Path]
+result.errors              # list[tuple[Path, Exception]]
+result.failed              # bool
+```
+
+Parameters:
+- `src` — single FIT file or directory
+- `dst` — output directory, `None` for in-place
+- `glob: str` — file discovery pattern (default: case-insensitive `**/*.fit`)
+- `overwrite: bool = False` — re-convert existing files
+- `workers: int = 1` — parallel processes, `-1` for all cores
+- `progress: bool = False` — show tqdm progress bar
+
+### ConvertResult
+
+```python
+@dataclass
+class ConvertResult:
+    converted: list[Path]
+    errors: list[tuple[Path, Exception]]
+    failed: bool  # property: True if any errors
+```
+
+## CLI
+
+### pyroparse convert
+
+```
+pyroparse convert <src> [-o <dst>] [flags]
+```
+
+| Flag | Description |
+|---|---|
+| `-o, --output PATH` | Output file or directory (default: `.parquet` next to source) |
+| `--overwrite` | Re-convert files whose output already exists |
+| `--glob PATTERN` | File discovery pattern (default: `**/*.[fF][iI][tT]`) |
+| `-w, --workers N` | Parallel workers; -1 = all cores (default: 1) |
+| `--no-progress` | Disable progress bar |
+
+```bash
+pyroparse convert ride.fit                         # -> ride.parquet
+pyroparse convert ride.fit -o /tmp/ride.parquet    # explicit output
+pyroparse convert ./activities/ -w -1              # batch, all cores
+pyroparse convert ./activities/ -o /tmp/parquet/   # mirror tree
+pyroparse convert ./activities/ --overwrite        # force re-convert
+```
+
+### pyroparse dump
+
+```
+pyroparse dump <src> [-o <file>] [flags]
+```
+
+| Flag | Description |
+|---|---|
+| `-o, --output FILE` | Write to file instead of stdout |
+| `--kind TYPE[,TYPE,...]` | Only include these message types |
+| `--exclude TYPE[,TYPE,...]` | Exclude these message types |
+| `--compact` | Single-line JSON (default: pretty-printed) |
+
+`--kind` and `--exclude` are mutually exclusive.
+
+```bash
+pyroparse dump ride.fit                            # all messages, pretty JSON
+pyroparse dump ride.fit --kind event,hr_zone       # filter by type
+pyroparse dump ride.fit --exclude record           # skip record messages
+pyroparse dump ride.fit --compact -o out.json      # compact, to file
+pyroparse dump ride.fit | jq '.[] | select(.kind == "session")'
+```
+
+Single file only — no batch mode. For batch: `for f in *.fit; do pyroparse dump "$f" -o "${f%.fit}.json"; done`

pyroparse-0.3.6/.agents/skills/pyroparse/integrations.md

@@ -0,0 +1,123 @@
+# Integrations
+
+## Polars
+
+Requires `polars` installed separately.
+
+```python
+import polars as pl
+import pyroparse.polars as ppl
+
+# Scan directory -> Polars DataFrame catalog
+catalog = ppl.scan_fit("~/data/")
+
+# Filter and load timeseries
+catalog.filter(pl.col("sport") == "cycling.road") \
+    .fit.load_data(columns=["timestamp", "power"])
+```
+
+### API
+
+```python
+ppl.scan_fit(path, *, recursive=True, errors="warn") -> pl.DataFrame
+ppl.scan_parquet(path, *, recursive=True, errors="warn") -> pl.DataFrame
+```
+
+The `.fit` namespace is registered on all Polars DataFrames:
+
+```python
+df.fit.load_data(*, columns=None, errors="warn") -> pl.DataFrame
+```
+
+Reads the `file_path` column, loads FIT files, returns concatenated DataFrame.
+
+### Zero-copy from PyArrow
+
+For single files, use `pl.from_arrow()` directly:
+
+```python
+import polars as pl
+import pyroparse as pp
+
+df = pl.from_arrow(pp.read_fit("ride.fit"))
+df.group_by("lap").agg(pl.col("power").mean())
+```
+
+## DuckDB
+
+Requires `duckdb` installed separately.
+
+```python
+import pyroparse.duckdb as ppdb
+
+# Scan -> DuckDB relation
+catalog = ppdb.scan_fit("~/data/")
+catalog.filter("sport = 'cycling.road'").fetchdf()
+
+# Load timeseries -> DuckDB relation
+paths = catalog.filter("sport = 'cycling.road'").fetchnumpy()["file_path"].tolist()
+data = ppdb.load_fit(paths, columns=["timestamp", "power"])
+data.filter("power > 300").fetchdf()
+```
+
+### API
+
+```python
+ppdb.scan_fit(path, *, recursive=True, errors="warn", con=None) -> DuckDBPyRelation
+ppdb.scan_parquet(path, *, recursive=True, errors="warn", con=None) -> DuckDBPyRelation
+ppdb.load_fit(paths, *, columns=None, errors="warn", con=None) -> DuckDBPyRelation
+```
+
+All accept an optional `con` parameter for a specific DuckDB connection.
+Defaults to `duckdb.default_connection`.
+
+### Direct Arrow scan
+
+```python
+import duckdb
+import pyroparse as pp
+
+activity = pp.Activity.load_fit("ride.fit")
+duckdb.from_arrow(activity.data).filter("power > 300").fetchdf()
+```
+
+### Parquet metadata queries
+
+Pyroparse stores metadata in Parquet schema under the `b"pyroparse"` key.
+Query it with DuckDB without reading row data:
+
+```sql
+SELECT filename, json_extract_string(value, '$.sport') AS sport
+FROM parquet_kv_metadata('activities/*.parquet')
+WHERE key = 'pyroparse'
+  AND json_extract_string(value, '$.sport') = 'cycling.road';
+```
+
+## CSV
+
+```python
+activity = pp.Activity.load_csv("export.csv", metadata={"sport": "cycling"})
+```
+
+CSV loading infers:
+- Timestamps from common column names
+- Duration from first/last timestamp
+- Available metrics from column names
+- Constant string columns are promoted to metadata
+
+No special dependencies. Use `metadata={}` override for sport and other
+values CSV cannot express.
+
+## pandas
+
+Use PyArrow's built-in conversion:
+
+```python
+import pyroparse as pp
+
+df = pp.read_fit("ride.fit").to_pandas()
+
+# Or from an Activity
+activity = pp.Activity.load_fit("ride.fit")
+df = activity.data.to_pandas()
+```

pyroparse-0.3.6/.agents/skills/pyroparse/metadata.md

@@ -0,0 +1,142 @@
+# Metadata, Devices & Sport
+
+## ActivityMetadata
+
+Dataclass extracted from FIT Session and DeviceInfo messages.
+
+```python
+@dataclass
+class ActivityMetadata:
+    sport: str | None               # "cycling.road", "running.trail", etc.
+    name: str | None                # User-given activity name
+    start_time: datetime | None     # UTC, timezone-aware
+    start_time_local: datetime | None  # Naive, local wall-clock time (no tz)
+    duration: float | None          # Seconds
+    distance: float | None          # Meters
+    metrics: set[str]               # {"heart_rate", "power", "speed", "cadence", "gps"}
+    devices: list[Device]           # Head unit + connected sensors
+    extra: dict                     # {"sub_sport": "road", ...}
+```
+
+### Methods
+
+```python
+meta.column_source("power")        # -> Device that produced the column, or None
+meta.to_dict()                     # -> JSON-serializable dict
+```
+
+### Metadata override
+
+All loaders accept `metadata={}` to override file-native values:
+
+```python
+activity = pp.Activity.load_fit("ride.fit", metadata={"sport": "gravel"})
+activity.metadata.sport       # "gravel" (overridden)
+activity.metadata.duration    # 3842.7  (preserved from FIT)
+```
+
+Override keys must match `ActivityMetadata` field names. Overrides merge on
+top — unspecified fields keep their file-native values.
+
+## Device
+
+```python
+@dataclass
+class Device:
+    name: str | None               # "garmin edge_540", "stryd Stryd"
+    manufacturer: str | None       # "garmin", "stryd", "wahoo_fitness"
+    product: str | None            # "edge_540", "Stryd"
+    serial_number: str | None      # String (may be numeric but stored as str)
+    device_type: str | None        # "creator", "sensor", or "developer"
+    columns: list[str]             # ["power", "cadence"] — columns this device produced
+```
+
+- `"creator"` = head unit (device_index 0 in FIT)
+- `"sensor"` = hardware sensor (ANT+/BLE)
+- `"developer"` = CIQ app (e.g. Stryd, CORE)
+
+`columns` lists the data columns attributed to this device. Pyroparse uses
+ANT+ device type and known manufacturer tables to attribute columns. For
+developer fields, it detects CIQ apps by UUID (Stryd, CORE, etc.).
+
+After column selection, `device.columns` is filtered to only include columns
+present in the final table.
+
+## CourseMetadata
+
+Dataclass for course/route files. Waypoints are embedded in metadata (not a
+separate table) since they're small, sparse annotations.
+
+```python
+@dataclass
+class CourseMetadata:
+    name: str | None              # Course name
+    distance: float | None        # Total distance in meters
+    ascent: float | None          # Total ascent in meters
+    descent: float | None         # Total descent in meters
+    waypoints: list[Waypoint]     # Turns, climbs, sprints, etc.
+```
+
+## Waypoint
+
+```python
+@dataclass
+class Waypoint:
+    name: str | None              # "km 0", "Sprint 1", "Road works"
+    type: str | None              # "generic", "sharp_left", "first_category", etc.
+    latitude: float | None        # Degrees
+    longitude: float | None       # Degrees
+    distance: float | None        # Meters along route
+```
+
+Types include: `generic`, `summit`, `valley`, `water`, `food`, `danger`,
+`left`, `right`, `sharp_left`, `sharp_right`, `slight_left`, `slight_right`,
+`sprint`, `first_category`, `second_category`, `third_category`,
+`fourth_category`, `hors_category`, and more.
+
+## Sport enum
+
+Hierarchical enum with dot-notation values.
+
+```python
+from pyroparse import Sport, classify_sport
+
+Sport.CYCLING              # "cycling"
+Sport.CYCLING_ROAD         # "cycling.road"
+Sport.CYCLING_TRACK_250M   # "cycling.track.250m"
+
+sport = classify_sport("cycling", "road", has_gps=True)
+# -> Sport.CYCLING_ROAD
+```
+
+### Methods
+
+```python
+sport.parent_sport()       # Sport.CYCLING (or None for root sports)
+sport.root_sport()         # Sport.CYCLING (walks to root)
+sport.is_root_sport()      # False
+sport.display_name()       # "Cycling > Road"
+
+sport.is_sub_sport_of(Sport.CYCLING)           # True
+sport.is_sub_sport_of([Sport.CYCLING, Sport.RUNNING])  # True (any match)
+```
+
+### classify_sport()
+
+```python
+pp.classify_sport(sport: str | None, sub_sport: str | None, has_gps: bool) -> Sport
+```
+
+Maps FIT `sport` + `sub_sport` strings to a `Sport` enum value. Uses `has_gps`
+to distinguish indoor/outdoor variants (e.g. cycling with GPS -> `cycling.road`,
+without -> `cycling.trainer`).
+
+Returns `Sport.UNKNOWN` for unrecognized combinations.
+
+### Available sports
+
+Root sports: `cycling`, `running`, `walking`, `swimming`, `rowing`,
+`cross_country_skiing`, `generic`, `unknown`.
+
+Each has sub-sports (e.g. `cycling.road`, `cycling.trainer`,
+`running.trail`, `swimming.pool.25m`). Up to 3 levels deep.