open-earable-python 0.0.1__tar.gz → 0.0.2__tar.gz

open_earable_python-0.0.2/CHANGELOG.md

@@ -0,0 +1,10 @@
+## 0.0.2
+
+* added access to raw microphone dataframes
+* added a dedicated `docs/` documentation set (getting started, data model, API reference)
+* updated `README.md` as a concise entry point and linked full docs
+
+## 0.0.1
+
+* Initial release of the Open Earable Python SDK.
+* parse Open Earable data files

open_earable_python-0.0.2/PKG-INFO

@@ -0,0 +1,70 @@
+Metadata-Version: 2.4
+Name: open-earable-python
+Version: 0.0.2
+Summary: Reader and utilities for multi-sensor OpenEarable recordings.
+Author-email: "Karlsruhe Institut of Technology (KIT)" <open-earable@lists.kit.edu>
+License-Expression: MIT
+Project-URL: Source, https://github.com/OpenEarable/open-earable-python
+Project-URL: Issues, https://github.com/OpenEarable/open-earable-python/issues
+Keywords: openearable,earable,sensors,imu,ppg,audio,wearables,.oe
+Classifier: Programming Language :: Python :: 3
+Classifier: Development Status :: 4 - Beta
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: ipython
+Requires-Dist: scipy
+Dynamic: license-file
+
+# Open Earable Python
+
+Python toolkit for parsing and analyzing multi-sensor OpenEarable recordings.
+
+## Installation
+
+```bash
+pip install open-earable-python
+```
+
+For local development:
+
+```bash
+git clone https://github.com/OpenEarable/open-earable-python.git
+cd open-earable-python
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+## Quick Example
+
+```python
+from open_earable_python import SensorDataset
+
+dataset = SensorDataset("recording.oe")
+
+# Combined time-indexed DataFrame of all parsed sensors
+df = dataset.get_dataframe()
+
+# Per-sensor views
+imu_df = dataset.imu.df
+ppg_red = dataset.ppg["ppg.red"]
+audio_df = dataset.get_audio_dataframe()
+```
+
+## Documentation
+
+- [Documentation index](docs/README.md)
+- [Getting started](docs/getting-started.md)
+- [Data model and sensor channels](docs/data-model.md)
+- [API reference](docs/api-reference.md)
+
+## License
+
+MIT. See `LICENSE`.

open_earable_python-0.0.2/README.md

@@ -0,0 +1,46 @@
+# Open Earable Python
+
+Python toolkit for parsing and analyzing multi-sensor OpenEarable recordings.
+
+## Installation
+
+```bash
+pip install open-earable-python
+```
+
+For local development:
+
+```bash
+git clone https://github.com/OpenEarable/open-earable-python.git
+cd open-earable-python
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+## Quick Example
+
+```python
+from open_earable_python import SensorDataset
+
+dataset = SensorDataset("recording.oe")
+
+# Combined time-indexed DataFrame of all parsed sensors
+df = dataset.get_dataframe()
+
+# Per-sensor views
+imu_df = dataset.imu.df
+ppg_red = dataset.ppg["ppg.red"]
+audio_df = dataset.get_audio_dataframe()
+```
+
+## Documentation
+
+- [Documentation index](docs/README.md)
+- [Getting started](docs/getting-started.md)
+- [Data model and sensor channels](docs/data-model.md)
+- [API reference](docs/api-reference.md)
+
+## License
+
+MIT. See `LICENSE`.

open_earable_python-0.0.2/docs/README.md

@@ -0,0 +1,16 @@
+# Open Earable Python Documentation
+
+`open-earable-python` parses `.oe` recordings into pandas DataFrames and exposes convenient accessors for OpenEarable sensor streams.
+
+## Contents
+
+- [Getting started](getting-started.md)
+- [Data model and sensor channels](data-model.md)
+- [API reference](api-reference.md)
+
+## Package Scope
+
+- Parse binary OpenEarable streams into structured sensor samples.
+- Build per-sensor and combined time-indexed DataFrames.
+- Decode microphone PCM samples and export/play audio.
+- Load one or multiple recordings with the same API.

open_earable_python-0.0.2/docs/api-reference.md

@@ -0,0 +1,125 @@
+# API Reference
+
+## Package Exports
+
+```python
+from open_earable_python import SensorDataset, load_recordings
+```
+
+## `SensorDataset`
+
+High-level API for loading and analyzing a single `.oe` recording.
+
+### Constructor
+
+```python
+SensorDataset(filename: str, verbose: bool = False)
+```
+
+- `filename`: path to `.oe` file.
+- `verbose`: enables parser diagnostic output.
+
+Parsing happens during initialization.
+
+### Attributes
+
+- `filename: str` source file path.
+- `verbose: bool` parser verbosity flag.
+- `parse_result: parser.ParseResult` raw parse output.
+- `sensor_dfs: Dict[int, pandas.DataFrame]` per-SID DataFrames.
+- `df: pandas.DataFrame` lazily built combined DataFrame.
+- `audio_stereo: Optional[numpy.ndarray]` stereo audio frames (`int16`, shape `(N, 2)`).
+- `audio_df: pandas.DataFrame` cached audio DataFrame.
+
+Sensor accessor attributes:
+
+- `dataset.imu`
+- `dataset.barometer`
+- `dataset.microphone`
+- `dataset.ppg`
+- `dataset.optical_temp`
+- `dataset.bone_acc`
+
+Each accessor supports grouped and channel-level access (see data model docs).
+
+### Methods
+
+#### `parse() -> None`
+
+Re-parses the recording file and updates `parse_result`.
+
+#### `list_sensors() -> List[str]`
+
+Returns sensor names with non-empty DataFrames.
+
+#### `get_sensor_dataframe(name: str) -> pandas.DataFrame`
+
+Returns one sensor DataFrame by name.
+
+- Valid names: `imu`, `barometer`, `microphone`, `ppg`, `optical_temp`, `bone_acc`
+- Raises `KeyError` for unknown names.
+
+#### `get_dataframe() -> pandas.DataFrame`
+
+Builds and caches a merged DataFrame across all non-empty sensor streams.
+
+#### `get_audio_dataframe(sampling_rate: int = 48000) -> pandas.DataFrame`
+
+Returns timestamp-indexed audio DataFrame with columns:
+
+- `mic.inner`
+- `mic.outer`
+
+Behavior:
+
+- Raises `ValueError` if `sampling_rate <= 0`.
+- Returns empty DataFrame with expected columns if no mic packets exist.
+- Caches by sampling rate.
+
+#### `export_csv() -> None`
+
+Writes combined DataFrame to `<recording_basename>.csv` by delegating to `save_csv()`.
+
+#### `save_csv(path: str) -> None`
+
+Saves the combined DataFrame to CSV if `self.df` is non-empty.
+
+Call `get_dataframe()` first to ensure `self.df` is populated.
+
+#### `play_audio(sampling_rate: int = 48000) -> None`
+
+Plays audio in IPython/Jupyter via `IPython.display.Audio`.
+
+#### `save_audio(path: str, sampling_rate: int = 48000) -> None`
+
+Writes WAV audio with `scipy.io.wavfile.write`.
+
+## `load_recordings`
+
+```python
+load_recordings(file_paths: Sequence[str]) -> List[SensorDataset]
+```
+
+Creates `SensorDataset` objects for existing files only.
+
+## Parser Module (`open_earable_python.parser`)
+
+Core classes and helpers for decoding binary packets:
+
+- `Parser`: stream parser over packetized binary data.
+- `PayloadParser`: base parser interface.
+- `SchemePayloadParser`: parser built from `SensorScheme`.
+- `MicPayloadParser`: parser for microphone payloads.
+- `ParseResult`: parse container with per-SID DataFrames and microphone artifacts.
+- `interleaved_mic_to_stereo(samples)`: converts interleaved samples to stereo.
+- `mic_packet_to_stereo_frames(packet, sampling_rate)`: timestamp + stereo frame conversion.
+
+## Scheme Module (`open_earable_python.scheme`)
+
+Defines sensor schema primitives:
+
+- `ParseType` enum
+- `SensorComponentScheme`
+- `SensorComponentGroupScheme`
+- `SensorScheme`
+- `build_default_sensor_schemes(sensor_sid)`

open_earable_python-0.0.2/docs/data-model.md

@@ -0,0 +1,61 @@
+# Data Model and Sensor Channels
+
+## Time Index
+
+All sensor DataFrames are indexed by `timestamp` in seconds (`float`), derived from packet timestamps in microseconds.
+
+## Sensor Streams
+
+`SensorDataset` defines these sensor streams:
+
+- `imu` (SID 0)
+- `barometer` (SID 1)
+- `microphone` (SID 2)
+- `ppg` (SID 4)
+- `optical_temp` (SID 6)
+- `bone_acc` (SID 7)
+
+## Default Columns by Sensor
+
+- `imu`: `acc.x`, `acc.y`, `acc.z`, `gyro.x`, `gyro.y`, `gyro.z`, `mag.x`, `mag.y`, `mag.z`
+- `barometer`: `barometer.temperature`, `barometer.pressure`
+- `ppg`: `ppg.red`, `ppg.ir`, `ppg.green`, `ppg.ambient`
+- `bone_acc`: `bone_acc.x`, `bone_acc.y`, `bone_acc.z`
+- `optical_temp`: `optical_temp`
+- `microphone`: `mic.inner`, `mic.outer`
+
+## Accessor Semantics
+
+Each sensor is exposed as a `_SensorAccessor` object:
+
+- `sensor.df` or `sensor.to_dataframe()` returns the full sensor DataFrame with original column names.
+- Group columns are available as sub-DataFrames:
+  - `dataset.imu.acc` -> columns `x`, `y`, `z`
+  - `dataset.imu.gyro` -> columns `x`, `y`, `z`
+  - `dataset.imu.mag` -> columns `x`, `y`, `z`
+  - `dataset.ppg.ppg` -> columns `red`, `ir`, `green`, `ambient`
+- Original columns remain directly accessible:
+  - `dataset.imu["acc.x"]`
+  - `dataset.ppg["ppg.red"]`
+
+## Combined DataFrame
+
+`get_dataframe()` merges all non-empty per-sensor DataFrames:
+
+- Creates a union of all sensor timestamps.
+- Reindexes each sensor DataFrame onto that common index.
+- Concatenates columns into one DataFrame.
+
+This preserves each stream while aligning them on time.
+
+## Microphone Data Details
+
+Microphone payloads are parsed as interleaved `int16` samples and converted to stereo frames:
+
+- Input interleaving: `[outer, inner, outer, inner, ...]`
+- Output stereo columns/order: `[inner, outer]`
+
+The audio DataFrame generated by `get_audio_dataframe()` uses:
+
+- index: `timestamp` in seconds
+- columns: `mic.inner`, `mic.outer`

open_earable_python-0.0.2/docs/getting-started.md

@@ -0,0 +1,96 @@
+# Getting Started
+
+## Requirements
+
+- Python 3.9+
+- `numpy`, `pandas`, `scipy`, `ipython` (installed automatically with this package)
+
+## Installation
+
+```bash
+pip install open-earable-python
+```
+
+From source:
+
+```bash
+git clone https://github.com/OpenEarable/open-earable-python.git
+cd open-earable-python
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+## Load a Recording
+
+```python
+from open_earable_python import SensorDataset
+
+dataset = SensorDataset("my_recording.oe")
+```
+
+`SensorDataset` parses the file immediately during initialization.
+
+## Work with Sensor Data
+
+```python
+# Combined DataFrame (all available non-empty sensor streams)
+df = dataset.get_dataframe()
+print(df.head())
+
+# List non-empty sensor streams
+print(dataset.list_sensors())
+
+# Access one sensor DataFrame directly
+imu_df = dataset.get_sensor_dataframe("imu")
+print(imu_df.columns)
+```
+
+## Access Channels via Accessors
+
+```python
+# Full IMU DataFrame (columns: acc.x, acc.y, ...)
+imu = dataset.imu.df
+
+# Group-level access (columns renamed to x, y, z)
+acc = dataset.imu.acc
+gyro = dataset.imu.gyro
+
+# Channel-level access
+acc_x = dataset.imu.acc["x"]
+mag_z = dataset.imu.mag.z
+```
+
+## Work with Audio
+
+```python
+# Timestamp-indexed stereo audio DataFrame
+audio_df = dataset.get_audio_dataframe()  # default 48_000 Hz
+print(audio_df.columns)  # mic.inner, mic.outer
+
+# Save WAV
+dataset.save_audio("recording.wav")
+
+# Play in Jupyter/IPython environments
+dataset.play_audio()
+```
+
+## Export CSV
+
+```python
+# Build combined DataFrame, then export it
+dataset.get_dataframe()
+dataset.save_csv("recording.csv")
+```
+
+`save_csv()` writes only if the combined DataFrame is already populated (for example after calling `get_dataframe()`).
+
+## Load Multiple Files
+
+```python
+from open_earable_python import load_recordings
+
+recordings = load_recordings(["session1.oe", "session2.oe"])
+for rec in recordings:
+    print(rec.filename, rec.list_sensors())
+```

{open_earable_python-0.0.1 → open_earable_python-0.0.2}/src/open_earable_python/dataset.py

@@ -20,6 +20,7 @@ LABELS: Dict[str, List[str]] = {
     "ppg": ["ppg.red", "ppg.ir", "ppg.green", "ppg.ambient"],
     "bone_acc": ["bone_acc.x", "bone_acc.y", "bone_acc.z"],
     "optical_temp": ["optical_temp"],
+    "microphone": ["mic.inner", "mic.outer"],
 }
 
 COLORS: Dict[str, List[str]] = {
@@ -131,107 +132,43 @@ class SensorDataset:
     def __init__(self, filename: str, verbose: bool = False):
         self.filename = filename
         self.verbose = verbose
-        self.parse_result: Dict[int, List] = defaultdict(list)
+        self.parse_result: parser.ParseResult = parser.ParseResult(
+            sensor_dfs={},
+            mic_samples=[],
+        )
         # Per-SID dataframes built in _build_accessors
         self.sensor_dfs: Dict[int, pd.DataFrame] = {}
         self.audio_stereo: Optional[np.ndarray] = None
+        self.audio_df: pd.DataFrame = pd.DataFrame()
+        self._audio_df_sampling_rate: Optional[int] = None
         self.bone_sound: Optional[np.ndarray] = None
         self.df: pd.DataFrame = pd.DataFrame()
 
-        self.imu = _SensorAccessor(pd.DataFrame(columns=LABELS["imu"]), LABELS["imu"])
-        self.barometer = _SensorAccessor(pd.DataFrame(columns=LABELS["barometer"]), LABELS["barometer"])
-        self.ppg = _SensorAccessor(pd.DataFrame(columns=LABELS["ppg"]), LABELS["ppg"])
-        self.bone_acc = _SensorAccessor(pd.DataFrame(columns=LABELS["bone_acc"]), LABELS["bone_acc"])
-        self.optical_temp = _SensorAccessor(pd.DataFrame(columns=LABELS["optical_temp"]), LABELS["optical_temp"])
-
-        self.parser: parser.Parser = parser.Parser({
-            self.SENSOR_SID["imu"]: parser.SchemePayloadParser(scheme.SensorScheme(
-                name='imu',
-                sid=self.SENSOR_SID["imu"],
-                groups=[
-                    scheme.SensorComponentGroupScheme(
-                        name='acc',
-                        components=[
-                            scheme.SensorComponentScheme('x', scheme.ParseType.FLOAT),
-                            scheme.SensorComponentScheme('y', scheme.ParseType.FLOAT),
-                            scheme.SensorComponentScheme('z', scheme.ParseType.FLOAT),
-                        ]
-                    ),
-                    scheme.SensorComponentGroupScheme(
-                        name='gyro',
-                        components=[
-                            scheme.SensorComponentScheme('x', scheme.ParseType.FLOAT),
-                            scheme.SensorComponentScheme('y', scheme.ParseType.FLOAT),
-                            scheme.SensorComponentScheme('z', scheme.ParseType.FLOAT),
-                        ]
-                    ),
-                    scheme.SensorComponentGroupScheme(
-                        name='mag',
-                        components=[
-                            scheme.SensorComponentScheme('x', scheme.ParseType.FLOAT),
-                            scheme.SensorComponentScheme('y', scheme.ParseType.FLOAT),
-                            scheme.SensorComponentScheme('z', scheme.ParseType.FLOAT),
-                        ]
-                    ),
-                ])),
-            self.SENSOR_SID["barometer"]: parser.SchemePayloadParser(scheme.SensorScheme(
-                name='barometer',
-                sid=self.SENSOR_SID["barometer"],
-                groups=[
-                    scheme.SensorComponentGroupScheme(
-                        name='barometer',
-                        components=[
-                            scheme.SensorComponentScheme('temperature', scheme.ParseType.FLOAT),
-                            scheme.SensorComponentScheme('pressure', scheme.ParseType.FLOAT),
-                        ]
-                    ),
-                ])),
-            self.SENSOR_SID["ppg"]: parser.SchemePayloadParser(scheme.SensorScheme(
-                name='ppg',
-                sid=self.SENSOR_SID["ppg"],
-                groups=[
-                    scheme.SensorComponentGroupScheme(
-                        name='ppg',
-                        components=[
-                            scheme.SensorComponentScheme('red', scheme.ParseType.UINT32),
-                            scheme.SensorComponentScheme('ir', scheme.ParseType.UINT32),
-                            scheme.SensorComponentScheme('green', scheme.ParseType.UINT32),
-                            scheme.SensorComponentScheme('ambient', scheme.ParseType.UINT32),
-                        ]
-                    ),
-                ])),
-            self.SENSOR_SID["optical_temp"]: parser.SchemePayloadParser(scheme.SensorScheme(
-                name='optical_temp',
-                sid=self.SENSOR_SID["optical_temp"],
-                groups=[
-                    scheme.SensorComponentGroupScheme(
-                        name='optical_temp',
-                        components=[
-                            scheme.SensorComponentScheme('optical_temp', scheme.ParseType.FLOAT),
-                        ]
-                    ),
-                ])),
-            self.SENSOR_SID["bone_acc"]: parser.SchemePayloadParser(scheme.SensorScheme(
-                name='bone_acc',
-                sid=self.SENSOR_SID["bone_acc"],
-                groups=[
-                    scheme.SensorComponentGroupScheme(
-                        name='bone_acc',
-                        components=[
-                            scheme.SensorComponentScheme('x', scheme.ParseType.INT16),
-                            scheme.SensorComponentScheme('y', scheme.ParseType.INT16),
-                            scheme.SensorComponentScheme('z', scheme.ParseType.INT16),
-                        ]
-                    ),
-                ])),
-            self.SENSOR_SID["microphone"]: parser.MicPayloadParser(
-                sample_count=48000,
-            ),
-        }, verbose=verbose)
+        for sensor_name, labels in LABELS.items():
+            setattr(
+                self,
+                sensor_name,
+                _SensorAccessor(pd.DataFrame(columns=labels), labels),
+            )
+
+        self.parser: parser.Parser = self._build_parser(verbose=verbose)
 
         self.parse()
         self._build_accessors()
 
+    @classmethod
+    def _build_parser(cls, verbose: bool = False) -> parser.Parser:
+        sensor_schemes = scheme.build_default_sensor_schemes(cls.SENSOR_SID)
+        dataset_parser = parser.Parser.from_sensor_schemes(
+            sensor_schemes=sensor_schemes,
+            verbose=verbose,
+        )
+        dataset_parser.parsers[cls.SENSOR_SID["microphone"]] = parser.MicPayloadParser(
+            sample_count=48000,
+            verbose=verbose,
+        )
+        return dataset_parser
+
     def parse(self) -> None:
         """Parse the binary recording file into structured sensor data."""
         with open(self.filename, "rb") as f:
@@ -245,10 +182,17 @@ class SensorDataset:
         The combined DataFrame over all sensors is built lazily in
         :meth:`get_dataframe`.
         """
+        self.audio_stereo = self.parse_result.audio_stereo
+        self.audio_df = pd.DataFrame()
+        self._audio_df_sampling_rate = None
+        self.sensor_dfs = {}
+
         data_dict = self.parse_result.sensor_dfs
         for name, sid in self.SENSOR_SID.items():
-            labels = LABELS.get(name, [f"val{i}" for i in range(0)])
-            if sid in data_dict and isinstance(data_dict[sid], pd.DataFrame):
+            labels = LABELS.get(name, [])
+            if name == "microphone":
+                df = self.get_audio_dataframe()
+            elif sid in data_dict and isinstance(data_dict[sid], pd.DataFrame):
                 df = data_dict[sid]
                 df = df[~df.index.duplicated(keep="first")]
             else:
@@ -263,8 +207,6 @@ class SensorDataset:
         # Clear combined dataframe; it will be built lazily on demand
         self.df = pd.DataFrame()
 
-        self.audio_stereo = self.parse_result.audio_stereo
-
     def list_sensors(self) -> List[str]:
         """Return a list of available sensor names in the dataset."""
         available_sensors = []
@@ -330,6 +272,64 @@ class SensorDataset:
 
         return self.df
 
+    def get_audio_dataframe(self, sampling_rate: int = 48000) -> pd.DataFrame:
+        """Return microphone audio as a timestamp-indexed stereo DataFrame.
+
+        The returned DataFrame has:
+        - index: ``timestamp`` in seconds
+        - columns: ``mic.inner`` and ``mic.outer`` (int16 PCM)
+        """
+        if sampling_rate <= 0:
+            raise ValueError(f"sampling_rate must be > 0, got {sampling_rate}")
+
+        if self._audio_df_sampling_rate == sampling_rate:
+            return self.audio_df
+
+        mic_packets = getattr(self.parse_result, "mic_packets", [])
+        if not mic_packets:
+            self.audio_df = pd.DataFrame(columns=["mic.inner", "mic.outer"])
+            self.audio_df.index.name = "timestamp"
+            self._audio_df_sampling_rate = sampling_rate
+            return self.audio_df
+
+        timestamps: List[np.ndarray] = []
+        stereo_frames: List[np.ndarray] = []
+
+        for packet in mic_packets:
+            ts, stereo = parser.mic_packet_to_stereo_frames(
+                packet=packet,
+                sampling_rate=sampling_rate,
+            )
+            if stereo.size == 0:
+                continue
+            timestamps.append(ts)
+            stereo_frames.append(stereo)
+
+        if not timestamps:
+            self.audio_df = pd.DataFrame(columns=["mic.inner", "mic.outer"])
+            self.audio_df.index.name = "timestamp"
+            self._audio_df_sampling_rate = sampling_rate
+            return self.audio_df
+
+        all_ts = np.concatenate(timestamps)
+        all_stereo = np.vstack(stereo_frames)
+
+        self.audio_df = pd.DataFrame(
+            {
+                "mic.inner": all_stereo[:, 0],
+                "mic.outer": all_stereo[:, 1],
+            },
+            index=all_ts,
+        )
+        self.audio_df.index.name = "timestamp"
+        self.audio_df = self.audio_df[~self.audio_df.index.duplicated(keep="first")]
+        self._audio_df_sampling_rate = sampling_rate
+
+        if sampling_rate == 48000:
+            self.sensor_dfs[self.SENSOR_SID["microphone"]] = self.audio_df
+
+        return self.audio_df
+
     def export_csv(self) -> None:
         base_filename, _ = os.path.splitext(self.filename)
         self.save_csv(base_filename + ".csv")