downsampler 0.1.0__tar.gz

downsampler-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 KNMI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

downsampler-0.1.0/PKG-INFO

@@ -0,0 +1,246 @@
+Metadata-Version: 2.4
+Name: downsampler
+Version: 0.1.0
+Summary: Timeseries DataFrame downsampling with LTTB, aggregation methods, gap handling, and fidelity testing
+Author-email: Eelco Doornbos <eelco.doornbos@knmi.nl>
+License: MIT
+Project-URL: Homepage, https://gitlab.com/KNMI-OSS/spaceweather/downsampler
+Project-URL: Repository, https://gitlab.com/KNMI-OSS/spaceweather/downsampler
+Keywords: timeseries,downsampling,lttb,pandas,visualization
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.20
+Requires-Dist: pandas>=1.3
+Requires-Dist: lttb>=0.3
+Requires-Dist: scipy>=1.7
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.5; extra == "viz"
+Requires-Dist: altair>=5.0; extra == "viz"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0; extra == "test"
+Provides-Extra: dev
+Requires-Dist: downsampler[test,viz]; extra == "dev"
+Dynamic: license-file
+
+# downsampler
+
+A Python package for time series DataFrame downsampling with LTTB, multiple aggregation methods, gap handling, and fidelity testing.
+
+## Features
+
+- **Multiple downsampling methods**: LTTB (visual fidelity), mean, median, min, max
+- **Gap-aware processing**: Automatically detects and handles gaps in time series
+- **Edge handling**: Flag, discard, or keep edge points
+- **Multi-aggregate output**: Generate min/mean/max columns in a single call
+- **Deferred fetching**: Lazy data loading with automatic edge buffering
+- **Fidelity testing**: Compare methods and measure visual accuracy
+
+## Installation
+
+```bash
+pip install downsampler
+```
+
+Or with visualization support:
+
+```bash
+pip install downsampler[viz]
+```
+
+## Quick Start
+
+### Basic Downsampling
+
+```python
+import pandas as pd
+from downsampler import downsample
+
+# Create sample data
+df = pd.DataFrame(
+    {'temperature': range(1000)},
+    index=pd.date_range('2024-01-01', periods=1000, freq='1s')
+)
+
+# Downsample to 1-minute cadence (default: mean)
+result = downsample(df, target_cadence='PT1M')
+```
+
+### Using Different Methods
+
+```python
+from downsampler import downsample, DownsampleConfig, AggregationMethod
+
+# Mean (default)
+result = downsample(df, '10min')
+
+# Maximum
+result = downsample(df, '10min', method='max')
+
+# LTTB for visual fidelity
+config = DownsampleConfig(
+    method=AggregationMethod.LTTB,
+    lttb_target_column='temperature'
+)
+result = downsample(df, '10min', config=config)
+```
+
+### Multi-Aggregate Downsampling
+
+Create min/mean/max columns for visualization with error bands:
+
+```python
+from downsampler import downsample_multi_aggregate
+
+result = downsample_multi_aggregate(
+    df,
+    target_cadence='1min',
+    variables=['temperature', 'pressure'],
+    aggregations=['min', 'mean', 'max']
+)
+# Result has columns: temperature_min, temperature_mean, temperature_max, etc.
+```
+
+### Handling Gaps
+
+```python
+from downsampler import DownsampleConfig, GapHandling
+
+config = DownsampleConfig(
+    gap_handling=GapHandling.SEGMENT,  # Process segments independently
+    gap_threshold='5min'  # Gaps > 5 min trigger segmentation
+)
+result = downsample(df, '1min', config=config)
+```
+
+### Deferred Data Fetching
+
+For data that needs to be fetched from an external source:
+
+```python
+from downsampler.deferred import deferred_downsample
+
+def fetch_from_api(start, end):
+    # Your data fetching logic here
+    return pd.DataFrame(...)
+
+result = deferred_downsample(
+    fetcher=fetch_from_api,
+    output_start=pd.Timestamp('2024-01-01'),
+    output_end=pd.Timestamp('2024-01-02'),
+    target_cadence='1H'
+)
+```
+
+### Fidelity Comparison
+
+Compare different methods to find the best one for your data:
+
+```python
+from downsampler.fidelity import FidelityComparison
+
+comp = FidelityComparison(original_df, 'signal')
+results = comp.compare_methods('10s')
+
+for r in results:
+    print(f"{r.method.value}: RMSE={r.metrics.rmse:.4f}")
+
+# Get recommendation
+config = comp.recommend_settings('10s', priority='visual')
+```
+
+## Configuration Options
+
+### DownsampleConfig
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `method` | AggregationMethod | MEAN | Downsampling method |
+| `lttb_target_column` | str | None | Column to optimize for LTTB |
+| `include_columns` | list[str] | [] | Columns to include (empty = all) |
+| `exclude_columns` | list[str] | [] | Columns to exclude |
+| `gap_handling` | GapHandling | SEGMENT | How to handle gaps |
+| `gap_threshold` | str/Timedelta | "auto" | Min duration for gaps |
+| `edge_handling` | EdgeHandling | FLAG | How to handle edges |
+| `edge_window` | int | 2 | Points at each edge |
+| `min_points_per_segment` | int | 3 | Min points for processing |
+
+### Aggregation Methods
+
+- `MEAN`: Arithmetic mean (best for general use)
+- `MEDIAN`: Median (robust to outliers)
+- `MIN`: Minimum value (preserves lows)
+- `MAX`: Maximum value (preserves highs)
+- `LTTB`: Largest Triangle Three Buckets (best visual fidelity)
+
+### Gap Handling
+
+- `SEGMENT`: Split at gaps, process independently (recommended)
+- `INTERPOLATE`: Fill gaps before processing
+- `IGNORE`: Treat as continuous data
+
+### Edge Handling
+
+- `KEEP`: Keep edge points as-is
+- `FLAG`: Add `_is_edge` column
+- `DISCARD`: Remove edge points
+
+## Examples
+
+See the `examples/` directory for complete examples:
+
+- `basic_downsampling.py`: Core downsampling features
+- `multi_aggregate.py`: Creating min/mean/max columns
+- `deferred_fetch.py`: Lazy data loading
+- `fidelity_comparison.marimo.py`: Interactive comparison notebook
+
+## API Reference
+
+### Core Functions
+
+```python
+downsample(df, target_cadence, config=None, **kwargs) -> DataFrame
+downsample_multi_aggregate(df, target_cadence, variables, aggregations, ...) -> DataFrame
+```
+
+### Gap Functions
+
+```python
+find_gap_indices(df, timedelta_max_gap) -> Series
+groupby_gaps(df, timedelta_max_gap) -> DataFrameGroupBy
+split_at_gaps(df, timedelta_max_gap) -> list[DataFrame]
+mark_gaps_in_dataframe(df, nominal_timedelta, ...) -> DataFrame
+```
+
+### Deferred Functions
+
+```python
+deferred_downsample(fetcher, output_start, output_end, target_cadence, ...) -> DataFrame
+batch_deferred_downsample(fetcher, ..., batch_size) -> DataFrame
+LazyDownsampler(fetcher, cache_buffer) -> LazyDownsampler
+```
+
+### Fidelity Functions
+
+```python
+compute_metrics(original, downsampled, column) -> FidelityMetrics
+FidelityComparison(original_df, column) -> FidelityComparison
+plot_comparison(original, downsampled, column, backend) -> Figure
+```
+
+## License
+
+MIT License - see LICENSE file for details.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit issues and pull requests.

downsampler-0.1.0/README.md

@@ -0,0 +1,212 @@
+# downsampler
+
+A Python package for time series DataFrame downsampling with LTTB, multiple aggregation methods, gap handling, and fidelity testing.
+
+## Features
+
+- **Multiple downsampling methods**: LTTB (visual fidelity), mean, median, min, max
+- **Gap-aware processing**: Automatically detects and handles gaps in time series
+- **Edge handling**: Flag, discard, or keep edge points
+- **Multi-aggregate output**: Generate min/mean/max columns in a single call
+- **Deferred fetching**: Lazy data loading with automatic edge buffering
+- **Fidelity testing**: Compare methods and measure visual accuracy
+
+## Installation
+
+```bash
+pip install downsampler
+```
+
+Or with visualization support:
+
+```bash
+pip install downsampler[viz]
+```
+
+## Quick Start
+
+### Basic Downsampling
+
+```python
+import pandas as pd
+from downsampler import downsample
+
+# Create sample data
+df = pd.DataFrame(
+    {'temperature': range(1000)},
+    index=pd.date_range('2024-01-01', periods=1000, freq='1s')
+)
+
+# Downsample to 1-minute cadence (default: mean)
+result = downsample(df, target_cadence='PT1M')
+```
+
+### Using Different Methods
+
+```python
+from downsampler import downsample, DownsampleConfig, AggregationMethod
+
+# Mean (default)
+result = downsample(df, '10min')
+
+# Maximum
+result = downsample(df, '10min', method='max')
+
+# LTTB for visual fidelity
+config = DownsampleConfig(
+    method=AggregationMethod.LTTB,
+    lttb_target_column='temperature'
+)
+result = downsample(df, '10min', config=config)
+```
+
+### Multi-Aggregate Downsampling
+
+Create min/mean/max columns for visualization with error bands:
+
+```python
+from downsampler import downsample_multi_aggregate
+
+result = downsample_multi_aggregate(
+    df,
+    target_cadence='1min',
+    variables=['temperature', 'pressure'],
+    aggregations=['min', 'mean', 'max']
+)
+# Result has columns: temperature_min, temperature_mean, temperature_max, etc.
+```
+
+### Handling Gaps
+
+```python
+from downsampler import DownsampleConfig, GapHandling
+
+config = DownsampleConfig(
+    gap_handling=GapHandling.SEGMENT,  # Process segments independently
+    gap_threshold='5min'  # Gaps > 5 min trigger segmentation
+)
+result = downsample(df, '1min', config=config)
+```
+
+### Deferred Data Fetching
+
+For data that needs to be fetched from an external source:
+
+```python
+from downsampler.deferred import deferred_downsample
+
+def fetch_from_api(start, end):
+    # Your data fetching logic here
+    return pd.DataFrame(...)
+
+result = deferred_downsample(
+    fetcher=fetch_from_api,
+    output_start=pd.Timestamp('2024-01-01'),
+    output_end=pd.Timestamp('2024-01-02'),
+    target_cadence='1H'
+)
+```
+
+### Fidelity Comparison
+
+Compare different methods to find the best one for your data:
+
+```python
+from downsampler.fidelity import FidelityComparison
+
+comp = FidelityComparison(original_df, 'signal')
+results = comp.compare_methods('10s')
+
+for r in results:
+    print(f"{r.method.value}: RMSE={r.metrics.rmse:.4f}")
+
+# Get recommendation
+config = comp.recommend_settings('10s', priority='visual')
+```
+
+## Configuration Options
+
+### DownsampleConfig
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `method` | AggregationMethod | MEAN | Downsampling method |
+| `lttb_target_column` | str | None | Column to optimize for LTTB |
+| `include_columns` | list[str] | [] | Columns to include (empty = all) |
+| `exclude_columns` | list[str] | [] | Columns to exclude |
+| `gap_handling` | GapHandling | SEGMENT | How to handle gaps |
+| `gap_threshold` | str/Timedelta | "auto" | Min duration for gaps |
+| `edge_handling` | EdgeHandling | FLAG | How to handle edges |
+| `edge_window` | int | 2 | Points at each edge |
+| `min_points_per_segment` | int | 3 | Min points for processing |
+
+### Aggregation Methods
+
+- `MEAN`: Arithmetic mean (best for general use)
+- `MEDIAN`: Median (robust to outliers)
+- `MIN`: Minimum value (preserves lows)
+- `MAX`: Maximum value (preserves highs)
+- `LTTB`: Largest Triangle Three Buckets (best visual fidelity)
+
+### Gap Handling
+
+- `SEGMENT`: Split at gaps, process independently (recommended)
+- `INTERPOLATE`: Fill gaps before processing
+- `IGNORE`: Treat as continuous data
+
+### Edge Handling
+
+- `KEEP`: Keep edge points as-is
+- `FLAG`: Add `_is_edge` column
+- `DISCARD`: Remove edge points
+
+## Examples
+
+See the `examples/` directory for complete examples:
+
+- `basic_downsampling.py`: Core downsampling features
+- `multi_aggregate.py`: Creating min/mean/max columns
+- `deferred_fetch.py`: Lazy data loading
+- `fidelity_comparison.marimo.py`: Interactive comparison notebook
+
+## API Reference
+
+### Core Functions
+
+```python
+downsample(df, target_cadence, config=None, **kwargs) -> DataFrame
+downsample_multi_aggregate(df, target_cadence, variables, aggregations, ...) -> DataFrame
+```
+
+### Gap Functions
+
+```python
+find_gap_indices(df, timedelta_max_gap) -> Series
+groupby_gaps(df, timedelta_max_gap) -> DataFrameGroupBy
+split_at_gaps(df, timedelta_max_gap) -> list[DataFrame]
+mark_gaps_in_dataframe(df, nominal_timedelta, ...) -> DataFrame
+```
+
+### Deferred Functions
+
+```python
+deferred_downsample(fetcher, output_start, output_end, target_cadence, ...) -> DataFrame
+batch_deferred_downsample(fetcher, ..., batch_size) -> DataFrame
+LazyDownsampler(fetcher, cache_buffer) -> LazyDownsampler
+```
+
+### Fidelity Functions
+
+```python
+compute_metrics(original, downsampled, column) -> FidelityMetrics
+FidelityComparison(original_df, column) -> FidelityComparison
+plot_comparison(original, downsampled, column, backend) -> Figure
+```
+
+## License
+
+MIT License - see LICENSE file for details.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit issues and pull requests.

downsampler-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "downsampler"
+version = "0.1.0"
+description = "Timeseries DataFrame downsampling with LTTB, aggregation methods, gap handling, and fidelity testing"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.10"
+authors = [
+    {name = "Eelco Doornbos", email = "eelco.doornbos@knmi.nl"}
+]
+keywords = ["timeseries", "downsampling", "lttb", "pandas", "visualization"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering",
+]
+dependencies = [
+    "numpy>=1.20",
+    "pandas>=1.3",
+    "lttb>=0.3",
+    "scipy>=1.7",
+]
+
+[project.optional-dependencies]
+viz = ["matplotlib>=3.5", "altair>=5.0"]
+test = ["pytest>=7.0", "pytest-cov>=4.0"]
+dev = ["downsampler[viz,test]"]
+
+[project.urls]
+Homepage = "https://gitlab.com/KNMI-OSS/spaceweather/downsampler"
+Repository = "https://gitlab.com/KNMI-OSS/spaceweather/downsampler"
+
+[tool.setuptools.packages.find]
+where = ["src"]

downsampler-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

downsampler-0.1.0/src/downsampler/__init__.py

@@ -0,0 +1,80 @@
+"""downsampler - Timeseries DataFrame downsampling with LTTB, aggregation methods, and fidelity testing.
+
+This package provides tools for downsampling time series data in pandas DataFrames,
+with support for:
+- LTTB (Largest Triangle Three Buckets) algorithm for visual fidelity
+- Multiple aggregation methods (mean, median, min, max)
+- Gap-aware processing
+- Edge handling strategies
+- Deferred/lazy data fetching
+- Fidelity testing and comparison
+
+Example:
+    >>> import pandas as pd
+    >>> from downsampler import downsample, DownsampleConfig, AggregationMethod
+    >>>
+    >>> # Create sample data
+    >>> df = pd.DataFrame(
+    ...     {'value': [1, 2, 3, 4, 5]},
+    ...     index=pd.date_range('2024-01-01', periods=5, freq='1min')
+    ... )
+    >>>
+    >>> # Downsample using mean
+    >>> result = downsample(df, target_cadence='5min')
+    >>>
+    >>> # Downsample using LTTB
+    >>> config = DownsampleConfig(
+    ...     method=AggregationMethod.LTTB,
+    ...     lttb_target_column='value'
+    ... )
+    >>> result = downsample(df, target_cadence='5min', config=config)
+"""
+
+from downsampler.config import (
+    AggregationMethod,
+    EdgeHandling,
+    GapHandling,
+    DownsampleConfig,
+)
+from downsampler.core import downsample, downsample_multi_aggregate
+from downsampler.gaps import (
+    find_gap_indices,
+    groupby_gaps,
+    wrap_in_nans,
+    mark_gaps_in_dataframe,
+)
+from downsampler.lttb import downsample_lttb
+from downsampler.aggregators import (
+    downsample_mean,
+    downsample_median,
+    downsample_min,
+    downsample_max,
+)
+from downsampler.deferred import deferred_downsample
+
+__version__ = "0.1.0"
+
+__all__ = [
+    # Config
+    "AggregationMethod",
+    "EdgeHandling",
+    "GapHandling",
+    "DownsampleConfig",
+    # Core
+    "downsample",
+    "downsample_multi_aggregate",
+    # Gaps
+    "find_gap_indices",
+    "groupby_gaps",
+    "wrap_in_nans",
+    "mark_gaps_in_dataframe",
+    # LTTB
+    "downsample_lttb",
+    # Aggregators
+    "downsample_mean",
+    "downsample_median",
+    "downsample_min",
+    "downsample_max",
+    # Deferred
+    "deferred_downsample",
+]