metdatapy 1.0.0__tar.gz

metdatapy-1.0.0/.coveragerc

@@ -0,0 +1,29 @@
+[run]
+source = metdatapy
+omit =
+    */tests/*
+    */test_*.py
+    */__pycache__/*
+    */site-packages/*
+
+[report]
+precision = 2
+show_missing = True
+skip_covered = False
+
+exclude_lines =
+    pragma: no cover
+    def __repr__
+    raise AssertionError
+    raise NotImplementedError
+    if __name__ == .__main__.:
+    if TYPE_CHECKING:
+    @abstractmethod
+    @abc.abstractmethod
+
+[html]
+directory = htmlcov
+
+[xml]
+output = coverage.xml
+

metdatapy-1.0.0/CITATION.cff

@@ -0,0 +1,28 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+type: software
+title: "MetDataPy: A Source-Agnostic Toolkit for Meteorological Time-Series Data"
+abstract: >
+  MetDataPy is a Python package for ingesting, normalizing, quality-controlling,
+  and preparing meteorological time-series data for machine learning applications.
+  It provides a unified canonical schema, robust quality control algorithms,
+  derived meteorological metrics, and ML-ready feature engineering with time-safe
+  data splitting and scaling.
+authors:
+  - family-names: "Kartas"
+    given-names: "Kyriakos"
+    orcid: "https://orcid.org/0009-0001-6477-4676"
+repository-code: "https://github.com/kkartas/MetDataPy"
+url: "https://github.com/kkartas/MetDataPy"
+keywords:
+  - meteorology
+  - weather data
+  - time series
+  - quality control
+  - machine learning
+  - data preprocessing
+  - climate science
+  - atmospheric science
+license: MIT
+version: 1.0.0
+date-released: "2025-10-25"

metdatapy-1.0.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,35 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment include:
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+
+Examples of unacceptable behavior include:
+- The use of sexualized language or imagery and unwelcome sexual attention or advances
+- Trolling, insulting/derogatory comments, and personal or political attacks
+- Public or private harassment
+- Other conduct which could reasonably be considered inappropriate
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying standards and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+## Scope
+
+This Code of Conduct applies within all project spaces and in public spaces when an individual is representing the project or its community.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting the maintainers. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances.
+
+## Attribution
+
+This Code of Conduct is adapted from the Contributor Covenant, version 2.1.

metdatapy-1.0.0/CONTRIBUTING.md

@@ -0,0 +1,60 @@
+# Contributing to MetDataPy
+
+Thank you for your interest in contributing! This guide explains how to set up your environment, propose changes, and follow project conventions.
+
+## Getting started
+
+- Fork the repo and create a feature branch from `main`.
+- Use recent Python (>=3.9). Install in editable mode:
+  ```bash
+  python -m venv .venv && source .venv/bin/activate  # Windows: .venv\Scripts\activate
+  python -m pip install -U pip
+  python -m pip install -e .
+  python -m pip install pytest mkdocs mkdocs-material
+  ```
+
+**Note:** For regular users, MetDataPy is available on PyPI: `pip install metdatapy`. This development setup is only needed for contributors.
+
+## Running tests and docs
+
+- Tests:
+  ```bash
+  python -m pytest -q
+  ```
+- Docs (MkDocs):
+  ```bash
+  mkdocs serve   # live preview
+  mkdocs build   # static site in site/
+  ```
+
+## Coding standards
+
+- Write clear, readable, and typed Python where reasonable.
+- Prefer vectorized `pandas`/`numpy` operations; avoid unnecessary loops.
+- Keep public APIs stable; add deprecations with care.
+- Lint/formatting enabled in CI: ruff, black, isort; type checking with mypy.
+- Match existing code style and avoid unrelated reformatting.
+
+## Pull requests
+
+- Scope PRs narrowly; include tests and docs for new features.
+- Update `README.md` and relevant pages in `docs/` when behavior changes.
+- Add entries/checkmarks to `ROADMAP.md` when features are implemented.
+- Explain motivation, approach, and any trade-offs in the PR description.
+
+## Issue reporting
+
+- Include minimal reproducible examples, data snippets, versions, and OS.
+- Label issues appropriately (bug, enhancement, docs, question).
+
+## Security and conduct
+
+- Please avoid sharing sensitive data in issues/PRs.
+- Be respectful and professional. See `CODE_OF_CONDUCT.md`.
+
+## Release process (outline)
+
+- Ensure tests/docs pass on CI; bump version in `pyproject.toml`.
+- Tag the release; publish wheels/sdist to PyPI; archive on Zenodo (planned).
+
+Thanks again for contributing!

metdatapy-1.0.0/LICENSE

@@ -0,0 +1,23 @@
+ MIT License
+
+ Copyright (c) 2025 MetDataPy
+
+ Permission is hereby granted, free of charge, to any person obtaining a copy
+ of this software and associated documentation files (the "Software"), to deal
+ in the Software without restriction, including without limitation the rights
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the Software is
+ furnished to do so, subject to the following conditions:
+
+ The above copyright notice and this permission notice shall be included in all
+ copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ SOFTWARE.
+
+

metdatapy-1.0.0/MANIFEST.in

@@ -0,0 +1,38 @@
+include README.md
+include LICENSE
+include CITATION.cff
+include CONTRIBUTING.md
+include CODE_OF_CONDUCT.md
+include pyproject.toml
+include .coveragerc
+
+recursive-include metdatapy *.py
+recursive-include tests *.py
+recursive-include docs *.md
+
+# Exclude data directory from package (used for examples only)
+prune data
+prune examples
+
+exclude .gitignore
+exclude TO-DO.md
+exclude mkdocs.yml
+exclude .github
+
+recursive-exclude * __pycache__
+recursive-exclude * *.py[co]
+recursive-exclude * *.pyc
+recursive-exclude * *.pyo
+recursive-exclude * .DS_Store
+recursive-exclude * *.so
+recursive-exclude * *.dylib
+prune site
+prune htmlcov
+prune .pytest_cache
+prune .mypy_cache
+prune .ruff_cache
+prune .coverage
+prune dist
+prune build
+prune *.egg-info
+

metdatapy-1.0.0/PKG-INFO

@@ -0,0 +1,285 @@
+Metadata-Version: 2.4
+Name: metdatapy
+Version: 1.0.0
+Summary: Source-agnostic toolkit for ingesting, cleaning, QC, and preparing meteorological time-series data for ML.
+Author: Kyriakos Kartas
+License: MIT
+Project-URL: homepage, https://github.com/kkartas/MetDataPy
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=2.0
+Requires-Dist: numpy>=1.23
+Requires-Dist: pyarrow>=13.0
+Requires-Dist: click>=8.1
+Requires-Dist: pydantic>=2.4
+Requires-Dist: PyYAML>=6.0
+Provides-Extra: ml
+Requires-Dist: scikit-learn>=1.2; extra == "ml"
+Requires-Dist: statsmodels>=0.13; extra == "ml"
+Provides-Extra: netcdf
+Requires-Dist: xarray>=2023.6.0; extra == "netcdf"
+Requires-Dist: netCDF4>=1.6; extra == "netcdf"
+Requires-Dist: cftime>=1.6; extra == "netcdf"
+Provides-Extra: extras
+Requires-Dist: astral>=3.2; extra == "extras"
+Requires-Dist: holidays>=0.36; extra == "extras"
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.5; extra == "viz"
+Requires-Dist: seaborn>=0.12; extra == "viz"
+Provides-Extra: all
+Requires-Dist: scikit-learn>=1.2; extra == "all"
+Requires-Dist: statsmodels>=0.13; extra == "all"
+Requires-Dist: xarray>=2023.6.0; extra == "all"
+Requires-Dist: netCDF4>=1.6; extra == "all"
+Requires-Dist: cftime>=1.6; extra == "all"
+Requires-Dist: astral>=3.2; extra == "all"
+Requires-Dist: holidays>=0.36; extra == "all"
+Requires-Dist: matplotlib>=3.5; extra == "all"
+Requires-Dist: seaborn>=0.12; extra == "all"
+Dynamic: license-file
+
+# MetDataPy
+
+[![PyPI version](https://img.shields.io/pypi/v/MetDataPy?style=flat)](https://pypi.org/project/MetDataPy/)
+[![CI](https://github.com/kkartas/MetDataPy/actions/workflows/ci.yml/badge.svg)](https://github.com/kkartas/MetDataPy/actions/workflows/ci.yml)
+[![Documentation Status](https://readthedocs.org/projects/metdatapy/badge/?version=latest)](https://metdatapy.readthedocs.io/en/latest/?badge=latest)
+[![codecov](https://codecov.io/gh/kkartas/MetDataPy/branch/main/graph/badge.svg)](https://codecov.io/gh/kkartas/MetDataPy)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Source-agnostic toolkit for ingesting, cleaning, QC-flagging, enriching, and preparing meteorological time-series data for machine learning.
+
+## Statement of Need
+
+Modern ML pipelines require clean, unit-consistent, well-flagged meteorological time series. MetDataPy provides a canonical schema, robust ingestion (with autodetection and an interactive mapping wizard), quality control, derived metrics, time-safe ML preparation, and reproducible exports.
+
+## Quickstart
+
+```bash
+# Install MetDataPy
+pip install metdatapy
+
+# Detect column mappings
+mdp ingest detect --csv path/to/file.csv --save mapping.yml
+
+# Apply mapping and ingest data
+mdp ingest apply --csv path/to/file.csv --map mapping.yml --out raw.parquet
+
+# Run quality control
+mdp qc run --in raw.parquet --out clean.parquet --report qc_report.json \
+  --config qc_config.yml
+```
+
+For detailed installation options (including optional features), see the [Installation](#installation) section below.
+
+## Installation
+
+### Basic Installation
+
+```bash
+pip install metdatapy
+```
+
+This installs MetDataPy with core dependencies only. The core package is compatible with both NumPy 1.x and 2.x.
+
+### Installation with Optional Features
+
+```bash
+# For machine learning features
+pip install "metdatapy[ml]"
+
+# For NetCDF export functionality
+pip install "metdatapy[netcdf]"
+
+# For visualization (examples/notebooks)
+pip install "metdatapy[viz]"
+
+# For all optional features
+pip install "metdatapy[all]"
+
+# Or combine specific features
+pip install "metdatapy[ml,netcdf]"
+```
+
+### Development Installation
+
+For developers or contributors who want to install from source:
+
+```bash
+git clone https://github.com/kkartas/MetDataPy.git
+cd MetDataPy
+pip install -e .
+```
+
+### Requirements
+
+**Python:** 3.9+
+
+**Core dependencies:** pandas ≥2.0, numpy ≥1.23, pyarrow ≥13.0, click ≥8.1, pydantic ≥2.4, PyYAML ≥6.0
+
+**Optional dependencies:**
+- ML: scikit-learn ≥1.2, statsmodels ≥0.13
+- NetCDF: xarray ≥2023.6.0, netCDF4 ≥1.6, cftime ≥1.6
+- Visualization: matplotlib ≥3.5, seaborn ≥0.12
+- Extras: astral ≥3.2, holidays ≥0.36
+
+### NumPy 2.x Compatibility
+
+**Core MetDataPy package:** Fully compatible with both NumPy 1.x and 2.x. All functionality works with either version.
+
+**Visualization dependencies:** Some visualization packages (matplotlib, seaborn) may have compatibility issues with NumPy 2.x on certain platforms. If you encounter errors like:
+
+```
+A module that was compiled using NumPy 1.x cannot be run in NumPy 2.x
+```
+
+**Solutions:**
+1. **For core usage** (data processing, QC, ML prep): No action needed - works with any NumPy version
+2. **For visualization** (running examples with plots):
+   ```bash
+   pip install 'numpy<2.0' matplotlib seaborn
+   ```
+3. **Alternative**: Wait for matplotlib/seaborn to release NumPy 2.x compatible builds
+
+**Note:** This issue only affects optional visualization features. The core MetDataPy functionality (ingestion, QC, derived metrics, ML preparation, NetCDF export) works perfectly with NumPy 2.x.
+
+## Documentation
+
+Full documentation is available on **[Read the Docs](https://metdatapy.readthedocs.io/)**.
+
+To build documentation locally:
+```bash
+pip install metdatapy[all]
+pip install mkdocs mkdocs-material
+mkdocs serve
+# Then open http://localhost:8000
+```
+
+Features
+- Canonical schema with UTC index and metric units
+- Ingestion from CSV with mapping autodetection and interactive mapping wizard
+- Unit normalization, rain accumulation fix-up, gap insertion with `gap` flag
+- WeatherSet resampling/aggregation, calendar features, exogenous joins
+- Derived: dew point, VPD, heat index, wind chill
+- ML prep: supervised table builder (lags, horizons), time-safe split, scaling (Standard/MinMax/Robust)
+- Export: Parquet and CF-compliant NetCDF with metadata
+- **Performance:** Processes 1 year of 10-min data in <0.5s (see `benchmarks/`)
+
+Quality Control
+- Range checks with boolean flags (`qc_<var>_range`)
+- Spike detection (rolling MAD z-score) and flatline detection (rolling variance)
+- Cross-variable consistency checks with aggregate `qc_any`
+- CLI supports a config file for thresholds:
+
+```bash
+mdp qc run --in raw.parquet --out clean.parquet \
+  --config qc_config.yml --report qc_report.json
+```
+
+Example `qc_config.yml`:
+```yaml
+spike:
+  window: 9
+  thresh: 6.0
+flatline:
+  window: 5
+  tol: 0.0
+```
+
+Python API example
+```python
+import pandas as pd
+from metdatapy.mapper import Mapper
+from metdatapy.core import WeatherSet
+from metdatapy.mlprep import make_supervised, time_split, fit_scaler, apply_scaler
+
+mapping = Mapper.load("mapping.yml")
+df = pd.read_csv("path/to/file.csv")
+ws = WeatherSet.from_mapping(df, mapping).to_utc().normalize_units(mapping)
+ws = ws.insert_missing().fix_accum_rain().qc_range().qc_spike().qc_flatline().qc_consistency()
+ws = ws.derive(["dew_point", "vpd", "heat_index", "wind_chill"]).resample("1H").calendar_features()
+clean = ws.to_dataframe()
+
+# Export to CF-compliant NetCDF
+ws.to_netcdf("weather_data.nc", metadata={"title": "Weather Station Data"}, 
+             station_metadata={"station_id": "AWS001", "lat": 40.7, "lon": -74.0})
+
+sup = make_supervised(clean, targets=["temp_c"], horizons=[1,3], lags=[1,2,3])
+splits = time_split(sup, train_end=pd.Timestamp("2025-01-15T00:00Z"))
+scaler = fit_scaler(splits["train"], method="standard")
+train_scaled = apply_scaler(splits["train"], scaler)
+```
+
+## Examples
+
+See the `examples/` directory for:
+
+**Jupyter Notebook:**
+- **`metdatapy_tutorial.ipynb`** - Publication-quality interactive tutorial  
+  [![GitHub](https://img.shields.io/badge/view-GitHub-181717?logo=github)](https://github.com/kkartas/MetDataPy/blob/main/examples/metdatapy_tutorial.ipynb)
+  [![nbviewer](https://img.shields.io/badge/render-nbviewer-orange?logo=jupyter)](https://nbviewer.org/github/kkartas/MetDataPy/blob/main/examples/metdatapy_tutorial.ipynb)
+  [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/kkartas/MetDataPy/main?filepath=examples/metdatapy_tutorial.ipynb)
+  
+  Comprehensive tutorial with:
+  - Step-by-step workflow with scientific references
+  - Publication-ready visualizations (QC flags, derived metrics)
+  - Mathematical formulas and physical validation
+  - Complete reproducible pipeline
+
+**Python Scripts:**
+- **`complete_workflow.py`** - Automated batch processing script
+- **`netcdf_export_example.py`** - CF-compliant NetCDF export demonstration
+
+**Additional Resources:**
+- **`README.md`** - Detailed usage guide
+- **Sample weather data** - `data/sample_weather_2024.csv` contains a full year (2024) of synthetic 10-minute weather station data (52,561 records) with realistic meteorological patterns. This dataset includes temperature (°F), relative humidity (%), pressure (mbar), wind speed/direction (mph/degrees), rainfall (mm), solar radiation (W/m²), and UV index. The data is used in all examples and can be used to test the full MetDataPy workflow.
+
+**Try the notebooks:**
+- 📁 **View on GitHub** - Click GitHub links above for native rendering (works immediately)
+- 🔍 **View on nbviewer** - Better rendering with MathJax support
+  - *Note: nbviewer may show 400 errors for new/recently updated files due to caching. If this happens, use GitHub view or try again in a few minutes.*
+- 🚀 **Run interactively** - Click Binder badge for live Jupyter environment (takes ~2 min to launch)
+
+**Or run locally:**
+```bash
+# Install MetDataPy with all optional features
+pip install metdatapy[all]
+
+# Clone the repository for examples
+git clone https://github.com/kkartas/MetDataPy.git
+cd MetDataPy/examples
+jupyter notebook metdatapy_tutorial.ipynb
+```
+
+**Automated workflow:**
+```bash
+# Install MetDataPy with all optional features
+pip install metdatapy[all]
+
+# Clone the repository for examples
+git clone https://github.com/kkartas/MetDataPy.git
+cd MetDataPy/examples
+python complete_workflow.py
+```
+
+## Citation
+
+If you use MetDataPy in your research, please cite it:
+
+```bibtex
+@software{metdatapy,
+  title = {MetDataPy: A Source-Agnostic Toolkit for Meteorological Time-Series Data},
+  author = {Kyriakos Kartas},
+  year = {2025},
+  url = {https://github.com/kkartas/MetDataPy},
+  version = {1.0.0}
+}
+```
+
+See `CITATION.cff` for machine-readable citation metadata.
+
+## License
+
+MIT License. See `LICENSE` for details.
+