aemo-mdff-reader 2.2.1__tar.gz → 4.0.1__tar.gz

{aemo_mdff_reader-2.2.1 → aemo_mdff_reader-4.0.1}/CHANGELOG.md

@@ -4,6 +4,80 @@ All notable changes are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
 project follows [Semantic Versioning](https://semver.org/).
 
+## [4.0.1](https://github.com/Utilified/aemo-mdff-reader/compare/v4.0.0...v4.0.1) (2026-11-02)
+
+
+### Bug Fixes
+
+* **deps:** bump certifi to 2026.10.19 for PYSEC-2026-4320 ([#145](https://github.com/Utilified/aemo-mdff-reader/issues/145)) ([7b66f2a](https://github.com/Utilified/aemo-mdff-reader/commit/7b66f2a4cd2b5e333c5258dcfd41277909c9ac62))
+* **deps:** bump msgpack to 1.2.4 for CVE-2026-114613 ([#148](https://github.com/Utilified/aemo-mdff-reader/issues/148)) ([06eb1ac](https://github.com/Utilified/aemo-mdff-reader/commit/06eb1ace6e8a94d4a87cad33211713b917a946e0))
+
+## [4.0.0](https://github.com/Utilified/aemo-mdff-reader/compare/v3.0.1...v4.0.0) (2026-09-01)
+
+
+### ⚠ BREAKING CHANGES
+
+* interval and accumulation reading values are now `float | None`. An
+  empty cell parses to `None` (missing) instead of `0.0` — a missing
+  reading must not be indistinguishable from a genuine zero in a
+  billing input. This affects `IntervalReading.value`, the `Value`
+  column in `to_columns` / `parse_to_columns` / the pandas and parquet
+  paths, and NEM13 `previous_register_read`, `current_register_read`
+  and `quantity`. `aggregate.daily_totals` skips `None` readings rather
+  than summing them as zero, excludes them from `interval_count`, and
+  reports them in the new `DailyTotal.missing_count` field.
+* the `aemo_mdff_reader.sql` subpackage and the `[mysql]` extra are
+  removed, along with `QueryBuilder`, `COLUMNS`, `Storer` and
+  `DBCredentials`. The subpackage referenced `NEMReader` attributes that
+  have not existed since v2, so every entry point raised
+  `AttributeError` on call, and its query builder interpolated
+  file-derived values into SQL unescaped. There is no replacement:
+  persist via `to_columns()` / `iter_columns_chunks()` and your own
+  parameterised inserts.
+* files that previously parsed silently now raise `NEM12ParseError`: a
+  300 row carrying more interval values than its parent 200 record's
+  IntervalLength allows, a non-numeric interval value or reason code, an
+  impossible date, and a 400 row whose StartInterval / EndInterval falls
+  outside the NMI's interval count. Trailing empty cells on a 300 row
+  (Excel padding) are tolerated and do not count against the field
+  bound.
+
+### Bug Fixes
+
+* raise on over-long 300 rows instead of silently truncating them. A
+  15-minute file under a stale 30-minute 200 header emitted 48 readings,
+  dropped the remaining 48, and took `quality_method` from a dropped
+  *value*. `validate_file` now checks 300 field counts too — it
+  previously returned clean for a 30-minute NMI carrying 288 values.
+* wrap the remaining raw `ValueError`s at the parse boundary as
+  `NEM12ParseError` (`_parse_int`, `_parse_float`, the columnar value
+  path, the 400 record's interval bounds, and the fixed-width
+  `datetime()` construction), each naming the offending field and value.
+* bound-check 400 StartInterval / EndInterval against the NMI's
+  intervals per day — `400,1,999,S` previously parsed against a
+  48-interval day.
+* correct the `spec.DIRECTION_INDICATORS` descriptions, which had Import
+  and Export the wrong way round.
+
+## [3.0.1](https://github.com/Utilified/aemo-mdff-reader/compare/v3.0.0...v3.0.1) (2026-07-18)
+
+
+### Bug Fixes
+
+* **fuzz:** install with --ignore-requires-python on the 3.11 OSS-Fuzz image ([#75](https://github.com/Utilified/aemo-mdff-reader/issues/75)) ([28605fe](https://github.com/Utilified/aemo-mdff-reader/commit/28605fe6c408891656bf106d7facdab1afe7dd8e))
+* tolerate unparseable 100-record DateTime instead of failing the file ([#73](https://github.com/Utilified/aemo-mdff-reader/issues/73)) ([ec84dd9](https://github.com/Utilified/aemo-mdff-reader/commit/ec84dd9a42c8c92445b0f6ef6fab7a1417ece220)), closes [#72](https://github.com/Utilified/aemo-mdff-reader/issues/72)
+
+## [3.0.0](https://github.com/Utilified/aemo-mdff-reader/compare/v2.2.1...v3.0.0) (2026-07-16)
+
+
+### ⚠ BREAKING CHANGES
+
+* minimum Python version is now 3.12. Users on 3.11 should pin to aemo-mdff-reader<2.3.
+
+### Features
+
+* drop Python 3.11 support ([#70](https://github.com/Utilified/aemo-mdff-reader/issues/70)) ([383ff6d](https://github.com/Utilified/aemo-mdff-reader/commit/383ff6de7430372a8a67bd1b1a8271aa849f53eb))
+
 ## [2.2.1](https://github.com/Utilified/aemo-mdff-reader/compare/v2.2.0...v2.2.1) (2026-05-10)
 
 

{aemo_mdff_reader-2.2.1 → aemo_mdff_reader-4.0.1}/CONTRIBUTING.md

@@ -36,16 +36,16 @@ To regenerate manually after editing them or `pyproject.toml`:
 
 ```bash
 pip install uv
-uv pip compile --generate-hashes --strip-extras --python-version 3.11 --output-file requirements/dev.txt   requirements/dev.in
-uv pip compile --generate-hashes --strip-extras --python-version 3.11 --output-file requirements/lint.txt  requirements/lint.in
-uv pip compile --generate-hashes --strip-extras --python-version 3.11 --output-file requirements/build.txt requirements/build.in
-uv pip compile --generate-hashes --strip-extras --python-version 3.11 --output-file requirements/audit.txt requirements/audit.in
+uv pip compile --generate-hashes --strip-extras --python-version 3.12 --output-file requirements/dev.txt   requirements/dev.in
+uv pip compile --generate-hashes --strip-extras --python-version 3.12 --output-file requirements/lint.txt  requirements/lint.in
+uv pip compile --generate-hashes --strip-extras --python-version 3.12 --output-file requirements/build.txt requirements/build.in
+uv pip compile --generate-hashes --strip-extras --python-version 3.12 --output-file requirements/audit.txt requirements/audit.in
 ```
 
 `uv` is used over `pip-tools` because its resolver handles the
-3.11–3.12 matrix without back-tracking failures, and it emits hashes
+3.12+ matrix without back-tracking failures, and it emits hashes
 for every wheel of each pinned version (so a single lock file works
-across the whole CI matrix). The `--python-version 3.11` flag pins
+across the whole CI matrix). The `--python-version 3.12` flag pins
 resolution to the lowest supported interpreter so packages whose
 latest releases dropped support for it are correctly stepped back
 to a compatible version.

{aemo_mdff_reader-2.2.1/aemo_mdff_reader.egg-info → aemo_mdff_reader-4.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aemo-mdff-reader
-Version: 2.2.1
+Version: 4.0.1
 Summary: Fast streaming reader for AEMO MDFF NEM12 / NEM13 metering files. Zero required dependencies.
 Author: Utilified Holdings Pty Ltd
 License-Expression: MIT
@@ -15,13 +15,12 @@ Classifier: Intended Audience :: Information Technology
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Office/Business :: Financial
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Utilities
-Requires-Python: >=3.11
+Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Provides-Extra: pandas
@@ -29,8 +28,6 @@ Requires-Dist: pandas>=2; extra == "pandas"
 Provides-Extra: parquet
 Requires-Dist: pandas>=2; extra == "parquet"
 Requires-Dist: pyarrow>=10; extra == "parquet"
-Provides-Extra: mysql
-Requires-Dist: PyMySQL>=1.0; extra == "mysql"
 Provides-Extra: dev
 Requires-Dist: pytest>=7; extra == "dev"
 Requires-Dist: pytest-cov>=4; extra == "dev"
@@ -47,7 +44,7 @@ Dynamic: license-file
 
 [![CI](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/ci.yml/badge.svg)](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/ci.yml)
 [![PyPI](https://img.shields.io/pypi/v/aemo-mdff-reader.svg)](https://pypi.org/project/aemo-mdff-reader/)
-[![Python versions](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://pypi.org/project/aemo-mdff-reader/)
+[![Python versions](https://img.shields.io/badge/python-3.12%20%7C%203.13-blue)](https://pypi.org/project/aemo-mdff-reader/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![CodeQL](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/codeql.yml/badge.svg)](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/codeql.yml)
 [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/Utilified/aemo-mdff-reader/badge)](https://scorecard.dev/viewer/?uri=github.com/Utilified/aemo-mdff-reader)
@@ -56,7 +53,7 @@ Fast, zero-dependency streaming reader for AEMO **NEM12** and **NEM13**
 metering files. Implements AEMO MDFF (Meter Data File Format) v2.6.
 
 - O(1) memory — iterate through millions of intervals.
-- Pure stdlib core; pandas / PyMySQL are opt-in extras.
+- Pure stdlib core; pandas / pyarrow are opt-in extras.
 - ~2 M readings/sec on the columnar fast path.
 - Includes an `aemo-mdff-reader` CLI.
 
@@ -66,8 +63,8 @@ metering files. Implements AEMO MDFF (Meter Data File Format) v2.6.
 pip install aemo-mdff-reader
 
 # optional extras
-pip install aemo-mdff-reader[pandas]   # to_dataframe() / parquet
-pip install aemo-mdff-reader[mysql]    # SQL persistence
+pip install aemo-mdff-reader[pandas]   # to_dataframe()
+pip install aemo-mdff-reader[parquet]  # parquet output (pandas + pyarrow)
 ```
 
 ## Use
@@ -84,8 +81,8 @@ Or as a flat CSV / DataFrame:
 ```python
 from aemo_mdff_reader import parse, write_csv, to_dataframe
 
-write_csv(parse("metering.csv"), "out.csv")     # no pandas
-df = to_dataframe("metering.csv")                # needs [pandas]
+write_csv(parse("metering.csv"), "out.csv")  # no pandas
+df = to_dataframe("metering.csv")  # needs [pandas]
 ```
 
 From the command line:
@@ -104,7 +101,7 @@ Each parsed record is a slots-based class with named attributes plus a
 
 ```python
 for r in parse("metering.csv"):
-    payload = r.to_dict()              # {"nmi": "...", "value": 0.12, ...}
+    payload = r.to_dict()  # {"nmi": "...", "value": 0.12, ...}
     print(r.quality_flag, r.method_flag)  # split of the QMM field, e.g. "S", "52"
 ```
 
@@ -156,6 +153,13 @@ Each `parse(...)` yields an `IntervalReading` with `nmi`,
 the type stubs (`from aemo_mdff_reader import IntervalReading`) for the
 exact signatures.
 
+`value` is `float | None`: an empty cell in the source file is a
+*missing* reading and stays `None` rather than becoming `0.0`, which
+would be indistinguishable from a genuine zero. The same holds for the
+NEM13 `previous_register_read`, `current_register_read` and `quantity`
+fields. `aggregate.daily_totals` skips missing readings and reports how
+many it saw in `DailyTotal.missing_count`.
+
 ## Notes
 
 - **Spec**: AEMO Meter Data File Format Specification NEM12 & NEM13,
@@ -166,8 +170,8 @@ exact signatures.
   as constants in `aemo_mdff_reader.spec` for callers that want stricter
   validation than the parser performs.
 - **Tolerant**: UTF-8 BOM is consumed silently, LF and CRLF both work,
-  and empty interval cells are coerced to `0.0` (use `quality_method`
-  to distinguish missing from zero). Datetime fields accept the spec
+  and empty interval cells parse to `None` (missing), never `0.0`.
+  Datetime fields accept the spec
   forms (`YYYYMMDD`, `YYYYMMDDhhmmss`) and a few common non-spec
   variants (`YYYY-MM-DD`, ISO `YYYY-MM-DDTHH:MM:SS`, with or without a
   `Z` / `±HH:MM` / `±HHMM` timezone suffix — the suffix is stripped
@@ -237,7 +241,7 @@ pip install -e .[dev]
 pytest
 ```
 
-CI runs ruff, mypy --strict, the test matrix on Python 3.11 → 3.12 /
+CI runs ruff, mypy --strict, the test matrix on Python 3.12 /
 Linux / macOS / Windows, `pip-audit`, `bandit`, CodeQL, OpenSSF
 Scorecard, and a wheel-install smoke test.
 

{aemo_mdff_reader-2.2.1 → aemo_mdff_reader-4.0.1}/README.md

@@ -2,7 +2,7 @@
 
 [![CI](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/ci.yml/badge.svg)](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/ci.yml)
 [![PyPI](https://img.shields.io/pypi/v/aemo-mdff-reader.svg)](https://pypi.org/project/aemo-mdff-reader/)
-[![Python versions](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)](https://pypi.org/project/aemo-mdff-reader/)
+[![Python versions](https://img.shields.io/badge/python-3.12%20%7C%203.13-blue)](https://pypi.org/project/aemo-mdff-reader/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![CodeQL](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/codeql.yml/badge.svg)](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/codeql.yml)
 [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/Utilified/aemo-mdff-reader/badge)](https://scorecard.dev/viewer/?uri=github.com/Utilified/aemo-mdff-reader)
@@ -11,7 +11,7 @@ Fast, zero-dependency streaming reader for AEMO **NEM12** and **NEM13**
 metering files. Implements AEMO MDFF (Meter Data File Format) v2.6.
 
 - O(1) memory — iterate through millions of intervals.
-- Pure stdlib core; pandas / PyMySQL are opt-in extras.
+- Pure stdlib core; pandas / pyarrow are opt-in extras.
 - ~2 M readings/sec on the columnar fast path.
 - Includes an `aemo-mdff-reader` CLI.
 
@@ -21,8 +21,8 @@ metering files. Implements AEMO MDFF (Meter Data File Format) v2.6.
 pip install aemo-mdff-reader
 
 # optional extras
-pip install aemo-mdff-reader[pandas]   # to_dataframe() / parquet
-pip install aemo-mdff-reader[mysql]    # SQL persistence
+pip install aemo-mdff-reader[pandas]   # to_dataframe()
+pip install aemo-mdff-reader[parquet]  # parquet output (pandas + pyarrow)
 ```
 
 ## Use
@@ -39,8 +39,8 @@ Or as a flat CSV / DataFrame:
 ```python
 from aemo_mdff_reader import parse, write_csv, to_dataframe
 
-write_csv(parse("metering.csv"), "out.csv")     # no pandas
-df = to_dataframe("metering.csv")                # needs [pandas]
+write_csv(parse("metering.csv"), "out.csv")  # no pandas
+df = to_dataframe("metering.csv")  # needs [pandas]
 ```
 
 From the command line:
@@ -59,7 +59,7 @@ Each parsed record is a slots-based class with named attributes plus a
 
 ```python
 for r in parse("metering.csv"):
-    payload = r.to_dict()              # {"nmi": "...", "value": 0.12, ...}
+    payload = r.to_dict()  # {"nmi": "...", "value": 0.12, ...}
     print(r.quality_flag, r.method_flag)  # split of the QMM field, e.g. "S", "52"
 ```
 
@@ -111,6 +111,13 @@ Each `parse(...)` yields an `IntervalReading` with `nmi`,
 the type stubs (`from aemo_mdff_reader import IntervalReading`) for the
 exact signatures.
 
+`value` is `float | None`: an empty cell in the source file is a
+*missing* reading and stays `None` rather than becoming `0.0`, which
+would be indistinguishable from a genuine zero. The same holds for the
+NEM13 `previous_register_read`, `current_register_read` and `quantity`
+fields. `aggregate.daily_totals` skips missing readings and reports how
+many it saw in `DailyTotal.missing_count`.
+
 ## Notes
 
 - **Spec**: AEMO Meter Data File Format Specification NEM12 & NEM13,
@@ -121,8 +128,8 @@ exact signatures.
   as constants in `aemo_mdff_reader.spec` for callers that want stricter
   validation than the parser performs.
 - **Tolerant**: UTF-8 BOM is consumed silently, LF and CRLF both work,
-  and empty interval cells are coerced to `0.0` (use `quality_method`
-  to distinguish missing from zero). Datetime fields accept the spec
+  and empty interval cells parse to `None` (missing), never `0.0`.
+  Datetime fields accept the spec
   forms (`YYYYMMDD`, `YYYYMMDDhhmmss`) and a few common non-spec
   variants (`YYYY-MM-DD`, ISO `YYYY-MM-DDTHH:MM:SS`, with or without a
   `Z` / `±HH:MM` / `±HHMM` timezone suffix — the suffix is stripped
@@ -192,7 +199,7 @@ pip install -e .[dev]
 pytest
 ```
 
-CI runs ruff, mypy --strict, the test matrix on Python 3.11 → 3.12 /
+CI runs ruff, mypy --strict, the test matrix on Python 3.12 /
 Linux / macOS / Windows, `pip-audit`, `bandit`, CodeQL, OpenSSF
 Scorecard, and a wheel-install smoke test.
 

{aemo_mdff_reader-2.2.1 → aemo_mdff_reader-4.0.1}/aemo_mdff_reader/__init__.py

@@ -48,7 +48,7 @@ from .types import (
     NMIDetails,
 )
 
-__version__ = "2.2.1"  # x-release-please-version
+__version__ = "4.0.1"  # x-release-please-version
 
 __all__ = [
     "INTERVAL_DATA_OUTPUT_HEADERS",

{aemo_mdff_reader-2.2.1 → aemo_mdff_reader-4.0.1}/aemo_mdff_reader/aggregate.py

@@ -61,11 +61,15 @@ class DailyTotal(NamedTuple):
     """A single channel's total energy / quantity for a calendar day.
 
     ``total`` is the sum of ``IntervalReading.value`` for every reading
-    that fell on ``interval_date``. ``interval_count`` is the number of
-    readings summed (useful for spotting partial days). ``unique_quality_flags``
-    is the set of distinct 1-character quality flags seen — a single
-    ``{"A"}`` means the day is fully Actual, anything else hints at
-    substituted or estimated data.
+    that fell on ``interval_date``. Readings with a ``None`` value are
+    *missing*, not zero: they are excluded from ``total`` and from
+    ``interval_count``, and counted in ``missing_count`` instead — so a
+    day with any ``missing_count > 0`` is an incomplete total, not a
+    lower one. ``interval_count`` is the number of readings summed
+    (useful for spotting partial days). ``unique_quality_flags`` is the
+    set of distinct 1-character quality flags seen — a single ``{"A"}``
+    means the day is fully Actual, anything else hints at substituted or
+    estimated data.
     """
 
     nmi: str
@@ -76,6 +80,7 @@ class DailyTotal(NamedTuple):
     total: float
     interval_count: int
     unique_quality_flags: frozenset[str]
+    missing_count: int
 
 
 def group_by_nmi(
@@ -111,13 +116,15 @@ def daily_totals(readings: Iterable[IntervalReading]) -> Iterator[DailyTotal]:
     O(1)-memory operation: we accumulate while ``(channel, date)`` stays
     constant and emit when it changes.
 
-    Empty cells in the source are coerced to ``0.0`` upstream (see
-    :class:`IntervalReading`); they are summed as zero.
+    Empty cells in the source parse to ``None`` (see
+    :class:`IntervalReading`). They are skipped rather than summed as
+    zero, and reported per day via ``DailyTotal.missing_count``.
     """
     current_key: Optional[Tuple[ChannelKey, datetime]] = None
     current_uom: str = ""
     total = 0.0
     count = 0
+    missing = 0
     flags: set[str] = set()
 
     for r in readings:
@@ -135,14 +142,19 @@ def daily_totals(readings: Iterable[IntervalReading]) -> Iterator[DailyTotal]:
                 total=total,
                 interval_count=count,
                 unique_quality_flags=frozenset(flags),
+                missing_count=missing,
             )
             current_key = key
             current_uom = r.uom
             total = 0.0
             count = 0
+            missing = 0
             flags = set()
-        total += r.value
-        count += 1
+        if r.value is None:
+            missing += 1
+        else:
+            total += r.value
+            count += 1
         if r.quality_method:
             flags.add(r.quality_method[:1])
 
@@ -156,6 +168,7 @@ def daily_totals(readings: Iterable[IntervalReading]) -> Iterator[DailyTotal]:
             total=total,
             interval_count=count,
             unique_quality_flags=frozenset(flags),
+            missing_count=missing,
         )