aemo-mdff-reader 3.0.0__tar.gz → 4.0.1__tar.gz

{aemo_mdff_reader-3.0.0 → aemo_mdff_reader-4.0.1}/CHANGELOG.md

@@ -4,6 +4,69 @@ All notable changes are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
 project follows [Semantic Versioning](https://semver.org/).
 
+## [4.0.1](https://github.com/Utilified/aemo-mdff-reader/compare/v4.0.0...v4.0.1) (2026-11-02)
+
+
+### Bug Fixes
+
+* **deps:** bump certifi to 2026.10.19 for PYSEC-2026-4320 ([#145](https://github.com/Utilified/aemo-mdff-reader/issues/145)) ([7b66f2a](https://github.com/Utilified/aemo-mdff-reader/commit/7b66f2a4cd2b5e333c5258dcfd41277909c9ac62))
+* **deps:** bump msgpack to 1.2.4 for CVE-2026-114613 ([#148](https://github.com/Utilified/aemo-mdff-reader/issues/148)) ([06eb1ac](https://github.com/Utilified/aemo-mdff-reader/commit/06eb1ace6e8a94d4a87cad33211713b917a946e0))
+
+## [4.0.0](https://github.com/Utilified/aemo-mdff-reader/compare/v3.0.1...v4.0.0) (2026-09-01)
+
+
+### ⚠ BREAKING CHANGES
+
+* interval and accumulation reading values are now `float | None`. An
+  empty cell parses to `None` (missing) instead of `0.0` — a missing
+  reading must not be indistinguishable from a genuine zero in a
+  billing input. This affects `IntervalReading.value`, the `Value`
+  column in `to_columns` / `parse_to_columns` / the pandas and parquet
+  paths, and NEM13 `previous_register_read`, `current_register_read`
+  and `quantity`. `aggregate.daily_totals` skips `None` readings rather
+  than summing them as zero, excludes them from `interval_count`, and
+  reports them in the new `DailyTotal.missing_count` field.
+* the `aemo_mdff_reader.sql` subpackage and the `[mysql]` extra are
+  removed, along with `QueryBuilder`, `COLUMNS`, `Storer` and
+  `DBCredentials`. The subpackage referenced `NEMReader` attributes that
+  have not existed since v2, so every entry point raised
+  `AttributeError` on call, and its query builder interpolated
+  file-derived values into SQL unescaped. There is no replacement:
+  persist via `to_columns()` / `iter_columns_chunks()` and your own
+  parameterised inserts.
+* files that previously parsed silently now raise `NEM12ParseError`: a
+  300 row carrying more interval values than its parent 200 record's
+  IntervalLength allows, a non-numeric interval value or reason code, an
+  impossible date, and a 400 row whose StartInterval / EndInterval falls
+  outside the NMI's interval count. Trailing empty cells on a 300 row
+  (Excel padding) are tolerated and do not count against the field
+  bound.
+
+### Bug Fixes
+
+* raise on over-long 300 rows instead of silently truncating them. A
+  15-minute file under a stale 30-minute 200 header emitted 48 readings,
+  dropped the remaining 48, and took `quality_method` from a dropped
+  *value*. `validate_file` now checks 300 field counts too — it
+  previously returned clean for a 30-minute NMI carrying 288 values.
+* wrap the remaining raw `ValueError`s at the parse boundary as
+  `NEM12ParseError` (`_parse_int`, `_parse_float`, the columnar value
+  path, the 400 record's interval bounds, and the fixed-width
+  `datetime()` construction), each naming the offending field and value.
+* bound-check 400 StartInterval / EndInterval against the NMI's
+  intervals per day — `400,1,999,S` previously parsed against a
+  48-interval day.
+* correct the `spec.DIRECTION_INDICATORS` descriptions, which had Import
+  and Export the wrong way round.
+
+## [3.0.1](https://github.com/Utilified/aemo-mdff-reader/compare/v3.0.0...v3.0.1) (2026-07-18)
+
+
+### Bug Fixes
+
+* **fuzz:** install with --ignore-requires-python on the 3.11 OSS-Fuzz image ([#75](https://github.com/Utilified/aemo-mdff-reader/issues/75)) ([28605fe](https://github.com/Utilified/aemo-mdff-reader/commit/28605fe6c408891656bf106d7facdab1afe7dd8e))
+* tolerate unparseable 100-record DateTime instead of failing the file ([#73](https://github.com/Utilified/aemo-mdff-reader/issues/73)) ([ec84dd9](https://github.com/Utilified/aemo-mdff-reader/commit/ec84dd9a42c8c92445b0f6ef6fab7a1417ece220)), closes [#72](https://github.com/Utilified/aemo-mdff-reader/issues/72)
+
 ## [3.0.0](https://github.com/Utilified/aemo-mdff-reader/compare/v2.2.1...v3.0.0) (2026-07-16)
 
 

{aemo_mdff_reader-3.0.0/aemo_mdff_reader.egg-info → aemo_mdff_reader-4.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aemo-mdff-reader
-Version: 3.0.0
+Version: 4.0.1
 Summary: Fast streaming reader for AEMO MDFF NEM12 / NEM13 metering files. Zero required dependencies.
 Author: Utilified Holdings Pty Ltd
 License-Expression: MIT
@@ -28,8 +28,6 @@ Requires-Dist: pandas>=2; extra == "pandas"
 Provides-Extra: parquet
 Requires-Dist: pandas>=2; extra == "parquet"
 Requires-Dist: pyarrow>=10; extra == "parquet"
-Provides-Extra: mysql
-Requires-Dist: PyMySQL>=1.0; extra == "mysql"
 Provides-Extra: dev
 Requires-Dist: pytest>=7; extra == "dev"
 Requires-Dist: pytest-cov>=4; extra == "dev"
@@ -55,7 +53,7 @@ Fast, zero-dependency streaming reader for AEMO **NEM12** and **NEM13**
 metering files. Implements AEMO MDFF (Meter Data File Format) v2.6.
 
 - O(1) memory — iterate through millions of intervals.
-- Pure stdlib core; pandas / PyMySQL are opt-in extras.
+- Pure stdlib core; pandas / pyarrow are opt-in extras.
 - ~2 M readings/sec on the columnar fast path.
 - Includes an `aemo-mdff-reader` CLI.
 
@@ -65,8 +63,8 @@ metering files. Implements AEMO MDFF (Meter Data File Format) v2.6.
 pip install aemo-mdff-reader
 
 # optional extras
-pip install aemo-mdff-reader[pandas]   # to_dataframe() / parquet
-pip install aemo-mdff-reader[mysql]    # SQL persistence
+pip install aemo-mdff-reader[pandas]   # to_dataframe()
+pip install aemo-mdff-reader[parquet]  # parquet output (pandas + pyarrow)
 ```
 
 ## Use
@@ -83,8 +81,8 @@ Or as a flat CSV / DataFrame:
 ```python
 from aemo_mdff_reader import parse, write_csv, to_dataframe
 
-write_csv(parse("metering.csv"), "out.csv")     # no pandas
-df = to_dataframe("metering.csv")                # needs [pandas]
+write_csv(parse("metering.csv"), "out.csv")  # no pandas
+df = to_dataframe("metering.csv")  # needs [pandas]
 ```
 
 From the command line:
@@ -103,7 +101,7 @@ Each parsed record is a slots-based class with named attributes plus a
 
 ```python
 for r in parse("metering.csv"):
-    payload = r.to_dict()              # {"nmi": "...", "value": 0.12, ...}
+    payload = r.to_dict()  # {"nmi": "...", "value": 0.12, ...}
     print(r.quality_flag, r.method_flag)  # split of the QMM field, e.g. "S", "52"
 ```
 
@@ -155,6 +153,13 @@ Each `parse(...)` yields an `IntervalReading` with `nmi`,
 the type stubs (`from aemo_mdff_reader import IntervalReading`) for the
 exact signatures.
 
+`value` is `float | None`: an empty cell in the source file is a
+*missing* reading and stays `None` rather than becoming `0.0`, which
+would be indistinguishable from a genuine zero. The same holds for the
+NEM13 `previous_register_read`, `current_register_read` and `quantity`
+fields. `aggregate.daily_totals` skips missing readings and reports how
+many it saw in `DailyTotal.missing_count`.
+
 ## Notes
 
 - **Spec**: AEMO Meter Data File Format Specification NEM12 & NEM13,
@@ -165,8 +170,8 @@ exact signatures.
   as constants in `aemo_mdff_reader.spec` for callers that want stricter
   validation than the parser performs.
 - **Tolerant**: UTF-8 BOM is consumed silently, LF and CRLF both work,
-  and empty interval cells are coerced to `0.0` (use `quality_method`
-  to distinguish missing from zero). Datetime fields accept the spec
+  and empty interval cells parse to `None` (missing), never `0.0`.
+  Datetime fields accept the spec
   forms (`YYYYMMDD`, `YYYYMMDDhhmmss`) and a few common non-spec
   variants (`YYYY-MM-DD`, ISO `YYYY-MM-DDTHH:MM:SS`, with or without a
   `Z` / `±HH:MM` / `±HHMM` timezone suffix — the suffix is stripped

{aemo_mdff_reader-3.0.0 → aemo_mdff_reader-4.0.1}/README.md

@@ -11,7 +11,7 @@ Fast, zero-dependency streaming reader for AEMO **NEM12** and **NEM13**
 metering files. Implements AEMO MDFF (Meter Data File Format) v2.6.
 
 - O(1) memory — iterate through millions of intervals.
-- Pure stdlib core; pandas / PyMySQL are opt-in extras.
+- Pure stdlib core; pandas / pyarrow are opt-in extras.
 - ~2 M readings/sec on the columnar fast path.
 - Includes an `aemo-mdff-reader` CLI.
 
@@ -21,8 +21,8 @@ metering files. Implements AEMO MDFF (Meter Data File Format) v2.6.
 pip install aemo-mdff-reader
 
 # optional extras
-pip install aemo-mdff-reader[pandas]   # to_dataframe() / parquet
-pip install aemo-mdff-reader[mysql]    # SQL persistence
+pip install aemo-mdff-reader[pandas]   # to_dataframe()
+pip install aemo-mdff-reader[parquet]  # parquet output (pandas + pyarrow)
 ```
 
 ## Use
@@ -39,8 +39,8 @@ Or as a flat CSV / DataFrame:
 ```python
 from aemo_mdff_reader import parse, write_csv, to_dataframe
 
-write_csv(parse("metering.csv"), "out.csv")     # no pandas
-df = to_dataframe("metering.csv")                # needs [pandas]
+write_csv(parse("metering.csv"), "out.csv")  # no pandas
+df = to_dataframe("metering.csv")  # needs [pandas]
 ```
 
 From the command line:
@@ -59,7 +59,7 @@ Each parsed record is a slots-based class with named attributes plus a
 
 ```python
 for r in parse("metering.csv"):
-    payload = r.to_dict()              # {"nmi": "...", "value": 0.12, ...}
+    payload = r.to_dict()  # {"nmi": "...", "value": 0.12, ...}
     print(r.quality_flag, r.method_flag)  # split of the QMM field, e.g. "S", "52"
 ```
 
@@ -111,6 +111,13 @@ Each `parse(...)` yields an `IntervalReading` with `nmi`,
 the type stubs (`from aemo_mdff_reader import IntervalReading`) for the
 exact signatures.
 
+`value` is `float | None`: an empty cell in the source file is a
+*missing* reading and stays `None` rather than becoming `0.0`, which
+would be indistinguishable from a genuine zero. The same holds for the
+NEM13 `previous_register_read`, `current_register_read` and `quantity`
+fields. `aggregate.daily_totals` skips missing readings and reports how
+many it saw in `DailyTotal.missing_count`.
+
 ## Notes
 
 - **Spec**: AEMO Meter Data File Format Specification NEM12 & NEM13,
@@ -121,8 +128,8 @@ exact signatures.
   as constants in `aemo_mdff_reader.spec` for callers that want stricter
   validation than the parser performs.
 - **Tolerant**: UTF-8 BOM is consumed silently, LF and CRLF both work,
-  and empty interval cells are coerced to `0.0` (use `quality_method`
-  to distinguish missing from zero). Datetime fields accept the spec
+  and empty interval cells parse to `None` (missing), never `0.0`.
+  Datetime fields accept the spec
   forms (`YYYYMMDD`, `YYYYMMDDhhmmss`) and a few common non-spec
   variants (`YYYY-MM-DD`, ISO `YYYY-MM-DDTHH:MM:SS`, with or without a
   `Z` / `±HH:MM` / `±HHMM` timezone suffix — the suffix is stripped

{aemo_mdff_reader-3.0.0 → aemo_mdff_reader-4.0.1}/aemo_mdff_reader/__init__.py

@@ -48,7 +48,7 @@ from .types import (
     NMIDetails,
 )
 
-__version__ = "3.0.0"  # x-release-please-version
+__version__ = "4.0.1"  # x-release-please-version
 
 __all__ = [
     "INTERVAL_DATA_OUTPUT_HEADERS",

{aemo_mdff_reader-3.0.0 → aemo_mdff_reader-4.0.1}/aemo_mdff_reader/aggregate.py

@@ -61,11 +61,15 @@ class DailyTotal(NamedTuple):
     """A single channel's total energy / quantity for a calendar day.
 
     ``total`` is the sum of ``IntervalReading.value`` for every reading
-    that fell on ``interval_date``. ``interval_count`` is the number of
-    readings summed (useful for spotting partial days). ``unique_quality_flags``
-    is the set of distinct 1-character quality flags seen — a single
-    ``{"A"}`` means the day is fully Actual, anything else hints at
-    substituted or estimated data.
+    that fell on ``interval_date``. Readings with a ``None`` value are
+    *missing*, not zero: they are excluded from ``total`` and from
+    ``interval_count``, and counted in ``missing_count`` instead — so a
+    day with any ``missing_count > 0`` is an incomplete total, not a
+    lower one. ``interval_count`` is the number of readings summed
+    (useful for spotting partial days). ``unique_quality_flags`` is the
+    set of distinct 1-character quality flags seen — a single ``{"A"}``
+    means the day is fully Actual, anything else hints at substituted or
+    estimated data.
     """
 
     nmi: str
@@ -76,6 +80,7 @@ class DailyTotal(NamedTuple):
     total: float
     interval_count: int
     unique_quality_flags: frozenset[str]
+    missing_count: int
 
 
 def group_by_nmi(
@@ -111,13 +116,15 @@ def daily_totals(readings: Iterable[IntervalReading]) -> Iterator[DailyTotal]:
     O(1)-memory operation: we accumulate while ``(channel, date)`` stays
     constant and emit when it changes.
 
-    Empty cells in the source are coerced to ``0.0`` upstream (see
-    :class:`IntervalReading`); they are summed as zero.
+    Empty cells in the source parse to ``None`` (see
+    :class:`IntervalReading`). They are skipped rather than summed as
+    zero, and reported per day via ``DailyTotal.missing_count``.
     """
     current_key: Optional[Tuple[ChannelKey, datetime]] = None
     current_uom: str = ""
     total = 0.0
     count = 0
+    missing = 0
     flags: set[str] = set()
 
     for r in readings:
@@ -135,14 +142,19 @@ def daily_totals(readings: Iterable[IntervalReading]) -> Iterator[DailyTotal]:
                 total=total,
                 interval_count=count,
                 unique_quality_flags=frozenset(flags),
+                missing_count=missing,
             )
             current_key = key
             current_uom = r.uom
             total = 0.0
             count = 0
+            missing = 0
             flags = set()
-        total += r.value
-        count += 1
+        if r.value is None:
+            missing += 1
+        else:
+            total += r.value
+            count += 1
         if r.quality_method:
             flags.add(r.quality_method[:1])
 
@@ -156,6 +168,7 @@ def daily_totals(readings: Iterable[IntervalReading]) -> Iterator[DailyTotal]:
             total=total,
             interval_count=count,
             unique_quality_flags=frozenset(flags),
+            missing_count=missing,
         )
 
 

{aemo_mdff_reader-3.0.0 → aemo_mdff_reader-4.0.1}/aemo_mdff_reader/parser.py

@@ -45,6 +45,12 @@ END_RECORD = "900"
 # Field counts per AEMO MDFF v2.6 (NEM12 / NEM13).
 ACCUMULATION_FIELDS = 23  # 250 record
 
+# A 300 record carries at most five trailing fields after the interval
+# values: QualityMethod, ReasonCode, ReasonDescription, UpdateDateTime,
+# MSATSLoadDateTime (MDFF v2.6 §4.4). Anything beyond that means the row
+# holds more values than the parent 200 record's IntervalLength implies.
+INTERVAL_TRAILER_FIELDS = 5
+
 MINUTES_PER_DAY = 24 * 60
 
 # Path-like is anything ``os.fspath`` can convert. Streams are anything
@@ -87,27 +93,33 @@ def _parse_datetime(value: str) -> Optional[datetime]:
         value = value[: m.start()]
 
     n = len(value)
-    if n == 14:
-        return datetime(
-            int(value[0:4]),
-            int(value[4:6]),
-            int(value[6:8]),
-            int(value[8:10]),
-            int(value[10:12]),
-            int(value[12:14]),
-        )
-    if n == 12:  # YYYYMMDDhhmm — seen in some retailer exports
-        return datetime(
-            int(value[0:4]),
-            int(value[4:6]),
-            int(value[6:8]),
-            int(value[8:10]),
-            int(value[10:12]),
-        )
-    if n == 8:
-        return datetime(int(value[0:4]), int(value[4:6]), int(value[6:8]))
-    if n == 10 and value[4] == "-" and value[7] == "-":
-        return datetime(int(value[0:4]), int(value[5:7]), int(value[8:10]))
+    # ``datetime(...)`` raises a bare ValueError on impossible components
+    # (e.g. ``20049999`` → "month must be in 1..12"); surface it as the
+    # advertised NEM12ParseError instead.
+    try:
+        if n == 14:
+            return datetime(
+                int(value[0:4]),
+                int(value[4:6]),
+                int(value[6:8]),
+                int(value[8:10]),
+                int(value[10:12]),
+                int(value[12:14]),
+            )
+        if n == 12:  # YYYYMMDDhhmm — seen in some retailer exports
+            return datetime(
+                int(value[0:4]),
+                int(value[4:6]),
+                int(value[6:8]),
+                int(value[8:10]),
+                int(value[10:12]),
+            )
+        if n == 8:
+            return datetime(int(value[0:4]), int(value[4:6]), int(value[6:8]))
+        if n == 10 and value[4] == "-" and value[7] == "-":
+            return datetime(int(value[0:4]), int(value[5:7]), int(value[8:10]))
+    except ValueError as exc:
+        raise NEM12ParseError(f"Invalid date/time value: {value!r} ({exc})") from exc
 
     # ISO 8601 date-time fall-through, e.g. ``2024-01-01T00:00:00`` or
     # ``2024-01-01 00:00:00`` (with optional microseconds). datetime's
@@ -128,21 +140,26 @@ def _parse_datetime(value: str) -> Optional[datetime]:
 def _parse_int(value: str) -> Optional[int]:
     if value == "" or value is None:
         return None
-    return int(value)
+    try:
+        return int(value)
+    except ValueError as exc:
+        raise NEM12ParseError(f"Expected an integer field, got {value!r}") from exc
 
 
-def _parse_float(value: str) -> float:
-    """Parse a numeric NEM12 cell.
+def _parse_float(value: str) -> Optional[float]:
+    """Parse a numeric NEM12 / NEM13 cell.
 
-    NEM12 uses decimal numbers without thousands separators. Empty cells
-    (sometimes seen for missing intervals) are coerced to ``0.0`` so a
-    single missing cell doesn't fail an entire row. Use the row's
-    ``QualityMethod`` flag to distinguish a real zero from a missing
-    reading — see :class:`IntervalReading` for the surfaced fields.
+    NEM12 uses decimal numbers without thousands separators. An empty
+    cell is a *missing* reading and returns ``None`` — never ``0.0``,
+    which would be indistinguishable from a genuine zero reading in a
+    billing input.
     """
     if value == "" or value is None:
-        return 0.0
-    return float(value)
+        return None
+    try:
+        return float(value)
+    except ValueError as exc:
+        raise NEM12ParseError(f"Expected a numeric field, got {value!r}") from exc
 
 
 class _ParserState:
@@ -286,13 +303,33 @@ def parse_header(source: RowSource) -> Optional[Header]:
     return None
 
 
-def _parse_header(row: Sequence[str]) -> Header:
+def _parse_header(row: Sequence[str], *, strict: bool = False) -> Header:
+    """Parse a 100 record.
+
+    The DateTime field is file provenance: nothing in this library reads
+    it back, and interval timestamps come from the 300 records. Files
+    round-tripped through a spreadsheet routinely arrive with it mangled
+    — Excel reads the 12/14-digit value as a number and rewrites it as
+    ``2.02607E+11``, which is lossy beyond recovery. Failing the whole
+    file over an unread field is the wrong trade, so an unparseable
+    value degrades to ``None`` here.
+
+    ``strict=True`` restores the raise, for :func:`validate_file` — the
+    opt-in pass whose job is to report exactly this kind of damage.
+    """
     # 100, VersionHeader, DateTime, FromParticipant, ToParticipant
     if len(row) < 3:
         raise NEM12ParseError("100 header row missing required fields")
+    file_datetime: Optional[datetime] = None
+    if len(row) > 2:
+        try:
+            file_datetime = _parse_datetime(row[2])
+        except NEM12ParseError:
+            if strict:
+                raise
     return Header(
         version=row[1] if len(row) > 1 else "",
-        datetime=_parse_datetime(row[2]) if len(row) > 2 else None,
+        datetime=file_datetime,
         from_participant=row[3] if len(row) > 3 else "",
         to_participant=row[4] if len(row) > 4 else "",
     )
@@ -330,6 +367,37 @@ def _parse_nmi(row: Sequence[str]) -> NMIDetails:
     )
 
 
+def _effective_row_len(row: Sequence[str], minimum: int) -> int:
+    """Length of *row* ignoring trailing empty cells, floored at *minimum*.
+
+    Excel round-trips pad every row to the widest row's width with empty
+    cells; those must not count against the field-count upper bound.
+    """
+    effective = len(row)
+    while effective > minimum and row[effective - 1] == "":
+        effective -= 1
+    return effective
+
+
+def _check_interval_value_count(row: Sequence[str], nmi: NMIDetails, n: int) -> None:
+    """Reject a 300 row carrying more values than its 200 record allows.
+
+    Only checking the lower bound lets a 15-minute row sitting under a
+    stale 30-minute 200 header parse as 48 readings, silently dropping
+    the rest and picking up a *reading* as the QualityMethod trailer.
+    Trailing empty cells (Excel padding) are ignored.
+    """
+    max_fields = 2 + n + INTERVAL_TRAILER_FIELDS
+    effective = _effective_row_len(row, 2 + n)
+    if effective > max_fields:
+        raise NEM12ParseError(
+            f"300 row for NMI {nmi.nmi} on {row[1]} has {effective - 2} fields "
+            f"after IntervalDate, expected {n} interval values plus at most "
+            f"{INTERVAL_TRAILER_FIELDS} trailer fields "
+            f"for IntervalLength={nmi.interval_length}"
+        )
+
+
 def _emit_intervals(row: Sequence[str], state: _ParserState) -> Iterator[IntervalReading]:
     nmi = state.current_nmi
     if nmi is None:
@@ -342,6 +410,7 @@ def _emit_intervals(row: Sequence[str], state: _ParserState) -> Iterator[Interva
             f"300 row has {len(row)} fields, expected at least {expected_min} "
             f"for IntervalLength={nmi.interval_length}"
         )
+    _check_interval_value_count(row, nmi, n)
 
     interval_date = _parse_datetime(row[1])
     if interval_date is None:
@@ -453,6 +522,7 @@ def parse_to_columns(source: RowSource) -> Columns:
                 raise NEM12ParseError(
                     f"300 row has {len(row)} fields, expected at least {expected_min}"
                 )
+            _check_interval_value_count(row, nmi_obj, n)
             interval_date = _parse_datetime(row[1])
             if interval_date is None:
                 raise NEM12ParseError("300 row missing IntervalDate")
@@ -484,7 +554,13 @@ def parse_to_columns(source: RowSource) -> Columns:
                 a_iend(end)
                 a_idx(i + 1)
                 cell = row[base + i]
-                a_val(0.0 if cell == "" else float(cell))
+                if cell == "":
+                    a_val(None)
+                else:
+                    try:
+                        a_val(float(cell))
+                    except ValueError as exc:
+                        raise NEM12ParseError(f"Expected a numeric field, got {cell!r}") from exc
                 a_q(quality)
                 a_rc(reason_code)
                 a_rd(reason_desc)
@@ -1023,13 +1099,38 @@ def _parse_interval_event(
     def _get(i: int) -> str:
         return row[i] if i < len(row) else ""
 
+    # An IntervalLength that does not divide the day has no well-defined interval
+    # count, so the ceiling is unknowable; the 1.. floor still applies. Mirrors the
+    # guard used wherever else intervals-per-day is derived.
+    il = nmi.interval_length
+    intervals_per_day = MINUTES_PER_DAY // il if il > 0 and MINUTES_PER_DAY % il == 0 else None
+
+    bounds = []
+    for idx, name in ((1, "StartInterval"), (2, "EndInterval")):
+        try:
+            bound = int(row[idx])
+        except ValueError as exc:
+            raise NEM12ParseError(f"400 event row has non-integer {name}: {row[idx]!r}") from exc
+        if bound < 1 or (intervals_per_day is not None and bound > intervals_per_day):
+            ceiling = "" if intervals_per_day is None else f"..{intervals_per_day}"
+            raise NEM12ParseError(
+                f"400 event row has {name}={bound}; must be in 1{ceiling} for IntervalLength={il}"
+            )
+        bounds.append(bound)
+
+    if bounds[0] > bounds[1]:
+        raise NEM12ParseError(
+            f"400 event row has StartInterval={bounds[0]} after EndInterval={bounds[1]}; "
+            "the range is inverted"
+        )
+
     return IntervalEvent(
         nmi=nmi.nmi,
         meter_serial_number=nmi.meter_serial_number,
         register_id=nmi.register_id,
         interval_date=interval_date,
-        start_interval=int(row[1]),
-        end_interval=int(row[2]),
+        start_interval=bounds[0],
+        end_interval=bounds[1],
         quality_method=_get(3),
         reason_code=_parse_int(_get(4)),
         reason_description=_get(5),
@@ -1189,7 +1290,7 @@ def validate_file(source: RowSource) -> List[str]:
             if line_no != 1:
                 issues.append(f"line {line_no}: 100 header should be the first row")
             try:
-                _parse_header(row)
+                _parse_header(row, strict=True)
             except NEM12ParseError as exc:
                 issues.append(f"line {line_no}: {exc}")
         elif rec == NMI_RECORD:
@@ -1203,6 +1304,21 @@ def validate_file(source: RowSource) -> List[str]:
         elif rec == INTERVAL_RECORD:
             if current_nmi is None:
                 issues.append(f"line {line_no}: 300 row encountered before any 200 NMI row")
+            elif MINUTES_PER_DAY % current_nmi.interval_length == 0:
+                n = MINUTES_PER_DAY // current_nmi.interval_length
+                if len(row) < 2 + n:
+                    issues.append(
+                        f"line {line_no}: 300 row has {len(row) - 2} interval values, "
+                        f"expected {n} for IntervalLength={current_nmi.interval_length}"
+                    )
+                elif _effective_row_len(row, 2 + n) > 2 + n + INTERVAL_TRAILER_FIELDS:
+                    issues.append(
+                        f"line {line_no}: 300 row has "
+                        f"{_effective_row_len(row, 2 + n) - 2} fields after "
+                        f"IntervalDate, expected {n} interval values plus at most "
+                        f"{INTERVAL_TRAILER_FIELDS} trailer fields "
+                        f"for IntervalLength={current_nmi.interval_length}"
+                    )
         elif rec == ACCUMULATION_RECORD:
             if len(row) < 20:
                 issues.append(

{aemo_mdff_reader-3.0.0 → aemo_mdff_reader-4.0.1}/aemo_mdff_reader/spec.py

@@ -41,8 +41,8 @@ ALLOWED_INTERVAL_LENGTHS: FrozenSet[int] = frozenset({5, 15, 30})
 
 #: DirectionIndicator allowed values per the spec.
 DIRECTION_INDICATORS: Dict[str, str] = {
-    "I": "Import (energy flows from the connection point to the grid)",
-    "E": "Export (energy flows from the grid to the connection point)",
+    "I": "Import (energy flows from the grid to the connection point)",
+    "E": "Export (energy flows from the connection point to the grid)",
 }
 
 

{aemo_mdff_reader-3.0.0 → aemo_mdff_reader-4.0.1}/aemo_mdff_reader/types.py

@@ -118,11 +118,11 @@ class IntervalReading:
 
     Notes
     -----
-    ``value`` is always a ``float``. Empty cells in the source NEM12
-    file are coerced to ``0.0`` so a single missing cell does not fail
-    an entire row. Use ``quality_method`` (e.g. ``"S"`` for substituted,
-    ``"F"`` for failed, ``"N"`` for null) to distinguish a real zero
-    from a missing reading.
+    ``value`` is ``None`` when the source cell was empty — a missing
+    reading, which must not be confused with a genuine ``0.0``. Callers
+    handling billing data should decide explicitly what to do with it;
+    ``quality_method`` (e.g. ``"S"`` for substituted, ``"F"`` for
+    failed) says why the reading is what it is.
     """
 
     __slots__ = (
@@ -156,7 +156,7 @@ class IntervalReading:
         interval_start: datetime,
         interval_end: datetime,
         interval_index: int,
-        value: float,
+        value: Optional[float],
         quality_method: str,
         reason_code: Optional[int],
         reason_description: str,
@@ -262,6 +262,8 @@ class AccumulationReading:
 
     Each 250 record represents a previous + current register read pair
     plus a calculated quantity (consumption between the two reads).
+    ``previous_register_read``, ``current_register_read`` and
+    ``quantity`` are ``None`` when the source cell was empty.
 
     .. note::
        ``direction_indicator`` is stored as the raw string from the
@@ -306,17 +308,17 @@ class AccumulationReading:
         mdm_data_stream_identifier: str,
         meter_serial_number: str,
         direction_indicator: str,
-        previous_register_read: float,
+        previous_register_read: Optional[float],
         previous_register_read_datetime: Optional[datetime],
         previous_quality_method: str,
         previous_reason_code: Optional[int],
         previous_reason_description: str,
-        current_register_read: float,
+        current_register_read: Optional[float],
         current_register_read_datetime: Optional[datetime],
         current_quality_method: str,
         current_reason_code: Optional[int],
         current_reason_description: str,
-        quantity: float,
+        quantity: Optional[float],
         uom: str,
         next_scheduled_read_date: Optional[datetime],
         update_datetime: Optional[datetime],