aemo-mdff-reader 2.0.2__tar.gz

aemo_mdff_reader-2.0.2/CHANGELOG.md

@@ -0,0 +1,182 @@
+# Changelog
+
+All notable changes are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
+project follows [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [2.0.2] — 2026-05-10
+
+First release published to PyPI via Trusted Publishing from the now-public
+GitHub repository. No source or behavioural changes versus 2.0.1 — this
+release packages the public-launch hardening described below.
+
+### Added
+
+- `CODE_OF_CONDUCT.md` (Contributor Covenant v2.1 adoption notice).
+- GitHub issue forms for bug reports and feature requests, plus a
+  pull request template.
+- CodeQL workflow for static analysis on push / PR / schedule.
+- OpenSSF Scorecard workflow for supply-chain posture reporting.
+
+### Changed
+
+- GitHub Actions across all workflows pinned to commit SHAs (managed
+  by Dependabot) for supply-chain hardening.
+
+### Fixed
+
+- GitHub org casing in repository URLs (`utilified` → `Utilified`).
+
+## [2.0.1] — 2026-05-10
+
+The 2.0.0 tag was cut before the project was renamed to
+`aemo-mdff-reader`. 2.0.1 is the first version published under the new
+name end-to-end. Source semantics are unchanged — this release exists
+to give consumers a single stable tag matching the
+`aemo_mdff_reader` import path and the `aemo-mdff-reader` PyPI dist
++ CLI command.
+
+### Changed
+
+- **Project renamed end-to-end** to `aemo-mdff-reader`: GitHub repo
+  `Utilified/aemo-mdff-reader`, PyPI distribution `aemo-mdff-reader`,
+  Python import `aemo_mdff_reader`, CLI command `aemo-mdff-reader`.
+  v1 callers using `from nem12_reader import …` must update their
+  imports; any references to the `nem12-reader` CLI command must be
+  updated. GitHub redirects the previous repo URL so existing git
+  pins (`git@github.com:Utilified/nem12-reader.git`) keep resolving.
+
+### Tooling
+
+- Bumped GitHub Actions used in `ci.yml` / `release.yml` to their
+  current major versions: `actions/setup-python@v6`,
+  `actions/upload-artifact@v7`, `actions/download-artifact@v8`,
+  `softprops/action-gh-release@v3` (Node 24 runtime),
+  `sigstore/gh-action-sigstore-python@v3.3.0`.
+- The release workflow's PyPI publish step is now gated on
+  `vars.PYPI_PUBLISH_ENABLED == 'true'`, so tag pushes run cleanly
+  before Trusted Publishing is configured (build + sign succeed;
+  publish + GitHub-release jobs skip).
+- Repository description, homepage, and topic tags configured for
+  PyPI / GitHub discoverability.
+
+## [2.0.0] — 2026-05-10
+
+### Highlights
+
+- **Renamed to `aemo-mdff-reader` end-to-end.** v1 shipped under
+  `nem12-reader` (GitHub repo, PyPI distribution, Python import, and
+  CLI command). v2 unifies all four surfaces under
+  `aemo-mdff-reader` / `aemo_mdff_reader` — the AEMO spec name (MDFF,
+  Meter Data File Format) covers both NEM12 (interval) and NEM13
+  (accumulation) data and disambiguates the project from the
+  unrelated `nem-reader` package on PyPI. GitHub redirects the
+  previous repo URL so existing git pins
+  (`git@github.com:Utilified/nem12-reader.git`) continue to resolve.
+  Source consumers must update their imports
+  (`from nem12_reader import …` → `from aemo_mdff_reader import …`)
+  and any references to the `nem12-reader` CLI command.
+- **Spec target: AEMO MDFF v2.6** (effective 29 September 2024).
+  Previous v1.x of this package targeted MDFF v1.02 (2017). All
+  record-type schemas, allowed values, and reason-code descriptions
+  are aligned to v2.6 — including the v2.6 reason codes added by the
+  ICF_054 substitution-type review (codes 100–109) and the v2.6
+  designation of obsolete reason codes (Appendix F). The full set is
+  exposed as constants in the new `aemo_mdff_reader.spec` module.
+- **Zero required dependencies**: pandas, PyMySQL, wrapt, six, tomli,
+  pytz, python-dateutil are no longer required. pandas / PyMySQL are
+  available as opt-in extras (`pip install aemo-mdff-reader[pandas]`,
+  `[mysql]`).
+- **Streaming parser** (`aemo_mdff_reader.parse`): O(1) memory, works on
+  arbitrarily large files.
+- **Columnar fast path** (`parse_to_columns`, `to_dataframe(path)`):
+  ~7× faster than the v1 `read_from_file` + `to_dataframe` flow.
+- **Full NEM13 (250) accumulation support**: `parse_accumulations`,
+  `to_accumulations_dataframe`, `write_accumulations_csv`. `parse_all`
+  emits both NEM12 and NEM13 records in file order.
+- **Full record-type coverage**: `parse_events()` (400 quality/event
+  rows), `parse_b2b()` (500 NEM12 + 550 NEM13 transaction details).
+- **Spec-conformance utilities**: `nmi_checksum()`, `validate_nmi()`,
+  `validate_file()` plus a `aemo-mdff-reader --validate` CLI flag.
+- **CLI** (`aemo-mdff-reader`): convert NEM12 / NEM13 → flat CSV / parquet
+  from the command line. Was previously documented but not implemented.
+  `--records intervals|accumulations` selects the record type;
+  `--validate` runs structural checks against the AEMO MDFF spec.
+
+### Added
+
+- `aemo_mdff_reader.parse(source)` — generator-based streaming parser
+  (NEM12 300 records).
+- `aemo_mdff_reader.parse_to_columns(source)` — single-pass columnar
+  parse without per-cell row-object allocation.
+- `aemo_mdff_reader.parse_header(source)` — header-only fast read.
+- `aemo_mdff_reader.parse_accumulations(source)` — NEM13 250 record
+  streaming parser.
+- `aemo_mdff_reader.parse_accumulations_to_columns(source)` — columnar
+  NEM13 fast path.
+- `aemo_mdff_reader.parse_all(source)` — unified iterator yielding both
+  300 and 250 records in file order.
+- `aemo_mdff_reader.to_dataframe(source)` and
+  `to_accumulations_dataframe(source)` — pandas DataFrame builders.
+- `aemo_mdff_reader.write_csv(...)` and `write_accumulations_csv(...)` —
+  flat-CSV writers that work without pandas.
+- `aemo_mdff_reader.types` — slots-based `Header`, `NMIDetails`,
+  `IntervalReading`, `IntervalEvent`, `AccumulationReading` classes.
+- `aemo_mdff_reader.cli.main` — `aemo-mdff-reader` console script with
+  `--records intervals|accumulations` and `--format csv|parquet`.
+- `pyproject.toml` (PEP 621) with full metadata, classifiers, and
+  optional extras (`pandas`, `parquet`, `mysql`, `dev`).
+- `mypy --strict` configuration; the public `aemo_mdff_reader` surface
+  is fully typed and passes `mypy --strict`.
+- `ruff` lint + format configuration.
+- `LICENSE` file (MIT).
+- `tests/` — 55-test pytest suite covering parser, edge cases,
+  datetime variants, CLI, NEM13, and lazy SQL imports.
+- `benchmarks/bench_parser.py` — repeatable performance benchmark.
+- GitHub Actions:
+  - CI matrix (Python 3.8 → 3.12, Linux/macOS/Windows).
+  - `ruff` lint job.
+  - `mypy --strict` job.
+  - Build job with `twine check --strict` and wheel-install smoke.
+  - Release workflow (`release.yml`) — publishes to PyPI on tag via
+    Trusted Publishing.
+- `dependabot.yml` — weekly bumps for GitHub Actions and pip.
+- `.pre-commit-config.yaml` — local lint + format + mypy hooks.
+
+### Changed
+
+- `NEMReader().to_dataframe()` now uses the columnar fast path when
+  the source filename is known. Output schema clarifies interval
+  start/end timestamps and adds an explicit Suffix column.
+- `aemo_mdff_reader.sql` no longer eagerly imports `pymysql`; the
+  dependency is only loaded when `Storer` is instantiated.
+
+### Removed
+
+- `aemo_mdff_reader.nemstructure` package (Header / NMIData / IntervalData
+  abstractions and the dynamically-built `NEMField` schema). The new
+  `aemo_mdff_reader.types` module replaces this surface with simpler,
+  faster slots classes.
+- Broken `aemo_mdff_reader.nemstructure.nem13` and `enem12` modules,
+  which called `Record.__init__(...)` with arguments the base class
+  never accepted.
+- Dead `requirements.txt` pins for `numpy`, `wrapt`, `six`, `tomli`,
+  `python-dateutil`, `pytz`. None were actually used in the codebase.
+- The unsanitized `test.csv` sample (contained a real NMI / meter
+  serial). Replaced with a synthesized fixture under
+  `tests/fixtures/sample_nem12.csv` and a generator script.
+
+### Fixed
+
+- Quadratic-time row consumption in the legacy parser
+  (`array = array[1:]` in a hot loop) that scaled poorly on
+  large files.
+- Per-`IntervalData`-row construction of ~289 `NEMField` objects
+  for 5-minute reads (eliminated entirely; schema is fixed in code).
+- Eager pandas import on package load.
+
+## [1.0.1] — 2024
+
+Initial public versions. See git history for detailed changes.

aemo_mdff_reader-2.0.2/CONTRIBUTING.md

@@ -0,0 +1,118 @@
+# Contributing
+
+Thanks for your interest in improving aemo-mdff-reader. The repository
+is small and the bar for contributions is straightforward: a pull
+request that passes CI and doesn't regress the public benchmark is
+welcome.
+
+Naming: everything (GitHub repo `Utilified/aemo-mdff-reader`, PyPI
+distribution `aemo-mdff-reader`, Python import `aemo_mdff_reader`,
+CLI `aemo-mdff-reader`) is consistently named after the AEMO spec —
+MDFF, the Meter Data File Format that covers both NEM12 (interval) and
+NEM13 (accumulation) formats. GitHub serves redirects from the
+previous `Utilified/nem12-reader` repo URL so existing git pins
+continue to work; v1 callers importing `from nem12_reader …` need to
+update to `from aemo_mdff_reader …`.
+
+## Setup
+
+```bash
+git clone https://github.com/Utilified/aemo-mdff-reader.git
+cd aemo-mdff-reader
+python -m venv .venv && source .venv/bin/activate
+pip install -e .[dev]
+pytest
+```
+
+## Running the test suite
+
+```bash
+pytest -v
+pytest --cov=aemo_mdff_reader --cov-report=term-missing
+```
+
+A small, sanitized NEM12 sample lives at
+`tests/fixtures/sample_nem12.csv`. To generate larger synthetic files
+for ad-hoc testing or benchmarking:
+
+```bash
+python tests/fixtures/generate.py /tmp/big.csv --nmis 4 --days 365 --interval-minutes 5
+```
+
+## Performance
+
+`benchmarks/bench_parser.py` is the canonical performance test. Please
+run it before and after changes that touch the hot path
+(`parser.py::_emit_intervals`, `parse_to_columns`, `to_columns`):
+
+```bash
+python benchmarks/bench_parser.py --nmis 4 --days 365 --interval-minutes 5
+```
+
+The streaming and columnar fast paths must remain at or above the
+current published throughput.
+
+## Code style
+
+- Pure stdlib in the core. **No** new required runtime dependencies.
+  Optional features go behind `[project.optional-dependencies]`.
+- Type hints on all public symbols.
+- Keep `aemo_mdff_reader.types` slots-based — per-row allocation is on
+  the hot path.
+
+## Reporting issues
+
+Please open an issue with:
+
+1. The Python version and OS.
+2. A minimal reproducer (a few NEM12 rows is usually enough).
+3. The expected vs actual behaviour.
+
+If you can share a sanitized sample file (no real NMIs / meter
+serials), please attach it to the issue.
+
+## Releasing
+
+Maintainers only. Releases are tagged and pushed to GitHub; the
+`Release` workflow then builds, signs (sigstore), publishes to PyPI
+(via Trusted Publishing), and attaches signatures to the GitHub Release.
+
+### One-time setup (per project)
+
+Before the first release, configure these on PyPI / GitHub:
+
+1. **PyPI Trusted Publishing**: register the project at
+   https://pypi.org/manage/account/publishing/ with:
+   - PyPI project name: `aemo-mdff-reader`
+   - Owner: `Utilified`
+   - Repository: `aemo-mdff-reader`
+   - Workflow filename: `release.yml`
+   - Environment name: `pypi`
+2. **GitHub Environment**: in *Settings → Environments*, create an
+   environment named `pypi`. Add required reviewers if you want
+   manual approval before publish (recommended for the first few
+   releases). Without this environment, the publish job will fail with
+   a confusing "environment not found" error.
+
+### Cutting a release
+
+```bash
+# Bump version in pyproject.toml AND aemo_mdff_reader/__init__.py.
+# Update CHANGELOG.md with the new section.
+git commit -am "release: vX.Y.Z"
+git tag vX.Y.Z
+git push origin main vX.Y.Z
+```
+
+The `Release` workflow takes over from here. Verify the published
+artefact at https://pypi.org/project/aemo-mdff-reader/ and the
+GitHub Release at https://github.com/Utilified/aemo-mdff-reader/releases.
+
+### Local dry run
+
+```bash
+python -m build
+twine check --strict dist/*
+python -m venv /tmp/smoke && /tmp/smoke/bin/pip install dist/*.whl
+/tmp/smoke/bin/aemo-mdff-reader --version
+```

aemo_mdff_reader-2.0.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023-2026 Utilified Holdings Pty Ltd
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

aemo_mdff_reader-2.0.2/MANIFEST.in

@@ -0,0 +1,8 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+include CONTRIBUTING.md
+include pyproject.toml
+include aemo_mdff_reader/py.typed
+recursive-include tests *.py *.csv
+recursive-include benchmarks *.py

aemo_mdff_reader-2.0.2/PKG-INFO

@@ -0,0 +1,247 @@
+Metadata-Version: 2.4
+Name: aemo-mdff-reader
+Version: 2.0.2
+Summary: Fast streaming reader for AEMO MDFF NEM12 / NEM13 metering files. Zero required dependencies.
+Author: Utilified Holdings Pty Ltd
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Utilified/aemo-mdff-reader
+Project-URL: Repository, https://github.com/Utilified/aemo-mdff-reader
+Project-URL: Issues, https://github.com/Utilified/aemo-mdff-reader/issues
+Project-URL: Changelog, https://github.com/Utilified/aemo-mdff-reader/blob/main/CHANGELOG.md
+Keywords: nem12,nem13,aemo,energy,metering,australia,mdff
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Office/Business :: Financial
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.3; extra == "pandas"
+Provides-Extra: parquet
+Requires-Dist: pandas>=1.3; extra == "parquet"
+Requires-Dist: pyarrow>=10; extra == "parquet"
+Provides-Extra: mysql
+Requires-Dist: PyMySQL>=1.0; extra == "mysql"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-cov>=4; extra == "dev"
+Requires-Dist: pandas>=1.3; extra == "dev"
+Requires-Dist: pyarrow>=10; extra == "dev"
+Requires-Dist: mypy>=1.8; extra == "dev"
+Requires-Dist: ruff>=0.5; extra == "dev"
+Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Requires-Dist: packaging>=24.2; extra == "dev"
+Dynamic: license-file
+
+# aemo-mdff-reader
+
+[![CI](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/ci.yml/badge.svg)](https://github.com/Utilified/aemo-mdff-reader/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/aemo-mdff-reader.svg)](https://pypi.org/project/aemo-mdff-reader/)
+[![Python versions](https://img.shields.io/pypi/pyversions/aemo-mdff-reader.svg)](https://pypi.org/project/aemo-mdff-reader/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+Fast, zero-dependency streaming reader for AEMO **NEM12** and **NEM13**
+metering files. Implements AEMO MDFF (Meter Data File Format) v2.6.
+
+- O(1) memory — iterate through millions of intervals.
+- Pure stdlib core; pandas / PyMySQL are opt-in extras.
+- ~2 M readings/sec on the columnar fast path.
+- Includes an `aemo-mdff-reader` CLI.
+
+## Install
+
+```bash
+pip install aemo-mdff-reader
+
+# optional extras
+pip install aemo-mdff-reader[pandas]   # to_dataframe() / parquet
+pip install aemo-mdff-reader[mysql]    # SQL persistence
+```
+
+## Use
+
+```python
+from aemo_mdff_reader import parse
+
+for r in parse("metering.csv"):
+    print(r.nmi, r.interval_start, r.value, r.uom)
+```
+
+Or as a flat CSV / DataFrame:
+
+```python
+from aemo_mdff_reader import parse, write_csv, to_dataframe
+
+write_csv(parse("metering.csv"), "out.csv")     # no pandas
+df = to_dataframe("metering.csv")                # needs [pandas]
+```
+
+From the command line:
+
+```bash
+aemo-mdff-reader metering.csv -o out.csv
+aemo-mdff-reader metering.csv --validate                       # spec check
+aemo-mdff-reader metering.csv --nmi NMI1234567 --start 2024-01-01 --end 2024-01-31
+aemo-mdff-reader manual.csv --records accumulations            # NEM13
+```
+
+## Working with the data
+
+Each parsed record is a slots-based class with named attributes plus a
+`to_dict()` for JSON / dict pipelines:
+
+```python
+for r in parse("metering.csv"):
+    payload = r.to_dict()              # {"nmi": "...", "value": 0.12, ...}
+    print(r.quality_flag, r.method_flag)  # split of the QMM field, e.g. "S", "52"
+```
+
+For aggregation, `aemo_mdff_reader.aggregate` provides streaming helpers:
+
+```python
+from aemo_mdff_reader import parse
+from aemo_mdff_reader.aggregate import group_by_nmi, daily_totals
+
+for key, group in group_by_nmi(parse("metering.csv")):
+    # key = ChannelKey(nmi, register_id, nmi_suffix)
+    intervals = list(group)
+
+for day in daily_totals(parse("metering.csv")):
+    # day.total, day.interval_count, day.unique_quality_flags
+    print(day.nmi, day.interval_date.date(), day.total, day.uom)
+```
+
+End-to-end recipes — load + inspect, daily roll-up, filter to pandas,
+spec validation — live in [`examples/`](examples/).
+
+## API at a glance
+
+| You want                              | Call                          |
+| ------------------------------------- | ----------------------------- |
+| 300 interval readings (NEM12)         | `parse(src)`                  |
+| 250 accumulations (NEM13)             | `parse_accumulations(src)`    |
+| Both, in file order                   | `parse_all(src)`              |
+| 400 quality / event flags             | `parse_events(src)`           |
+| 500 / 550 B2B transactions            | `parse_b2b(src)`              |
+| Just the 100 header                   | `parse_header(src)`           |
+| Build a pandas DataFrame              | `to_dataframe(src)`           |
+| Write a flat CSV (no pandas)          | `write_csv(rows, out)`        |
+| Validate against AEMO MDFF v2.6       | `validate_file(src)`          |
+| Compute / verify an NMI checksum      | `nmi_checksum`, `validate_nmi`|
+| Group readings by NMI / channel       | `aggregate.group_by_nmi(rows)`|
+| Roll up to daily totals               | `aggregate.daily_totals(rows)`|
+| Convert any record to a plain dict    | `r.to_dict()`                 |
+
+`src` can be a path, a file-like object, an iterable of CSV lines, or
+an iterable of pre-split rows. The v1 `NEMReader` facade
+(`read_from_file`, `to_dataframe`, `to_csv`) still works.
+
+Each `parse(...)` yields an `IntervalReading` with `nmi`,
+`meter_serial_number`, `register_id`, `nmi_suffix`, `uom`,
+`interval_length`, `interval_date`, `interval_start`, `interval_end`,
+`interval_index`, `value`, `quality_method`, `reason_code`,
+`reason_description`, `update_datetime`, `msats_load_datetime`. See
+the type stubs (`from aemo_mdff_reader import IntervalReading`) for the
+exact signatures.
+
+## Notes
+
+- **Spec**: AEMO Meter Data File Format Specification NEM12 & NEM13,
+  **v2.6** (effective 29 September 2024). Records `100`, `200`, `250`,
+  `300`, `400`, `500`, `550`, `900` are all surfaced; unknown indicators
+  are ignored. Allowed values for quality flags, transaction codes,
+  reason codes, units of measure, and direction indicators are exposed
+  as constants in `aemo_mdff_reader.spec` for callers that want stricter
+  validation than the parser performs.
+- **Tolerant**: UTF-8 BOM is consumed silently, LF and CRLF both work,
+  and empty interval cells are coerced to `0.0` (use `quality_method`
+  to distinguish missing from zero). Datetime fields accept the spec
+  forms (`YYYYMMDD`, `YYYYMMDDhhmmss`) and a few common non-spec
+  variants (`YYYY-MM-DD`, ISO `YYYY-MM-DDTHH:MM:SS`, with or without a
+  `Z` / `±HH:MM` / `±HHMM` timezone suffix — the suffix is stripped
+  and parsed datetimes are returned naive). `direction_indicator` on
+  250 records passes through whatever the file emits; the spec set is
+  `spec.DIRECTION_INDICATORS = {"I", "E"}` but `B` and `N` appear in
+  the wild. The parser also accepts non-spec IntervalLengths (1, 60,
+  etc.); strict callers should compare against
+  `spec.ALLOWED_INTERVAL_LENGTHS` (= `{5, 15, 30}`).
+- **Migration from v1**: `NEMReader` still works. The internal
+  `aemo_mdff_reader.nemstructure` package is gone — see the API table
+  above. `pandas` is now opt-in. See `CHANGELOG.md` for details.
+
+## Performance
+
+420,480 readings (4 NMIs × 365 days × 5-min, 2.8 MiB CSV), Python 3.11:
+
+| Operation                          | Time   |
+| ---------------------------------- | -----: |
+| `for r in parse(path): ...`        | 0.45 s |
+| `parse_to_columns(path)`           | 0.21 s |
+| `to_dataframe(path)` (pandas)      | 0.76 s |
+
+~2.7× faster than v1 end-to-end; reproduce with
+`python benchmarks/bench_parser.py`.
+
+## Large files
+
+The parser is built to scale to gigabyte-class NEM12 files without
+loading them into RAM. Measured peak memory delta on a synthetic
+**10.5 M-reading file** (100 NMIs × 365 days × 5-min, 71 MiB CSV),
+Python 3.12:
+
+| API                                      | Memory profile | Peak Δ |
+| ---------------------------------------- | -------------- | -----: |
+| `for r in parse(path): ...`              | streaming      | **1.3 MiB** |
+| `daily_totals(parse(path))`              | streaming      | **0 MiB** |
+| `write_csv(parse(path), out)`            | streaming      | **0 MiB** |
+| `iter_dataframes(path, chunk_size=N)`    | bounded O(N)   | **~30 MiB / 100k** |
+| `iter_columns_chunks(path, chunk_size=N)`| bounded O(N)   | **~10 MiB / 100k** |
+| `parse_to_columns(path)`                 | full materialise | ~600 MiB |
+| `list(parse(path))` / `to_dataframe(path)` / `NEMReader.read_from_file()` | full materialise | ~2.5 GiB |
+
+Rule of thumb: stay on the streaming or chunked APIs for any file
+larger than a few hundred MiB. The chunked variants make pandas-based
+workflows safe on arbitrarily large inputs:
+
+```python
+from aemo_mdff_reader import iter_dataframes
+
+# Process a multi-GiB file 50,000 readings at a time.
+for df in iter_dataframes("huge.csv", chunk_size=50_000):
+    daily = df.groupby(["NMI", "IntervalDate"])["Value"].sum()
+    daily.to_csv("out.csv", mode="a", header=False)
+```
+
+The `NEMReader` facade and `to_dataframe(path)` materialise their
+inputs by design (so `len(reader)` and `df.iloc[...]` work). Avoid
+them for files that won't fit in RAM.
+
+## Development
+
+```bash
+git clone https://github.com/Utilified/aemo-mdff-reader.git
+cd aemo-mdff-reader
+pip install -e .[dev]
+pytest
+```
+
+CI runs ruff, mypy --strict, the test matrix on Python 3.9 → 3.12 /
+Linux / macOS / Windows, `pip-audit`, `bandit`, and a wheel-install
+smoke test. Releases are signed with sigstore and published to PyPI
+via Trusted Publishing on tag.
+
+## License
+
+MIT — see [LICENSE](LICENSE).