liander-open-data 0.1.0__tar.gz

liander_open_data-0.1.0/.gitignore

@@ -0,0 +1,22 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+
+# Local data (too large for git / PyPI; see docs/installation.md)
+data/Profiles/
+*.db
+*.db.tmp
+
+# Tooling
+site/
+.pytest_cache/
+.ruff_cache/
+.coverage
+coverage.xml

liander_open_data-0.1.0/CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
+[Semantic Versioning](https://semver.org/).
+
+## [0.1.0] - 2026-10-10
+
+### Added
+
+- `build_database()` / `liander-open-data build`: build a single SQLite file
+  from the Liander profile CSVs and the SBI dictionary.
+- `ProfileDatabase` query API: `get_profile`, `get_profiles`, `lookup`,
+  `recommended_profile`, `dictionary`, `list_profiles` and `query`.
+- Parser for the Liander *Overzicht met per SBI-code een aanbevolen profiel*
+  workbook, with a cleaned copy bundled in the package.
+- Command line interface with `build`, `info`, `list`, `lookup`, `search`,
+  `export` and `dictionary` commands.

liander_open_data-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Lingkang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

liander_open_data-0.1.0/PKG-INFO

@@ -0,0 +1,122 @@
+Metadata-Version: 2.5
+Name: liander-open-data
+Version: 0.1.0
+Summary: Lightweight SQLite database and Python API for Liander's open SBI load profiles.
+Project-URL: Homepage, https://github.com/lingkang95/Liander_open_data
+Project-URL: Documentation, https://github.com/lingkang95/Liander_open_data/tree/master/docs
+Project-URL: Source, https://github.com/lingkang95/Liander_open_data
+Project-URL: Issues, https://github.com/lingkang95/Liander_open_data/issues
+Project-URL: Changelog, https://github.com/lingkang95/Liander_open_data/blob/master/CHANGELOG.md
+Author-email: Lingkang <l.jin@tue.nl>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: electricity,energy,liander,load profiles,open data,sbi,sqlite
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.23
+Requires-Dist: pandas>=1.5
+Provides-Extra: all
+Requires-Dist: openpyxl>=3.1; extra == 'all'
+Requires-Dist: pyarrow>=12; extra == 'all'
+Provides-Extra: excel
+Requires-Dist: openpyxl>=3.1; extra == 'excel'
+Provides-Extra: parquet
+Requires-Dist: pyarrow>=12; extra == 'parquet'
+Description-Content-Type: text/markdown
+
+# liander-open-data
+
+[![PyPI](https://img.shields.io/pypi/v/liander-open-data.svg)](https://pypi.org/project/liander-open-data/)
+[![Python](https://img.shields.io/pypi/pyversions/liander-open-data.svg)](https://pypi.org/project/liander-open-data/)
+[![CI](https://github.com/lingkang95/Liander_open_data/actions/workflows/ci.yml/badge.svg)](https://github.com/lingkang95/Liander_open_data/actions/workflows/ci.yml)
+
+A lightweight **SQLite database** and Python API for the
+[Liander open data](https://www.liander.nl/over-ons/open-data) **SBI load profiles**:
+15-minute electricity profiles per SBI code (Dutch Standard Industrial
+Classification), together with Liander's dictionary of the *recommended
+profile per SBI code*.
+
+- One portable `.db` file holding all profiles and the SBI dictionary, which
+  any SQLite tool can open.
+- A small Python API that returns `pandas` DataFrames.
+- A `liander-open-data` command line tool.
+- The SBI dictionary (1,427 codes) ships with the package and works without
+  any download.
+
+## Installation
+
+```bash
+pip install liander-open-data
+# optional extras: read the original .xlsx dictionary / export Parquet
+pip install "liander-open-data[excel,parquet]"
+```
+
+## Quick start
+
+The profile CSVs (`<SBI code>.csv`, ~1.2 GB) are not bundled. Download them
+from Liander and build the database once:
+
+```bash
+liander-open-data build data/Profiles -o liander_profiles.db
+```
+
+Then query it from Python:
+
+```python
+from liander_open_data import ProfileDatabase
+
+with ProfileDatabase("liander_profiles.db") as db:
+    db.lookup("8411")                        # dictionary entry for an SBI code
+    df = db.get_profile("8411", columns=["mean", "p90"])
+    rec = db.recommended_profile("84111")    # Liander's recommended profile
+    wide = db.get_profiles(["01", "8411"], column="mean", start="2023-06-01")
+```
+
+Or from the command line:
+
+```bash
+liander-open-data lookup 8411
+liander-open-data search onderwijs --with-profile
+liander-open-data export 8411 -o 8411.csv --columns mean p90
+```
+
+Or with plain SQL:
+
+```sql
+SELECT code, AVG(mean) AS avg_load
+FROM profile_timeseries
+WHERE timestamp BETWEEN '2023-07-01' AND '2023-08-01'
+GROUP BY code ORDER BY avg_load DESC;
+```
+
+## Documentation
+
+Full documentation, covering the data model, CLI reference, API reference and
+the release guide, is in the [`docs/`](docs/) folder. To browse it as a
+website locally, run `uv run mkdocs serve`.
+
+## Development
+
+```bash
+uv sync --all-groups
+uv run pytest
+uv run mkdocs serve      # live documentation preview
+```
+
+## Data and license
+
+The code is released under the MIT license. The profiles and SBI dictionary
+are published by Liander N.V. as open data; check Liander's terms before
+redistributing them.

liander_open_data-0.1.0/README.md

@@ -0,0 +1,84 @@
+# liander-open-data
+
+[![PyPI](https://img.shields.io/pypi/v/liander-open-data.svg)](https://pypi.org/project/liander-open-data/)
+[![Python](https://img.shields.io/pypi/pyversions/liander-open-data.svg)](https://pypi.org/project/liander-open-data/)
+[![CI](https://github.com/lingkang95/Liander_open_data/actions/workflows/ci.yml/badge.svg)](https://github.com/lingkang95/Liander_open_data/actions/workflows/ci.yml)
+
+A lightweight **SQLite database** and Python API for the
+[Liander open data](https://www.liander.nl/over-ons/open-data) **SBI load profiles**:
+15-minute electricity profiles per SBI code (Dutch Standard Industrial
+Classification), together with Liander's dictionary of the *recommended
+profile per SBI code*.
+
+- One portable `.db` file holding all profiles and the SBI dictionary, which
+  any SQLite tool can open.
+- A small Python API that returns `pandas` DataFrames.
+- A `liander-open-data` command line tool.
+- The SBI dictionary (1,427 codes) ships with the package and works without
+  any download.
+
+## Installation
+
+```bash
+pip install liander-open-data
+# optional extras: read the original .xlsx dictionary / export Parquet
+pip install "liander-open-data[excel,parquet]"
+```
+
+## Quick start
+
+The profile CSVs (`<SBI code>.csv`, ~1.2 GB) are not bundled. Download them
+from Liander and build the database once:
+
+```bash
+liander-open-data build data/Profiles -o liander_profiles.db
+```
+
+Then query it from Python:
+
+```python
+from liander_open_data import ProfileDatabase
+
+with ProfileDatabase("liander_profiles.db") as db:
+    db.lookup("8411")                        # dictionary entry for an SBI code
+    df = db.get_profile("8411", columns=["mean", "p90"])
+    rec = db.recommended_profile("84111")    # Liander's recommended profile
+    wide = db.get_profiles(["01", "8411"], column="mean", start="2023-06-01")
+```
+
+Or from the command line:
+
+```bash
+liander-open-data lookup 8411
+liander-open-data search onderwijs --with-profile
+liander-open-data export 8411 -o 8411.csv --columns mean p90
+```
+
+Or with plain SQL:
+
+```sql
+SELECT code, AVG(mean) AS avg_load
+FROM profile_timeseries
+WHERE timestamp BETWEEN '2023-07-01' AND '2023-08-01'
+GROUP BY code ORDER BY avg_load DESC;
+```
+
+## Documentation
+
+Full documentation, covering the data model, CLI reference, API reference and
+the release guide, is in the [`docs/`](docs/) folder. To browse it as a
+website locally, run `uv run mkdocs serve`.
+
+## Development
+
+```bash
+uv sync --all-groups
+uv run pytest
+uv run mkdocs serve      # live documentation preview
+```
+
+## Data and license
+
+The code is released under the MIT license. The profiles and SBI dictionary
+are published by Liander N.V. as open data; check Liander's terms before
+redistributing them.

liander_open_data-0.1.0/docs/changelog.md

@@ -0,0 +1 @@
+--8<-- "CHANGELOG.md"

liander_open_data-0.1.0/docs/cli.md

@@ -0,0 +1,89 @@
+# Command line
+
+Installing the package provides the `liander-open-data` command (also
+available as `python -m liander_open_data`). Commands that read a database
+accept `--db PATH` (default: `liander_profiles.db` in the current
+directory).
+
+```text
+liander-open-data [--version] [-v] COMMAND ...
+```
+
+| Command      | Purpose                                                   |
+|--------------|-----------------------------------------------------------|
+| `build`      | Build the database from the profile CSVs                  |
+| `info`       | Show build metadata                                       |
+| `list`       | List all stored profiles                                  |
+| `lookup`     | Show the dictionary entry of one SBI code                 |
+| `search`     | Search codes by description text or code prefix           |
+| `export`     | Export a profile to CSV or Parquet                        |
+| `dictionary` | Convert the SBI workbook (or the bundled copy) to CSV     |
+
+## `build`
+
+```bash
+liander-open-data build PROFILES_DIR [-o OUTPUT] [-d DICTIONARY] [--overwrite] [-q]
+```
+
+- `PROFILES_DIR`: directory containing `<SBI code>.csv` files.
+- `-o, --output`: database file to create (default `liander_profiles.db`).
+- `-d, --dictionary`: `.xlsx` workbook or `.csv` export. Defaults to the
+  bundled dictionary.
+- `--overwrite`: replace an existing database.
+- `-q, --quiet`: hide the progress bar.
+
+## `lookup`
+
+```bash
+$ liander-open-data lookup 8411
+                  code: 8411
+           description: Algemeen overheidsbestuur
+               warning: None
+          n_timeseries: 638
+          best_profile: 841
+      standard_profile: KO_KANTOOR_ONDERWIJS
+     corr_best_profile: 0.3404778207107602
+ corr_standard_profile: 0.2784444209758644
+            ko_profile: KO_KANTOOR_ONDERWIJS
+           has_profile: True
+   recommended_profile: 841
+```
+
+Unknown codes fall back to the nearest parent code.
+
+## `search`
+
+```bash
+liander-open-data search onderwijs --with-profile
+liander-open-data search 85
+```
+
+## `export`
+
+```bash
+liander-open-data export CODE -o FILE [-c COL ...] [--start TS] [--end TS] [--recommended]
+```
+
+- The file extension of `FILE` sets the format: `.parquet` needs the
+  `parquet` extra, anything else is written as CSV.
+- `--recommended` exports the profile Liander recommends for `CODE` instead
+  of `CODE`'s own profile.
+
+```bash
+liander-open-data export 8411 -o 8411_summer.csv -c mean p90 \
+    --start 2023-06-01 --end 2023-09-01
+```
+
+## `dictionary`
+
+```bash
+# convert a (newer) workbook to the clean CSV format
+liander-open-data dictionary "Overzicht met per SBI-code een aanbevolen profiel.xlsx" -o sbi.csv
+# or dump the bundled copy
+liander-open-data dictionary -o sbi.csv
+```
+
+## Exit status
+
+`0` on success. `1` on an expected error such as a missing file, an unknown
+code, or a bad column; the message is printed to stderr.

liander_open_data-0.1.0/docs/contributing.md

@@ -0,0 +1,52 @@
+# Contributing
+
+## Set up
+
+```bash
+git clone https://github.com/lingkang95/Liander_open_data.git
+cd Liander_open_data
+uv sync --all-groups
+```
+
+## Everyday commands
+
+```bash
+uv run pytest                  # tests + doctests
+uv run pytest --cov            # with coverage
+uv run ruff check .            # lint
+uv run ruff format .           # format
+uv run mkdocs serve            # docs at http://127.0.0.1:8000
+```
+
+The tests use small synthetic CSVs and need no real Liander data.
+
+## Project layout
+
+```text
+src/liander_open_data/
+├── __init__.py          # public API
+├── database.py          # schema, build_database, ProfileDatabase
+├── dictionary.py        # SBI workbook parsing, SBICode
+├── cli.py               # liander-open-data command
+└── data/
+    └── sbi_dictionary.csv   # bundled, cleaned dictionary
+tests/                   # pytest suite
+docs/                    # this documentation (MkDocs Material)
+```
+
+## Updating the bundled dictionary
+
+When Liander publishes a new workbook:
+
+```bash
+uv run liander-open-data dictionary path/to/new.xlsx -o src/liander_open_data/data/sbi_dictionary.csv
+uv run pytest
+```
+
+Review the diff of the CSV, add a changelog entry and release a new version.
+
+## Changing the schema
+
+If you change the layout of the tables in `database.py`, bump
+`SCHEMA_VERSION`, update [Data model](data-model.md), and note the change in
+the changelog.

liander_open_data-0.1.0/docs/data-model.md

@@ -0,0 +1,114 @@
+# Data model
+
+The database is a plain SQLite 3 file. You can open it with
+`ProfileDatabase`, with `sqlite3.connect()`, or with any SQL tool.
+
+```mermaid
+erDiagram
+    sbi_codes ||--o| profiles : "has"
+    profiles ||--|{ profile_values : "contains"
+    timestamps ||--|{ profile_values : "indexes"
+
+    sbi_codes {
+        TEXT code PK
+        TEXT description
+        INTEGER level
+        TEXT parent_code
+        TEXT warning
+        INTEGER n_timeseries
+        TEXT best_profile
+        TEXT standard_profile
+        REAL corr_best_profile
+        REAL corr_standard_profile
+        TEXT ko_profile
+        INTEGER has_profile
+    }
+    profiles {
+        TEXT code PK
+        INTEGER n_points
+        TEXT start
+        TEXT end
+        INTEGER interval_minutes
+        TEXT source_file
+    }
+    timestamps {
+        INTEGER slot PK
+        TEXT timestamp
+    }
+    profile_values {
+        TEXT code PK
+        INTEGER slot PK
+        REAL min
+        REAL p10
+        REAL median
+        REAL mean
+        REAL p90
+        REAL max
+        REAL median_renorm
+        REAL mean_renorm
+        REAL p90_renorm
+    }
+```
+
+## Tables
+
+### `sbi_codes`
+
+The full SBI dictionary, with one row per code. See [The data](data.md#sbi-dictionary)
+for the meaning of each column. Two columns are derived:
+
+- `level`: number of digits (2 to 5).
+- `parent_code`: the code one level up (`8411` → `841`), or `NULL` for
+  2-digit codes.
+
+### `profiles`
+
+One row per profile loaded from a CSV: number of points, first and last
+timestamp, and the detected interval in minutes.
+
+### `timestamps`
+
+The time axis shared by all profiles. Storing an integer `slot` per value
+instead of the full timestamp string keeps the database compact.
+Timestamps use ISO format `YYYY-MM-DD HH:MM:SS`, so string comparison
+matches chronological order.
+
+### `profile_values`
+
+The measurements, one row per `(code, slot)`. This is a `WITHOUT ROWID`
+table clustered on its primary key, so reading one profile is a single range
+scan.
+
+### `profile_timeseries` (view)
+
+`profile_values` joined with `timestamps`, for convenient ad-hoc SQL:
+
+```sql
+SELECT timestamp, mean, p90
+FROM profile_timeseries
+WHERE code = '8411' AND timestamp >= '2023-12-25' AND timestamp < '2023-12-27';
+```
+
+### `metadata`
+
+Key/value pairs describing the build: `schema_version`, `package_version`,
+`built_at`, `n_profiles`, `n_sbi_codes`, `n_timestamps` and `value_columns`.
+
+## Size and performance
+
+For the 2023 dataset (203 profiles × 35,040 points × 9 statistics):
+
+| Item                                | Value       |
+|-------------------------------------|-------------|
+| Source CSVs                         | ≈ 1.2 GB    |
+| SQLite database                     | ≈ 717 MB    |
+| Build time                          | < 1 min     |
+| `get_profile()` (one full year)     | ≈ 0.1 s     |
+| `get_profiles()` (all, one column)  | ≈ 5 s       |
+
+## Schema versioning
+
+`SCHEMA_VERSION` is stored in the `metadata` table. When a new release
+changes the layout incompatibly, the version is bumped and
+`ProfileDatabase` logs a warning when it opens an older file. Rebuild with
+`--overwrite` to upgrade.

liander_open_data-0.1.0/docs/data.md

@@ -0,0 +1,71 @@
+# The data
+
+## Profile CSVs
+
+Each file `<SBI code>.csv` contains one calendar year (2023) of 15-minute
+values, 35,040 rows in total. Each row holds statistics over all metered
+connections of businesses registered under that SBI code:
+
+| Column          | Meaning                                                        |
+|-----------------|----------------------------------------------------------------|
+| *(index)*       | Timestamp of the start of the 15-minute interval               |
+| `min`           | Minimum over all connections                                   |
+| `p10`           | 10th percentile                                                |
+| `median`        | Median                                                         |
+| `mean`          | Mean                                                           |
+| `p90`           | 90th percentile                                                |
+| `max`           | Maximum                                                        |
+| `median_renorm` | Median, renormalised                                           |
+| `mean_renorm`   | Mean, renormalised                                             |
+| `p90_renorm`    | 90th percentile, renormalised                                  |
+
+Values are normalised and dimensionless, in the range 0 to 1. Use them as a
+*shape* and scale it to an actual consumption (see [Recipes](recipes.md)).
+
+!!! info "Timestamps"
+    The source files contain exactly 365 × 96 = 35,040 points with no
+    daylight-saving gaps or duplicates. Timestamps are stored as given,
+    without a time zone.
+
+SBI codes are hierarchical: `84` → `842` → `8423` → `84232`. Profiles exist
+at several levels, and a profile at a higher level aggregates more
+connections.
+
+## SBI dictionary
+
+The workbook *Overzicht met per SBI-code een aanbevolen profiel.xlsx* lists
+all 1,427 SBI codes. The package renames its columns to English:
+
+| Workbook column                     | Field                   | Description                                                  |
+|-------------------------------------|-------------------------|--------------------------------------------------------------|
+| `code`                              | `code`                  | SBI code (string, leading zeros kept)                        |
+| `description`                       | `description`           | Dutch description                                            |
+| `Warning`                           | `warning`               | Set when < 10 time series were available (less reliable)     |
+| `Tijdseries in deze SBI`            | `n_timeseries`          | Number of connections behind the profile                     |
+| `Profiel hoogste correlatie`        | `best_profile`          | Profile with the highest correlation (SBI code or `KO_*`)    |
+| `Standaard profiel`                 | `standard_profile`      | Standard `KO_*` profile for the sector                       |
+| `correlatie geaggregeerde profiel`  | `corr_best_profile`     | Correlation with `best_profile`                              |
+| `correlatie standaard profiel`      | `corr_standard_profile` | Correlation with `standard_profile`                          |
+| `KO_profiel`                        | `ko_profile`            | Recommended standard profile                                 |
+| `Profiel aanwezig`                  | `has_profile`           | A measured profile exists for this exact code                |
+
+### Cleaning rules
+
+The parser applies these rules to the raw workbook:
+
+- Codes are kept as strings. Codes that lost their leading zero in Excel
+  (`2`, `6`, `8`, `9`) are restored to `02`, `06`, `08`, `09`.
+- `"NA"`, `"nan"`, empty cells and the placeholder `0` in text columns become
+  `None`.
+- A correlation of exactly `0` is a placeholder for *not computed* and
+  becomes `None`.
+- Leading and trailing whitespace is stripped from descriptions.
+- In the database, `has_profile` is recomputed from the CSV files actually
+  loaded, so it always matches the `profiles` table.
+
+### Standard profiles (`KO_*`)
+
+Values such as `KO_INDUSTRIE`, `KO_KANTOOR_ONDERWIJS`, `KO_AGRARIER`,
+`KO_GLASTUINBOUW`, `KO_LOGISTIEK` and `KO_OVERIG` refer to standard
+profiles. They are **not** included in the profile CSVs. `Onbekend` means
+*unknown*, and `Onbekend, geen eans` means *unknown, no connections (EANs)*.

liander_open_data-0.1.0/docs/index.md

@@ -0,0 +1,51 @@
+# liander-open-data
+
+**liander-open-data** packs the
+[Liander open data](https://www.liander.nl/over-ons/open-data) *SBI load
+profiles* into a single, lightweight **SQLite database** and gives you a
+small Python API and command line tool to query it.
+
+Liander, a Dutch distribution system operator, publishes 15-minute
+electricity consumption profiles aggregated per **SBI code** (*Standaard
+Bedrijfsindeling*, the Dutch Standard Industrial Classification), plus a
+workbook that tells you **which profile to use for each SBI code**. This
+package combines both.
+
+## Highlights
+
+- **One file, no server.** All 203 profiles (≈7 million rows) and 1,427 SBI
+  codes go into one portable `.db` file. Python, the `sqlite3` CLI,
+  DBeaver, DuckDB and R can all open it.
+- **Lossless.** Values are stored as 64-bit floats and round-trip exactly to
+  the source CSVs.
+- **Fast enough.** One full-year profile loads in about 0.1 s, and all
+  profiles for a single statistic load in about 5 s.
+- **Batteries included.** The cleaned SBI dictionary ships with the package,
+  so `load_dictionary()` works without any download.
+- **Typed, tested, documented.**
+
+## At a glance
+
+```python
+from liander_open_data import ProfileDatabase
+
+with ProfileDatabase("liander_profiles.db") as db:
+    entry = db.lookup("8411")
+    print(entry.description, "->", entry.recommended_profile)
+    # Algemeen overheidsbestuur -> 841
+
+    df = db.recommended_profile("8411", columns=["mean", "p90"])
+    df.resample("D").mean().plot()
+```
+
+```mermaid
+flowchart LR
+    A["Profiles/*.csv<br/>(1.2 GB)"] --> C[build_database]
+    B["SBI dictionary<br/>(.xlsx / bundled)"] --> C
+    C --> D[("liander_profiles.db<br/>SQLite")]
+    D --> E[ProfileDatabase API]
+    D --> F[liander-open-data CLI]
+    D --> G[Any SQL tool]
+```
+
+Next: [Installation](installation.md) · [Quick start](quickstart.md)