flextool 4.0.0__tar.gz

flextool-4.0.0/LICENSE.txt

@@ -0,0 +1,19 @@
+© International Renewable Energy Agency 2018-2022
+
+The FlexTool is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License
+as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+
+The FlexTool is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; 
+without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public License along with the FlexTool.  
+If not, see <https://www.gnu.org/licenses/>.
+
+Author: Juha Kiviluoma (2017-2022), VTT Technical Research Centre of Finland
+
+
+
+
+-----------------------------------------------
+IRENA FlexTool uses the HiGHS solver, which has its own license.
+HiGHS is licensed with MIT license. Please see https://github.com/ERGO-Code/HiGHS for more information.

flextool-4.0.0/PKG-INFO

@@ -0,0 +1,217 @@
+Metadata-Version: 2.4
+Name: flextool
+Version: 4.0.0
+Summary: IRENA FlexTool is an energy and power systems model for understanding the role of variable power generation in future energy systems.
+Author: IRENA FlexTool contributors
+License: © International Renewable Energy Agency 2018-2022
+        
+        The FlexTool is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License
+        as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+        
+        The FlexTool is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; 
+        without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more details.
+        
+        You should have received a copy of the GNU Lesser General Public License along with the FlexTool.  
+        If not, see <https://www.gnu.org/licenses/>.
+        
+        Author: Juha Kiviluoma (2017-2022), VTT Technical Research Centre of Finland
+        
+        
+        
+        
+        -----------------------------------------------
+        IRENA FlexTool uses the HiGHS solver, which has its own license.
+        HiGHS is licensed with MIT license. Please see https://github.com/ERGO-Code/HiGHS for more information.
+Project-URL: Documentation, https://irena-flextool.github.io/flextool/
+Project-URL: Repository, https://github.com/irena-flextool/flextool
+Project-URL: Issues, https://github.com/irena-flextool/flextool/issues
+Keywords: energy-systems,optimization,linear-programming,mixed-integer-programming,power-systems,investment-planning,highs,spine
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: polar-high>=3.7.0
+Requires-Dist: highspy<=1.14.0
+Requires-Dist: polars>=1.40
+Requires-Dist: pandas
+Requires-Dist: spinedb-api
+Requires-Dist: openpyxl
+Requires-Dist: odfpy
+Requires-Dist: python-calamine
+Requires-Dist: xlsxwriter
+Requires-Dist: pyyaml
+Requires-Dist: matplotlib
+Requires-Dist: psutil
+Requires-Dist: sv-ttk
+Requires-Dist: screeninfo
+Requires-Dist: pywin32; sys_platform == "win32"
+Provides-Extra: toolbox
+Requires-Dist: spinetoolbox; extra == "toolbox"
+Provides-Extra: viewer
+Requires-Dist: tsdownsample>=0.1.3; extra == "viewer"
+Requires-Dist: networkx>=3.0; extra == "viewer"
+Provides-Extra: engine-polars
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-timeout>=2.3; extra == "dev"
+Requires-Dist: pytest-xdist>=3.0; extra == "dev"
+Requires-Dist: ruff>=0.5; extra == "dev"
+Dynamic: license-file
+
+[![PyPI version](https://img.shields.io/pypi/v/flextool.svg)](https://pypi.org/project/flextool/)
+[![Documentation Status](https://img.shields.io/badge/Documentation-passing-brightgreen)](https://irena-flextool.github.io/flextool/)
+[![Python](https://img.shields.io/badge/python-3.11%20|%203.12-blue.svg)](https://www.python.org/downloads/release/python-3120/)
+
+![IRENA FlexTool logo](./docs/img/flextool_logo.png)
+
+IRENA FlexTool is an energy and power systems model for understanding the role of variable power generation in future energy systems. It performs capacity expansion planning as well as operational planning.
+
+> [!NOTE]
+> The previous `master` branch is deprecated and no longer updated — `main` is
+> now FlexTool. If you still run FlexTool from `master`, migrate with a fresh
+> parallel install (separate directory and venv); see
+> [Installation](#installation) so your existing setup is left untouched.
+
+## What's new
+
+- **A lot faster.** The model build and pre-processing now run as pure Python
+  instead of the previous GLPSOL-based text pre-processing and AMPL-style model
+  translation. There is no LP file written to disk and re-read between build and
+  solve.
+- **Automatic problem scaling.** The `polar-high` matrix layer automatically
+  scales the optimisation problem before it reaches the solver. Better numerical
+  conditioning often speeds up the HiGHS solve considerably — sometimes
+  dramatically — so total run time usually improves well beyond the faster build
+  alone. The solver is still HiGHS; the gain comes from handing it a
+  better-conditioned problem.
+- **A new, easy-to-use interface.** The **FlexTool GUI** is a standalone
+  application (`python -m flextool.gui`) that manages projects and input
+  sources, runs scenarios, and lets you **browse the results** — a result viewer
+  reads outputs directly and supports single-scenario, scenario-comparison, and
+  network-graph views with keyboard navigation. See the
+  [FlexTool GUI guide](https://irena-flextool.github.io/flextool/flextool_gui_interface/).
+- **Spine Toolbox is still a parallel interface.** Advanced users who build
+  multi-tool workflows or integrate FlexTool with other models can keep using
+  the [Spine Toolbox workflow](https://irena-flextool.github.io/flextool/spine_toolbox/).
+- **The same model and data.** Existing input databases continue to work; the
+  change is in how the matrix is generated and solved, not in the model
+  formulation.
+
+This is IRENA FlexTool v4 (see the current version in CHANGELOG.md). Report any bugs or difficulties in the [issue tracker](https://github.com/irena-flextool/flextool/issues). 
+The previous version of IRENA FlexTool can be found in https://www.irena.org/energytransition/Energy-System-Models-and-Data/IRENA-FlexTool.
+
+## Under the hood: Rust-based matrix generation
+
+FlexTool is a thin Python layer over a Rust engine. It reads its input data and
+builds the optimisation matrix as [polars](https://pola.rs/) DataFrames via
+[`polar-high`](https://github.com/nodal-tools/polar-high), then solves with
+[HiGHS](https://highs.dev/). Variables and parameters are polars frames, so
+multiplications become joins and aggregations become group-bys — the heavy
+coefficient work runs inside polars' Rust core rather than as per-coefficient
+Python objects, and the matrix goes straight to HiGHS through `highspy` with no
+intermediate LP/MPS file. `polar-high` is a general-purpose, domain-free
+modelling layer published separately (Apache-2.0, `pip install polar-high`); it
+also applies the automatic problem scaling noted above, and its
+[benchmark](https://nodal-tools.fi/polar-high/compare/benchmark/) compares it
+against linopy and Pyomo on the same HiGHS solver.
+
+## Installation
+
+### Check your Python (and possibly install it)
+
+FlexTool requires **Python 3.11+**.
+
+Open a terminal (Windows: Command Prompt or PowerShell; macOS: Terminal; Linux: your shell)
+and check which Python you have:
+
+| Platform | Command |
+|---|---|
+| Windows | `py --version` (falls back to `python --version`) |
+| macOS / Linux | `python3 --version` |
+
+The output looks like `Python 3.12.4`. If the command is not found, prints
+something below 3.11, or (on Windows) opens the Microsoft Store, install Python
+from https://www.python.org/downloads/ — on Windows tick
+**"Add python.exe to PATH"** in the installer. Then close and reopen the
+terminal so the new PATH takes effect.
+
+> On Windows, use `py` in place of `python` in the commands below if `python`
+> is not recognised. If several versions are installed, `py -3.11 --version`
+> (or `-3.12`, …) picks a specific one, and `py -0` lists them all.
+> On macOS/Linux, `python` often means Python 2 or does not exist at all, so
+> use `python3`. Inside an activated virtual environment, plain `python` always
+> means that environment's Python on every platform.
+
+
+### Install from PyPI (recommended)
+
+Install FlexTool into a virtual environment to keep its packages separate from
+other Python applications. If you want to install into the base Python (perhaps
+because you do not otherwise use Python), just run the last command from below,
+`pip install flextool`.
+
+```
+python -m venv .venv        # Windows: py -m venv .venv
+                            # macOS/Linux: python3 -m venv .venv
+# activate — Windows PowerShell:  .\.venv\Scripts\Activate.ps1
+#            Windows CMD:         .\.venv\Scripts\activate.bat
+#            Linux / macOS:       . .venv/bin/activate
+python -m pip install --upgrade pip
+pip install flextool
+```
+
+Your prompt should now show `(.venv)`. Verify with `python --version` — it
+should report 3.11 or newer.
+
+(If PowerShell blocks the activate script, run once per user:
+`Set-ExecutionPolicy -Scope CurrentUser RemoteSigned`.)
+
+Launch the FlexTool GUI:
+
+```
+flextool
+```
+
+— equivalently `python -m flextool.gui`. The GUI creates and manages your
+projects and input databases; to seed the bundled template and example
+databases (handy for the CLI or Spine Toolbox workflows) run `flextool-update`.
+
+### Install from source (developers / latest `main`)
+
+To participate in FlexTool development — or to run the not-yet-released `main`
+— clone the repo and install it editable:
+
+```powershell
+git clone https://github.com/irena-flextool/flextool.git
+cd flextool
+python -m venv .venv
+.\.venv\Scripts\Activate.ps1     # Linux / macOS:  . .venv/bin/activate
+python -m pip install --upgrade pip
+pip install -e .
+```
+
+The editable install picks up source changes on `git pull`; re-run
+`pip install -e .` only when a dependency pin in `pyproject.toml` changes.
+
+### Update FlexTool:
+
+```
+# If using virtual environment (venv), then activate venv first:
+#   Windows PowerShell:  .\.venv\Scripts\Activate.ps1
+#   Windows CMD:         .\.venv\Scripts\activate.bat
+#   Linux / macOS:       . .venv/bin/activate
+pip install --upgrade flextool
+```
+
+## [Documentation](https://irena-flextool.github.io/flextool/) and [installation](https://irena-flextool.github.io/flextool/install_toolbox/)
+
+> [!IMPORTANT]
+> Installation, user guide and ***documentation*** can be found at: https://irena-flextool.github.io/flextool/.

flextool-4.0.0/README.md

@@ -0,0 +1,149 @@
+[![PyPI version](https://img.shields.io/pypi/v/flextool.svg)](https://pypi.org/project/flextool/)
+[![Documentation Status](https://img.shields.io/badge/Documentation-passing-brightgreen)](https://irena-flextool.github.io/flextool/)
+[![Python](https://img.shields.io/badge/python-3.11%20|%203.12-blue.svg)](https://www.python.org/downloads/release/python-3120/)
+
+![IRENA FlexTool logo](./docs/img/flextool_logo.png)
+
+IRENA FlexTool is an energy and power systems model for understanding the role of variable power generation in future energy systems. It performs capacity expansion planning as well as operational planning.
+
+> [!NOTE]
+> The previous `master` branch is deprecated and no longer updated — `main` is
+> now FlexTool. If you still run FlexTool from `master`, migrate with a fresh
+> parallel install (separate directory and venv); see
+> [Installation](#installation) so your existing setup is left untouched.
+
+## What's new
+
+- **A lot faster.** The model build and pre-processing now run as pure Python
+  instead of the previous GLPSOL-based text pre-processing and AMPL-style model
+  translation. There is no LP file written to disk and re-read between build and
+  solve.
+- **Automatic problem scaling.** The `polar-high` matrix layer automatically
+  scales the optimisation problem before it reaches the solver. Better numerical
+  conditioning often speeds up the HiGHS solve considerably — sometimes
+  dramatically — so total run time usually improves well beyond the faster build
+  alone. The solver is still HiGHS; the gain comes from handing it a
+  better-conditioned problem.
+- **A new, easy-to-use interface.** The **FlexTool GUI** is a standalone
+  application (`python -m flextool.gui`) that manages projects and input
+  sources, runs scenarios, and lets you **browse the results** — a result viewer
+  reads outputs directly and supports single-scenario, scenario-comparison, and
+  network-graph views with keyboard navigation. See the
+  [FlexTool GUI guide](https://irena-flextool.github.io/flextool/flextool_gui_interface/).
+- **Spine Toolbox is still a parallel interface.** Advanced users who build
+  multi-tool workflows or integrate FlexTool with other models can keep using
+  the [Spine Toolbox workflow](https://irena-flextool.github.io/flextool/spine_toolbox/).
+- **The same model and data.** Existing input databases continue to work; the
+  change is in how the matrix is generated and solved, not in the model
+  formulation.
+
+This is IRENA FlexTool v4 (see the current version in CHANGELOG.md). Report any bugs or difficulties in the [issue tracker](https://github.com/irena-flextool/flextool/issues). 
+The previous version of IRENA FlexTool can be found in https://www.irena.org/energytransition/Energy-System-Models-and-Data/IRENA-FlexTool.
+
+## Under the hood: Rust-based matrix generation
+
+FlexTool is a thin Python layer over a Rust engine. It reads its input data and
+builds the optimisation matrix as [polars](https://pola.rs/) DataFrames via
+[`polar-high`](https://github.com/nodal-tools/polar-high), then solves with
+[HiGHS](https://highs.dev/). Variables and parameters are polars frames, so
+multiplications become joins and aggregations become group-bys — the heavy
+coefficient work runs inside polars' Rust core rather than as per-coefficient
+Python objects, and the matrix goes straight to HiGHS through `highspy` with no
+intermediate LP/MPS file. `polar-high` is a general-purpose, domain-free
+modelling layer published separately (Apache-2.0, `pip install polar-high`); it
+also applies the automatic problem scaling noted above, and its
+[benchmark](https://nodal-tools.fi/polar-high/compare/benchmark/) compares it
+against linopy and Pyomo on the same HiGHS solver.
+
+## Installation
+
+### Check your Python (and possibly install it)
+
+FlexTool requires **Python 3.11+**.
+
+Open a terminal (Windows: Command Prompt or PowerShell; macOS: Terminal; Linux: your shell)
+and check which Python you have:
+
+| Platform | Command |
+|---|---|
+| Windows | `py --version` (falls back to `python --version`) |
+| macOS / Linux | `python3 --version` |
+
+The output looks like `Python 3.12.4`. If the command is not found, prints
+something below 3.11, or (on Windows) opens the Microsoft Store, install Python
+from https://www.python.org/downloads/ — on Windows tick
+**"Add python.exe to PATH"** in the installer. Then close and reopen the
+terminal so the new PATH takes effect.
+
+> On Windows, use `py` in place of `python` in the commands below if `python`
+> is not recognised. If several versions are installed, `py -3.11 --version`
+> (or `-3.12`, …) picks a specific one, and `py -0` lists them all.
+> On macOS/Linux, `python` often means Python 2 or does not exist at all, so
+> use `python3`. Inside an activated virtual environment, plain `python` always
+> means that environment's Python on every platform.
+
+
+### Install from PyPI (recommended)
+
+Install FlexTool into a virtual environment to keep its packages separate from
+other Python applications. If you want to install into the base Python (perhaps
+because you do not otherwise use Python), just run the last command from below,
+`pip install flextool`.
+
+```
+python -m venv .venv        # Windows: py -m venv .venv
+                            # macOS/Linux: python3 -m venv .venv
+# activate — Windows PowerShell:  .\.venv\Scripts\Activate.ps1
+#            Windows CMD:         .\.venv\Scripts\activate.bat
+#            Linux / macOS:       . .venv/bin/activate
+python -m pip install --upgrade pip
+pip install flextool
+```
+
+Your prompt should now show `(.venv)`. Verify with `python --version` — it
+should report 3.11 or newer.
+
+(If PowerShell blocks the activate script, run once per user:
+`Set-ExecutionPolicy -Scope CurrentUser RemoteSigned`.)
+
+Launch the FlexTool GUI:
+
+```
+flextool
+```
+
+— equivalently `python -m flextool.gui`. The GUI creates and manages your
+projects and input databases; to seed the bundled template and example
+databases (handy for the CLI or Spine Toolbox workflows) run `flextool-update`.
+
+### Install from source (developers / latest `main`)
+
+To participate in FlexTool development — or to run the not-yet-released `main`
+— clone the repo and install it editable:
+
+```powershell
+git clone https://github.com/irena-flextool/flextool.git
+cd flextool
+python -m venv .venv
+.\.venv\Scripts\Activate.ps1     # Linux / macOS:  . .venv/bin/activate
+python -m pip install --upgrade pip
+pip install -e .
+```
+
+The editable install picks up source changes on `git pull`; re-run
+`pip install -e .` only when a dependency pin in `pyproject.toml` changes.
+
+### Update FlexTool:
+
+```
+# If using virtual environment (venv), then activate venv first:
+#   Windows PowerShell:  .\.venv\Scripts\Activate.ps1
+#   Windows CMD:         .\.venv\Scripts\activate.bat
+#   Linux / macOS:       . .venv/bin/activate
+pip install --upgrade flextool
+```
+
+## [Documentation](https://irena-flextool.github.io/flextool/) and [installation](https://irena-flextool.github.io/flextool/install_toolbox/)
+
+> [!IMPORTANT]
+> Installation, user guide and ***documentation*** can be found at: https://irena-flextool.github.io/flextool/.

flextool-4.0.0/flextool/__init__.py

@@ -0,0 +1,41 @@
+"""FlexTool energy system optimization model package."""
+import os as _os
+# jemalloc decay config — set BEFORE the first ``import polars`` (which
+# happens transitively in the imports just below).  polars statically
+# links jemalloc; jemalloc reads ``_RJEM_MALLOC_CONF`` exactly once at
+# .so load.  Its default ``dirty_decay_ms`` (~10 s) is longer than the
+# gap between rolling-solve rolls, so each roll's freed GB-scale
+# coefficient frames stay resident as dirty pages when the next roll
+# starts allocating — the per-roll ``priv_dirty`` floor ratchets up
+# (~+2 GB/roll on the 9-roll DES run).  ``dirty_decay_ms:1000`` purges a
+# roll's freed pages ~1 s after they go idle (well before the next roll,
+# which takes minutes) without eagerly madvise-ing on every transient
+# free inside a hot operation.  ``muzzy_decay_ms:0`` forces MADV_DONTNEED
+# (so Private_Dirty actually drops) rather than leaving pages as
+# reclaimable-but-resident MADV_FREE.  glibc knobs (MALLOC_ARENA_MAX,
+# malloc_trim) do NOT touch polars memory — this is the lever that does.
+# ``setdefault`` so a shell-provided value still wins (A/B profiling).
+_os.environ.setdefault("_RJEM_MALLOC_CONF", "dirty_decay_ms:1000,muzzy_decay_ms:0")
+import warnings as _warnings
+# Silence requests' RequestsDependencyWarning.  The FlexTool venv ships a
+# ``requests`` whose pinned urllib3/charset_normalizer version ranges lag the
+# (working) installed versions, so ``import requests`` warns once at import
+# time.  It is harmless but leaks into Spine Toolbox's Tool console for any
+# path that pulls in requests (e.g. an ``http://`` Spine DB-server URL).
+# Match by message — set BEFORE requests is imported transitively below — so
+# we needn't import requests here just to reference its warning class, and the
+# filter stays robust across the venv's exact version numbers.
+_warnings.filterwarnings(
+    "ignore",
+    message=r"urllib3 .*doesn't match a supported version",
+)
+__all__ = [
+    'write_outputs',
+    'migrate_database',
+    'initialize_database',
+    'update_flextool',
+]
+from flextool.process_outputs import write_outputs  # noqa: E402  (after env/warning config above)
+from flextool.update_flextool import migrate_database, initialize_database, update_flextool  # noqa: E402
+
+name = "flextool"

flextool-4.0.0/flextool/_mem_sampler.py

@@ -0,0 +1,193 @@
+"""In-process memory sampler — timestamped, allocator-aware memory trace.
+
+RSS is a poor signal for this workload: it conflates shared, clean, and
+allocator-reserved-but-free pages with genuinely-live private data.  This
+sampler additionally records, per sample:
+
+* ``pss_gb`` / ``priv_dirty_gb`` / ``proc_swap_gb`` — from
+  ``/proc/self/smaps_rollup``.  ``Private_Dirty + Swap`` is the
+  allocator-agnostic "what this process truly owns" number (counts dirty
+  anon pages whether they live in glibc, mimalloc, or the C++ heap, and
+  whether resident or swapped).
+* ``glibc_inuse_gb`` / ``glibc_free_gb`` / ``glibc_mmap_gb`` — from
+  glibc ``mallinfo2()`` (``uordblks`` / ``fordblks`` / ``hblkhd``).  The
+  *live-vs-reserved* discriminator: ``fordblks`` is memory glibc has
+  freed but is holding on its arena free-list rather than returning to
+  the OS (what ``malloc_trim`` would release).  NOTE: only sees the glibc
+  allocator — HiGHS/C++ go through glibc, but polars (Rust) may use its
+  own allocator, so a large ``priv_dirty`` with small ``glibc_inuse``
+  points at the non-glibc (polars) side.
+* ``mem_used_gb`` — system used (``MemTotal - MemAvailable``), the number
+  most desktop system monitors plot.  ``available_gb`` / ``swap_used_gb``
+  kept for back-compat.
+
+Gated by ``FLEXTOOL_MEM_SAMPLER``; interval via
+``FLEXTOOL_MEM_SAMPLER_INTERVAL_MS`` (default 100 ms).
+"""
+from __future__ import annotations
+
+import ctypes
+import datetime
+import os
+import sys
+import threading
+import time
+
+
+class _Mallinfo2(ctypes.Structure):
+    # glibc >= 2.33.  All fields size_t (the legacy ``mallinfo`` used int
+    # and overflows past 2 GB — useless here).
+    _fields_ = [
+        (n, ctypes.c_size_t)
+        for n in (
+            "arena", "ordblks", "smblks", "hblks", "hblkhd", "usmblks",
+            "fsmblks", "uordblks", "fordblks", "keepcost",
+        )
+    ]
+
+
+def _make_mallinfo2():
+    """Return a callable giving (inuse, free, mmap) bytes, or None."""
+    try:
+        libc = ctypes.CDLL("libc.so.6")
+        fn = libc.mallinfo2  # AttributeError on glibc < 2.33
+        fn.restype = _Mallinfo2
+        fn.argtypes = []
+    except (OSError, AttributeError):
+        return None
+
+    def _read():
+        mi = fn()
+        return mi.uordblks, mi.fordblks, mi.hblkhd
+
+    return _read
+
+
+def start_mem_sampler() -> None:
+    """Start a daemon thread that samples memory at a fixed interval if FLEXTOOL_MEM_SAMPLER is set."""
+    if os.environ.get("FLEXTOOL_MEM_SAMPLER", "").strip().lower() in ("", "0", "false"):
+        return
+
+    pid = os.getpid()
+    log_path = os.environ.get("FLEXTOOL_MEM_SAMPLER_LOG", f"/tmp/flextool_mem_sampler_{pid}.log")
+    try:
+        interval_ms = max(20, min(10000, int(os.environ.get("FLEXTOOL_MEM_SAMPLER_INTERVAL_MS", "100"))))
+    except ValueError:
+        interval_ms = 100
+    interval_s = interval_ms / 1000.0
+
+    if not sys.platform.startswith("linux"):
+        with open(log_path, "a", buffering=1) as _f:
+            _f.write(
+                f"# flextool memory sampler pid={pid} WARNING: platform is not "
+                f"linux — sampler disabled\n"
+            )
+        return
+
+    _GB = float(2 ** 30)
+    _mallinfo2 = _make_mallinfo2()
+
+    def _read_kv_bytes(path: str, keys: tuple[str, ...]) -> dict[str, int]:
+        """Parse ``Key:   N kB`` lines from a /proc file → bytes."""
+        out: dict[str, int] = {}
+        want = set(keys)
+        with open(path) as fh:
+            for line in fh:
+                k = line.split(":", 1)[0]
+                if k in want:
+                    out[k] = int(line.split()[1]) * 1024
+                    if len(out) == len(want):
+                        break
+        return out
+
+    def _sample() -> dict[str, float]:
+        # Per-process private cost — allocator-agnostic.
+        rss = pss = priv_dirty = proc_swap = 0
+        try:
+            r = _read_kv_bytes(
+                "/proc/self/smaps_rollup",
+                ("Rss", "Pss", "Private_Dirty", "Swap"),
+            )
+            rss = r.get("Rss", 0)
+            pss = r.get("Pss", 0)
+            priv_dirty = r.get("Private_Dirty", 0)
+            proc_swap = r.get("Swap", 0)
+        except Exception:
+            pass
+        if rss == 0:
+            # smaps_rollup unavailable — fall back to VmRSS.
+            try:
+                rss = _read_kv_bytes("/proc/self/status", ("VmRSS",)).get("VmRSS", 0)
+            except Exception:
+                pass
+
+        # System-wide.
+        mem_total = mem_avail = swap_total = swap_free = 0
+        try:
+            m = _read_kv_bytes(
+                "/proc/meminfo",
+                ("MemTotal", "MemAvailable", "SwapTotal", "SwapFree"),
+            )
+            mem_total = m.get("MemTotal", 0)
+            mem_avail = m.get("MemAvailable", 0)
+            swap_total = m.get("SwapTotal", 0)
+            swap_free = m.get("SwapFree", 0)
+        except Exception:
+            pass
+
+        # glibc allocator breakdown — live vs reserved-free.
+        g_inuse = g_free = g_mmap = -1
+        if _mallinfo2 is not None:
+            try:
+                g_inuse, g_free, g_mmap = _mallinfo2()
+            except Exception:
+                pass
+
+        return {
+            "rss_gb": rss / _GB,
+            "pss_gb": pss / _GB,
+            "priv_dirty_gb": priv_dirty / _GB,
+            "proc_swap_gb": proc_swap / _GB,
+            "available_gb": mem_avail / _GB,
+            "mem_used_gb": (mem_total - mem_avail) / _GB,
+            "swap_used_gb": (swap_total - swap_free) / _GB,
+            "glibc_inuse_gb": g_inuse / _GB if g_inuse >= 0 else -1.0,
+            "glibc_free_gb": g_free / _GB if g_free >= 0 else -1.0,
+            "glibc_mmap_gb": g_mmap / _GB if g_mmap >= 0 else -1.0,
+        }
+
+    started_at = datetime.datetime.now(datetime.UTC).isoformat()
+    # Stable column order so post-hoc parsing is trivial.
+    _cols = (
+        "rss_gb", "pss_gb", "priv_dirty_gb", "proc_swap_gb",
+        "available_gb", "mem_used_gb", "swap_used_gb",
+        "glibc_inuse_gb", "glibc_free_gb", "glibc_mmap_gb",
+    )
+
+    def _loop(log_path: str, interval_s: float) -> None:
+        mono_start = time.monotonic()
+        with open(log_path, "a", buffering=1) as f:
+            f.write(
+                f"# flextool memory sampler pid={pid} started_at={started_at} "
+                f"interval_ms={interval_ms} mallinfo2={'yes' if _mallinfo2 else 'no'}\n"
+            )
+            f.write("# priv_dirty_gb+proc_swap_gb = true private cost; "
+                    "glibc_free_gb = held-free (malloc_trim-able); "
+                    "mem_used_gb = system used (monitor metric)\n")
+            while True:
+                ts = datetime.datetime.now(datetime.UTC).isoformat()
+                mono_s = time.monotonic() - mono_start
+                try:
+                    s = _sample()
+                    parts = "\t".join(f"{c}={s[c]:.4f}" for c in _cols)
+                    line = f"ts={ts}\tmono_s={mono_s:.3f}\tepoch={time.time():.3f}\t{parts}\n"
+                except Exception as exc:
+                    line = f"ts={ts}\tmono_s={mono_s:.3f}\tepoch={time.time():.3f}\terror={repr(exc)}\n"
+                f.write(line)
+                # Explicit flush so the last samples before SIGKILL reach disk;
+                # line-buffered mode alone won't flush until the process exits cleanly.
+                f.flush()
+                time.sleep(interval_s)
+
+    t = threading.Thread(target=_loop, args=(log_path, interval_s), name="flextool-mem-sampler", daemon=True)
+    t.start()

flextool-4.0.0/flextool/_resources.py

@@ -0,0 +1,43 @@
+"""Locate bundled FlexTool package data files.
+
+Static FlexTool resources — schema JSONs, YAML/text templates, and
+canonical-database sources (all under ``flextool/schemas/``), plus the HiGHS
+options template (``flextool/bin/``) — must be reached through
+:mod:`importlib.resources` rather than ``Path(__file__).resolve()``
+walks.  After ``pip install flextool`` the package is the only thing
+on ``sys.path``; the historical layout where these dirs sat at the
+*repo root* alongside the package no longer exists.
+
+Two helpers are exposed:
+
+``package_data_path(relative)``
+    Returns a :class:`pathlib.Path` to a resource inside the
+    ``flextool`` package, e.g.
+    ``package_data_path("schemas/default_plots.yaml")``.
+    Editable installs return the real on-disk path; wheel installs
+    return the unpacked site-packages path (importlib.resources
+    materialises zipped data on demand, but we never ship zipped so
+    in practice this is always a stable filesystem path).
+
+``package_data_text(relative)``
+    Convenience wrapper that returns the file contents as a string —
+    handy when the caller doesn't actually need a path (YAML/JSON
+    parsing).
+"""
+
+from __future__ import annotations
+
+from importlib import resources
+from pathlib import Path
+
+import flextool
+
+
+def package_data_path(relative: str) -> Path:
+    """Path to ``flextool/<relative>`` inside the installed package."""
+    return Path(resources.files(flextool).joinpath(relative))
+
+
+def package_data_text(relative: str) -> str:
+    """Text contents of ``flextool/<relative>``."""
+    return resources.files(flextool).joinpath(relative).read_text(encoding="utf-8")