rootfig 0.8.0__tar.gz → 0.9.0__tar.gz

{rootfig-0.8.0 → rootfig-0.9.0}/CONTRIBUTING.md

@@ -62,12 +62,12 @@ docstrings and full type hints.
 
 ## Test data
 
-Almost all tests generate their ROOT files with uproot on the fly. The one
-committed file, `tests/data/split_collection.root`, has a split
-`std::vector<struct>` branch and a `TParameter` the way podio/EDM4hep and
-FCCAnalyses write them. uproot cannot write such a file, so it was produced
-with PyROOT (ROOT 6.40) by this script; rerun it only if the layout has to
-change, and keep the file small:
+Almost all tests generate their ROOT files with uproot on the fly. Two files
+that uproot cannot write are committed, produced with ROOT 6.40; rerun their
+scripts only if the layout has to change, and keep the files small.
+
+`tests/data/split_collection.root` has a split `std::vector<struct>` branch and
+a `TParameter` the way podio/EDM4hep and FCCAnalyses write them (PyROOT):
 
 ```python
 import ROOT
@@ -95,6 +95,31 @@ f.Write()
 f.Close()
 ```
 
+`tests/data/embedded_basket.root` holds a tree whose baskets are embedded in its
+branches, without keys of their own, as a tree filled in memory and written
+afterwards keeps them (`root -l -b -q embedded.C`):
+
+```cpp
+// embedded.C
+void embedded() {
+   TTree t("events", "events");
+   t.SetDirectory(nullptr);  // in memory: nothing is flushed while filling
+   Float_t x;
+   std::vector<double> y;
+   t.Branch("x", &x);
+   t.Branch("y", &y);
+   TRandom3 r(1);
+   for (int i = 0; i < 500; ++i) {
+      x = r.Gaus(0, 1);
+      y.assign(i % 4, r.Uniform());
+      t.Fill();
+   }
+   TFile f("embedded_basket.root", "RECREATE");
+   f.WriteObject(&t, "events");
+   f.Close();
+}
+```
+
 ## Figures and the gallery
 
 The `examples/gallery` package is both the showcase and the image-regression

{rootfig-0.8.0 → rootfig-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rootfig
-Version: 0.8.0
+Version: 0.9.0
 Summary: Publication-quality figures straight from ROOT trees and histograms, without ROOT: uproot + Awkward + hist + mplhep with a TTree::Draw-like API.
 Project-URL: Homepage, https://github.com/jbeirer/rootfig
 Project-URL: Documentation, https://jbeirer.github.io/rootfig/

{rootfig-0.8.0 → rootfig-0.9.0}/docs/composable.md

@@ -52,7 +52,7 @@ mem = rf.Sample({"x": awkward_array, "w": weights}, label="in memory")
   trees; naming the histogram in place of the branch reads it (see
   [stored histograms](plotting.md#histograms-already-in-root-files)).
 - `entry_start`/`entry_stop` restrict reading for quick looks at large files
-  (a plot reads every needed branch of every file into memory at once). For a
+  (a plot reads every file of the sample, a chunk of entries at a time). For a
   ready-made `FileSource`, give the range to the source itself; passing it to
   `Sample` afterwards raises a [`SourceError`][rootfig.SourceError].
 - `systematics={name: variation}` lists the sample's sources of systematic

{rootfig-0.8.0 → rootfig-0.9.0}/docs/ecosystem.md

@@ -72,8 +72,9 @@ can discover shared histograms with `rf.ALL`. See the
   goodness-of-fit panel): [plothist](https://plothist.readthedocs.io).
 - You need a full columnar analysis framework with lazy, distributed
   processing: [coffea](https://coffeateam.github.io/coffea/). rootfig reads
-  every branch a plot needs from every file into memory at once (only the
-  required branches, optionally an entry range), which suits quick looks and
-  moderate ntuples, not multi-terabyte datasets.
+  the branches a plot needs a chunk of entries at a time, in threads, and holds
+  the values it fills (see [large inputs](plotting.md#large-inputs)), all in one
+  process: it suits quick looks and ntuples that one machine reads in minutes,
+  not multi-terabyte datasets.
 - You want to build fit templates and workspaces: [cabinetry](https://cabinetry.readthedocs.io).
 - You want a quick terminal look at a branch: [histoprint](https://github.com/scikit-hep/histoprint).

{rootfig-0.8.0 → rootfig-0.9.0}/docs/expressions.md

@@ -120,7 +120,10 @@ weights (`hist` `Weight` storage).
 ## Missing and non-finite values
 
 `None` values (e.g. `max(Muon_pt)` of an event with no muons) never enter a
-histogram; in a selection they count as `False`. `nan` and `inf` values are
+histogram; in a selection they count as `False`. A whole collection that is
+`None` for an event, whether the variable's, the weight's or the one a
+per-object selection is evaluated on, drops that event, and its objects are
+counted as missing. `nan` and `inf` values are
 dropped with a [`RootfigWarning`][rootfig.RootfigWarning] telling you how
 many, or raise with `nonfinite="error"`. Both counts are recorded in the
 histogram's `stats`.

{rootfig-0.8.0 → rootfig-0.9.0}/docs/plotting.md

@@ -466,6 +466,34 @@ panel alike.
   is not needed. End the call with `;` to hide the `Plot` repr, and use
   `Plot.close()` in loops that make many figures.
 
+## Large inputs
+
+A plot reads only the branches its variable, selection, weight and systematics
+use, a chunk of entries (about 32 MB of arrays) at a time. The chunks are
+decompressed and prepared (expressions, selection, weights) in parallel
+threads, and only the values that fill the histograms are kept, so the peak
+memory follows those values rather than every branch read. The histograms and
+their statistics are the same as from reading everything at once.
+
+- Decompressing and preparing share up to eight worker threads, fewer on a
+  machine with fewer cores. Set `ROOTFIG_THREADS` to choose the number, e.g. the
+  cores a batch job was given; `ROOTFIG_THREADS=1` decompresses and prepares in
+  the calling thread. uproot fetches the file contents in a thread or two of its
+  own either way, which do little but wait for the data.
+- An explicit range (`bins=(50, 0, 200)`, or `range=(low, high)`) skips range
+  inference, which takes two medians over all the values of every sample.
+- Many plots of the same files are fastest as a [`PlotBook`](batch.md), which
+  reads each file once per batch of variables: a tree with thousands of
+  branches costs a noticeable fraction of a second to open, every time.
+- LZMA-compressed files (the NanoAOD default) take several times longer to
+  decompress than ZLIB, LZ4 or ZSTD ones; the threads help most there.
+- A sample whose files hold a branch in different types (`float32` in one,
+  `float64` in another) is read whole once the difference shows, and evaluated
+  in the wider type. The first file's chunks may already have been evaluated in
+  their own type by then, so NumPy can warn about an overflow that the wider
+  type avoids. The histograms are the same either way, also when NumPy is set
+  to raise such errors (`np.errstate(over="raise")`, a warnings filter).
+
 ## 2D histograms and correlations
 
 ```python

{rootfig-0.8.0 → rootfig-0.9.0}/pyproject.toml

@@ -29,8 +29,9 @@ classifiers = [
 # Floors are the oldest versions the test suite passes with (CI job "minimum-versions"),
 # kept at or below what the Key4hep nightlies ship (CI workflow "key4hep"), which take
 # every dependency from the stack: uproot 5.7.1 reads nested RNTuple fields and entry
-# ranges across files; mplhep 1.2.1 has exp_label(text=) and hist2dplot(mask=); hist 2.9
-# exposes the axis metadata stored histograms are renamed through.
+# ranges across files (a floor of 5.7.3 deletes src/rootfig/io/compat.py, see there);
+# mplhep 1.2.1 has exp_label(text=) and hist2dplot(mask=); hist 2.9 exposes the axis
+# metadata stored histograms are renamed through.
 dependencies = [
   "uproot>=5.7.1",
   "awkward>=2.8",

{rootfig-0.8.0 → rootfig-0.9.0}/src/rootfig/__init__.py

@@ -58,7 +58,7 @@ from rootfig.histograms import (
 from rootfig.model import Cut, Group, Sample, Style, Systematic, Variable, log_bins
 from rootfig.plotting import Plot, dark_theme, use_style
 
-__version__ = "0.8.0"
+__version__ = "0.9.0"
 
 __all__ = [
     "ALL",

rootfig-0.9.0/src/rootfig/_threads.py

@@ -0,0 +1,65 @@
+"""How many threads rootfig reads and prepares event data in, and the pool they share."""
+
+from __future__ import annotations
+
+import os
+from collections.abc import Iterator
+from concurrent.futures import ThreadPoolExecutor
+from contextlib import contextmanager
+from contextvars import ContextVar
+
+__all__ = ["THREADS", "worker_pool"]
+
+
+def _threads() -> int:
+    given = os.environ.get("ROOTFIG_THREADS")
+    if given is not None:
+        try:
+            return max(1, int(given))
+        except ValueError:
+            msg = f"ROOTFIG_THREADS must be a whole number of threads, got {given!r}"
+            raise ValueError(msg) from None
+    available = (
+        len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count() or 1
+    )
+    return min(8, available)
+
+
+THREADS: int = _threads()
+"""Worker threads that decompress baskets and prepare chunks of events, together.
+
+Both release the GIL for most of their work, so a few threads make reading and
+preparing several times faster; more add little and would crowd a shared
+machine. ``ROOTFIG_THREADS`` sets the number (``1`` decompresses and prepares in
+the calling thread), for instance to match the cores a batch job was given.
+uproot fetches the file contents in threads of its own either way.
+"""
+
+_LENT: ContextVar[ThreadPoolExecutor | None] = ContextVar("rootfig_worker_pool", default=None)
+
+
+@contextmanager
+def worker_pool(*, lend: bool = False) -> Iterator[ThreadPoolExecutor | None]:
+    """Give the block :data:`THREADS` worker threads; ``None`` for one (the calling thread).
+
+    With ``lend``, every block inside takes this pool instead of starting its own:
+    chunks are read while earlier ones are prepared, and one pool keeps both to
+    :data:`THREADS` threads together. A pool lives for its block only, so no thread
+    outlives it (and none is inherited, dead, by a forked process). Only a plain
+    function may lend: a generator would lend to its caller between its yields.
+    """
+    if THREADS == 1:
+        yield None
+        return
+    lent = _LENT.get()
+    if lent is not None:
+        yield lent
+        return
+    pool = ThreadPoolExecutor(max_workers=THREADS)
+    token = _LENT.set(pool) if lend else None
+    try:
+        yield pool
+    finally:
+        if token is not None:
+            _LENT.reset(token)
+        pool.shutdown(cancel_futures=True)

{rootfig-0.8.0 → rootfig-0.9.0}/src/rootfig/histograms/pipeline.py

@@ -10,9 +10,10 @@ of their histograms.
 
 from __future__ import annotations
 
-from collections.abc import Mapping, Sequence
+from collections.abc import Iterator, Mapping, Sequence
 from contextlib import AbstractContextManager
 from dataclasses import dataclass
+from itertools import pairwise
 from os import PathLike
 from typing import Any
 
@@ -20,14 +21,20 @@ import awkward as ak
 import numpy as np
 
 from rootfig._typing import Hist
-from rootfig.errors import MissingBranchError, SourceError, SystematicError, annotate
+from rootfig.errors import (
+    MissingBranchError,
+    RootfigError,
+    SourceError,
+    SystematicError,
+    annotate,
+)
 from rootfig.expressions import Expression, parse
 from rootfig.histograms.build import Histogram, fill, from_sample, mirror
 from rootfig.histograms.groups import regroup_histograms
 from rootfig.histograms.sources import shared_source
 from rootfig.histograms.stats import summarize
 from rootfig.histograms.stored import read_stored, stored_mode
-from rootfig.io import ArraySource, FileSource, ReadCache, Source, as_source
+from rootfig.io import CHUNK_BYTES, ArraySource, FileSource, ReadCache, Source, as_source
 from rootfig.io.sources import resolve_files
 from rootfig.model.binning import Axis, resolve_axis
 from rootfig.model.cuts import Cut, CutLike, as_cut
@@ -36,7 +43,7 @@ from rootfig.model.inputs import leaf_samples
 from rootfig.model.samples import Sample
 from rootfig.model.systematics import Systematic, SystematicLike, as_systematics
 from rootfig.model.variables import Variable, as_variable
-from rootfig.selection import Columns, NonFinitePolicy, prepare
+from rootfig.selection import Columns, NonFinitePolicy, Request, prepare_chunks
 
 __all__ = [
     "branch_names",
@@ -47,6 +54,7 @@ __all__ = [
     "load_columns",
     "load_columns_each",
     "read_arrays",
+    "read_chunks",
 ]
 
 
@@ -102,6 +110,143 @@ def read_arrays(
     return arrays, source_length(source)
 
 
+def read_chunks(
+    sample: Sample, expressions: Sequence[Any], *, cache: ReadCache | None = None
+) -> Iterator[tuple[dict[str, Any], int]]:
+    """Return the branches ``expressions`` need from ``sample``, a chunk of events at a time.
+
+    Each chunk comes with its number of events, and the chunks, joined in order,
+    are what :func:`read_arrays` returns. Files are read chunk by chunk
+    (:meth:`~rootfig.io.FileSource.iterate`), so only the chunks being prepared
+    are held, never all branches of all events; arrays already in memory (in-memory
+    data, a ``cache``) are cut into slices of about as many bytes, views that copy
+    nothing, so that they are prepared in parallel alike. The branch names are
+    checked before this returns, and nothing is read until the first chunk is
+    asked for. A chunk holds each branch in the type its file does, which the
+    whole read promotes to one type for all files; a chunk whose types differ from
+    the first chunk's raises :class:`_MixedTypesError` (see :func:`_prepared`).
+    """
+    source = sample.source
+    if cache is None and isinstance(source, FileSource):
+        needed = branch_names(source, expressions)
+        if needed:
+            return _same_types(source.iterate(needed))
+    arrays, n_events = read_arrays(sample, expressions, cache=cache)
+    return _slices(arrays, n_events)
+
+
+class _MixedTypesError(Exception):
+    """A chunk holds a branch in another type than the first chunk did."""
+
+
+def _same_types(chunks: Iterator[dict[str, Any]]) -> Iterator[tuple[dict[str, Any], int]]:
+    """Yield ``chunks`` with their numbers of events, as long as their types stay the first's."""
+    try:
+        first = None
+        for arrays in chunks:
+            types = [array.type.content for array in arrays.values()]
+            if first is None:
+                first = types
+            elif types != first:
+                raise _MixedTypesError
+            yield arrays, len(next(iter(arrays.values())))
+    finally:
+        close = getattr(chunks, "close", None)
+        if close is not None:
+            close()
+
+
+def _prepared(
+    sample: Sample,
+    expressions: Sequence[Any],
+    requests: Sequence[Request],
+    *,
+    cache: ReadCache | None = None,
+) -> list[Columns]:
+    """Run ``requests`` on the chunks of ``sample`` that :func:`read_chunks` reads.
+
+    Where the files hold a branch in different types (``int32`` in one, ``int64``
+    in another), the sample is read whole instead (:func:`read_arrays`), which
+    promotes them to one type before anything is evaluated: ``x * x`` would
+    otherwise overflow in the ``int32`` file's chunks, and the result would depend
+    on where the chunks start and on whether a cache was used. The difference shows
+    at the first chunk of another type, so the chunks before it have been prepared
+    in their own type. Their columns are discarded, and an error they raise that
+    rootfig did not raise itself (NumPy's under ``np.errstate``, a warning turned
+    into an error, a NumPy error handler's, Awkward's) sends the sample to the whole
+    read as well if its files' types differ, which is only then looked up: the
+    sample fails or succeeds as the whole read does. Running out of memory does not,
+    since the whole read takes more. Their warnings are not taken back. Knowing
+    every file's types beforehand would cost opening each file's metadata once
+    more, a quarter of a second for a tree of 1 000 branches.
+    """
+    try:
+        return prepare_chunks(read_chunks(sample, expressions, cache=cache), requests)
+    except _MixedTypesError:
+        pass
+    except Exception as exc:
+        chain = list(_chain(exc))
+        if (
+            cache is not None
+            or isinstance(exc, SourceError)
+            or all(isinstance(each, RootfigError) for each in chain)
+            or any(isinstance(each, MemoryError) for each in chain)  # a whole read needs more
+        ):
+            raise
+        try:
+            differ = _types_differ(sample, expressions)
+        except Exception:  # a file that cannot tell: the error stands as raised
+            differ = False
+        if not differ:
+            raise
+    return prepare_chunks(_slices(*read_arrays(sample, expressions, cache=cache)), requests)
+
+
+def _chain(exc: BaseException) -> Iterator[BaseException]:
+    """Yield ``exc`` and the exceptions it was raised from or while handling, once each."""
+    seen: set[int] = set()
+    cause: BaseException | None = exc
+    while cause is not None and id(cause) not in seen:
+        yield cause
+        seen.add(id(cause))
+        cause = cause.__cause__ or cause.__context__
+
+
+def _types_differ(sample: Sample, expressions: Sequence[Any]) -> bool:
+    """Return whether the files of ``sample`` hold the branches read in different types.
+
+    The types come from each file's metadata (:meth:`~rootfig.io.FileSource.branch_forms`),
+    so an empty tree has them too.
+    """
+    source = sample.source
+    if not isinstance(source, FileSource) or len(source.files) < 2:
+        return False
+    needed = branch_names(source, expressions)
+    tree = source.resolved_tree()
+
+    def types(path: str) -> list[Any]:
+        forms = FileSource(path, tree).branch_forms()
+        return [forms[name].type if name in forms else None for name in needed]
+
+    first = types(source.files[0])
+    return any(types(path) != first for path in source.files[1:])
+
+
+def _slices(arrays: dict[str, Any], n_events: int) -> Iterator[tuple[dict[str, Any], int]]:
+    """Cut ``arrays`` into consecutive slices of at most about :data:`~rootfig.io.CHUNK_BYTES`.
+
+    With nothing read (every expression a constant), each event still becomes a
+    ``float64`` once a constant is broadcast to it, and those are what is counted.
+    """
+    size = sum(getattr(array, "nbytes", 0) for array in arrays.values()) or 8 * n_events
+    count = max(1, min(n_events, -(-size // CHUNK_BYTES)))
+    if count == 1:
+        yield arrays, n_events
+        return
+    for start, stop in pairwise(n_events * part // count for part in range(count + 1)):
+        yield {name: array[start:stop] for name, array in arrays.items()}, stop - start
+
+
 def branch_names(source: Source, expressions: Sequence[Any]) -> list[str]:
     """Return the branches of ``source`` that ``expressions`` need, once each, in order of use.
 
@@ -154,17 +299,16 @@ def load_columns(
     var_exprs = [as_variable(v).expression for v in variables]
     cut = combined_selection(sample, selection)
     weight_expr = combined_weight(sample, weight)
-    arrays, n_events = _read_for(sample, var_exprs, cut, weight_expr, cache=cache)
-    return prepare(
-        arrays,
-        var_exprs,
+    request = Request(
+        tuple(var_exprs),
         selection=None if cut is None else cut.expression,
         weight=weight_expr,
         scale=sample.scale * sample.lumi_scale(lumi),
         nonfinite=nonfinite,
         context=sample.label,
-        n_events=n_events,
     )
+    expressions = _expressions(var_exprs, cut, weight_expr)
+    return _prepared(sample, expressions, [request], cache=cache)[0]
 
 
 def load_columns_each(
@@ -186,38 +330,31 @@ def load_columns_each(
     var_exprs = [as_variable(v).expression for v in variables]
     cut = combined_selection(sample, selection)
     weight_expr = combined_weight(sample, weight)
-    arrays, n_events = _read_for(sample, var_exprs, cut, weight_expr)
     scale = sample.scale * sample.lumi_scale(lumi)
-    return [
-        prepare(
-            arrays,
-            [expression],
+    requests = [
+        Request(
+            (expression,),
             selection=None if cut is None else cut.expression,
             weight=weight_expr,
             scale=scale,
             nonfinite=nonfinite,
             context=sample.label,
-            n_events=n_events,
         )
         for expression in var_exprs
     ]
+    return _prepared(sample, _expressions(var_exprs, cut, weight_expr), requests)
 
 
-def _read_for(
-    sample: Sample,
-    var_exprs: Sequence[str],
-    cut: Cut | None,
-    weight_expr: str | None,
-    *,
-    cache: ReadCache | None = None,
-) -> tuple[dict[str, Any], int]:
-    """Read the branches needed by the variables, the selection and the weight."""
+def _expressions(
+    var_exprs: Sequence[str], cut: Cut | None, weight_expr: str | None
+) -> list[Expression]:
+    """Return the parsed variables, selection and weight, whose branches are read."""
     expressions = [parse(v) for v in var_exprs]
     if cut is not None:
         expressions.append(cut.parsed())
     if weight_expr is not None:
         expressions.append(parse(weight_expr))
-    return read_arrays(sample, expressions, cache=cache)
+    return expressions
 
 
 def build_histograms(
@@ -343,28 +480,46 @@ def _load_with_variations(
 ) -> _Loaded:
     """Prepare the nominal columns and every variation, reading the sample's branches once.
 
-    Weight and branch variations are evaluated on the arrays read for the
-    nominal; variations from other data read those. A variation is a
-    :class:`Columns` to fill, a factor for the nominal histogram, or ``None``
-    for a down direction mirrored from the up one.
+    Weight and branch variations are prepared from the chunks read for the
+    nominal (:func:`read_chunks`), chunk by chunk alongside it; variations from
+    other data read those. A variation is a :class:`Columns` to fill, a factor
+    for the nominal histogram, or ``None`` for a down direction mirrored from the
+    up one.
     """
     cut = combined_selection(sample, selection)
     cut_text = None if cut is None else cut.expression
     weight_expr = combined_weight(sample, weight)
     scale = sample.scale * sample.lumi_scale(lumi)
     expressions, used = _read_plan(sample, var, systematics, selection=selection, weight=weight)
-    arrays, n_events = read_arrays(sample, expressions, cache=cache)
 
-    nominal = prepare(
-        arrays,
-        [var.expression],
-        selection=cut_text,
-        weight=weight_expr,
-        scale=scale,
-        nonfinite=nonfinite,
-        context=sample.label,
-        n_events=n_events,
-    )
+    def request(context: str, weight: str | None, **extra: Any) -> Request:
+        return Request(
+            (var.expression,),
+            selection=cut_text,
+            weight=weight,
+            scale=scale,
+            nonfinite=nonfinite,
+            context=context,
+            **extra,
+        )
+
+    requests = [request(sample.label, weight_expr)]
+    shifted: dict[tuple[str, str], int] = {}  # the request of each weight or branch variation
+    for name, syst in systematics.items():
+        for direction, spec in (("up", syst.up), ("down", syst.down)):
+            if spec is None or syst.kind not in ("weight", "replace"):
+                continue
+            context = f"{sample.label} [{name} {direction}]"
+            shifted[name, direction] = len(requests)
+            note = _variation_note(context)
+            if syst.kind == "weight":
+                requests.append(request(context, _product(spec, weight), note=note))
+            else:
+                # the replacing branches take the place of the replaced ones, so the
+                # variable, selection and weight all see the shifted values
+                replaced = {old: new for old, new in spec.items() if old in used}
+                requests.append(request(context, weight_expr, substitutes=replaced, note=note))
+    prepared = _prepared(sample, expressions, requests, cache=cache)
 
     variations: dict[str, tuple[Columns | float | None, Columns | float | None]] = {}
     for name, syst in systematics.items():
@@ -390,27 +545,9 @@ def _load_with_variations(
                         )
                     )
                 else:
-                    if syst.kind == "weight":
-                        varied_arrays, varied_weight = arrays, _product(spec, weight)
-                    else:
-                        # the replacing branches take the place of the replaced ones, so the
-                        # variable, selection and weight all see the shifted values
-                        replaced = {old: arrays[new] for old, new in spec.items() if old in used}
-                        varied_arrays, varied_weight = {**arrays, **replaced}, weight_expr
-                    shifts.append(
-                        prepare(
-                            varied_arrays,
-                            [var.expression],
-                            selection=cut_text,
-                            weight=varied_weight,
-                            scale=scale,
-                            nonfinite=nonfinite,
-                            context=context,
-                            n_events=n_events,
-                        )
-                    )
+                    shifts.append(prepared[shifted[name, direction]])
         variations[name] = (shifts[0], shifts[1])
-    return _Loaded(nominal, variations)
+    return _Loaded(prepared[0], variations)
 
 
 def _read_plan(
@@ -472,7 +609,12 @@ def _read_plan(
 
 def _in_variation(context: str) -> AbstractContextManager[None]:
     """Name the variation (sample, source, direction) in any rootfig error raised inside."""
-    return annotate(f"while evaluating the systematic variation {context}")
+    return annotate(_variation_note(context))
+
+
+def _variation_note(context: str) -> str:
+    """Return the note naming the variation ``context`` (sample, source, direction)."""
+    return f"while evaluating the systematic variation {context}"
 
 
 def _variant_sample(

{rootfig-0.8.0 → rootfig-0.9.0}/src/rootfig/histograms/stats.py

@@ -82,7 +82,7 @@ def _fmt(value: float, precision: int) -> str:
 def summarize(columns: Columns, index: int = 0) -> Summary:
     """Compute :class:`Summary` statistics for column ``index`` of ``columns``."""
     values = columns.arrays[index]
-    weights = columns.effective_weights()
+    weights = columns.weights
     n = int(values.size)
     common: dict[str, Any] = {
         "entries": n,
@@ -92,8 +92,11 @@ def summarize(columns: Columns, index: int = 0) -> Summary:
         "n_nonfinite": columns.n_nonfinite,
         "per_object": columns.per_object,
     }
-    sum_w = float(weights.sum())
-    sum_w2 = float((weights**2).sum())
+    if weights is None:  # unit weights sum to the entry count
+        sum_w = sum_w2 = float(n)
+    else:
+        sum_w = float(weights.sum())
+        sum_w2 = float(np.square(weights).sum())
     if n == 0 or sum_w == 0:
         nan = float("nan")
         return Summary(
@@ -107,14 +110,22 @@ def summarize(columns: Columns, index: int = 0) -> Summary:
             _sum_w2=sum_w2,
             **common,
         )
-    mean = float(np.average(values, weights=weights))
+
+    def average(array: np.ndarray) -> float:
+        # np.average, without summing the weights again (or a column of ones) every time
+        total = array.sum() if weights is None else np.multiply(array, weights).sum()
+        return float(total / sum_w)
+
+    mean = average(values)
     centred = values - mean
-    variance = float(np.average(centred**2, weights=weights))
+    power = np.square(centred)
+    variance = average(power)
     # negative weights can make the weighted second moment negative: undefined, not an error
     std = math.sqrt(variance) if variance >= 0 else float("nan")
     n_eff = sum_w**2 / sum_w2 if sum_w2 else 0.0
     sem = std / math.sqrt(n_eff) if n_eff > 0 else float("nan")
-    third = float(np.average(centred**3, weights=weights))
+    power *= centred  # the cube from the square: centred**3 calls pow() for every value
+    third = average(power)
     skewness = third / std**3 if std > 0 else float("nan")
     return Summary(
         sum_weights=sum_w,

{rootfig-0.8.0 → rootfig-0.9.0}/src/rootfig/io/__init__.py

@@ -2,6 +2,7 @@
 
 from rootfig.io.cache import ReadCache
 from rootfig.io.sources import (
+    CHUNK_BYTES,
     ArraySource,
     FilesLike,
     FileSource,
@@ -11,6 +12,7 @@ from rootfig.io.sources import (
 )
 
 __all__ = [
+    "CHUNK_BYTES",
     "ArraySource",
     "FileSource",
     "FilesLike",