rootfig 0.9.0__tar.gz → 0.11.0__tar.gz

{rootfig-0.9.0 → rootfig-0.11.0}/CONTRIBUTING.md

@@ -62,9 +62,10 @@ docstrings and full type hints.
 
 ## Test data
 
-Almost all tests generate their ROOT files with uproot on the fly. Two files
-that uproot cannot write are committed, produced with ROOT 6.40; rerun their
-scripts only if the layout has to change, and keep the files small.
+Almost all tests generate their ROOT files with uproot on the fly. A few files
+are committed, produced with ROOT 6.40: two that uproot cannot write, and one
+whose point is that ROOT wrote it. Rerun their scripts only if the layout has
+to change, and keep the files small.
 
 `tests/data/split_collection.root` has a split `std::vector<struct>` branch and
 a `TParameter` the way podio/EDM4hep and FCCAnalyses write them (PyROOT):
@@ -120,6 +121,69 @@ void embedded() {
 }
 ```
 
+`tests/data/error_options.root` holds the same counts (0, 1 and 4) saved with
+each of ROOT's statistical error options, and weighted entries saved with
+`kPoisson`, which ROOT itself draws with `sqrt(sum w^2)` (PyROOT):
+
+```python
+import ROOT
+
+f = ROOT.TFile("error_options.root", "RECREATE")
+for name, option in (
+    ("normal", ROOT.TH1.kNormal),
+    ("poisson", ROOT.TH1.kPoisson),
+    ("poisson2", ROOT.TH1.kPoisson2),
+):
+    h = ROOT.TH1D(name, name, 3, 0.0, 3.0)
+    for x, n in ((1.5, 1), (2.5, 4)):
+        for _ in range(n):
+            h.Fill(x)
+    h.SetBinErrorOption(option)
+    h.Write()
+w = ROOT.TH1D("weighted", "weighted", 3, 0.0, 3.0)
+w.Sumw2()
+w.Fill(1.5, 2.0)
+w.Fill(2.5, 1.0)
+w.SetBinErrorOption(ROOT.TH1.kPoisson)
+w.Write()
+f.Close()
+```
+
+`tests/test_goodness.py` pins ROOT 6.40's `TH1::Chi2TestX` and
+`TH1::KolmogorovTest` for the histograms defined at its top (`COUNTS_A`, ...).
+Rerun to check them against another ROOT version (PyROOT):
+
+```python
+import ctypes, math
+import ROOT
+
+
+def th1(name, values, variances=None):  # Sumw2 only for weighted contents
+    h = ROOT.TH1D(name, name, len(values), 0.0, float(len(values)))
+    if variances is not None:
+        h.Sumw2()
+    for i, v in enumerate(values, start=1):
+        h.SetBinContent(i, v)
+        if variances is not None:
+            h.SetBinError(i, math.sqrt(variances[i - 1]))
+    return h
+
+
+def chi2_test(h1, h2, option):  # (chi2, ndf, p-value, igood)
+    chi2, ndf, igood = ctypes.c_double(), ctypes.c_int(), ctypes.c_int()
+    p = h1.Chi2TestX(h2, chi2, ndf, igood, option)
+    return chi2.value, ndf.value, p, igood.value
+
+
+a, b = th1("a", COUNTS_A), th1("b", COUNTS_B)
+data, mc = th1("data", DATA), th1("mc", MC, MC_VARIANCES)
+print(chi2_test(a, b, "UU"), chi2_test(data, mc, "UW"))
+scaled = th1("bs", [0.37 * n for n in COUNTS_B], [0.37**2 * n for n in COUNTS_B])
+print(chi2_test(a, scaled, "UU NORM"))
+print(chi2_test(th1("w1", W1, W1_VARIANCES), th1("w2", W2, W2_VARIANCES), "WW"))
+print(a.KolmogorovTest(b), a.KolmogorovTest(b, "M"))  # likewise for the other pairs
+```
+
 ## Figures and the gallery
 
 The `examples/gallery` package is both the showcase and the image-regression

{rootfig-0.9.0 → rootfig-0.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rootfig
-Version: 0.9.0
+Version: 0.11.0
 Summary: Publication-quality figures straight from ROOT trees and histograms, without ROOT: uproot + Awkward + hist + mplhep with a TTree::Draw-like API.
 Project-URL: Homepage, https://github.com/jbeirer/rootfig
 Project-URL: Documentation, https://jbeirer.github.io/rootfig/
@@ -28,6 +28,7 @@ Requires-Dist: hist>=2.9
 Requires-Dist: matplotlib>=3.11
 Requires-Dist: mplhep>=1.2.1
 Requires-Dist: numpy>=1.26
+Requires-Dist: scipy>=1.11.2
 Requires-Dist: uproot>=5.7.1
 Description-Content-Type: text/markdown
 
@@ -141,6 +142,7 @@ p = rf.plot(
     selection=baseline,
     stack=True,
     panel="ratio",
+    data_errors="auto",  # Poisson intervals for unweighted data
     logy=True,
     style=style,
 )
@@ -175,12 +177,14 @@ book.save_pdf("zh.pdf")  # rf.ALL in place of the list plots every histogram the
 ## What you can do
 
 - **Select events and objects with readable expressions.** Write cuts such as
-  `count(Jet_pt) >= 2` or `Muon_pt > 20`; event and object selections have
-  explicit rules, and event weights carry through to each selected object.
-- **Compare samples with a few keywords.** Overlays, stacks, data points and a
-  lower panel (ratio, difference, relative difference, pull or significance,
-  against a reference you name) share binning and propagate histogram
-  uncertainties; bin edges
+  `count(Jet_pt) >= 2` or `Muon_pt > 20`, calling functions of your own where the
+  built-in ones end (`functions={"delta_r": delta_r}`); event and object selections
+  have explicit rules, and event weights carry through to each selected object.
+- **Compare samples with a few keywords.** Overlays, stacks, data points (`√N`
+  as ROOT draws them, or Poisson intervals) and a lower panel (ratio, difference, relative difference, asymmetry, pull or
+  significance, against a reference you name) share binning and propagate histogram
+  uncertainties; ROOT's χ² and Kolmogorov tests quote the agreement
+  (`goodness_of_fit=True`). Bin edges
   and `(n, low, high)` are used as given, while a range inferred from the data
   ignores far outliers, so `-999` sentinels do not set the axis. Normalise to
   unity, density, bin width or luminosity; stack some samples and overlay the rest.
@@ -227,6 +231,7 @@ stack, building on familiar libraries:
 | Jagged arrays | [Awkward Array](https://github.com/scikit-hep/awkward) | the per-event/per-object rules for cuts and weights |
 | Histograms | [hist](https://github.com/scikit-hep/hist) / boost-histogram | shared binning, robust automatic ranges, normalisation, ratios |
 | Drawing | [mplhep](https://github.com/scikit-hep/mplhep) + matplotlib | overlays, stacks, ratio panels, labels and legends with good defaults |
+| Intervals | [SciPy](https://scipy.org) | ROOT's Poisson and efficiency intervals, kept through scaling and normalisation |
 
 If your histograms already exist, `rf.plot` draws them too: name a `TH1`
 stored in the file instead of a branch (`rf.plot("zh_histo.root", "m_recoil")`),

{rootfig-0.9.0 → rootfig-0.11.0}/README.md

@@ -108,6 +108,7 @@ p = rf.plot(
     selection=baseline,
     stack=True,
     panel="ratio",
+    data_errors="auto",  # Poisson intervals for unweighted data
     logy=True,
     style=style,
 )
@@ -142,12 +143,14 @@ book.save_pdf("zh.pdf")  # rf.ALL in place of the list plots every histogram the
 ## What you can do
 
 - **Select events and objects with readable expressions.** Write cuts such as
-  `count(Jet_pt) >= 2` or `Muon_pt > 20`; event and object selections have
-  explicit rules, and event weights carry through to each selected object.
-- **Compare samples with a few keywords.** Overlays, stacks, data points and a
-  lower panel (ratio, difference, relative difference, pull or significance,
-  against a reference you name) share binning and propagate histogram
-  uncertainties; bin edges
+  `count(Jet_pt) >= 2` or `Muon_pt > 20`, calling functions of your own where the
+  built-in ones end (`functions={"delta_r": delta_r}`); event and object selections
+  have explicit rules, and event weights carry through to each selected object.
+- **Compare samples with a few keywords.** Overlays, stacks, data points (`√N`
+  as ROOT draws them, or Poisson intervals) and a lower panel (ratio, difference, relative difference, asymmetry, pull or
+  significance, against a reference you name) share binning and propagate histogram
+  uncertainties; ROOT's χ² and Kolmogorov tests quote the agreement
+  (`goodness_of_fit=True`). Bin edges
   and `(n, low, high)` are used as given, while a range inferred from the data
   ignores far outliers, so `-999` sentinels do not set the axis. Normalise to
   unity, density, bin width or luminosity; stack some samples and overlay the rest.
@@ -194,6 +197,7 @@ stack, building on familiar libraries:
 | Jagged arrays | [Awkward Array](https://github.com/scikit-hep/awkward) | the per-event/per-object rules for cuts and weights |
 | Histograms | [hist](https://github.com/scikit-hep/hist) / boost-histogram | shared binning, robust automatic ranges, normalisation, ratios |
 | Drawing | [mplhep](https://github.com/scikit-hep/mplhep) + matplotlib | overlays, stacks, ratio panels, labels and legends with good defaults |
+| Intervals | [SciPy](https://scipy.org) | ROOT's Poisson and efficiency intervals, kept through scaling and normalisation |
 
 If your histograms already exist, `rf.plot` draws them too: name a `TH1`
 stored in the file instead of a branch (`rf.plot("zh_histo.root", "m_recoil")`),

{rootfig-0.9.0 → rootfig-0.11.0}/docs/api.md

@@ -16,6 +16,7 @@
 ::: rootfig.profile
 ::: rootfig.evaluate
 ::: rootfig.compare
+::: rootfig.goodness_of_fit
 ::: rootfig.log_bins
 ::: rootfig.use_style
 ::: rootfig.dark_theme
@@ -28,6 +29,8 @@
 ::: rootfig.Cut
 ::: rootfig.Style
 ::: rootfig.Systematic
+::: rootfig.Bayesian
+::: rootfig.ONE_SIGMA
 
 ## Batch plotting
 
@@ -43,6 +46,7 @@
 ::: rootfig.Plot
 ::: rootfig.Histogram
 ::: rootfig.Comparison
+::: rootfig.GoodnessOfFit
 ::: rootfig.Uncertainty
 ::: rootfig.Summary
 ::: rootfig.SummaryTable
@@ -65,6 +69,7 @@ above.
     options:
       members:
         - FileSource
+        - NpzSource
         - ArraySource
         - Source
         - ReadCache
@@ -92,6 +97,8 @@ above.
         - fill
         - sum_histograms
         - uncertainty
+        - poisson_interval
+        - shape_covariance
         - summarize
         - read_stored
         - compatible_binning

{rootfig-0.9.0 → rootfig-0.11.0}/docs/batch.md

@@ -78,7 +78,10 @@ prepared histograms. With the signal last and `panel="s/sqrt(b)"` in
 `plot_kwargs`, both show the same significance panel.
 
 Every other keyword of `rf.plot` (`observed=`, `normalize=`, `panel=`,
-`systematics=`, `style=`, ...) goes into `plot_kwargs` or a variant. The keyword
+`systematics=`, `style=`, ...) goes into `plot_kwargs` or a variant, also
+`functions=`: with `plot_kwargs={"functions": functions}` every variable and
+selection of the book may call [functions of your own](expressions.md#functions-of-your-own),
+and the branches they are called on are read with the rest of the batch. The keyword
 names are checked against `rf.plot`'s signature when the book is built, so a
 misspelt `log_y` is reported, with `logy` as the suggestion, before anything is
 drawn rather than after the first tasks have written their files; the values are

{rootfig-0.9.0 → rootfig-0.11.0}/docs/composable.md

@@ -26,6 +26,11 @@ mem = rf.Sample({"x": awkward_array, "w": weights}, label="in memory")
 - `data` may be a path, glob, `"path:tree"`, a list of those, a mapping of
   arrays, an Awkward record array, a NumPy structured array, or any object
   implementing the [`Source`][rootfig.io.Source] protocol.
+- A NumPy `.npz` file (`np.savez`) is read as one branch per array
+  ([`NpzSource`][rootfig.io.NpzSource]): `rf.plot("events.npz", "Muon_pt", bins=50)`.
+  A flat array is per-event, an `(events, n)` array per-object; arrays of no
+  dimension (a scalar saved beside the events) are not branches, and object
+  arrays (ragged lists) are refused, since NumPy loads them only by unpickling.
 - `selection` and `weight` belong to the sample and combine with the ones
   given to `plot()` (`&` and `*` respectively).
 - `is_data=True` draws points with error bars (in the style's text colour

{rootfig-0.9.0 → rootfig-0.11.0}/docs/ecosystem.md

@@ -1,4 +1,4 @@
-# How rootfig relates to uproot, Awkward, hist, mplhep and matplotlib
+# How rootfig relates to uproot, Awkward, hist, mplhep, matplotlib and SciPy
 
 rootfig is a thin, opinionated layer over the Scientific Python HEP stack. It
 owns no file format, array type, histogram type or drawing primitive of its
@@ -42,7 +42,11 @@ ships the style sheets and label helpers of the LHC experiments. rootfig
 draws every histogram through `mplhep.histplot`/`hist2dplot` and uses
 mplhep's per-experiment label functions when a `Style(experiment=...)` is
 given. The experiment-neutral default style and the lower panel (ratios,
-differences, pulls, significances) are rootfig's.
+differences, asymmetries, pulls and significances of histograms; ratios,
+differences, asymmetries and pulls of efficiencies and profiles) are rootfig's,
+and so are the Poisson intervals of data: mplhep's (and `hist.intervals`) give
+an empty bin of scaled counts the scale of a neighbour, where rootfig keeps the
+bin's own (see [SciPy](#scipy)).
 
 ## matplotlib
 
@@ -51,6 +55,21 @@ applies its style only inside a `plt.style.context` while drawing, so it does
 not change global rcParams (unless you call `rf.use_style`). Anything you
 would do to a matplotlib figure, you can do to `Plot.fig` and `Plot.ax`.
 
+## SciPy
+
+[SciPy](https://scipy.org) supplies the quantiles behind rootfig's intervals, from
+`scipy.special`: the inverse incomplete gamma functions for the Garwood interval
+of data (ROOT's `TH1::kPoisson`) and the inverse incomplete beta function for the
+Clopper–Pearson and Bayesian intervals of efficiencies (`TEfficiency`'s default
+and its Beta priors); `scipy.optimize.brentq` finds the bounds that have no
+closed form (the shortest Bayesian interval). rootfig decides what the
+quantiles are taken of (the counts behind scaled or normalised contents, which
+interval ROOT would choose) and imports SciPy only when it computes one.
+`hist.intervals` holds the same Garwood, Clopper–Pearson and Poisson-ratio
+intervals, and rootfig's tests check that both agree to round-off. rootfig does
+not call it at run time: it imports `scipy.stats`, which takes 0.4 s or more on
+the first interval of every session, for a computation of a few lines.
+
 ## Trees and histogram files
 
 rootfig reads trees and stored `TH1`/`TH2` histograms without ROOT, independently
@@ -68,8 +87,8 @@ can discover shared histograms with `rf.ALL`. See the
 ## When to use something else
 
 - You have flat NumPy columns rather than ROOT files or histograms, or want a
-  comparison rootfig's lower panel does not offer (an asymmetry, efficiency or
-  goodness-of-fit panel): [plothist](https://plothist.readthedocs.io).
+  comparison rootfig's lower panel does not offer (a goodness-of-fit panel):
+  [plothist](https://plothist.readthedocs.io).
 - You need a full columnar analysis framework with lazy, distributed
   processing: [coffea](https://coffeateam.github.io/coffea/). rootfig reads
   the branches a plot needs a chunk of entries at a time, in threads, and holds

{rootfig-0.9.0 → rootfig-0.11.0}/docs/expressions.md

@@ -19,8 +19,8 @@ syntax evaluated with NumPy/Awkward semantics over the branches of the tree.
 operations, so they work on arrays. Dotted names such as
 `ReconstructedParticles.momentum.x` are read as one branch name (see below);
 any other attribute access, lambdas, comprehensions, string literals and calls
-to anything but the functions below are rejected at parse time with an
-[`ExpressionError`][rootfig.ExpressionError].
+to anything but the functions below (built in, or [your own](#functions-of-your-own))
+are rejected at parse time with an [`ExpressionError`][rootfig.ExpressionError].
 An unknown branch raises [`MissingBranchError`][rootfig.MissingBranchError]
 with close-match suggestions.
 
@@ -57,6 +57,64 @@ Kinematics from Cartesian components, as stored by EDM4hep (`momentum.x/y/z`,
 
 Reductions applied to a flat (per-event) branch raise an error.
 
+## Functions of your own
+
+Helpers that are not built in are passed with `functions=`, a mapping from the
+name the expressions use to a callable. Every call that evaluates expressions
+takes it (`plot`, `histogram`, `histograms`, `load`, `summarize`, `cutflow`,
+`efficiency`, `profile`, `plot2d`, `correlation`, `evaluate`; a
+[`PlotBook`](batch.md) in `plot_kwargs` or a variant), for the variable, the
+selection and the weights alike:
+
+```python
+import awkward as ak
+import numpy as np
+
+
+def delta_phi(phi1, phi2):
+    return (phi1 - phi2 + np.pi) % (2 * np.pi) - np.pi
+
+
+def delta_r(eta1, phi1, eta2, phi2):
+    return np.hypot(eta1 - eta2, delta_phi(phi1, phi2))
+
+
+def nth(x, n):
+    """The n-th object of each event (from 0); None where an event has fewer."""
+    return ak.pad_none(x, n + 1, axis=1)[:, n]
+
+
+functions = {"delta_phi": delta_phi, "delta_r": delta_r, "nth": nth}
+dr = "delta_r(nth(Muon_eta, 0), nth(Muon_phi, 0), nth(Muon_eta, 1), nth(Muon_phi, 1))"
+
+rf.plot(
+    sample,
+    "delta_phi(nth(Muon_phi, 0), nth(Muon_phi, 1))",
+    selection=f"{dr} > 0.4",
+    functions=functions,
+)
+```
+
+- A function receives the branch arrays (NumPy or Awkward, flat per event or
+  jagged per object) and returns an array or a number. Its result follows the
+  [per-event and per-object rules](#per-event-versus-per-object) like any other,
+  and a `None` in it counts as a [missing value](#missing-and-non-finite-values).
+- rootfig evaluates expressions a chunk of events at a time, in several threads at
+  once (see [Large inputs](plotting.md#large-inputs)). A function must therefore
+  treat each event on its own and change nothing outside it: `x / ak.max(x)`
+  would divide by the largest value of the chunk, not of the sample.
+- The names are looked up when a call uses the expression, so a `Variable` or `Cut`
+  calling `delta_r` can be built before any `functions=` is given. A name that a
+  built-in function or constant already has is refused, and so is a mapping that
+  is not one of names to callables.
+- Only the functions given to a call are reachable from its expressions; a later
+  call without `functions=` cannot call them. The functions themselves are ordinary
+  Python run as you: the expression language limits what an expression can reach,
+  not what the functions handed to it do, so pass only code you trust.
+
+Names beginning with `__rootfig_` are rootfig's own; a branch so named is written
+in backticks: `` `__rootfig_x` ``.
+
 ## Sub-branches of object collections
 
 podio/EDM4hep and other files with split object branches list their leaves