atlas-ftag-tools 0.2.10__py3-none-any.whl → 0.2.11__py3-none-any.whl

atlas_ftag_tools-0.2.11.dist-info/METADATA

@@ -0,0 +1,53 @@
+Metadata-Version: 2.4
+Name: atlas-ftag-tools
+Version: 0.2.11
+Summary: ATLAS Flavour Tagging Tools
+Author: Sam Van Stroud, Philipp Gadow
+License: MIT
+Project-URL: Homepage, https://github.com/umami-hep/atlas-ftag-tools/
+Requires-Python: <3.12,>=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: h5py>=3.0
+Requires-Dist: numpy>=2.2.3
+Requires-Dist: PyYAML>=5.1
+Requires-Dist: scipy>=1.15.2
+Provides-Extra: dev
+Requires-Dist: ruff==0.6.2; extra == "dev"
+Requires-Dist: mypy==1.11.2; extra == "dev"
+Requires-Dist: pre-commit==3.1.1; extra == "dev"
+Requires-Dist: pytest==7.2.2; extra == "dev"
+Requires-Dist: pytest-cov==4.0.0; extra == "dev"
+Requires-Dist: pytest_notebook==0.10.0; extra == "dev"
+Requires-Dist: ipykernel==6.21.3; extra == "dev"
+Dynamic: license-file
+
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Docs](https://img.shields.io/badge/info-documentation-informational)](https://umami-hep.github.io/atlas-ftag-tools/main)
+[![PyPI version](https://badge.fury.io/py/atlas-ftag-tools.svg)](https://badge.fury.io/py/atlas-ftag-tools)
+[![codecov](https://codecov.io/gh/umami-hep/atlas-ftag-tools/branch/main/graph/badge.svg?token=MBHLIYYQ7I)](https://codecov.io/gh/umami-hep/atlas-ftag-tools)
+
+# ATLAS FTAG Python Tools
+
+This is a collection of Python tools for working with files produced with the FTAG [ntuple dumper](https://gitlab.cern.ch/atlas-flavor-tagging-tools/training-dataset-dumper/).
+The code is intended to be used a [library](https://iscinumpy.dev/post/app-vs-library/) for other projects.
+Please see the [example notebook](ftag/example.ipynb) for usage.
+
+# Quickstart 
+
+## Installation
+
+If you want to use this package without modification, you can install from [pypi](https://pypi.org/project/atlas-ftag-tools/) using `pip`.
+
+```bash
+pip install atlas-ftag-tools
+```
+
+To additionally install the development dependencies (for formatting and linting) use
+```bash
+pip install atlas-ftag-tools[dev]
+```
+
+## Usage
+
+Extensive examples are given in the [Examples](https://umami-hep.github.io/atlas-ftag-tools/main/examples/index.html)

{atlas_ftag_tools-0.2.10.dist-info → atlas_ftag_tools-0.2.11.dist-info}/RECORD

@@ -1,8 +1,9 @@
-ftag/__init__.py,sha256=v9emuK48Hhd-_TCiirfCNMsZSzk52frz1zEOgk9PViQ,787
+atlas_ftag_tools-0.2.11.dist-info/licenses/LICENSE,sha256=R4o6bZfajQ1KxwcIeavTC00qYTdL33YGNe1hzfV53gM,11349
+ftag/__init__.py,sha256=BGQ1MtuhqCHFXRAh9S9f_ZnOCLWB5RA0ZtL9lW2tofs,748
 ftag/cli_utils.py,sha256=w3TtQmUHSyAKChS3ewvOtcSDAUJAZGIIomaNi8f446U,298
 ftag/cuts.py,sha256=9_ooLZHaO3SnIQBNxwbaPZn-qptGdKnB27FdKQGTiTY,2933
 ftag/flavours.py,sha256=ShH4M2UjQZpZ_NlCctTm2q1tJbzYxjmGteioQ2GcqEU,114
-ftag/flavours.yaml,sha256=5Lo9KWe-2KzmGMbc7o_X9gzwUyTl0Q5uVHYExduZ6T4,9502
+ftag/flavours.yaml,sha256=CrVTJKndHeL15LT2nkjPodi6Ck9mk_oUtdRby6X_Rcc,9921
 ftag/fraction_optimization.py,sha256=IlMEJe5fD0soX40f-LO4dYAYld2gMqgZRuBLctoPn9A,5566
 ftag/git_check.py,sha256=Y-XqM80CVXZ5ZKrDdZcYOJt3X64uU6W3OP6Z0D7AZU0,1663
 ftag/labeller.py,sha256=IXUgU9UBir39PxVWRKs5r5fqI66Tv0x7nJD3-RYpbrg,2780
@@ -12,19 +13,20 @@ ftag/region.py,sha256=ANv0dGI2W6NJqD9fp7EfqAUReH4FOjc1gwl_Qn8llcM,360
 ftag/sample.py,sha256=3N0FrRcu9l1sX8ohuGOHuMYGD0See6gMO4--7NzR2tE,2538
 ftag/track_selector.py,sha256=fJNk_kIBQriBqV4CPT_3ReJbOUnavDDzO-u3EQlRuyk,2654
 ftag/transform.py,sha256=uEGGJSnqoKOzLYQv650XdK_kDNw4Aw-5dc60z9Dp_y0,3963
-ftag/vds.py,sha256=nRViQZQIORB95nC7NZsW3KsSoGkLzEdOsuCViH5h8-U,3296
+ftag/vds.py,sha256=wqj1cA6mIJ4enk8inkearo7ccTw5KCbvuNo2oon51fc,4565
 ftag/working_points.py,sha256=RJws2jPMEDQDspCbXUZBifS1CCBmlMJ5ax0eMyDzCRA,15949
-ftag/hdf5/__init__.py,sha256=LFDNxVOCp58SvLHwQhdT68Q-KBMS_i6jBrbXoRpHzbM,354
+ftag/hdf5/__init__.py,sha256=8yzVQITge-HKkBQQ60eJwWmWDycYZjgVs-qVg4ShVr0,385
+ftag/hdf5/h5add_col.py,sha256=htS5wn4Tm4S3U6mrJ8s24VUnbI7o28Z6Ll-J_V68xTA,12558
 ftag/hdf5/h5move.py,sha256=oYpRu0IDCIJIQ2ML52HBAdoyDxmKkHTeM9JdbPEgKfI,947
 ftag/hdf5/h5reader.py,sha256=i31pDAqmOSaxdeRhc4iSBlld8xJ0pmp4rNd7CugNzw0,13706
 ftag/hdf5/h5split.py,sha256=4Wy6Xc3J58MdD9aBaSZHf5ZcVFnJSkWsm42R5Pgo-R4,2448
 ftag/hdf5/h5utils.py,sha256=-4zKTMtNCrDZr_9Ww7uzfsB7M7muBKpmm_1IkKJnHOI,3222
-ftag/hdf5/h5writer.py,sha256=9FkClV__UbBqmFsq_h2jwiZnbWVm8QFRL_4mDZZBbTs,5316
-ftag/utils/__init__.py,sha256=C0PgaA6Nk5WVpFqKhBhrHgj2mwsKJbSxoO6Cl67RsaI,544
+ftag/hdf5/h5writer.py,sha256=2gBztierWdwZIqcFItoYz8oua_7hphOI8mbDg7xBdPs,5784
+ftag/utils/__init__.py,sha256=U3YyLY77-FzxRUbudxciieDoy_mnLlY3OfBquA3PnTE,524
 ftag/utils/logging.py,sha256=54NaQiC9Bh4vSznSqzoPfR-7tj1PXfmoH7yKgv_ZHZk,3192
 ftag/utils/metrics.py,sha256=zQI4nPeRDSyzqKpdOPmu0GU560xSWoW1wgL13rrja-I,12664
-atlas_ftag_tools-0.2.10.dist-info/METADATA,sha256=VUhrtQML6_bUKlmZNFlUXxTTt5YBzNYupTrdlaF5IAw,5190
-atlas_ftag_tools-0.2.10.dist-info/WHEEL,sha256=CmyFI0kx5cdEMTLiONQRbGQwjIoR1aIYB7eCAQ4KPJ0,91
-atlas_ftag_tools-0.2.10.dist-info/entry_points.txt,sha256=b46bVP_O8Mg6aSdPmyjGgVkaXSdyXZMeKAsofh2IDeA,133
-atlas_ftag_tools-0.2.10.dist-info/top_level.txt,sha256=qiYQuKcAvMim-31FwkT3MTQu7WQm0s58tPAia5KKWqs,5
-atlas_ftag_tools-0.2.10.dist-info/RECORD,,
+atlas_ftag_tools-0.2.11.dist-info/METADATA,sha256=DVmllPN7YQNNmyDcTs3hEGo8mX8ogSReXq9gs6MwUR0,2152
+atlas_ftag_tools-0.2.11.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+atlas_ftag_tools-0.2.11.dist-info/entry_points.txt,sha256=acr7WwxMIJ3x2I7AheNxNnpWE7sS8XE9MA1eUJGcU5A,169
+atlas_ftag_tools-0.2.11.dist-info/top_level.txt,sha256=qiYQuKcAvMim-31FwkT3MTQu7WQm0s58tPAia5KKWqs,5
+atlas_ftag_tools-0.2.11.dist-info/RECORD,,

{atlas_ftag_tools-0.2.10.dist-info → atlas_ftag_tools-0.2.11.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (78.1.0)
+Generator: setuptools (80.9.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

{atlas_ftag_tools-0.2.10.dist-info → atlas_ftag_tools-0.2.11.dist-info}/entry_points.txt

@@ -1,4 +1,5 @@
 [console_scripts]
+h5addcol = ftag.hdf5.h5add_col:main
 h5move = ftag.hdf5.h5move:main
 h5split = ftag.hdf5.h5split:main
 vds = ftag.vds:main

atlas_ftag_tools-0.2.11.dist-info/licenses/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [2025] [Alexander Froch]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

ftag/__init__.py

@@ -2,18 +2,18 @@
 
 from __future__ import annotations
 
-__version__ = "v0.2.10"
+__version__ = "v0.2.11"
 
-from ftag import hdf5, utils
-from ftag.cuts import Cuts
-from ftag.flavours import Flavours
-from ftag.fraction_optimization import calculate_best_fraction_values
-from ftag.labeller import Labeller
-from ftag.labels import Label, LabelContainer
-from ftag.mock import get_mock_file
-from ftag.sample import Sample
-from ftag.transform import Transform
-from ftag.working_points import get_working_points
+from . import hdf5, utils
+from .cuts import Cuts
+from .flavours import Flavours
+from .fraction_optimization import calculate_best_fraction_values
+from .labeller import Labeller
+from .labels import Label, LabelContainer
+from .mock import get_mock_file
+from .sample import Sample
+from .transform import Transform
+from .working_points import get_working_points
 
 __all__ = [
     "Cuts",

ftag/flavours.yaml

@@ -107,42 +107,47 @@
   colour: "#38761D"
   category: xbb
 - name: qcdbb
-  label: QCD->bb
-  cuts: ["R10TruthLabel_R22v1 == 10", "GhostBHadronsFinalCount == 2"]
+  label: $\mathrm{QCD} \rightarrow b \bar{b}$
+  cuts: ["R10TruthLabel_R22v1 == 10", "GhostBHadronsFinalCount >= 2"]
   colour: "red"
   category: xbb
 - name: qcdnonbb
-  label: QCD
+  label: $\mathrm{QCD} \rightarrow \mathrm{non-} b \bar{b}$
   cuts: ["R10TruthLabel_R22v1 == 10", "GhostBHadronsFinalCount != 2"]
   colour: "silver"
   category: xbb
 - name: qcdbx
-  label: QCD
+  label: $\mathrm{QCD} \rightarrow bX$
   cuts: ["R10TruthLabel_R22v1 == 10", "GhostBHadronsFinalCount == 1"]
   colour: "gold"
   category: xbb
 - name: qcdcx
-  label: QCD
+  label: $\mathrm{QCD} \rightarrow cX$
   cuts: ["R10TruthLabel_R22v1 == 10", "GhostCHadronsFinalCount >= 1", "GhostBHadronsFinalCount == 0"]
   colour: "pink"
   category: xbb
 - name: qcdll
-  label: QCD
+  label: $\mathrm{QCD} \rightarrow ll$
   cuts: ["R10TruthLabel_R22v1 == 10", "GhostBHadronsFinalCount == 0", "GhostCHadronsFinalCount == 0"]
   colour: "green"
   category: xbb
-- name: htauel
-  label: $H \rightarrow \tau e$
+- name: Wqq
+  label: $W \rightarrow q\bar{q}$
+  cuts: ["R10TruthLabel_R22v1 == 2", "GhostBHadronsFinalCount < 2", "GhostCHadronsFinalCount < 2"]
+  colour: "purple"
+  category: xbb
+- name: htautauel
+  label: $H \rightarrow \tau_{\mathrm{had}} \tau_{e}$
   cuts: ["R10TruthLabel_R22v1 == 14"]
   colour: "#b40612"
   category: xbb
-- name: htaumu
-  label: $H \rightarrow \tau\mu$
+- name: htautaumu
+  label: $H \rightarrow \tau_{\mathrm{had}} \tau_{\mu}$
   cuts: ["R10TruthLabel_R22v1 == 15"]
   colour: "#b40657"
   category: xbb
-- name: htauhad
-  label: $H \rightarrow \tau\tau$
+- name: htautauhad
+  label: $H \rightarrow \tau_{\mathrm{had}} \tau_{\mathrm{had}}$
   cuts: ["R10TruthLabel_R22v1 == 16"]
   colour: "#b406a0"
   category: xbb

ftag/hdf5/__init__.py

@@ -1,14 +1,16 @@
 from __future__ import annotations
 
-from ftag.hdf5.h5reader import H5Reader
-from ftag.hdf5.h5utils import cast_dtype, get_dtype, join_structured_arrays, structured_from_dict
-from ftag.hdf5.h5writer import H5Writer
+from .h5add_col import h5_add_column
+from .h5reader import H5Reader
+from .h5utils import cast_dtype, get_dtype, join_structured_arrays, structured_from_dict
+from .h5writer import H5Writer
 
 __all__ = [
     "H5Reader",
     "H5Writer",
     "cast_dtype",
     "get_dtype",
+    "h5_add_column",
     "join_structured_arrays",
     "structured_from_dict",
 ]

ftag/hdf5/h5add_col.py

@@ -0,0 +1,391 @@
+# Utils to take an input h5 file, and append one or more columns to it
+from __future__ import annotations
+
+import argparse
+import importlib.util
+from pathlib import Path
+from typing import Callable
+
+import h5py
+import numpy as np
+
+from ftag.hdf5.h5reader import H5Reader
+from ftag.hdf5.h5writer import H5Writer
+
+
+def merge_dicts(dicts: list[dict[str, dict[str, np.ndarray]]]) -> dict[str, dict[str, np.ndarray]]:
+    """Merges a list of dictionaries.
+
+    Each dict is of the form:
+     {
+        group1: {
+            variable_1: np.array
+            variable_2: np.array
+        },
+        group2: {
+            variable_1: np.array
+            variable_2: np.array
+        }
+     }
+
+     E.g.
+
+     dict1 = {
+        "jets": {
+            "pt": np.array([1, 2, 3]),
+            "eta": np.array([4, 5, 6])
+        },
+    }
+    dict2 = {
+        "jets": {
+            "phi": np.array([7, 8, 9]),
+            "energy": np.array([10, 11, 12])
+        },
+    }
+
+    merged = {
+        "jets": {
+            "pt": np.array([1, 2, 3]),
+            "eta": np.array([4, 5, 6]),
+            "phi": np.array([7, 8, 9]),
+            "energy": np.array([10, 11, 12])
+        }
+    }
+
+    Parameters
+    ----------
+    dicts : list[dict[str, dict[str, np.ndarray]]]
+        List of dictionaries to merge. Each dictionary should be of the form:
+
+    Returns
+    -------
+    dict[str, dict[str, np.ndarray]]
+        Merged dictionary of the form:
+        {
+            group1: {
+                variable_1: np.array
+                variable_2: np.array
+            },
+            group2: {
+                variable_1: np.array
+                variable_2: np.array
+            }
+        }
+
+    Raises
+    ------
+    ValueError
+        If a variable already exists in the merged dictionary.
+    """
+    merged: dict[str, dict[str, np.ndarray]] = {}
+    for d in dicts:
+        for group, variables in d.items():
+            if group not in merged:
+                merged[group] = {}
+            for variable, data in variables.items():
+                if variable not in merged[group]:
+                    merged[group][variable] = data
+                else:
+                    raise ValueError(f"Variable {variable} already exists in group {group}.")
+    return merged
+
+
+def get_shape(num_jets: int, batch: dict[str, np.ndarray]) -> dict[str, tuple[int, ...]]:
+    """Returns a dictionary with the correct output shapes for the H5Writer.
+
+    Parameters
+    ----------
+    num_jets : int
+        Number of jets to write in total
+    batch : dict[str, np.ndarray]
+        Dictionary representing the batch
+
+    Returns
+    -------
+    dict[str, tuple[int, ...]]
+        Dictionary with the shapes of the output arrays
+    """
+    shape: dict[str, tuple[int, ...]] = {}
+
+    for key, values in batch.items():
+        if values.ndim == 1:
+            shape[key] = (num_jets,)
+        else:
+            shape[key] = (num_jets,) + values.shape[1:]
+    return shape
+
+
+def get_all_groups(file: Path | str) -> dict[str, None]:
+    """Returns a dictionary with all the groups in the h5 file.
+
+    Parameters
+    ----------
+    file : Path | str
+        Path to the h5 file
+
+    Returns
+    -------
+    dict[str, None]
+        A dictionary with all the groups in the h5 file as keys and None as values,
+        such that h5read.stream(all_groups) will return all the groups in the file.
+    """
+    with h5py.File(file, "r") as f:
+        groups = list(f.keys())
+        return dict.fromkeys(groups)
+
+
+def h5_add_column(
+    input_file: str | Path,
+    output_file: str | Path,
+    append_function: Callable | list[Callable],
+    num_jets: int = -1,
+    input_groups: list[str] | None = None,
+    output_groups: list[str] | None = None,
+    reader_kwargs: dict | None = None,
+    writer_kwargs: dict | None = None,
+    overwrite: bool = False,
+) -> None:
+    """Appends one or more columns to one or more groups in an h5 file.
+
+    Parameters
+    ----------
+    input_file : str | Path
+        Input h5 file to read from.
+    output_file : str | Path
+        Output h5 file to write to.
+    append_function : callable | list[callable]
+        A function, or list of functions, which take a batch from H5Reader and returns a dictionary
+        of the form:
+            {
+                group1 : {
+                    new_column1 : data,
+                    new_column2 : data,
+                },
+                group2 : {
+                    new_column3 : data,
+                    new_column4 : data,
+                },
+                ...
+            }
+    num_jets : int, optional
+        Number of jets to read from the input file. If -1, reads all jets. By default -1.
+    input_groups : list[str] | None, optional
+        List of groups to read from the input file. If None, reads all groups. By default None.
+    output_groups : list[str] | None, optional
+        List of groups to write to the output file. If None, writes all groups. By default None.
+        Note that this is a subset of the input groups, and must include all groups that the
+        append functions wish to write to.
+    reader_kwargs : dict, optional
+        Additional arguments to pass to the H5Reader. By default None.
+    writer_kwargs : dict, optional
+        Additional arguments to pass to the H5Writer. By default None.
+    overwrite : bool, optional
+        If True, will overwrite the output file if it exists. By default False.
+        If False, will raise a FileExistsError if the output file exists.
+        If None, will check if the output file exists and raise an error if it does unless
+        overwrite is True.
+
+    Raises
+    ------
+    FileNotFoundError
+        If the input file does not exist.
+    FileExistsError
+        If the output file exists and overwrite is False.
+    ValueError
+        If the new variable already exists, shape is incorrect, or the output group is not in
+        the input groups.
+
+    """
+    input_file = Path(input_file)
+    output_file = Path(output_file) if output_file is not None else None
+
+    if not input_file.exists():
+        raise FileNotFoundError(f"Input file {input_file} does not exist.")
+    if output_file is not None and output_file.exists() and not overwrite:
+        raise FileExistsError(
+            f"Output file {output_file} already exists. Please choose a different name."
+        )
+    if not reader_kwargs:
+        reader_kwargs = {}
+    if not writer_kwargs:
+        writer_kwargs = {}
+    if output_file is None:
+        output_file = input_file.with_name(input_file.name.replace(".h5", "_additional.h5"))
+
+    if not isinstance(append_function, list):
+        append_function = [append_function]
+
+    reader = H5Reader(input_file, shuffle=False, **reader_kwargs)
+    if "precision" not in writer_kwargs:
+        writer_kwargs["precision"] = "full"
+
+    njets = reader.num_jets if num_jets == -1 else num_jets
+    writer = None
+
+    input_variables = (
+        get_all_groups(input_file) if input_groups is None else dict.fromkeys(input_groups)
+    )
+    if output_groups is None:
+        output_groups = list(input_variables.keys())
+
+    assert all(
+        o in input_variables for o in output_groups
+    ), f"Output groups {output_groups} not in input groups {input_variables.keys()}"
+
+    num_batches = njets // reader.batch_size + 1
+    for i, batch in enumerate(reader.stream(input_variables, num_jets=njets)):
+        if (i + 1) % 10 == 0:
+            print(f"Processing batch {i + 1}/{num_batches} ({(i + 1) / num_batches * 100:.2f}%)")
+
+        to_append = merge_dicts([af(batch) for af in append_function])
+        for k, newvars in to_append.items():
+            if k not in output_groups:
+                raise ValueError(f"Trying to output to {k} but only {output_groups} are allowed")
+            for newkey, newval in newvars.items():
+                if newkey in batch[k].dtype.names:
+                    raise ValueError(
+                        f"Trying to append {newkey} to {k} but it already exists in batch"
+                    )
+                if newval.shape != batch[k].shape:
+                    raise ValueError(
+                        f"Trying to append {newkey} to {k} but the shape is not correct"
+                    )
+
+        to_write = {}
+
+        for key, str_array in batch.items():
+            if key not in output_groups:
+                continue
+            if key in to_append:
+                combined = np.lib.recfunctions.append_fields(
+                    str_array,
+                    list(to_append[key].keys()),
+                    list(to_append[key].values()),
+                    usemask=False,
+                )
+                to_write[key] = combined
+            else:
+                to_write[key] = str_array
+        if writer is None:
+            writer = H5Writer(
+                output_file,
+                dtypes={key: str_array.dtype for key, str_array in to_write.items()},
+                shapes=get_shape(njets, to_write),
+                shuffle=False,
+                **writer_kwargs,
+            )
+
+        writer.write(to_write)
+
+
+def parse_append_function(func_path: str) -> Callable:
+    """Attempts to load the function specified by func_path.
+    The function should be specified as 'path/to/file.py:function_name'.
+
+    Parameters
+    ----------
+    func_path : str
+        Path to the function to load. Should be of the form 'path/to/file.py:function_name'.
+
+    Returns
+    -------
+    Callable
+        The function specified by func_path.
+
+    Raises
+    ------
+    ValueError
+        If the function path is not of the form 'path/to/file.py:function_name'.
+    FileNotFoundError
+        If the file does not exist.
+    ImportError
+        If the file cannot be imported.
+    AttributeError
+        If the function does not exist in the file.
+    """
+    if isinstance(func_path, Path):
+        func_path = str(func_path)
+    if ":" not in func_path:
+        print(func_path)
+        raise ValueError("Function should be specified as 'path/to/file.py:function_name'")
+
+    file_str, func_name = func_path.split(":")
+    file_path = Path(file_str).resolve()
+
+    if not file_path.is_file():
+        raise FileNotFoundError(f"No such file: {file_path}")
+
+    module_name = file_path.stem  # Just the filename without extension
+
+    spec = importlib.util.spec_from_file_location(module_name, str(file_path))
+    if spec is None or spec.loader is None:
+        raise ImportError(f"Cannot load spec for {file_path}")
+
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+
+    if not hasattr(module, func_name):
+        raise AttributeError(f"Module {module_name} has no attribute {func_name}")
+
+    return getattr(module, func_name)
+
+
+def get_args(args):
+    parser = argparse.ArgumentParser(description="Append columns to an h5 file.")
+    parser.add_argument("--input", "-i", type=str, required=True, help="Input h5 file")
+    parser.add_argument(
+        "--append_function",
+        type=str,
+        nargs="+",
+        help="Function to append to the h5 file. Can be a list of functions.",
+        required=True,
+    )
+    parser.add_argument("--output", type=str, help="Output h5 file")
+    parser.add_argument(
+        "--num_jets", type=int, default=-1, help="Number of jets to read from the input file"
+    )
+    parser.add_argument(
+        "--input_groups",
+        type=str,
+        nargs="+",
+        default=None,
+        help="List of groups to read from the input file",
+    )
+    parser.add_argument(
+        "--output_groups",
+        type=str,
+        nargs="+",
+        default=None,
+        help="List of groups to write to the output file",
+    )
+    parser.add_argument(
+        "--reader_kwargs", type=dict, default=None, help="Additional arguments for H5Reader"
+    )
+    parser.add_argument(
+        "--writer_kwargs", type=dict, default=None, help="Additional arguments for H5Writer"
+    )
+    parser.add_argument(
+        "--overwrite", action="store_true", help="Overwrite the output file if it exists"
+    )
+
+    return parser.parse_args(args)
+
+
+def main(args=None):
+    args = get_args(args)
+    append_function = [
+        parse_append_function(func_path) if isinstance(func_path, str) else func_path
+        for func_path in args.append_function
+    ]
+
+    h5_add_column(
+        args.input,
+        args.output,
+        append_function,
+        num_jets=args.num_jets,
+        input_groups=args.input_groups,
+        output_groups=args.output_groups,
+        reader_kwargs=args.reader_kwargs,
+        writer_kwargs=args.writer_kwargs,
+        overwrite=args.overwrite,
+    )

ftag/hdf5/h5writer.py

@@ -31,8 +31,11 @@ class H5Writer:
         Compression algorithm to use. Default is "lzf".
     precision : str | None, optional
         Precision to use. Default is None.
+    full_precision_vars : list[str] | None, optional
+        List of variables to store in full precision. Default is None.
     shuffle : bool, optional
         Whether to shuffle the jets before writing. Default is True.
+
     """
 
     dst: Path | str
@@ -42,6 +45,7 @@ class H5Writer:
     add_flavour_label: bool = False
     compression: str = "lzf"
     precision: str = "full"
+    full_precision_vars: list[str] | None = None
     shuffle: bool = True
 
     def __post_init__(self):
@@ -85,8 +89,15 @@ class H5Writer:
             dtype = np.dtype([*dtype.descr, ("flavour_label", "i4")])
 
         # adjust dtype based on specified precision
+        full_precision_vars = [] if self.full_precision_vars is None else self.full_precision_vars
+        # If the field is in full_precision_vars, use the full precision dtype
         dtype = np.dtype([
-            (field, self.fp_dtype if np.issubdtype(dt, np.floating) else dt)
+            (
+                field,
+                self.fp_dtype
+                if field not in full_precision_vars and np.issubdtype(dt, np.floating)
+                else dt,
+            )
             for field, dt in dtype.descr
         ])
 

ftag/utils/__init__.py

@@ -1,7 +1,7 @@
 from __future__ import annotations
 
-from ftag.utils.logging import logger, set_log_level
-from ftag.utils.metrics import (
+from .logging import logger, set_log_level
+from .metrics import (
     calculate_efficiency,
     calculate_efficiency_error,
     calculate_rejection,

ftag/vds.py

@@ -2,6 +2,9 @@ from __future__ import annotations
 
 import argparse
 import glob
+import os
+import re
+import sys
 from pathlib import Path
 
 import h5py
@@ -13,6 +16,8 @@ def parse_args(args):
     )
     parser.add_argument("pattern", type=Path, help="quotes-enclosed glob pattern of files to merge")
     parser.add_argument("output", type=Path, help="path to output virtual file")
+    parser.add_argument("--use_regex", help="if provided pattern is a regex", action="store_true")
+    parser.add_argument("--regex_path", type=str, required="--regex" in sys.argv, default=None)
     return parser.parse_args(args)
 
 
@@ -43,13 +48,36 @@ def get_virtual_layout(fnames: list[str], group: str):
     return layout
 
 
+def glob_re(pattern, regex_path):
+    return list(filter(re.compile(pattern).match, os.listdir(regex_path)))
+
+
+def regex_files_from_dir(reg_matched_fnames, regex_path):
+    parent_dir = regex_path or str(Path.cwd())
+    full_paths = [parent_dir + "/" + fname for fname in reg_matched_fnames]
+    paths_to_glob = [fname + "/*.h5" if Path(fname).is_dir() else fname for fname in full_paths]
+    nested_fnames = [glob.glob(fname) for fname in paths_to_glob]
+    return sum(nested_fnames, [])
+
+
 def create_virtual_file(
-    pattern: Path | str, out_fname: Path | None = None, overwrite: bool = False
+    pattern: Path | str,
+    out_fname: Path | None = None,
+    use_regex: bool = False,
+    regex_path: str | None = None,
+    overwrite: bool = False,
 ):
     # get list of filenames
-    fnames = glob.glob(str(pattern))
+    pattern_str = str(pattern)
+    if use_regex:
+        reg_matched_fnames = glob_re(pattern_str, regex_path)
+        print("reg matched fnames: ", reg_matched_fnames)
+        fnames = regex_files_from_dir(reg_matched_fnames, regex_path)
+    else:
+        fnames = glob.glob(pattern_str)
     if not fnames:
         raise FileNotFoundError(f"No files matched pattern {pattern}")
+    print("Files to merge to vds: ", fnames)
 
     # infer output path if not given
     if out_fname is None:
@@ -94,8 +122,15 @@ def create_virtual_file(
 
 def main(args=None) -> None:
     args = parse_args(args)
-    print(f"Globbing {args.pattern}...")
-    create_virtual_file(args.pattern, args.output, overwrite=True)
+    matching_mode = "Applying regex to" if args.use_regex else "Globbing"
+    print(f"{matching_mode} {args.pattern}...")
+    create_virtual_file(
+        args.pattern,
+        args.output,
+        use_regex=args.use_regex,
+        regex_path=args.regex_path,
+        overwrite=True,
+    )
     with h5py.File(args.output) as f:
         key = next(iter(f.keys()))
         num = len(f[key])

atlas_ftag_tools-0.2.10.dist-info/METADATA

@@ -1,151 +0,0 @@
-Metadata-Version: 2.4
-Name: atlas-ftag-tools
-Version: 0.2.10
-Summary: ATLAS Flavour Tagging Tools
-Author: Sam Van Stroud, Philipp Gadow
-License: MIT
-Project-URL: Homepage, https://github.com/umami-hep/atlas-ftag-tools/
-Requires-Python: <3.12,>=3.8
-Description-Content-Type: text/markdown
-Requires-Dist: h5py>=3.0
-Requires-Dist: numpy>=2.2.3
-Requires-Dist: PyYAML>=5.1
-Requires-Dist: scipy>=1.15.2
-Provides-Extra: dev
-Requires-Dist: ruff==0.6.2; extra == "dev"
-Requires-Dist: mypy==1.11.2; extra == "dev"
-Requires-Dist: pre-commit==3.1.1; extra == "dev"
-Requires-Dist: pytest==7.2.2; extra == "dev"
-Requires-Dist: pytest-cov==4.0.0; extra == "dev"
-Requires-Dist: pytest_notebook==0.10.0; extra == "dev"
-Requires-Dist: ipykernel==6.21.3; extra == "dev"
-
-[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-[![PyPI version](https://badge.fury.io/py/atlas-ftag-tools.svg)](https://badge.fury.io/py/atlas-ftag-tools)
-[![codecov](https://codecov.io/gh/umami-hep/atlas-ftag-tools/branch/main/graph/badge.svg?token=MBHLIYYQ7I)](https://codecov.io/gh/umami-hep/atlas-ftag-tools)
-
-# ATLAS FTAG Python Tools
-
-This is a collection of Python tools for working with files produced with the FTAG [ntuple dumper](https://gitlab.cern.ch/atlas-flavor-tagging-tools/training-dataset-dumper/).
-The code is intended to be used a [library](https://iscinumpy.dev/post/app-vs-library/) for other projects.
-Please see the [example notebook](ftag/example.ipynb) for usage.
-
-# Quickstart 
-
-## Installation
-
-If you want to use this package without modification, you can install from [pypi](https://pypi.org/project/atlas-ftag-tools/) using `pip`.
-
-```bash
-pip install atlas-ftag-tools
-```
-
-To additionally install the development dependencies (for formatting and linting) use
-```bash
-pip install atlas-ftag-tools[dev]
-```
-
-## Development
-
-If you plan on making changes to teh code, instead clone the repository and install the package from source in editable mode with
-
-```bash
-python -m pip install -e .
-```
-
-Include development dependencies with
-
-```bash
-python -m pip install -e ".[dev]"
-```
-
-You can set up and run pre-commit hooks with
-
-```bash
-pre-commit install
-pre-commmit run --all-files
-```
-
-To run the tests you can use the `pytest` or `coverage` command, for example
-
-```bash
-coverage run --source ftag -m pytest --show-capture=stdout
-```
-
-Running `coverage report` will display the test coverage.
-
-
-# Usage
-
-Please see the [example notebook](ftag/example.ipynb) for full usage.
-Additional functionality is also documented below.
-
-## Calculate WPs
-
-This package contains a script to calculate tagger working points (WPs).
-The script is `working_points.py` and can be run after installing this package with
-
-```
-wps \
-    --ttbar "path/to/ttbar/*.h5" \
-    --tagger GN2v01 \
-    --fc 0.1
-```
-
-Both the `--tagger` and `--fc` options accept a list if you want to get the WPs for multiple taggers.
-If you are doing c-tagging or xbb-tagging, dedicated fx arguments are available ()you can find them all with `-h`.
-
-If you want to use the `ttbar` WPs get the efficiencies and rejections for the `zprime` sample, you can add `--zprime "path/to/zprime/*.h5"` to the command.
-Note that a default selection of $p_T > 250 ~GeV$ to jets in the `zprime` sample.
-
-If instead of defining the working points for a series of signal efficiencies, you wish to calculate a WP corresponding to a specific background rejection, the `--rejection` option can be given along with the desired background.
-
-By default the working points are printed to the terminal, but you can save the results to a YAML file with the `--outfile` option.
-
-See `wps --help` for more options and information.
-
-## Calculate efficiency at discriminant cut 
-
-The same script can be used to calculate the efficiency and rejection values at a given discriminant cut value.
-The script `working_points.py` can be run after intalling this package as follows
-
-```
-wps \
-    --ttbar "path/to/ttbar/*.h5" \
-    --tagger GN2v01 \
-    --fx 0.1
-    --disc_cuts 1.0 1.5
-```
-The `--tagger`, `--fx`, and `--outfile` follow the same procedure as in the 'Calculate WPs' script as described above.
-
-## H5 Utils
-
-### Create virtual file
-
-This package contains a script to easily merge a set of H5 files.
-A virtual file is a fast and lightweight way to wrap a set of files.
-See the [h5py documentation](https://docs.h5py.org/en/stable/vds.html) for more information on virtual datasets.
-
-The script is `vds.py` and can be run after installing this package with
-
-```
-vds <pattern> <output path>
-```
-
-The `<pattern>` argument should be a quotes enclosed [glob pattern](https://en.wikipedia.org/wiki/Glob_(programming)), for example `"dsid/path/*.h5"`
-
-See `vds --help` for more options and information.
-
-
-### [h5move](ftag/hdf5/h5move.py)
-
-A script to move/rename datasets inside an h5file.
-Useful for correcting discrepancies between group names.
-See [h5move.py](ftag/hdf5/h5move.py) for more info.
-
-
-### [h5split](ftag/hdf5/h5split.py)
-
-A script to split a large h5 file into several smaller files.
-Useful if output files are too large for EOS/grid storage.
-See [h5split.py](ftag/hdf5/h5split.py) for more info.