hepyy-workflows 0.1.0__tar.gz

hepyy_workflows-0.1.0/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.4
+Name: hepyy-workflows
+Version: 0.1.0
+Summary: Explicit, portable HEP data workflows
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: hepyy>=0.2.18
+Requires-Dist: jsonschema>=4.23
+Requires-Dist: numpy>=1.26
+Requires-Dist: pyarrow>=18
+Requires-Dist: PyYAML>=6
+Provides-Extra: parquet
+Provides-Extra: root
+Requires-Dist: uproot>=5; extra == "root"
+Requires-Dist: awkward>=2; extra == "root"
+Provides-Extra: hdf5
+Requires-Dist: h5py>=3; extra == "hdf5"
+Provides-Extra: hepmc
+Requires-Dist: hepyy>=0.2.18; extra == "hepmc"
+Provides-Extra: pythia
+Requires-Dist: hepyy>=0.2.18; extra == "pythia"
+Provides-Extra: web
+Requires-Dist: fastapi>=0.115; extra == "web"
+Requires-Dist: uvicorn>=0.30; extra == "web"
+Provides-Extra: tui
+Requires-Dist: textual>=0.80; extra == "tui"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: ruff>=0.9; extra == "dev"
+
+# hepyy-workflows
+
+An explicit HEP workflow package. [SPECS.md](SPECS.md) defines the full project;
+this checkout currently implements **milestones 1 through 6**. The package name is
+`hepyy_workflows` and the CLI is `hwf`. Python 3.11+ is required.
+
+The package provides Arrow dataset/source contracts, strict public reader and
+recipe schemas, a starter project, and bounded inspect/draft/validate/preview
+workflows for Parquet, ROOT TTrees, explicit HDF5 layouts, and HepMC3 ASCII.
+Recipes now run generic views, selections, histograms and cutflows and export a
+standalone file-analysis runner, self-contained reports, and local web/terminal
+authoring tools.
+
+```bash
+henv -n YOUR_ENV -x python -m pip install -e '.[dev]'
+henv -n YOUR_ENV -x python -m pytest
+```
+
+Create a starter project **from any directory** (choose a destination that does
+not yet exist):
+
+```bash
+henv -n YOUR_ENV -x hwf project init /path/to/my-analysis
+cd /path/to/my-analysis
+henv -n YOUR_ENV -x hwf schema validate reader reader.yaml --json
+henv -n YOUR_ENV -x hwf schema validate recipe recipe.yaml --json
+henv -n YOUR_ENV -x hwf reader inspect data/particles.parquet --json
+henv -n YOUR_ENV -x hwf reader validate reader.yaml --input data/particles.parquet --json
+henv -n YOUR_ENV -x hwf reader preview reader.yaml --input data/particles.parquet --events 2 --json
+henv -n YOUR_ENV -x hwf recipe run recipe.yaml --input data/particles.parquet --output results --json
+henv -n YOUR_ENV -x hwf recipe export recipe.yaml --input data/particles.parquet --output standalone --json
+```
+
+The last command reads the starter project's sample file. See
+[working-in-a-project.md](docs/working-in-a-project.md) for what the starter
+files mean and [examples/readers](examples/readers/README.md) for each format.
+For choosing event boundaries, collection paths, units and labels in an unfamiliar
+file, see [FILE_FORMAT_README.md](FILE_FORMAT_README.md).
+See [analysis.md](docs/analysis.md) for recipe operations, histogram products,
+and standalone execution; the [Lund](examples/lund/README.md) and
+[dilepton](examples/dilepton/README.md) examples are complete workflows.
+
+Build an offline report from completed results, or inspect them remotely in a
+terminal:
+
+```bash
+henv -n YOUR_ENV -x hwf report results --output results.html --json
+henv -n YOUR_ENV -x hwf tui results
+henv -n YOUR_ENV -x hwf web recipe.yaml --port 8765
+```
+
+The report is one HTML file with embedded data and scripts; it works from
+`file://` without a server or network connection. The web editor listens on
+localhost and validates edits through the same service as the CLI.
+
+Agents can use the same operations through line-delimited JSON:
+
+```bash
+printf '%s\n' '{"id":1,"method":"inspect_input","params":{"input":"data/particles.parquet"}}' \
+  | henv -n YOUR_ENV -x hwf agent serve
+henv -n YOUR_ENV -x hwf reader scaffold my_reader --output src/my_reader
+```
+
+See [agent-workflow.md](docs/agent-workflow.md) for the typed method contract
+and the recommended inspect → draft → validate → preview → run → export flow.
+
+Recipes may declare a visible Python preprocessing module. It receives the
+ordered input paths and a Parquet output path, and its source and dependencies
+are bundled by standalone export. See [processing.md](docs/processing.md).
+
+`hwf schema validate` accepts `reader-draft`, `reader`, or `recipe`; failed
+validation exits nonzero and reports stable diagnostic codes and YAML paths.
+Paths passed to the CLI are resolved from your current directory; use absolute
+paths when invoking it from elsewhere. A missing file reports `FILE_NOT_FOUND`.
+An omitted momentum-field unit resolves to GeV/c and appears in `assumed_units`
+in JSON output. Declare a different source unit explicitly when needed.
+Schema files are available under `schemas/` and ship inside the wheel.
+
+The package requires NumPy, PyArrow, JSON Schema and PyYAML. Pure histogramming does not import hepyy,
+FastJet, PYTHIA, HepMC, uproot, h5py or a web server. Optional extras in
+`pyproject.toml` reserve these dependency boundaries. For all milestone 2
+readers, install `.[root,hdf5,hepmc]` in the analysis environment. See
+[architecture.md](docs/architecture.md) and [provenance.md](docs/provenance.md).

hepyy_workflows-0.1.0/README.md

@@ -0,0 +1,83 @@
+# hepyy-workflows
+
+An explicit HEP workflow package. [SPECS.md](SPECS.md) defines the full project;
+this checkout currently implements **milestones 1 through 6**. The package name is
+`hepyy_workflows` and the CLI is `hwf`. Python 3.11+ is required.
+
+The package provides Arrow dataset/source contracts, strict public reader and
+recipe schemas, a starter project, and bounded inspect/draft/validate/preview
+workflows for Parquet, ROOT TTrees, explicit HDF5 layouts, and HepMC3 ASCII.
+Recipes now run generic views, selections, histograms and cutflows and export a
+standalone file-analysis runner, self-contained reports, and local web/terminal
+authoring tools.
+
+```bash
+henv -n YOUR_ENV -x python -m pip install -e '.[dev]'
+henv -n YOUR_ENV -x python -m pytest
+```
+
+Create a starter project **from any directory** (choose a destination that does
+not yet exist):
+
+```bash
+henv -n YOUR_ENV -x hwf project init /path/to/my-analysis
+cd /path/to/my-analysis
+henv -n YOUR_ENV -x hwf schema validate reader reader.yaml --json
+henv -n YOUR_ENV -x hwf schema validate recipe recipe.yaml --json
+henv -n YOUR_ENV -x hwf reader inspect data/particles.parquet --json
+henv -n YOUR_ENV -x hwf reader validate reader.yaml --input data/particles.parquet --json
+henv -n YOUR_ENV -x hwf reader preview reader.yaml --input data/particles.parquet --events 2 --json
+henv -n YOUR_ENV -x hwf recipe run recipe.yaml --input data/particles.parquet --output results --json
+henv -n YOUR_ENV -x hwf recipe export recipe.yaml --input data/particles.parquet --output standalone --json
+```
+
+The last command reads the starter project's sample file. See
+[working-in-a-project.md](docs/working-in-a-project.md) for what the starter
+files mean and [examples/readers](examples/readers/README.md) for each format.
+For choosing event boundaries, collection paths, units and labels in an unfamiliar
+file, see [FILE_FORMAT_README.md](FILE_FORMAT_README.md).
+See [analysis.md](docs/analysis.md) for recipe operations, histogram products,
+and standalone execution; the [Lund](examples/lund/README.md) and
+[dilepton](examples/dilepton/README.md) examples are complete workflows.
+
+Build an offline report from completed results, or inspect them remotely in a
+terminal:
+
+```bash
+henv -n YOUR_ENV -x hwf report results --output results.html --json
+henv -n YOUR_ENV -x hwf tui results
+henv -n YOUR_ENV -x hwf web recipe.yaml --port 8765
+```
+
+The report is one HTML file with embedded data and scripts; it works from
+`file://` without a server or network connection. The web editor listens on
+localhost and validates edits through the same service as the CLI.
+
+Agents can use the same operations through line-delimited JSON:
+
+```bash
+printf '%s\n' '{"id":1,"method":"inspect_input","params":{"input":"data/particles.parquet"}}' \
+  | henv -n YOUR_ENV -x hwf agent serve
+henv -n YOUR_ENV -x hwf reader scaffold my_reader --output src/my_reader
+```
+
+See [agent-workflow.md](docs/agent-workflow.md) for the typed method contract
+and the recommended inspect → draft → validate → preview → run → export flow.
+
+Recipes may declare a visible Python preprocessing module. It receives the
+ordered input paths and a Parquet output path, and its source and dependencies
+are bundled by standalone export. See [processing.md](docs/processing.md).
+
+`hwf schema validate` accepts `reader-draft`, `reader`, or `recipe`; failed
+validation exits nonzero and reports stable diagnostic codes and YAML paths.
+Paths passed to the CLI are resolved from your current directory; use absolute
+paths when invoking it from elsewhere. A missing file reports `FILE_NOT_FOUND`.
+An omitted momentum-field unit resolves to GeV/c and appears in `assumed_units`
+in JSON output. Declare a different source unit explicitly when needed.
+Schema files are available under `schemas/` and ship inside the wheel.
+
+The package requires NumPy, PyArrow, JSON Schema and PyYAML. Pure histogramming does not import hepyy,
+FastJet, PYTHIA, HepMC, uproot, h5py or a web server. Optional extras in
+`pyproject.toml` reserve these dependency boundaries. For all milestone 2
+readers, install `.[root,hdf5,hepmc]` in the analysis environment. See
+[architecture.md](docs/architecture.md) and [provenance.md](docs/provenance.md).

hepyy_workflows-0.1.0/pyproject.toml

@@ -0,0 +1,47 @@
+[build-system]
+requires = ["setuptools>=69", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hepyy-workflows"
+version = "0.1.0"
+description = "Explicit, portable HEP data workflows"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = ["hepyy>=0.2.18", "jsonschema>=4.23", "numpy>=1.26", "pyarrow>=18", "PyYAML>=6"]
+
+[project.optional-dependencies]
+parquet = []
+root = ["uproot>=5", "awkward>=2"]
+hdf5 = ["h5py>=3"]
+hepmc = ["hepyy>=0.2.18"]
+pythia = ["hepyy>=0.2.18"]
+web = ["fastapi>=0.115", "uvicorn>=0.30"]
+tui = ["textual>=0.80"]
+dev = ["pytest>=8", "ruff>=0.9"]
+
+[project.scripts]
+hwf = "hepyy_workflows.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+hepyy_workflows = ["schemas/*.json", "data/*.parquet", "starter/*.yaml", "starter/*.md"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+target-version = "py311"
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+
+[tool.ruff.lint.per-file-ignores]
+"src/hepyy_workflows/reporting/report.py" = ["E501"]
+"src/hepyy_workflows/ui/web.py" = ["E501"]
+"src/hepyy_workflows/agent/service.py" = ["E501"]
+"src/hepyy_workflows/readers/scaffold.py" = ["E501"]
+"tests/test_agent.py" = ["E501"]

hepyy_workflows-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

hepyy_workflows-0.1.0/src/hepyy_workflows/__init__.py

@@ -0,0 +1,3 @@
+"""Source-neutral contracts and services for HEP workflows."""
+
+__version__ = "0.1.0.dev0"

hepyy_workflows-0.1.0/src/hepyy_workflows/agent/__init__.py

@@ -0,0 +1,5 @@
+"""Optional narrow agent integration."""
+
+from .service import dispatch
+
+__all__ = ["dispatch"]

hepyy_workflows-0.1.0/src/hepyy_workflows/agent/service.py

@@ -0,0 +1,80 @@
+"""Stable JSON operations for agents and ordinary automation."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from hepyy_workflows.analysis.runner import prepare_recipe, run_recipe
+from hepyy_workflows.export.standalone import export_recipe
+from hepyy_workflows.readers.scaffold import scaffold_reader
+from hepyy_workflows.readers.service import (
+    draft_reader,
+    inspect_input,
+    preview_reader,
+    validate_reader,
+    write_draft,
+)
+from hepyy_workflows.reporting.report import build_report
+
+
+def _paths(values: list[str] | None) -> list[Path]:
+    return [Path(value) for value in values or []]
+
+
+def _error(error: Exception, code: str = "AGENT_FAILED") -> dict[str, Any]:
+    return {
+        "valid": False,
+        "diagnostics": [
+            {
+                "code": getattr(error, "code", code),
+                "path": getattr(error, "path", "/"),
+                "message": str(error),
+            }
+        ],
+    }
+
+
+def dispatch(method: str, params: dict[str, Any]) -> dict[str, Any]:
+    try:
+        if method == "inspect_input":
+            return inspect_input(Path(params["input"]))
+        if method == "draft_reader":
+            if "output" in params:
+                return write_draft(Path(params["input"]), Path(params["output"]))
+            draft, inspection = draft_reader(Path(params["input"]))
+            return {**inspection, "draft": draft}
+        if method == "validate_reader":
+            return validate_reader(Path(params["reader"]), _paths(params.get("inputs")))
+        if method == "preview_reader":
+            return preview_reader(
+                Path(params["reader"]), _paths(params.get("inputs")), int(params.get("events", 10))
+            )
+        if method == "validate_recipe":
+            _, _, prepared = prepare_recipe(Path(params["recipe"]), params.get("inputs"))
+            return {
+                "valid": True,
+                "inputs": {k: [str(p) for p in v] for k, v in prepared["paths"].items()},
+            }
+        if method == "run_analysis":
+            return run_recipe(
+                Path(params["recipe"]),
+                Path(params["output"]),
+                params.get("inputs"),
+                params.get("max_events"),
+            )
+        if method == "export_standalone":
+            return export_recipe(
+                Path(params["recipe"]), Path(params["output"]), params.get("inputs")
+            )
+        if method == "build_report":
+            return build_report(Path(params["results"]), Path(params["output"]))
+        if method == "scaffold_reader":
+            return scaffold_reader(params["name"], Path(params["output"]))
+        if method == "get_status":
+            return {"valid": True, "status": "idle", "diagnostics": []}
+        if method == "cancel_run":
+            return _error(ValueError("no asynchronous run is active"), "NO_ACTIVE_RUN")
+        return _error(ValueError(f"unknown method: {method}"), "METHOD_UNKNOWN")
+    except Exception as error:
+        return _error(error)

hepyy_workflows-0.1.0/src/hepyy_workflows/agent/stdio.py

@@ -0,0 +1,54 @@
+"""Line-delimited JSON agent interface; no shell or arbitrary code endpoint."""
+
+from __future__ import annotations
+
+import json
+import sys
+
+from .service import dispatch
+
+_METHODS = (
+    "inspect_input",
+    "draft_reader",
+    "validate_reader",
+    "preview_reader",
+    "validate_recipe",
+    "run_analysis",
+    "export_standalone",
+    "build_report",
+    "scaffold_reader",
+    "get_status",
+    "cancel_run",
+)
+
+
+def serve_stdio() -> int:
+    for line in sys.stdin:
+        if not line.strip():
+            continue
+        try:
+            request = json.loads(line)
+            if not isinstance(request, dict) or not isinstance(request.get("method"), str):
+                raise ValueError("request needs a method")
+            method = request["method"]
+            if method == "initialize":
+                result = {
+                    "protocolVersion": "2025-03-26",
+                    "serverInfo": {"name": "hwf", "version": "1"},
+                }
+            elif method == "tools/list":
+                result = {
+                    "tools": [
+                        {"name": name, "description": f"HWF {name} operation"} for name in _METHODS
+                    ]
+                }
+            elif method == "tools/call":
+                params = request.get("params", {})
+                result = dispatch(params["name"], params.get("arguments", {}))
+            else:
+                result = dispatch(method, request.get("params", {}))
+            response = {"id": request.get("id"), "result": result}
+        except Exception as error:
+            response = {"id": None, "error": {"code": "REQUEST_INVALID", "message": str(error)}}
+        print(json.dumps(response, default=str), flush=True)
+    return 0

hepyy_workflows-0.1.0/src/hepyy_workflows/analysis/__init__.py

@@ -0,0 +1,5 @@
+"""Generic recipe analysis services."""
+
+from .runner import RecipeError, prepare_recipe, run_recipe
+
+__all__ = ["RecipeError", "prepare_recipe", "run_recipe"]

hepyy_workflows-0.1.0/src/hepyy_workflows/analysis/expressions.py

@@ -0,0 +1,184 @@
+"""Small, side-effect-free expression evaluator for recipe values."""
+
+from __future__ import annotations
+
+import ast
+import math
+import operator
+from functools import lru_cache
+from typing import Any
+
+
+class ExpressionError(ValueError):
+    pass
+
+
+def wrapped_angle(value: float) -> float:
+    return math.atan2(math.sin(value), math.cos(value))
+
+
+_FUNCTIONS = {
+    "abs": abs,
+    "sqrt": math.sqrt,
+    "log": math.log,
+    "log10": math.log10,
+    "sin": math.sin,
+    "cos": math.cos,
+    "tan": math.tan,
+    "atan2": math.atan2,
+    "hypot": math.hypot,
+    "wrapped_angle": wrapped_angle,
+    "isfinite": math.isfinite,
+    "min": min,
+    "max": max,
+}
+_BINARY = {
+    ast.Add: operator.add,
+    ast.Sub: operator.sub,
+    ast.Mult: operator.mul,
+    ast.Div: operator.truediv,
+    ast.Pow: math.pow,
+    ast.Mod: operator.mod,
+}
+_COMPARE = {
+    ast.Eq: operator.eq,
+    ast.NotEq: operator.ne,
+    ast.Lt: operator.lt,
+    ast.LtE: operator.le,
+    ast.Gt: operator.gt,
+    ast.GtE: operator.ge,
+    ast.Is: operator.is_,
+    ast.IsNot: operator.is_not,
+}
+
+
+@lru_cache(maxsize=1024)
+def compile_expression(source: str) -> ast.Expression:
+    try:
+        tree = ast.parse(source, mode="eval")
+    except SyntaxError as exc:
+        raise ExpressionError(f"invalid expression {source!r}: {exc.msg}") from exc
+    _check(tree.body)
+    return tree
+
+
+def referenced_names(source: str) -> set[str]:
+    """Return field/parameter roots, excluding whitelisted function names."""
+
+    tree = compile_expression(source)
+    names = {node.id for node in ast.walk(tree) if isinstance(node, ast.Name)}
+    calls = {
+        node.func.id
+        for node in ast.walk(tree)
+        if isinstance(node, ast.Call) and isinstance(node.func, ast.Name)
+    }
+    return names - calls
+
+
+def _path(node: ast.AST) -> str | None:
+    if isinstance(node, ast.Name):
+        return node.id
+    if isinstance(node, ast.Attribute):
+        parent = _path(node.value)
+        return None if parent is None else parent + "." + node.attr
+    return None
+
+
+def _check(node: ast.AST) -> None:
+    if isinstance(node, ast.Constant):
+        if not isinstance(node.value, (int, float, str, bool, type(None))):
+            raise ExpressionError("unsupported literal")
+    elif isinstance(node, (ast.Name, ast.Attribute)):
+        if _path(node) is None or any(part.startswith("_") for part in _path(node).split(".")):
+            raise ExpressionError("private or dynamic field access is forbidden")
+    elif isinstance(node, ast.BinOp) and type(node.op) in _BINARY:
+        _check(node.left)
+        _check(node.right)
+    elif isinstance(node, ast.UnaryOp) and isinstance(node.op, (ast.USub, ast.UAdd, ast.Not)):
+        _check(node.operand)
+    elif isinstance(node, ast.BoolOp) and isinstance(node.op, (ast.And, ast.Or)):
+        for value in node.values:
+            _check(value)
+    elif isinstance(node, ast.Compare) and all(type(op) in _COMPARE for op in node.ops):
+        _check(node.left)
+        for item in node.comparators:
+            _check(item)
+    elif isinstance(node, ast.Call) and isinstance(node.func, ast.Name):
+        if node.func.id not in _FUNCTIONS or node.keywords:
+            raise ExpressionError("function is not allowed")
+        for item in node.args:
+            _check(item)
+    else:
+        raise ExpressionError(f"unsupported expression element: {type(node).__name__}")
+
+
+def field(context: dict, path: str) -> Any:
+    if path in context:
+        return context[path]
+    parts = path.split(".")
+    for start in range(len(parts), 0, -1):
+        prefix = ".".join(parts[:start])
+        if prefix in context:
+            value = context[prefix]
+            for part in parts[start:]:
+                if not isinstance(value, dict):
+                    return None
+                value = value.get(part)
+            return value
+    return None
+
+
+def evaluate(source: str, context: dict) -> Any:
+    tree = compile_expression(source)
+
+    def visit(node: ast.AST) -> Any:
+        if isinstance(node, ast.Constant):
+            return node.value
+        if isinstance(node, (ast.Name, ast.Attribute)):
+            return field(context, _path(node))
+        if isinstance(node, ast.BinOp):
+            left, right = visit(node.left), visit(node.right)
+            if left is None or right is None:
+                return None
+            if not isinstance(left, (int, float)) or not isinstance(right, (int, float)):
+                return None
+            try:
+                value = _BINARY[type(node.op)](left, right)
+            except (ArithmeticError, OverflowError, TypeError, ValueError):
+                return None
+            return value if not isinstance(value, float) or math.isfinite(value) else None
+        if isinstance(node, ast.UnaryOp):
+            value = visit(node.operand)
+            if isinstance(node.op, ast.Not):
+                return not bool(value)
+            return None if value is None else (-value if isinstance(node.op, ast.USub) else +value)
+        if isinstance(node, ast.BoolOp):
+            if isinstance(node.op, ast.And):
+                return all(bool(visit(item)) for item in node.values)
+            return any(bool(visit(item)) for item in node.values)
+        if isinstance(node, ast.Compare):
+            left = visit(node.left)
+            for op, right_node in zip(node.ops, node.comparators, strict=True):
+                right = visit(right_node)
+                if left is None or right is None:
+                    if not isinstance(op, (ast.Is, ast.IsNot, ast.Eq, ast.NotEq)):
+                        return False
+                try:
+                    if not _COMPARE[type(op)](left, right):
+                        return False
+                except TypeError:
+                    return False
+                left = right
+            return True
+        if isinstance(node, ast.Call):
+            args = [visit(item) for item in node.args]
+            if any(item is None for item in args):
+                return None
+            try:
+                value = _FUNCTIONS[node.func.id](*args)
+            except (ArithmeticError, OverflowError, TypeError, ValueError):
+                return None
+            return value if not isinstance(value, float) or math.isfinite(value) else None
+        raise AssertionError("expression was not checked")
+
+    return visit(tree.body)