benchcraft-core 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- benchcraft_core-0.1.0.dist-info/METADATA +203 -0
- benchcraft_core-0.1.0.dist-info/RECORD +11 -0
- benchcraft_core-0.1.0.dist-info/WHEEL +4 -0
- lazycore/__init__.py +100 -0
- lazycore/data.py +285 -0
- lazycore/licensing.py +288 -0
- lazycore/sandbox/__init__.py +92 -0
- lazycore/sandbox/base.py +356 -0
- lazycore/sandbox/linux_stub.py +140 -0
- lazycore/sandbox/seatbelt.py +1315 -0
- lazycore/telemetry.py +310 -0
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: benchcraft-core
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Benchcraft Core: the thin, shared substrate used by all Benchcraft modules (data-tier conventions, OTel GenAI telemetry helpers, license-isolation policy).
|
|
5
|
+
Author: Benchcraft
|
|
6
|
+
License: MIT
|
|
7
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
8
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
9
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
10
|
+
Requires-Python: >=3.11
|
|
11
|
+
Requires-Dist: opentelemetry-api>=1.20.0
|
|
12
|
+
Provides-Extra: dev
|
|
13
|
+
Requires-Dist: pandas>=2.0; extra == 'dev'
|
|
14
|
+
Requires-Dist: polars>=0.20; extra == 'dev'
|
|
15
|
+
Requires-Dist: pyarrow>=14.0; extra == 'dev'
|
|
16
|
+
Requires-Dist: pytest>=7.0; extra == 'dev'
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
|
|
19
|
+
# lazycore
|
|
20
|
+
|
|
21
|
+
The thin, shared substrate underneath every Benchcraft module (AutoML,
|
|
22
|
+
LazyClean, LazyForecast, LazyGraph, LazyVision, LazyTune, LazyRed,
|
|
23
|
+
LazyAgent). It exists so that modules working on largely the same
|
|
24
|
+
underlying data, telemetry, and licensing concerns don't each invent their
|
|
25
|
+
own conventions -- without forcing premature unification of things that
|
|
26
|
+
genuinely differ per module.
|
|
27
|
+
|
|
28
|
+
## This package is intentionally thin. Here's why.
|
|
29
|
+
|
|
30
|
+
Benchcraft's packaging model (architecture doc §2.7) is Hugging Face-style:
|
|
31
|
+
independently-versioned, separate packages per module, sharing one thin
|
|
32
|
+
`lazycore` package for common schemas/interfaces with near-zero
|
|
33
|
+
dependencies -- explicitly **not** one monorepo with pip extras. The reason
|
|
34
|
+
is concrete, not aesthetic: PyTorch-heavy modules (LazyTune, LazyVision,
|
|
35
|
+
LazyGraph), the deliberately PyTorch-free LazyClean (which stays under a
|
|
36
|
+
100MB footprint by using ONNX Runtime instead of PyTorch), and the
|
|
37
|
+
Node-adjacent tooling implied by LazyRed's Promptfoo integration have
|
|
38
|
+
genuinely conflicting dependency universes that pip's extras resolver
|
|
39
|
+
cannot cleanly reconcile. If `lazycore` pulled in pandas, polars, or torch
|
|
40
|
+
as hard dependencies, every module -- including the ones explicitly
|
|
41
|
+
designed to avoid those dependencies -- would be forced to install them
|
|
42
|
+
just to get `import lazycore` to work. That defeats the entire point of
|
|
43
|
+
per-module packaging.
|
|
44
|
+
|
|
45
|
+
The architecture doc is equally explicit that formal, typed interface
|
|
46
|
+
contracts between modules (in the style of MLflow's model "flavors") are
|
|
47
|
+
**deferred** until at least two real modules actually need to exchange
|
|
48
|
+
data (§2.9) -- so `lazycore` does not attempt to be a data-contract system.
|
|
49
|
+
What it provides instead are lightweight, checked-in *conventions*:
|
|
50
|
+
type-hint-only interfaces, small helper functions, policy tables encoded as
|
|
51
|
+
data. Nothing here assumes what shape a future formal contract should take.
|
|
52
|
+
|
|
53
|
+
Concretely, `lazycore`'s runtime dependency footprint is exactly one
|
|
54
|
+
non-stdlib package: `opentelemetry-api` (never the SDK). It does not depend
|
|
55
|
+
on pandas, polars, torch, or any ML framework -- where those are referenced
|
|
56
|
+
for type hints, the imports are guarded behind `typing.TYPE_CHECKING` or
|
|
57
|
+
done lazily inside a function body, so importing `lazycore` never forces
|
|
58
|
+
any of them to be installed.
|
|
59
|
+
|
|
60
|
+
## What's in here
|
|
61
|
+
|
|
62
|
+
Everything in this package maps directly to a locked decision in Part 2 of
|
|
63
|
+
`Benchcraft_Unified_Architecture.md`. Nothing module-specific belongs here
|
|
64
|
+
(see the repo's `CLAUDE.md`, "lazycore stays thin").
|
|
65
|
+
|
|
66
|
+
### `lazycore.data` -- three-tier data/tensor conventions (§2.1)
|
|
67
|
+
|
|
68
|
+
- **Tier 1 (dense tabular/text/time-series):** small conversion/validation
|
|
69
|
+
helpers over Arrow-backed pandas 2.x (`ArrowDtype`) and Polars, since both
|
|
70
|
+
are confirmed near-zero-cost, interchangeable front-ends over the same
|
|
71
|
+
Arrow buffers. `is_arrow_backed_pandas`, `pandas_arrow_dtypes`,
|
|
72
|
+
`to_polars_zero_copy`, `from_polars_zero_copy`.
|
|
73
|
+
- **Tier 2 (sparse graph tensors):** `SparseGraphTensorAdapter`, an abstract
|
|
74
|
+
base class describing the COO/CSR-CSC conversion boundary DLPack cannot
|
|
75
|
+
represent. No graph-library dependency; LazyGraph provides concrete PyG-
|
|
76
|
+
and DGL-facing implementations.
|
|
77
|
+
- **Tier 3 (dense image/audio):** `DenseMediaPipeline`, an abstract base
|
|
78
|
+
class describing an FFCV-style decode → augment → DLPack-handoff
|
|
79
|
+
pipeline shape (decode/augment are compute-bound, not copy-bound, so
|
|
80
|
+
DLPack only applies at the final dense-tensor stage). LazyVision provides
|
|
81
|
+
concrete implementations.
|
|
82
|
+
|
|
83
|
+
### `lazycore.telemetry` -- OpenTelemetry GenAI semantic conventions (§2.6)
|
|
84
|
+
|
|
85
|
+
Shared attribute-name constants (`security.severity`, `owasp.mapping`,
|
|
86
|
+
`ml.metric.*`) and a thin wrapper (`genai_span`, `set_security_finding`,
|
|
87
|
+
`set_ml_metric`, `add_transcript_event`) over `opentelemetry-api` so
|
|
88
|
+
LazyRed's security-audit reports, LazyAgent's execution trajectories, and
|
|
89
|
+
the ML leaderboards from AutoML/LazyForecast/LazyGraph/LazyVision all
|
|
90
|
+
report through the same schema. Depends only on `opentelemetry-api`; if the
|
|
91
|
+
calling application hasn't configured an SDK TracerProvider, spans are
|
|
92
|
+
OTel's documented no-ops -- safe to call, nothing exported until a real
|
|
93
|
+
exporter is wired up by whichever module needs one.
|
|
94
|
+
|
|
95
|
+
### `lazycore.licensing` -- license-isolation and model-allowlist policy (§2.2, §2.10)
|
|
96
|
+
|
|
97
|
+
- `RiskType` / `Mitigation` / `RISK_MITIGATIONS`: the §2.2 License Isolation
|
|
98
|
+
Policy decision table, encoded as data rather than prose, so module
|
|
99
|
+
owners can look up the exact required mitigation for a given dependency
|
|
100
|
+
risk (GPL-at-build-time, restrictive optional build dep,
|
|
101
|
+
AGPL/GPL-at-runtime-internal, AGPL/GPL-network-facing,
|
|
102
|
+
source-available-non-compete, non-commercial weights).
|
|
103
|
+
- `ModelTier` / `ModelLicenseEntry` / `Allowlist`: the §2.10 mechanism every
|
|
104
|
+
module uses to register and check model checkpoints against Tier 1
|
|
105
|
+
(permissive, auto-usable) or Tier 2 (restricted, opt-in-gated) --
|
|
106
|
+
including the runtime guard that raises `RestrictedLicenseNotAcceptedError`
|
|
107
|
+
for a Tier 2 checkpoint unless the caller explicitly passes
|
|
108
|
+
`accept_restricted_licenses=True`.
|
|
109
|
+
|
|
110
|
+
This is a **policy and mechanism**, not a populated list. Every `Allowlist`
|
|
111
|
+
starts empty; populating and maintaining the actual per-module allowlists
|
|
112
|
+
(which specific model checkpoints go in Tier 1 vs Tier 2) is called out in
|
|
113
|
+
the architecture doc (Part 6) as an ongoing, per-module maintenance task,
|
|
114
|
+
not something `lazycore` does on a module's behalf.
|
|
115
|
+
|
|
116
|
+
### `lazycore.sandbox` -- shared sandbox executor + adapter base class (§2.3, §2.3.1)
|
|
117
|
+
|
|
118
|
+
LazyRed and LazyAgent contain the same kernel-level threat class --
|
|
119
|
+
arbitrary code execution by a red-team target or a benchmarked agent -- so
|
|
120
|
+
per §2.3, LazyCore provides **one** shared sandbox executor and **one**
|
|
121
|
+
generic policy dataclass, with mode-specific policy *values* layered on
|
|
122
|
+
top by each module when it's built (LazyRed's "red-team target sandbox",
|
|
123
|
+
LazyAgent's "benchmark task sandbox"). Nothing module-specific (e.g. an
|
|
124
|
+
OWASP mapping, a benchmark-task allowlist) lives in this package -- only
|
|
125
|
+
the generic shape both modes share.
|
|
126
|
+
|
|
127
|
+
- `SandboxPolicy` -- generic, frozen dataclass config: `allow_network`,
|
|
128
|
+
`allowed_read_paths`, `allowed_write_paths`, `allowed_executables`,
|
|
129
|
+
plus env/timeout/cwd knobs. The exact same dataclass is meant to be
|
|
130
|
+
instantiated with different values for LazyRed vs. LazyAgent.
|
|
131
|
+
- `BaseSandboxExecutor` -- the shared ABC (`is_available`, `run_command`,
|
|
132
|
+
`run_callable`, returning a structured `SandboxResult`).
|
|
133
|
+
- `SeatbeltSandboxExecutor` -- the real, tested macOS backend. Generates an
|
|
134
|
+
SBPL (Sandbox Profile Language) profile from a `SandboxPolicy` and runs
|
|
135
|
+
the target command under `/usr/bin/sandbox-exec -f <profile> -- ...`.
|
|
136
|
+
- `LinuxNamespaceSandboxExecutor` -- a **documented stub**, not a real
|
|
137
|
+
implementation. It satisfies the same ABC and reports availability via
|
|
138
|
+
`shutil.which("bwrap"/"unshare")`, but every actual run method raises
|
|
139
|
+
`SandboxBackendUnavailableError`. This repository's reference/dev
|
|
140
|
+
environment is macOS, so a real gVisor/Firecracker/namespace-based
|
|
141
|
+
backend (the intended Linux implementation per §2.3, since Linux has no
|
|
142
|
+
VM-boundary GPU problem to design around) cannot be meaningfully built or
|
|
143
|
+
verified here. Do not treat this stub as production-ready.
|
|
144
|
+
- `get_default_executor()` -- returns `SeatbeltSandboxExecutor` on macOS
|
|
145
|
+
(when `/usr/bin/sandbox-exec` exists) or `LinuxNamespaceSandboxExecutor`
|
|
146
|
+
on Linux; raises `SandboxBackendUnavailableError` on anything else.
|
|
147
|
+
|
|
148
|
+
**Why GPU/Metal/MPS access is never sandboxed here, on either platform**
|
|
149
|
+
(§2.3.1): 2026 research confirmed no Mac container/VM runtime (Docker
|
|
150
|
+
Desktop, Podman+libkrun, Apple's `container`/Containerization framework)
|
|
151
|
+
exposes Metal/MPS passthrough into an isolation boundary, and -- more
|
|
152
|
+
fundamentally -- Seatbelt itself **cannot** gate GPU/Metal/Cocoa access
|
|
153
|
+
even in principle, because those are system-level services outside SBPL's
|
|
154
|
+
reach. The locked v1 design is a **split-trust architecture**: GPU-bound
|
|
155
|
+
model inference always runs unsandboxed (there's no adversarial surface in
|
|
156
|
+
local weights/forward-pass compute), and only the CPU-bound tool-calling/
|
|
157
|
+
code-execution layer -- shell commands, file I/O, network egress, the
|
|
158
|
+
actual adversarial surface in both LazyRed's and LazyAgent's threat models
|
|
159
|
+
-- is what this package ever constrains. `SeatbeltSandboxExecutor` does not
|
|
160
|
+
implement, and will never implement, any GPU-blocking rule; see its module
|
|
161
|
+
docstring for the exact technical reasoning.
|
|
162
|
+
|
|
163
|
+
Runtime dependencies added by this subpackage: **none** -- it uses only
|
|
164
|
+
`subprocess`, `shutil`, `tempfile`, `pickle`, `platform`, `dataclasses`,
|
|
165
|
+
and `abc` from the standard library.
|
|
166
|
+
|
|
167
|
+
## What's deliberately NOT in here
|
|
168
|
+
|
|
169
|
+
- A *real* Linux sandbox backend -- `LinuxNamespaceSandboxExecutor` is a
|
|
170
|
+
documented stub only; see `lazycore.sandbox` above.
|
|
171
|
+
- Any LLM router or `BaseTarget`/multi-provider abstraction (§2.8) --
|
|
172
|
+
explicitly and permanently excluded platform-wide.
|
|
173
|
+
- Formal inter-module data contracts / MLflow "flavors"-style manifests
|
|
174
|
+
(§2.9) -- deferred until two real modules need to exchange data.
|
|
175
|
+
- pandas, polars, torch, or any ML framework as a hard dependency.
|
|
176
|
+
|
|
177
|
+
## Installation
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
cd packages/lazycore
|
|
181
|
+
pip install -e . # runtime only
|
|
182
|
+
pip install -e ".[dev]" # + pytest for running the test suite
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
## Running tests
|
|
186
|
+
|
|
187
|
+
```bash
|
|
188
|
+
pytest packages/lazycore/tests
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
The Tier 1 data-helper tests additionally require `pandas` and `polars` to
|
|
192
|
+
be installed in the test environment (they are not runtime dependencies of
|
|
193
|
+
`lazycore` itself, so those tests are skipped via `pytest.importorskip` if
|
|
194
|
+
unavailable).
|
|
195
|
+
|
|
196
|
+
`tests/sandbox/test_seatbelt.py` actually invokes `/usr/bin/sandbox-exec`
|
|
197
|
+
against real temp directories to demonstrate enforcement (an allowed write
|
|
198
|
+
succeeds, a write outside the policy is denied, network egress is denied
|
|
199
|
+
by default) -- it is skipped entirely on non-macOS hosts via
|
|
200
|
+
`pytest.mark.skipif`, rather than mocked. `tests/sandbox/test_linux_stub.py`
|
|
201
|
+
only verifies the documented-stub behavior of `LinuxNamespaceSandboxExecutor`
|
|
202
|
+
on this non-Linux machine (it raises `SandboxBackendUnavailableError`); it
|
|
203
|
+
cannot and does not validate real Linux namespace isolation.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
lazycore/__init__.py,sha256=VbVQ5MhyG4Cq7uHFM4AArUlSSIvi4GkPvEQj6FO8_bI,2475
|
|
2
|
+
lazycore/data.py,sha256=6HqEV_ekFK4awAeEw5kF-OOt2Mph0vVRK2tP0vgO_8A,11544
|
|
3
|
+
lazycore/licensing.py,sha256=pNbpEQzUhIoFog3qexKd8a3DlQov9g2kKQVQ2KlFWpY,11804
|
|
4
|
+
lazycore/telemetry.py,sha256=FpM50x6gFXU7UZ4QJg3_B9H8N7MI3-D0vp2GZCdCjlA,13669
|
|
5
|
+
lazycore/sandbox/__init__.py,sha256=9ICIpsVD37Gk3RW9CYGpY7qiIzJVYgVCMh_Cfst42cs,3794
|
|
6
|
+
lazycore/sandbox/base.py,sha256=hKVv347wBrDnP04XDfrF5pF9Vf_wziwDHn2VEgG593w,17948
|
|
7
|
+
lazycore/sandbox/linux_stub.py,sha256=MisAM7U9OHnhsmpvdfZeriKx3_bEtyzzm9xgwsvsAc0,6735
|
|
8
|
+
lazycore/sandbox/seatbelt.py,sha256=-a51U72lUgHGlhLdvHGzWgsH7qVBXKp0n2JzsO7J9yc,66482
|
|
9
|
+
benchcraft_core-0.1.0.dist-info/METADATA,sha256=fv5mK8e-kWEEMLZdMWplQlQ1f3jaBUUFDyOV0tFgZKo,10923
|
|
10
|
+
benchcraft_core-0.1.0.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
|
|
11
|
+
benchcraft_core-0.1.0.dist-info/RECORD,,
|
lazycore/__init__.py
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
"""lazycore: the thin, shared substrate underneath every Benchcraft module.
|
|
2
|
+
|
|
3
|
+
See the package README for scope and the "why thin" rationale. This module
|
|
4
|
+
re-exports the small public surface of ``lazycore.data``,
|
|
5
|
+
``lazycore.telemetry``, ``lazycore.licensing``, and ``lazycore.sandbox`` so
|
|
6
|
+
most consumers only need ``import lazycore``.
|
|
7
|
+
"""
|
|
8
|
+
|
|
9
|
+
from lazycore.data import (
|
|
10
|
+
ArrowBackedFrame,
|
|
11
|
+
DenseMediaPipeline,
|
|
12
|
+
SparseFormat,
|
|
13
|
+
SparseGraphTensorAdapter,
|
|
14
|
+
from_polars_zero_copy,
|
|
15
|
+
is_arrow_backed_pandas,
|
|
16
|
+
pandas_arrow_dtypes,
|
|
17
|
+
to_polars_zero_copy,
|
|
18
|
+
)
|
|
19
|
+
from lazycore.licensing import (
|
|
20
|
+
Allowlist,
|
|
21
|
+
Mitigation,
|
|
22
|
+
ModelLicenseEntry,
|
|
23
|
+
ModelTier,
|
|
24
|
+
RestrictedLicenseNotAcceptedError,
|
|
25
|
+
RISK_MITIGATIONS,
|
|
26
|
+
RiskType,
|
|
27
|
+
)
|
|
28
|
+
from lazycore.sandbox import (
|
|
29
|
+
BaseSandboxExecutor,
|
|
30
|
+
LinuxNamespaceSandboxExecutor,
|
|
31
|
+
SandboxBackendUnavailableError,
|
|
32
|
+
SandboxError,
|
|
33
|
+
SandboxPolicy,
|
|
34
|
+
SandboxPolicyViolationError,
|
|
35
|
+
SandboxResult,
|
|
36
|
+
SeatbeltSandboxExecutor,
|
|
37
|
+
get_default_executor,
|
|
38
|
+
)
|
|
39
|
+
from lazycore.telemetry import (
|
|
40
|
+
ATTR_GENAI_EVENT_CONTENT,
|
|
41
|
+
ATTR_GENAI_EVENT_ROLE,
|
|
42
|
+
ATTR_ML_METRIC_ACCURACY,
|
|
43
|
+
ATTR_ML_METRIC_PREFIX,
|
|
44
|
+
ATTR_OWASP_MAPPING,
|
|
45
|
+
ATTR_SECURITY_SEVERITY,
|
|
46
|
+
SecuritySeverity,
|
|
47
|
+
add_transcript_event,
|
|
48
|
+
genai_span,
|
|
49
|
+
get_tracer,
|
|
50
|
+
ml_metric_attribute,
|
|
51
|
+
set_ml_metric,
|
|
52
|
+
set_security_finding,
|
|
53
|
+
)
|
|
54
|
+
|
|
55
|
+
__version__ = "0.1.0"
|
|
56
|
+
|
|
57
|
+
__all__ = [
|
|
58
|
+
"__version__",
|
|
59
|
+
# data.py
|
|
60
|
+
"ArrowBackedFrame",
|
|
61
|
+
"DenseMediaPipeline",
|
|
62
|
+
"SparseFormat",
|
|
63
|
+
"SparseGraphTensorAdapter",
|
|
64
|
+
"from_polars_zero_copy",
|
|
65
|
+
"is_arrow_backed_pandas",
|
|
66
|
+
"pandas_arrow_dtypes",
|
|
67
|
+
"to_polars_zero_copy",
|
|
68
|
+
# licensing.py
|
|
69
|
+
"Allowlist",
|
|
70
|
+
"Mitigation",
|
|
71
|
+
"ModelLicenseEntry",
|
|
72
|
+
"ModelTier",
|
|
73
|
+
"RestrictedLicenseNotAcceptedError",
|
|
74
|
+
"RISK_MITIGATIONS",
|
|
75
|
+
"RiskType",
|
|
76
|
+
# sandbox/__init__.py
|
|
77
|
+
"BaseSandboxExecutor",
|
|
78
|
+
"LinuxNamespaceSandboxExecutor",
|
|
79
|
+
"SandboxBackendUnavailableError",
|
|
80
|
+
"SandboxError",
|
|
81
|
+
"SandboxPolicy",
|
|
82
|
+
"SandboxPolicyViolationError",
|
|
83
|
+
"SandboxResult",
|
|
84
|
+
"SeatbeltSandboxExecutor",
|
|
85
|
+
"get_default_executor",
|
|
86
|
+
# telemetry.py
|
|
87
|
+
"ATTR_GENAI_EVENT_CONTENT",
|
|
88
|
+
"ATTR_GENAI_EVENT_ROLE",
|
|
89
|
+
"ATTR_ML_METRIC_ACCURACY",
|
|
90
|
+
"ATTR_ML_METRIC_PREFIX",
|
|
91
|
+
"ATTR_OWASP_MAPPING",
|
|
92
|
+
"ATTR_SECURITY_SEVERITY",
|
|
93
|
+
"SecuritySeverity",
|
|
94
|
+
"add_transcript_event",
|
|
95
|
+
"genai_span",
|
|
96
|
+
"get_tracer",
|
|
97
|
+
"ml_metric_attribute",
|
|
98
|
+
"set_ml_metric",
|
|
99
|
+
"set_security_finding",
|
|
100
|
+
]
|
lazycore/data.py
ADDED
|
@@ -0,0 +1,285 @@
|
|
|
1
|
+
"""Shared three-tier data/tensor conventions (architecture doc §2.1).
|
|
2
|
+
|
|
3
|
+
LazyCore does not ship a data-processing library. It documents and lightly
|
|
4
|
+
implements the three-tier convention that every Benchcraft module agrees to
|
|
5
|
+
follow so that modules sharing "the same underlying data" don't each invent
|
|
6
|
+
their own representation:
|
|
7
|
+
|
|
8
|
+
Tier 1 - Dense tabular / text / time-series
|
|
9
|
+
Canonical format: Apache Arrow, fronted by pandas 2.x ``ArrowDtype``
|
|
10
|
+
columns and/or Polars DataFrames. pandas 2.x and Polars are both
|
|
11
|
+
zero-copy front-ends over the same Arrow buffers, so this tier is a set
|
|
12
|
+
of small conversion/validation helpers, not a wrapping library. Neither
|
|
13
|
+
pandas nor polars is a hard dependency of ``lazycore`` -- consumers
|
|
14
|
+
import them themselves, and these helpers import them lazily so that a
|
|
15
|
+
PyTorch-free or otherwise minimal module never pays for that import
|
|
16
|
+
unless it actually calls into this tier.
|
|
17
|
+
|
|
18
|
+
Tier 2 - Sparse graph tensors
|
|
19
|
+
DLPack's spec is explicitly limited to dense/strided arrays and cannot
|
|
20
|
+
represent sparsity, so there is no zero-copy shortcut here -- a real
|
|
21
|
+
adapter is required. LazyCore only defines the *shape* of that adapter
|
|
22
|
+
(an abstract interface for COO / CSR-CSC exchange) via
|
|
23
|
+
:class:`SparseGraphTensorAdapter`. Concrete implementations bridging
|
|
24
|
+
PyTorch Geometric's native COO and DGL's native CSR/CSC formats belong
|
|
25
|
+
in LazyGraph, not here -- lazycore never depends on torch, PyG, or DGL.
|
|
26
|
+
|
|
27
|
+
Tier 3 - Dense image / audio
|
|
28
|
+
FFCV-style workloads are bottlenecked by decode+augmentation *compute*,
|
|
29
|
+
not by data-copy/serialization overhead, so the shared convention is a
|
|
30
|
+
pipeline shape (decode -> augment -> DLPack handoff only at the final
|
|
31
|
+
dense-tensor stage), defined here as :class:`DenseMediaPipeline`.
|
|
32
|
+
Concrete decode/augment implementations belong in LazyVision.
|
|
33
|
+
|
|
34
|
+
Nothing in this module imports pandas, polars, torch, or any other heavy
|
|
35
|
+
runtime dependency at module import time. Where such libraries are needed
|
|
36
|
+
for type-checking only, imports are guarded behind ``TYPE_CHECKING``.
|
|
37
|
+
"""
|
|
38
|
+
|
|
39
|
+
from __future__ import annotations
|
|
40
|
+
|
|
41
|
+
import abc
|
|
42
|
+
from typing import TYPE_CHECKING, Any, Iterable, Protocol, runtime_checkable
|
|
43
|
+
|
|
44
|
+
if TYPE_CHECKING: # pragma: no cover - type-checking-only imports
|
|
45
|
+
import pandas as pd
|
|
46
|
+
import polars as pl
|
|
47
|
+
|
|
48
|
+
__all__ = [
|
|
49
|
+
"ArrowBackedFrame",
|
|
50
|
+
"is_arrow_backed_pandas",
|
|
51
|
+
"pandas_arrow_dtypes",
|
|
52
|
+
"to_polars_zero_copy",
|
|
53
|
+
"from_polars_zero_copy",
|
|
54
|
+
"SparseGraphTensorAdapter",
|
|
55
|
+
"SparseFormat",
|
|
56
|
+
"DenseMediaPipeline",
|
|
57
|
+
]
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
# ---------------------------------------------------------------------------
|
|
61
|
+
# Tier 1: Arrow-backed dense tabular / text / time-series helpers
|
|
62
|
+
# ---------------------------------------------------------------------------
|
|
63
|
+
|
|
64
|
+
#: Duck-typed alias for "something Arrow-shaped": a pandas DataFrame backed
|
|
65
|
+
#: by ArrowDtype columns, or a Polars DataFrame. Not a runtime-enforced type
|
|
66
|
+
#: (that would require importing pandas/polars eagerly); it exists purely
|
|
67
|
+
#: for readability in module type hints via ``TYPE_CHECKING``.
|
|
68
|
+
if TYPE_CHECKING: # pragma: no cover
|
|
69
|
+
ArrowBackedFrame = "pd.DataFrame | pl.DataFrame"
|
|
70
|
+
else:
|
|
71
|
+
ArrowBackedFrame = Any
|
|
72
|
+
|
|
73
|
+
|
|
74
|
+
def _require_pandas() -> "pd": # pragma: no cover - trivial import guard
|
|
75
|
+
try:
|
|
76
|
+
import pandas as pd
|
|
77
|
+
except ImportError as exc: # pragma: no cover
|
|
78
|
+
raise ImportError(
|
|
79
|
+
"This lazycore.data helper requires pandas to be installed in "
|
|
80
|
+
"the calling module's environment. lazycore itself does not "
|
|
81
|
+
"depend on pandas."
|
|
82
|
+
) from exc
|
|
83
|
+
return pd
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
def _require_polars() -> "pl": # pragma: no cover - trivial import guard
|
|
87
|
+
try:
|
|
88
|
+
import polars as pl
|
|
89
|
+
except ImportError as exc: # pragma: no cover
|
|
90
|
+
raise ImportError(
|
|
91
|
+
"This lazycore.data helper requires polars to be installed in "
|
|
92
|
+
"the calling module's environment. lazycore itself does not "
|
|
93
|
+
"depend on polars."
|
|
94
|
+
) from exc
|
|
95
|
+
return pl
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
def pandas_arrow_dtypes(frame: "pd.DataFrame") -> dict[str, str]:
|
|
99
|
+
"""Return the subset of ``frame``'s columns backed by ``ArrowDtype``.
|
|
100
|
+
|
|
101
|
+
Maps column name -> the string form of its pyarrow type. Useful for a
|
|
102
|
+
module to quickly check which columns are already on the Tier-1
|
|
103
|
+
zero-copy path versus which are still plain numpy-backed pandas columns
|
|
104
|
+
that would force a conversion.
|
|
105
|
+
"""
|
|
106
|
+
pd = _require_pandas()
|
|
107
|
+
result: dict[str, str] = {}
|
|
108
|
+
for name, dtype in frame.dtypes.items():
|
|
109
|
+
if isinstance(dtype, pd.ArrowDtype):
|
|
110
|
+
result[str(name)] = str(dtype.pyarrow_dtype)
|
|
111
|
+
return result
|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
def is_arrow_backed_pandas(frame: "pd.DataFrame") -> bool:
|
|
115
|
+
"""True if every column in ``frame`` uses a pandas 2.x ``ArrowDtype``.
|
|
116
|
+
|
|
117
|
+
An empty DataFrame (no columns) is considered trivially Arrow-backed.
|
|
118
|
+
"""
|
|
119
|
+
pd = _require_pandas()
|
|
120
|
+
dtypes = list(frame.dtypes)
|
|
121
|
+
if not dtypes:
|
|
122
|
+
return True
|
|
123
|
+
return all(isinstance(dtype, pd.ArrowDtype) for dtype in dtypes)
|
|
124
|
+
|
|
125
|
+
|
|
126
|
+
def to_polars_zero_copy(frame: "pd.DataFrame") -> "pl.DataFrame":
|
|
127
|
+
"""Convert an Arrow-backed pandas DataFrame to Polars.
|
|
128
|
+
|
|
129
|
+
Per §2.1, pandas 2.x (``ArrowDtype``) and Polars are interchangeable
|
|
130
|
+
front-ends over the same Arrow buffers, so this conversion is
|
|
131
|
+
near-zero-cost when ``frame`` is already Arrow-backed. If it is not,
|
|
132
|
+
this still works correctly but pandas/polars will do the necessary
|
|
133
|
+
materialization themselves -- this helper does not silently hide that
|
|
134
|
+
cost, it just delegates to ``pl.from_pandas`` either way.
|
|
135
|
+
"""
|
|
136
|
+
pl = _require_polars()
|
|
137
|
+
return pl.from_pandas(frame)
|
|
138
|
+
|
|
139
|
+
|
|
140
|
+
def from_polars_zero_copy(frame: "pl.DataFrame") -> "pd.DataFrame":
|
|
141
|
+
"""Convert a Polars DataFrame to an Arrow-backed pandas DataFrame.
|
|
142
|
+
|
|
143
|
+
Uses pandas 2.x's ``dtype_backend="pyarrow"`` so the resulting frame
|
|
144
|
+
stays on the Tier-1 Arrow representation rather than falling back to
|
|
145
|
+
numpy-backed columns.
|
|
146
|
+
"""
|
|
147
|
+
_require_pandas() # ensure a clear error if pandas is missing at all
|
|
148
|
+
return frame.to_pandas(use_pyarrow_extension_array=True)
|
|
149
|
+
|
|
150
|
+
|
|
151
|
+
# ---------------------------------------------------------------------------
|
|
152
|
+
# Tier 2: Sparse graph tensor adapter (interface only)
|
|
153
|
+
# ---------------------------------------------------------------------------
|
|
154
|
+
|
|
155
|
+
|
|
156
|
+
class SparseFormat:
|
|
157
|
+
"""Sparse tensor storage format names used across the Tier-2 adapter.
|
|
158
|
+
|
|
159
|
+
These correspond to the "COO / CSR-CSC / SciPy sparse bridge" family
|
|
160
|
+
named in §2.1 (modeled on NVIDIA's "Universal Sparse Tensor" concept).
|
|
161
|
+
"""
|
|
162
|
+
|
|
163
|
+
COO = "coo"
|
|
164
|
+
CSR = "csr"
|
|
165
|
+
CSC = "csc"
|
|
166
|
+
|
|
167
|
+
|
|
168
|
+
class SparseGraphTensorAdapter(abc.ABC):
|
|
169
|
+
"""Abstract interface for a Tier-2 sparse graph tensor adapter.
|
|
170
|
+
|
|
171
|
+
DLPack cannot represent sparsity, so there is no zero-copy shortcut for
|
|
172
|
+
graph tensors -- a real conversion step between sparse formats (COO,
|
|
173
|
+
CSR/CSC) is unavoidable. LazyCore defines only the *shape* of that
|
|
174
|
+
conversion boundary here; it does not implement it and does not depend
|
|
175
|
+
on any graph library (PyTorch Geometric, DGL, SciPy, etc.). LazyGraph
|
|
176
|
+
is expected to provide concrete subclasses that actually bridge PyG's
|
|
177
|
+
native COO representation and DGL's native CSR/CSC representation.
|
|
178
|
+
|
|
179
|
+
Implementations should treat every method below as doing real work
|
|
180
|
+
(not just re-labelling metadata) since a genuine conversion step is
|
|
181
|
+
happening.
|
|
182
|
+
"""
|
|
183
|
+
|
|
184
|
+
@property
|
|
185
|
+
@abc.abstractmethod
|
|
186
|
+
def native_format(self) -> str:
|
|
187
|
+
"""The sparse format (one of :class:`SparseFormat`) this adapter
|
|
188
|
+
instance currently holds its data in."""
|
|
189
|
+
raise NotImplementedError
|
|
190
|
+
|
|
191
|
+
@property
|
|
192
|
+
@abc.abstractmethod
|
|
193
|
+
def shape(self) -> tuple[int, int]:
|
|
194
|
+
"""The dense (rows, cols) shape represented by this sparse tensor."""
|
|
195
|
+
raise NotImplementedError
|
|
196
|
+
|
|
197
|
+
@abc.abstractmethod
|
|
198
|
+
def to_coo(self) -> "SparseGraphTensorAdapter":
|
|
199
|
+
"""Return an adapter view/copy in COO format (PyG-native)."""
|
|
200
|
+
raise NotImplementedError
|
|
201
|
+
|
|
202
|
+
@abc.abstractmethod
|
|
203
|
+
def to_csr(self) -> "SparseGraphTensorAdapter":
|
|
204
|
+
"""Return an adapter view/copy in CSR format (DGL-native for many
|
|
205
|
+
aggregation ops)."""
|
|
206
|
+
raise NotImplementedError
|
|
207
|
+
|
|
208
|
+
@abc.abstractmethod
|
|
209
|
+
def to_csc(self) -> "SparseGraphTensorAdapter":
|
|
210
|
+
"""Return an adapter view/copy in CSC format."""
|
|
211
|
+
raise NotImplementedError
|
|
212
|
+
|
|
213
|
+
|
|
214
|
+
# ---------------------------------------------------------------------------
|
|
215
|
+
# Tier 3: Dense image / audio pipeline (interface only)
|
|
216
|
+
# ---------------------------------------------------------------------------
|
|
217
|
+
|
|
218
|
+
|
|
219
|
+
@runtime_checkable
|
|
220
|
+
class _SupportsDLPack(Protocol):
|
|
221
|
+
"""Structural type for "something that can hand off via DLPack" --
|
|
222
|
+
i.e. exposes ``__dlpack__``/``__dlpack_device__`` per the DLPack
|
|
223
|
+
protocol, without lazycore needing to depend on torch/numpy to say so.
|
|
224
|
+
"""
|
|
225
|
+
|
|
226
|
+
def __dlpack__(self, stream: Any | None = ...) -> Any:
|
|
227
|
+
"""Export this tensor via the DLPack protocol for zero-copy handoff."""
|
|
228
|
+
... # pragma: no cover
|
|
229
|
+
|
|
230
|
+
def __dlpack_device__(self) -> tuple[int, int]:
|
|
231
|
+
"""Return the ``(device_type, device_id)`` pair per the DLPack spec."""
|
|
232
|
+
... # pragma: no cover
|
|
233
|
+
|
|
234
|
+
|
|
235
|
+
class DenseMediaPipeline(abc.ABC):
|
|
236
|
+
"""Abstract interface for a Tier-3 FFCV-style decode+augment pipeline.
|
|
237
|
+
|
|
238
|
+
Image/audio workloads are bottlenecked by decode+augmentation
|
|
239
|
+
*compute*, not by data-copy/serialization overhead (this is why FFCV
|
|
240
|
+
outperforms both the PyTorch DataLoader and NVIDIA DALI). The shared
|
|
241
|
+
convention is therefore about pipeline *shape*, not about a shared
|
|
242
|
+
zero-copy buffer: decode and augment happen natively (however the
|
|
243
|
+
concrete implementation wants), and only the final, already-dense
|
|
244
|
+
tensor is handed off via DLPack.
|
|
245
|
+
|
|
246
|
+
LazyCore does not implement decode/augment logic and does not depend on
|
|
247
|
+
any image/audio/tensor library. LazyVision is expected to provide
|
|
248
|
+
concrete subclasses.
|
|
249
|
+
"""
|
|
250
|
+
|
|
251
|
+
@abc.abstractmethod
|
|
252
|
+
def decode(self, raw: bytes) -> Any:
|
|
253
|
+
"""Decode a raw media payload (e.g. JPEG/PNG/WAV bytes) into an
|
|
254
|
+
intermediate in-memory representation. Not required to be a dense
|
|
255
|
+
tensor yet -- this stage is decode-compute-bound, and
|
|
256
|
+
implementations are free to use whatever intermediate
|
|
257
|
+
representation is fastest for their decoder."""
|
|
258
|
+
raise NotImplementedError
|
|
259
|
+
|
|
260
|
+
@abc.abstractmethod
|
|
261
|
+
def augment(self, decoded: Any) -> Any:
|
|
262
|
+
"""Apply augmentation to a decoded sample. Still not required to be
|
|
263
|
+
a DLPack-compatible dense tensor -- augmentation is also
|
|
264
|
+
compute-bound, not copy-bound."""
|
|
265
|
+
raise NotImplementedError
|
|
266
|
+
|
|
267
|
+
@abc.abstractmethod
|
|
268
|
+
def to_dense_tensor(self, augmented: Any) -> _SupportsDLPack:
|
|
269
|
+
"""Produce the final dense tensor for this sample. The return value
|
|
270
|
+
must support the DLPack protocol (``__dlpack__`` /
|
|
271
|
+
``__dlpack_device__``) -- this is the one and only point in the
|
|
272
|
+
Tier-3 pipeline where a zero-copy handoff (e.g. to PyTorch) is
|
|
273
|
+
expected to occur."""
|
|
274
|
+
raise NotImplementedError
|
|
275
|
+
|
|
276
|
+
def run(self, raw: bytes) -> _SupportsDLPack:
|
|
277
|
+
"""Convenience driver: decode -> augment -> to_dense_tensor.
|
|
278
|
+
|
|
279
|
+
Concrete pipelines may override this if they need to fuse stages
|
|
280
|
+
for performance, but the default composition documents the
|
|
281
|
+
expected shape of the pipeline.
|
|
282
|
+
"""
|
|
283
|
+
decoded = self.decode(raw)
|
|
284
|
+
augmented = self.augment(decoded)
|
|
285
|
+
return self.to_dense_tensor(augmented)
|