mlflow-at1 0.1.0__tar.gz

mlflow_at1-0.1.0/PKG-INFO

@@ -0,0 +1,64 @@
+Metadata-Version: 2.4
+Name: mlflow-at1
+Version: 0.1.0
+Summary: MLflow artifact repository backed by verified, compressed, byte-exact AT-1 containers
+Author: TinyFiles / AT-1
+License: Apache-2.0
+Project-URL: Homepage, https://tinyfiles.io
+Project-URL: Documentation, https://tinyfiles.io/docs/mlflow
+Keywords: mlflow,artifacts,compression,provenance,verified,lossless
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: mlflow-skinny>=2.0
+Provides-Extra: full
+Requires-Dist: mlflow>=2.0; extra == "full"
+
+# mlflow-at1 — verified, compressed MLflow artifacts
+
+Store MLflow models, checkpoints, and datasets as **verified, byte-exact AT-1 containers** instead of raw
+files. Every artifact is compressed losslessly (never-worse vs xz), carries an embedded SHA-256, and is
+**byte-exact-verified on download** — a tampered or corrupted artifact is *refused*, not silently returned.
+
+That makes an artifact in your MLflow store **provably the exact bytes you logged** — the reproducibility /
+provenance hook that regulated AI (EU AI Act, NIST AI RMF, FDA 21 CFR Part 11) needs: "this model is the
+exact model that was trained, untampered."
+
+It's a **pure-glue plugin** — no engine code ships here. It registers the `at1://` artifact-repository
+scheme and shells out to the `at1` CLI.
+
+## Install
+```bash
+npm i -g @tinyfiles/cli          # the at1 binary (the codec; no Python engine needed)
+pip install mlflow-at1           # this plugin (registers the at1:// scheme)
+#   pip install mlflow-at1[full] # if you want full MLflow rather than mlflow-skinny
+```
+
+## Use
+Point an experiment's **artifact location** at an `at1://` path; everything else is normal MLflow:
+```python
+import mlflow
+exp = mlflow.create_experiment("regulated-model", artifact_location="at1:///data/mlartifacts")
+with mlflow.start_run(experiment_id=exp):
+    mlflow.log_artifact("model.bin")      # -> stored as a verified, compressed AT-1 container
+    # mlflow.sklearn.log_model(...) etc. all route through AT-1 too
+
+# later — download is a byte-exact, SHA-256-verified reconstruction:
+path = mlflow.artifacts.download_artifacts(run_id=run_id, artifact_path="model.bin")
+```
+
+`at1://<path>` is a local/mounted filesystem base directory for the containers (S3/GCS backing is a
+follow-on). Set `AT1_BIN` to point at a specific `at1` binary if it isn't on `PATH`.
+
+## What you get
+- **Smaller** — lossless, never-worse than xz; typically a real reduction on weights/checkpoints/datasets.
+- **Verified** — every container embeds a SHA-256; `download` refuses anything that doesn't reconstruct
+  byte-for-byte. Explicit proof per artifact: `repo.integrity("model.bin")` → `integrity OK (sha256 …)`.
+- **Addressable** — one `.at1` per artifact; tabular artifacts stay queryable in place via the `at1` CLI.
+- **Zero lock-in** — the container is an ordinary file with a tiny open decoder; you can always get the
+  exact original back, with or without this plugin.
+
+## Test
+```bash
+pip install mlflow-skinny pytest && python -m pytest tests/ -q   # needs the `at1` CLI on PATH
+```
+5 tests: log→download byte-exact, logical listing, directory trees, integrity + tamper-rejection, delete.

mlflow_at1-0.1.0/README.md

@@ -0,0 +1,49 @@
+# mlflow-at1 — verified, compressed MLflow artifacts
+
+Store MLflow models, checkpoints, and datasets as **verified, byte-exact AT-1 containers** instead of raw
+files. Every artifact is compressed losslessly (never-worse vs xz), carries an embedded SHA-256, and is
+**byte-exact-verified on download** — a tampered or corrupted artifact is *refused*, not silently returned.
+
+That makes an artifact in your MLflow store **provably the exact bytes you logged** — the reproducibility /
+provenance hook that regulated AI (EU AI Act, NIST AI RMF, FDA 21 CFR Part 11) needs: "this model is the
+exact model that was trained, untampered."
+
+It's a **pure-glue plugin** — no engine code ships here. It registers the `at1://` artifact-repository
+scheme and shells out to the `at1` CLI.
+
+## Install
+```bash
+npm i -g @tinyfiles/cli          # the at1 binary (the codec; no Python engine needed)
+pip install mlflow-at1           # this plugin (registers the at1:// scheme)
+#   pip install mlflow-at1[full] # if you want full MLflow rather than mlflow-skinny
+```
+
+## Use
+Point an experiment's **artifact location** at an `at1://` path; everything else is normal MLflow:
+```python
+import mlflow
+exp = mlflow.create_experiment("regulated-model", artifact_location="at1:///data/mlartifacts")
+with mlflow.start_run(experiment_id=exp):
+    mlflow.log_artifact("model.bin")      # -> stored as a verified, compressed AT-1 container
+    # mlflow.sklearn.log_model(...) etc. all route through AT-1 too
+
+# later — download is a byte-exact, SHA-256-verified reconstruction:
+path = mlflow.artifacts.download_artifacts(run_id=run_id, artifact_path="model.bin")
+```
+
+`at1://<path>` is a local/mounted filesystem base directory for the containers (S3/GCS backing is a
+follow-on). Set `AT1_BIN` to point at a specific `at1` binary if it isn't on `PATH`.
+
+## What you get
+- **Smaller** — lossless, never-worse than xz; typically a real reduction on weights/checkpoints/datasets.
+- **Verified** — every container embeds a SHA-256; `download` refuses anything that doesn't reconstruct
+  byte-for-byte. Explicit proof per artifact: `repo.integrity("model.bin")` → `integrity OK (sha256 …)`.
+- **Addressable** — one `.at1` per artifact; tabular artifacts stay queryable in place via the `at1` CLI.
+- **Zero lock-in** — the container is an ordinary file with a tiny open decoder; you can always get the
+  exact original back, with or without this plugin.
+
+## Test
+```bash
+pip install mlflow-skinny pytest && python -m pytest tests/ -q   # needs the `at1` CLI on PATH
+```
+5 tests: log→download byte-exact, logical listing, directory trees, integrity + tamper-rejection, delete.

mlflow_at1-0.1.0/mlflow_at1/__init__.py

@@ -0,0 +1,8 @@
+"""mlflow-at1 — store MLflow artifacts as verified, compressed, byte-exact AT-1 containers.
+
+Registers the ``at1://`` artifact-repository scheme. See README for usage.
+"""
+from mlflow_at1.at1_artifact_repo import AT1ArtifactRepository
+
+__all__ = ["AT1ArtifactRepository"]
+__version__ = "0.1.0"

mlflow_at1-0.1.0/mlflow_at1/at1_artifact_repo.py

@@ -0,0 +1,123 @@
+"""AT-1 artifact repository for MLflow — store models, checkpoints, and datasets as VERIFIED,
+compressed, byte-exact AT-1 containers instead of raw files.
+
+Register a tracking/artifact URI with the ``at1://`` scheme and MLflow routes every ``log_artifact`` /
+model log through this repository: each file is compressed losslessly (never-worse vs xz), carries an
+embedded SHA-256, and is **byte-exact-verified on download** (a tampered or corrupted artifact is refused,
+not silently returned). So an artifact in your MLflow store is provably the exact bytes you logged — the
+provenance hook regulated AI (EU AI Act / NIST AI RMF) needs.
+
+Pure glue: it shells out to the ``at1`` CLI (``npm i -g @tinyfiles/cli``). No engine code lives here.
+"""
+import os
+import shutil
+import posixpath
+import subprocess
+
+from mlflow.store.artifact.artifact_repo import ArtifactRepository
+from mlflow.entities import FileInfo
+
+_SUFFIX = ".at1"
+_AT1_BIN = os.environ.get("AT1_BIN", "at1")
+
+
+def _resolve_bin():
+    """Return the argv prefix to invoke the at1 CLI, handling a Windows .cmd/.bat launcher (npm shim)."""
+    resolved = shutil.which(_AT1_BIN) or _AT1_BIN
+    if os.name == "nt" and resolved.lower().endswith((".cmd", ".bat")):
+        return ["cmd", "/c", resolved]
+    return [resolved]
+
+
+_BIN = _resolve_bin()
+
+
+def _run(args):
+    r = subprocess.run(_BIN + args, capture_output=True, text=True)
+    if r.returncode != 0:
+        raise RuntimeError(f"AT-1 CLI failed: {' '.join(args)}\n{(r.stderr or r.stdout).strip()}")
+    return r
+
+
+def _uri_to_base(artifact_uri):
+    """at1://<path> -> a local filesystem base directory for the .at1 containers."""
+    p = artifact_uri
+    for pre in ("at1://", "at1:"):
+        if p.startswith(pre):
+            p = p[len(pre):]
+            break
+    # Windows drive paths arrive as /C:/... -> strip the leading slash.
+    if os.name == "nt" and len(p) >= 3 and p[0] == "/" and p[2] == ":":
+        p = p[1:]
+    return os.path.normpath(p) if p else "."
+
+
+class AT1ArtifactRepository(ArtifactRepository):
+    """MLflow ArtifactRepository backed by verified AT-1 containers (one ``.at1`` per artifact)."""
+
+    def __init__(self, artifact_uri, *args, **kwargs):
+        # MLflow passes tracking_uri/registry_uri kwargs in newer versions; forward what the installed
+        # base accepts and fall back to the legacy single-arg signature otherwise.
+        try:
+            super().__init__(artifact_uri, *args, **kwargs)
+        except TypeError:
+            super().__init__(artifact_uri)
+        self._base = _uri_to_base(artifact_uri)
+
+    # ---- write paths -------------------------------------------------------
+    def log_artifact(self, local_file, artifact_path=None):
+        dest_dir = self._base if not artifact_path else os.path.join(self._base, artifact_path)
+        os.makedirs(dest_dir, exist_ok=True)
+        out = os.path.join(dest_dir, os.path.basename(local_file) + _SUFFIX)
+        # 'auto' picks the best lossless codec for the file and is provably never-worse than xz.
+        _run(["compress", "auto", local_file, out])
+
+    def log_artifacts(self, local_dir, artifact_path=None):
+        local_dir = os.path.abspath(local_dir)
+        for root, _dirs, files in os.walk(local_dir):
+            rel = os.path.relpath(root, local_dir)
+            if rel == ".":
+                sub = artifact_path
+            else:
+                sub = posixpath.join(artifact_path, rel.replace(os.sep, "/")) if artifact_path else rel.replace(os.sep, "/")
+            for f in files:
+                self.log_artifact(os.path.join(root, f), sub)
+
+    # ---- read paths --------------------------------------------------------
+    def list_artifacts(self, path=None):
+        d = self._base if not path else os.path.join(self._base, path)
+        if not os.path.isdir(d):
+            return []
+        infos = []
+        for name in sorted(os.listdir(d)):
+            full = os.path.join(d, name)
+            rel = name if not path else posixpath.join(path, name)
+            if os.path.isdir(full):
+                infos.append(FileInfo(rel, True, None))
+            elif name.endswith(_SUFFIX):
+                # present the LOGICAL artifact name (without the .at1 container suffix). file_size is the
+                # on-disk compressed size; MLflow only uses it for display, and the byte-exact original is
+                # what download reconstructs.
+                infos.append(FileInfo(rel[: -len(_SUFFIX)], False, os.path.getsize(full)))
+        return infos
+
+    def _download_file(self, remote_file_path, local_path):
+        src = os.path.join(self._base, remote_file_path) + _SUFFIX
+        parent = os.path.dirname(local_path)
+        if parent:
+            os.makedirs(parent, exist_ok=True)          # at1 decompress writes into an existing dir
+        # decompress re-checks the embedded SHA-256 and REFUSES a tampered/corrupt container, so a
+        # download is a verified, byte-exact reconstruction of exactly what was logged.
+        _run(["decompress", src, local_path])
+
+    def integrity(self, remote_file_path):
+        """Explicit integrity proof for one stored artifact (SHA-256, byte-for-byte). Returns the CLI line."""
+        src = os.path.join(self._base, remote_file_path) + _SUFFIX
+        return _run(["integrity", src]).stdout.strip()
+
+    def delete_artifacts(self, artifact_path=None):
+        d = self._base if not artifact_path else os.path.join(self._base, artifact_path)
+        if os.path.isdir(d):
+            shutil.rmtree(d, ignore_errors=True)
+        elif os.path.isfile(d + _SUFFIX):
+            os.remove(d + _SUFFIX)

mlflow_at1-0.1.0/mlflow_at1.egg-info/PKG-INFO

@@ -0,0 +1,64 @@
+Metadata-Version: 2.4
+Name: mlflow-at1
+Version: 0.1.0
+Summary: MLflow artifact repository backed by verified, compressed, byte-exact AT-1 containers
+Author: TinyFiles / AT-1
+License: Apache-2.0
+Project-URL: Homepage, https://tinyfiles.io
+Project-URL: Documentation, https://tinyfiles.io/docs/mlflow
+Keywords: mlflow,artifacts,compression,provenance,verified,lossless
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: mlflow-skinny>=2.0
+Provides-Extra: full
+Requires-Dist: mlflow>=2.0; extra == "full"
+
+# mlflow-at1 — verified, compressed MLflow artifacts
+
+Store MLflow models, checkpoints, and datasets as **verified, byte-exact AT-1 containers** instead of raw
+files. Every artifact is compressed losslessly (never-worse vs xz), carries an embedded SHA-256, and is
+**byte-exact-verified on download** — a tampered or corrupted artifact is *refused*, not silently returned.
+
+That makes an artifact in your MLflow store **provably the exact bytes you logged** — the reproducibility /
+provenance hook that regulated AI (EU AI Act, NIST AI RMF, FDA 21 CFR Part 11) needs: "this model is the
+exact model that was trained, untampered."
+
+It's a **pure-glue plugin** — no engine code ships here. It registers the `at1://` artifact-repository
+scheme and shells out to the `at1` CLI.
+
+## Install
+```bash
+npm i -g @tinyfiles/cli          # the at1 binary (the codec; no Python engine needed)
+pip install mlflow-at1           # this plugin (registers the at1:// scheme)
+#   pip install mlflow-at1[full] # if you want full MLflow rather than mlflow-skinny
+```
+
+## Use
+Point an experiment's **artifact location** at an `at1://` path; everything else is normal MLflow:
+```python
+import mlflow
+exp = mlflow.create_experiment("regulated-model", artifact_location="at1:///data/mlartifacts")
+with mlflow.start_run(experiment_id=exp):
+    mlflow.log_artifact("model.bin")      # -> stored as a verified, compressed AT-1 container
+    # mlflow.sklearn.log_model(...) etc. all route through AT-1 too
+
+# later — download is a byte-exact, SHA-256-verified reconstruction:
+path = mlflow.artifacts.download_artifacts(run_id=run_id, artifact_path="model.bin")
+```
+
+`at1://<path>` is a local/mounted filesystem base directory for the containers (S3/GCS backing is a
+follow-on). Set `AT1_BIN` to point at a specific `at1` binary if it isn't on `PATH`.
+
+## What you get
+- **Smaller** — lossless, never-worse than xz; typically a real reduction on weights/checkpoints/datasets.
+- **Verified** — every container embeds a SHA-256; `download` refuses anything that doesn't reconstruct
+  byte-for-byte. Explicit proof per artifact: `repo.integrity("model.bin")` → `integrity OK (sha256 …)`.
+- **Addressable** — one `.at1` per artifact; tabular artifacts stay queryable in place via the `at1` CLI.
+- **Zero lock-in** — the container is an ordinary file with a tiny open decoder; you can always get the
+  exact original back, with or without this plugin.
+
+## Test
+```bash
+pip install mlflow-skinny pytest && python -m pytest tests/ -q   # needs the `at1` CLI on PATH
+```
+5 tests: log→download byte-exact, logical listing, directory trees, integrity + tamper-rejection, delete.

mlflow_at1-0.1.0/mlflow_at1.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+README.md
+pyproject.toml
+mlflow_at1/__init__.py
+mlflow_at1/at1_artifact_repo.py
+mlflow_at1.egg-info/PKG-INFO
+mlflow_at1.egg-info/SOURCES.txt
+mlflow_at1.egg-info/dependency_links.txt
+mlflow_at1.egg-info/entry_points.txt
+mlflow_at1.egg-info/requires.txt
+mlflow_at1.egg-info/top_level.txt
+tests/test_at1_artifact_repo.py

mlflow_at1-0.1.0/mlflow_at1.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

mlflow_at1-0.1.0/mlflow_at1.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[mlflow.artifact_repository]
+at1 = mlflow_at1.at1_artifact_repo:AT1ArtifactRepository

mlflow_at1-0.1.0/mlflow_at1.egg-info/requires.txt

@@ -0,0 +1,4 @@
+mlflow-skinny>=2.0
+
+[full]
+mlflow>=2.0

mlflow_at1-0.1.0/mlflow_at1.egg-info/top_level.txt

@@ -0,0 +1 @@
+mlflow_at1

mlflow_at1-0.1.0/pyproject.toml

@@ -0,0 +1,31 @@
+[build-system]
+requires = ["setuptools>=61", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "mlflow-at1"
+version = "0.1.0"
+description = "MLflow artifact repository backed by verified, compressed, byte-exact AT-1 containers"
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "Apache-2.0" }
+authors = [{ name = "TinyFiles / AT-1" }]
+keywords = ["mlflow", "artifacts", "compression", "provenance", "verified", "lossless"]
+# Pure glue: depends on MLflow (the host) and the `at1` CLI on PATH (npm i -g @tinyfiles/cli).
+# No engine code ships here, so this connector is Apache-2.0 and publishable. mlflow-skinny is the
+# lightweight tracking/store/entities subset this plugin needs; `pip install mlflow-at1[full]` pulls full MLflow.
+dependencies = ["mlflow-skinny>=2.0"]
+
+[project.optional-dependencies]
+full = ["mlflow>=2.0"]
+
+[project.urls]
+Homepage = "https://tinyfiles.io"
+Documentation = "https://tinyfiles.io/docs/mlflow"
+
+# THE plugin hook: register the `at1://` scheme so MLflow routes artifacts through AT-1.
+[project.entry-points."mlflow.artifact_repository"]
+at1 = "mlflow_at1.at1_artifact_repo:AT1ArtifactRepository"
+
+[tool.setuptools]
+packages = ["mlflow_at1"]

mlflow_at1-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

mlflow_at1-0.1.0/tests/test_at1_artifact_repo.py

@@ -0,0 +1,85 @@
+"""Tests for the AT-1 MLflow artifact repository. Needs the `at1` CLI on PATH (npm i -g @tinyfiles/cli)
+and mlflow-skinny. Skips cleanly if either is absent."""
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+pytest.importorskip("mlflow")
+from mlflow_at1.at1_artifact_repo import AT1ArtifactRepository, _BIN, _run  # noqa: E402
+
+
+def _at1_available():
+    try:
+        import subprocess
+        return subprocess.run(_BIN + ["--version"], capture_output=True).returncode == 0
+    except Exception:
+        return False
+
+
+if not _at1_available():
+    pytest.skip("at1 CLI not on PATH (npm i -g @tinyfiles/cli)", allow_module_level=True)
+
+
+def _repo(tmp_path):
+    return AT1ArtifactRepository(f"at1://{tmp_path.as_posix()}")
+
+
+def _write(p, data):
+    p.write_bytes(data)
+    return str(p)
+
+
+def test_log_download_byte_exact(tmp_path):
+    repo = _repo(tmp_path / "store")
+    src = _write(tmp_path / "model.bin", os.urandom(1000) + b"weights" * 4000)
+    repo.log_artifact(src)
+    # stored as a .at1 container, not the raw file
+    assert os.path.exists(os.path.join(repo._base, "model.bin.at1"))
+    out = tmp_path / "dl" / "model.bin"
+    repo._download_file("model.bin", str(out))
+    assert out.read_bytes() == open(src, "rb").read()          # byte-exact reconstruction
+
+
+def test_list_shows_logical_names(tmp_path):
+    repo = _repo(tmp_path / "store")
+    repo.log_artifact(_write(tmp_path / "a.pkl", b"A" * 5000))
+    repo.log_artifact(_write(tmp_path / "b.csv", b"id,v\n1,2\n3,4\n" * 500), artifact_path="data")
+    top = {fi.path: fi.is_dir for fi in repo.list_artifacts()}
+    assert "a.pkl" in top and top["a.pkl"] is False           # logical name, no .at1 suffix
+    assert top.get("data") is True                             # subdir surfaces
+    assert {fi.path for fi in repo.list_artifacts("data")} == {"data/b.csv"}
+
+
+def test_log_artifacts_directory_tree(tmp_path):
+    repo = _repo(tmp_path / "store")
+    d = tmp_path / "run"; (d / "sub").mkdir(parents=True)
+    _write(d / "top.bin", b"T" * 3000); _write(d / "sub" / "deep.bin", b"D" * 3000)
+    repo.log_artifacts(str(d), artifact_path="artifacts")
+    names = {fi.path for fi in repo.list_artifacts("artifacts")}
+    assert "artifacts/top.bin" in names
+    out = tmp_path / "deep.bin"
+    repo._download_file("artifacts/sub/deep.bin", str(out))
+    assert out.read_bytes() == b"D" * 3000
+
+
+def test_integrity_and_tamper_rejection(tmp_path):
+    repo = _repo(tmp_path / "store")
+    repo.log_artifact(_write(tmp_path / "w.bin", os.urandom(500) + b"w" * 3000))
+    assert "integrity OK" in repo.integrity("w.bin")
+    # flip a byte inside the stored container -> download must REFUSE, not return wrong bytes
+    cont = os.path.join(repo._base, "w.bin.at1")
+    raw = bytearray(open(cont, "rb").read()); raw[len(raw) // 2] ^= 1
+    open(cont, "wb").write(raw)
+    with pytest.raises(RuntimeError):
+        repo._download_file("w.bin", str(tmp_path / "bad.out"))
+
+
+def test_delete(tmp_path):
+    repo = _repo(tmp_path / "store")
+    repo.log_artifact(_write(tmp_path / "x.bin", b"X" * 4000))
+    assert repo.list_artifacts()
+    repo.delete_artifacts("x.bin")
+    assert not os.path.exists(os.path.join(repo._base, "x.bin.at1"))