ntrk 0.1.0__tar.gz

ntrk-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 tim-hudelmaier
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ntrk-0.1.0/PKG-INFO

@@ -0,0 +1,99 @@
+Metadata-Version: 2.4
+Name: ntrk
+Version: 0.1.0
+Summary: Wrap a command; later, ask how any file was made. A minimal experiment & lineage tracker.
+Keywords: lineage,provenance,experiment,reproducibility,cli
+Author: tim-hudelmaier
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/tim-hudelmaier/ntrk
+Project-URL: Repository, https://github.com/tim-hudelmaier/ntrk
+Description-Content-Type: text/markdown
+
+# ntrk
+
+**Wrap a command. Later, ask how any file was made.**
+
+ntrk records how each output was produced — the exact command, the git commit, and md5
+hashes of every input and output — so you can point at a file weeks later and see its full
+lineage, back to the raw inputs. Pure Python standard library, no dependencies, two commands.
+
+## Install
+
+```bash
+uv tool install ntrk
+```
+
+Commands: `nt` (alias `ntrk`). Run `uv tool update-shell` once if `nt` isn't on your `PATH`.
+
+## Add to Claude
+
+This repo is also a Claude Code plugin marketplace shipping the `ntrk` skill, so your coding
+agent knows when to record runs and trace outputs:
+
+```
+/plugin marketplace add tim-hudelmaier/ntrk
+/plugin install ntrk@ntrk
+```
+
+## Use
+
+Two commands, no flags.
+
+```bash
+nt track python main.py -i in.csv -o out.csv     # run it, record how
+nt trace out.csv                                 # how was this made?
+```
+
+`nt track` refuses to run on a dirty repo, so every result maps to a real commit. `nt trace`
+prints one line per step — raw inputs first, the file you asked about last — so you can pipe it
+through `head` / `grep` / `awk`. A `[modified]` marker appears if a file changed since it was made:
+
+```console
+$ nt trace out_final.csv
+in.csv -> step1.py @ 9c4b2a1 -> out.csv
+out.csv -> step2.py @ 3a1f9c8 -> out_final.csv
+```
+
+Inputs and outputs are detected from the conventional flags
+`-i -o --in --out --input --output` (repeated, space-, or comma-separated lists work). If your
+scripts use other flag names, add a small `ntrk.toml`:
+
+```toml
+[flags]
+inputs  = ["--source"]
+outputs = ["--dest"]
+```
+
+## How it works
+
+Each run is appended as one JSON line to `.ntrk/runs.jsonl` (commit it — it's your lineage
+history), and each output also gets an invisible self-contained sidecar (`.NAME.ntrk`) so a file
+stays traceable even when copied outside the repo. `trace` walks the chain by content hash, so it
+still works after intermediate files are renamed or deleted.
+
+## Development
+
+```bash
+uv sync          # set up the environment
+uv run pytest    # run the tests
+```
+
+## Releasing (maintainer)
+
+CI (`.github/workflows/ci.yml`) runs the test matrix on every push/PR and publishes to PyPI on
+push to `main` via **trusted publishing** (OIDC, no API token), using `skip-existing` so an
+unchanged version is a no-op. One-time PyPI setup (a *pending publisher*, since the project is not
+yet on PyPI):
+
+1. PyPI → *Your account* → *Publishing* → add a pending GitHub publisher:
+   - Project: `ntrk` · Owner: `tim-hudelmaier` · Repo: `ntrk`
+   - Workflow: `ci.yml` · Environment: `pypi`
+2. In GitHub → *Settings → Environments*, create an environment named `pypi`.
+3. Bump the version in `pyproject.toml` and push to `main` — the first successful publish creates
+   the project and converts the pending publisher.
+
+## License
+
+MIT

ntrk-0.1.0/README.md

@@ -0,0 +1,86 @@
+# ntrk
+
+**Wrap a command. Later, ask how any file was made.**
+
+ntrk records how each output was produced — the exact command, the git commit, and md5
+hashes of every input and output — so you can point at a file weeks later and see its full
+lineage, back to the raw inputs. Pure Python standard library, no dependencies, two commands.
+
+## Install
+
+```bash
+uv tool install ntrk
+```
+
+Commands: `nt` (alias `ntrk`). Run `uv tool update-shell` once if `nt` isn't on your `PATH`.
+
+## Add to Claude
+
+This repo is also a Claude Code plugin marketplace shipping the `ntrk` skill, so your coding
+agent knows when to record runs and trace outputs:
+
+```
+/plugin marketplace add tim-hudelmaier/ntrk
+/plugin install ntrk@ntrk
+```
+
+## Use
+
+Two commands, no flags.
+
+```bash
+nt track python main.py -i in.csv -o out.csv     # run it, record how
+nt trace out.csv                                 # how was this made?
+```
+
+`nt track` refuses to run on a dirty repo, so every result maps to a real commit. `nt trace`
+prints one line per step — raw inputs first, the file you asked about last — so you can pipe it
+through `head` / `grep` / `awk`. A `[modified]` marker appears if a file changed since it was made:
+
+```console
+$ nt trace out_final.csv
+in.csv -> step1.py @ 9c4b2a1 -> out.csv
+out.csv -> step2.py @ 3a1f9c8 -> out_final.csv
+```
+
+Inputs and outputs are detected from the conventional flags
+`-i -o --in --out --input --output` (repeated, space-, or comma-separated lists work). If your
+scripts use other flag names, add a small `ntrk.toml`:
+
+```toml
+[flags]
+inputs  = ["--source"]
+outputs = ["--dest"]
+```
+
+## How it works
+
+Each run is appended as one JSON line to `.ntrk/runs.jsonl` (commit it — it's your lineage
+history), and each output also gets an invisible self-contained sidecar (`.NAME.ntrk`) so a file
+stays traceable even when copied outside the repo. `trace` walks the chain by content hash, so it
+still works after intermediate files are renamed or deleted.
+
+## Development
+
+```bash
+uv sync          # set up the environment
+uv run pytest    # run the tests
+```
+
+## Releasing (maintainer)
+
+CI (`.github/workflows/ci.yml`) runs the test matrix on every push/PR and publishes to PyPI on
+push to `main` via **trusted publishing** (OIDC, no API token), using `skip-existing` so an
+unchanged version is a no-op. One-time PyPI setup (a *pending publisher*, since the project is not
+yet on PyPI):
+
+1. PyPI → *Your account* → *Publishing* → add a pending GitHub publisher:
+   - Project: `ntrk` · Owner: `tim-hudelmaier` · Repo: `ntrk`
+   - Workflow: `ci.yml` · Environment: `pypi`
+2. In GitHub → *Settings → Environments*, create an environment named `pypi`.
+3. Bump the version in `pyproject.toml` and push to `main` — the first successful publish creates
+   the project and converts the pending publisher.
+
+## License
+
+MIT

ntrk-0.1.0/pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "ntrk"
+version = "0.1.0"
+description = "Wrap a command; later, ask how any file was made. A minimal experiment & lineage tracker."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "tim-hudelmaier" }]
+keywords = ["lineage", "provenance", "experiment", "reproducibility", "cli"]
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/tim-hudelmaier/ntrk"
+Repository = "https://github.com/tim-hudelmaier/ntrk"
+
+[project.scripts]
+nt = "ntrk.cli:main"
+ntrk = "ntrk.cli:main"
+
+[build-system]
+requires = ["uv_build>=0.11.3,<0.12"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = ["pytest>=8"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]

ntrk-0.1.0/src/ntrk/__init__.py

@@ -0,0 +1,3 @@
+"""ntrk — wrap a command; later, ask how any file was made."""
+
+__version__ = "0.1.0"

ntrk-0.1.0/src/ntrk/__main__.py

@@ -0,0 +1,6 @@
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

ntrk-0.1.0/src/ntrk/cli.py

@@ -0,0 +1,206 @@
+"""ntrk CLI: two verbs, no options.
+
+    nt track <command...>    run a command, record how each output was made
+    nt trace <file>          print how a file was made, back to raw inputs
+"""
+
+import subprocess
+import sys
+from pathlib import Path
+
+from . import config, gitutil, hashing, parse, record, store
+
+USAGE = (
+    "usage: nt <track|trace> ...\n"
+    "  nt track <command...>   run a command and record its lineage\n"
+    "  nt trace <file>         show how a file was made\n"
+)
+
+
+def main(argv=None):
+    argv = list(sys.argv[1:] if argv is None else argv)
+    if not argv:
+        sys.stderr.write(USAGE)
+        return 2
+    cmd, rest = argv[0], argv[1:]
+    if cmd == "track":
+        return cmd_track(rest)
+    if cmd == "trace":
+        return cmd_trace(rest)
+    sys.stderr.write(USAGE)
+    return 2
+
+
+# --- track -----------------------------------------------------------------
+
+def _to_rel(root, given):
+    """Resolve a command path argument (relative to cwd) to a root-relative
+    posix path. Returns (rel_or_None, abspath)."""
+    ap = (Path.cwd() / given).resolve()
+    try:
+        return ap.relative_to(Path(root).resolve()).as_posix(), ap
+    except ValueError:
+        return None, ap
+
+
+def _collect(root, tokens, kind):
+    """Hash existing in-tree files among ``tokens``; warn-and-skip the rest."""
+    items = []
+    for tok in tokens:
+        rel, ap = _to_rel(root, tok)
+        if rel is None:
+            sys.stderr.write(f"nt: {kind} outside repo, not tracked: {tok}\n")
+            continue
+        if not ap.exists():
+            sys.stderr.write(f"nt: {kind} not found, not tracked: {tok}\n")
+            continue
+        try:
+            md5, size = hashing.md5_file(ap)
+        except OSError as e:
+            sys.stderr.write(f"nt: cannot read {kind} {tok}: {e}\n")
+            continue
+        items.append({"path": rel, "md5": md5, "size": size})
+    return items
+
+
+def cmd_track(rest):
+    if not rest:
+        sys.stderr.write("usage: nt track <command...>\n")
+        return 2
+
+    try:
+        root = gitutil.repo_root()
+    except gitutil.GitError:
+        sys.stderr.write("nt: not a git repository\n")
+        return 1
+
+    dirty = gitutil.dirty_paths(root)
+    if dirty:
+        sys.stderr.write(
+            "nt: refusing to run — commit your changes first "
+            "(every run maps to a commit):\n"
+        )
+        for p in dirty:
+            sys.stderr.write(f"    {p}\n")
+        return 1
+
+    try:
+        commit = gitutil.head_commit(root)
+        branch_name = gitutil.branch(root)
+    except gitutil.GitError:
+        sys.stderr.write("nt: repository has no commits yet\n")
+        return 1
+
+    cfg = config.load(root)
+    parsed = parse.parse(rest, cfg)
+
+    inputs = _collect(root, parsed.inputs, "input")
+
+    proc = subprocess.run(rest)
+    if proc.returncode != 0:
+        return proc.returncode
+
+    outputs = _collect(root, parsed.outputs, "output")
+    if not outputs:
+        sys.stderr.write("nt: no tracked outputs produced; nothing recorded.\n")
+        return proc.returncode
+
+    script_meta = None
+    if parsed.script:
+        rel, ap = _to_rel(root, parsed.script)
+        if rel is not None and ap.exists():
+            md5, _ = hashing.md5_file(ap)
+            script_meta = {"path": rel, "blob": gitutil.blob_hash(root, rel), "md5": md5}
+        else:
+            script_meta = {"path": parsed.script, "blob": None, "md5": None}
+
+    cwd_rel = _to_rel(root, ".")[0] or "."
+    rec = record.build(commit, branch_name, cwd_rel, script_meta, rest, inputs, outputs)
+    store.append(root, rec)
+    for o in outputs:
+        store.write_sidecar(root, o["path"], rec)
+
+    sys.stderr.write(
+        f"nt: recorded {rec['id'][:8]} ({len(outputs)} output(s)); "
+        "commit .ntrk/runs.jsonl to keep it.\n"
+    )
+    return 0
+
+
+# --- trace -----------------------------------------------------------------
+
+def cmd_trace(rest):
+    if len(rest) != 1:
+        sys.stderr.write("usage: nt trace <file>\n")
+        return 2
+    target = rest[0]
+    try:
+        root = gitutil.repo_root()
+    except gitutil.GitError:
+        root = None
+
+    lines = _trace_chain(root, target)
+    if not lines:
+        sys.stderr.write(f"nt: no lineage found for {target}\n")
+        return 1
+    for line in lines:
+        sys.stdout.write(line + "\n")
+    return 0
+
+
+def _trace_chain(root, target):
+    records = store.read_records(root)
+    first, out_rel = store.resolve(root, target)
+    if first is None:
+        return []
+
+    index_by_id = {r.get("id"): i for i, r in enumerate(records)}
+    start_idx = index_by_id.get(first.get("id"), len(records))
+
+    steps = []          # (record, output_rel), discovered newest -> oldest
+    visited = set()
+
+    def walk(rec, output_rel, idx):
+        key = (rec.get("id"), output_rel)
+        if key in visited:
+            return
+        visited.add(key)
+        steps.append((rec, output_rel))
+        for inp in rec.get("inputs", []):
+            prod, prod_out, prod_idx = store.producer_before(
+                records, inp.get("md5"), idx, exclude_id=rec.get("id")
+            )
+            if prod is not None:
+                walk(prod, prod_out, prod_idx)
+
+    walk(first, out_rel, start_idx)
+    steps.reverse()  # roots first -> reads left-to-right, top-to-bottom
+    return [_format_step(root, rec, output_rel) for rec, output_rel in steps]
+
+
+def _marker(root, path_rel, recorded_md5):
+    if root is None or path_rel is None or recorded_md5 is None:
+        return ""
+    ap = Path(root) / path_rel
+    if not ap.exists():
+        return " [missing]"
+    try:
+        cur, _ = hashing.md5_file(ap)
+    except OSError:
+        return ""
+    return "" if cur == recorded_md5 else " [modified]"
+
+
+def _format_step(root, rec, output_rel):
+    ins = rec.get("inputs", [])
+    if ins:
+        in_str = ", ".join(
+            i.get("path", "?") + _marker(root, i.get("path"), i.get("md5"))
+            for i in ins
+        )
+    else:
+        in_str = "(no tracked inputs)"
+    script = (rec.get("script") or {}).get("path") or "?"
+    commit = (rec.get("git") or {}).get("commit", "")[:7]
+    out_str = (output_rel or "?") + _marker(root, output_rel, store.md5_for_output(rec, output_rel))
+    return f"{in_str} -> {script} @ {commit} -> {out_str}"

ntrk-0.1.0/src/ntrk/config.py

@@ -0,0 +1,38 @@
+"""Built-in input/output flag convention, plus optional per-repo extensions.
+
+ntrk knows the conventional flags out of the box. A repo only needs a
+``ntrk.toml`` if its scripts use other flag names:
+
+    [flags]
+    inputs  = ["--source"]
+    outputs = ["--dest"]
+
+These are *added* to the built-ins (the defaults always keep working). There is
+no other configuration: strict-on-dirty and md5 are not negotiable.
+"""
+
+import tomllib
+from pathlib import Path
+
+DEFAULT_INPUT_FLAGS = ("-i", "--in", "--input")
+DEFAULT_OUTPUT_FLAGS = ("-o", "--out", "--output")
+
+
+class Config:
+    def __init__(self, input_flags, output_flags):
+        self.input_flags = set(input_flags)
+        self.output_flags = set(output_flags)
+
+
+def load(root):
+    """Return a Config: built-in flags merged with optional ``ntrk.toml``."""
+    inputs = set(DEFAULT_INPUT_FLAGS)
+    outputs = set(DEFAULT_OUTPUT_FLAGS)
+    cfg = Path(root) / "ntrk.toml"
+    if cfg.exists():
+        with open(cfg, "rb") as f:
+            data = tomllib.load(f)
+        flags = data.get("flags", {})
+        inputs |= set(flags.get("inputs", []))
+        outputs |= set(flags.get("outputs", []))
+    return Config(inputs, outputs)

ntrk-0.1.0/src/ntrk/gitutil.py

@@ -0,0 +1,84 @@
+"""Thin, read-only git interface. ntrk never mutates git state."""
+
+import subprocess
+from pathlib import Path
+
+
+class GitError(Exception):
+    pass
+
+
+def _git(args, cwd):
+    result = subprocess.run(
+        ["git", *args], cwd=str(cwd), capture_output=True, text=True
+    )
+    if result.returncode != 0:
+        raise GitError(result.stderr.strip() or f"git {' '.join(args)} failed")
+    return result.stdout
+
+
+def repo_root(cwd="."):
+    """Absolute path to the git work-tree root. Raises GitError outside a repo."""
+    out = _git(["rev-parse", "--show-toplevel"], cwd)
+    return Path(out.strip())
+
+
+def head_commit(root):
+    """40-char commit sha of HEAD. Raises GitError if there are no commits."""
+    return _git(["rev-parse", "HEAD"], root).strip()
+
+
+def branch(root):
+    return _git(["rev-parse", "--abbrev-ref", "HEAD"], root).strip()
+
+
+def blob_hash(root, relpath):
+    """git blob sha of ``relpath`` at HEAD, or None if untracked / no commits."""
+    try:
+        return _git(["rev-parse", f"HEAD:{relpath}"], root).strip()
+    except GitError:
+        return None
+
+
+def _is_ntrk_file(path):
+    # Only ntrk's own artifacts: the .ntrk/ dir and dot-prefixed
+    # `.NAME.ntrk` sidecars. A genuine experiment file named e.g. `model.ntrk`
+    # is NOT excluded — it must still count toward a dirty tree.
+    name = path.rsplit("/", 1)[-1]
+    return (
+        path == ".ntrk"
+        or path.startswith(".ntrk/")
+        or (name.startswith(".") and name.endswith(".ntrk"))
+    )
+
+
+def dirty_paths(root):
+    """Porcelain paths that count as 'dirty', excluding ntrk's own files.
+
+    Uses ``--porcelain -z`` so paths with spaces/unicode aren't quoted/escaped
+    and rename entries are unambiguous (the original path follows in its own
+    NUL-separated field)."""
+    # -uall lists untracked files individually (not collapsed to a dir) so a
+    # sidecar in a new directory can be excluded by name.
+    out = _git(["status", "--porcelain", "-z", "-uall"], root)
+    fields = out.split("\0")
+    dirty = []
+    i = 0
+    while i < len(fields):
+        entry = fields[i]
+        if not entry:
+            i += 1
+            continue
+        status, path = entry[:2], entry[3:]
+        # rename/copy: the next field is the original path; skip it.
+        if status[:1] in ("R", "C") or status[1:2] in ("R", "C"):
+            i += 2
+        else:
+            i += 1
+        if not _is_ntrk_file(path):
+            dirty.append(path)
+    return dirty
+
+
+def is_clean(root):
+    return not dirty_paths(root)

ntrk-0.1.0/src/ntrk/hashing.py

@@ -0,0 +1,15 @@
+"""md5 file fingerprints. md5 is a content fingerprint, not tamper-evidence."""
+
+import hashlib
+from pathlib import Path
+
+
+def md5_file(path):
+    """Return ``(hexdigest, size_bytes)`` for ``path``, streamed.
+
+    Raises FileNotFoundError if the path does not exist.
+    """
+    p = Path(path)
+    with open(p, "rb") as f:
+        digest = hashlib.file_digest(f, lambda: hashlib.md5(usedforsecurity=False))
+    return digest.hexdigest(), p.stat().st_size

ntrk-0.1.0/src/ntrk/parse.py

@@ -0,0 +1,83 @@
+"""Classify a wrapped command's tokens into inputs, outputs, and the script.
+
+An input/output flag is *greedy*: it consumes every following token up to the
+next ``-``-prefixed token, and each consumed token is also split on commas. So
+``-i a.csv b.csv,c.csv -o out1,out2`` -> inputs ``[a,b,c]``, outputs ``[out1,out2]``.
+
+Accepted caveat: a filename containing a comma, or starting with ``-``, can't be
+expressed.
+"""
+
+INTERPRETERS = {
+    "python", "python3", "python2", "Rscript", "bash", "sh",
+    "node", "ruby", "perl", "julia",
+}
+SOURCE_EXTS = (".py", ".R", ".r", ".sh", ".jl", ".rb", ".js", ".pl")
+
+
+class ParsedCommand:
+    def __init__(self, command, script, inputs, outputs):
+        self.command = command  # list[str], verbatim
+        self.script = script    # str or None
+        self.inputs = inputs    # list[str]
+        self.outputs = outputs  # list[str]
+
+
+def _role(flag, config):
+    if flag in config.input_flags:
+        return "in"
+    if flag in config.output_flags:
+        return "out"
+    return None
+
+
+def _split(token):
+    return [part for part in token.split(",") if part]
+
+
+def parse(argv, config):
+    inputs, outputs = [], []
+    i, n = 0, len(argv)
+    while i < n:
+        flag, eq, inline = argv[i].partition("=")
+        role = _role(flag, config)
+        if role is None:
+            i += 1
+            continue
+        bucket = inputs if role == "in" else outputs
+        if eq:  # --flag=value : self-contained, comma-split, no greedy consume
+            bucket.extend(_split(inline))
+            i += 1
+            continue
+        # bare flag: greedily consume until the next '-'-prefixed token
+        i += 1
+        while i < n and not argv[i].startswith("-"):
+            bucket.extend(_split(argv[i]))
+            i += 1
+    classified = set(inputs) | set(outputs)
+    return ParsedCommand(list(argv), _detect_script(argv, classified), inputs, outputs)
+
+
+def _is_interpreter(tok):
+    base = tok.rsplit("/", 1)[-1]
+    return base in INTERPRETERS or base.startswith("python") or base.startswith("Rscript")
+
+
+def _detect_script(argv, classified=()):
+    classified = set(classified)
+    # 1. first non-input/output token ending in a known source extension
+    for tok in argv:
+        if tok not in classified and tok.endswith(SOURCE_EXTS):
+            return tok
+    # 2. first non-flag, non-input/output token after a known interpreter
+    for idx, tok in enumerate(argv):
+        if _is_interpreter(tok):
+            for nxt in argv[idx + 1:]:
+                if not nxt.startswith("-") and nxt not in classified:
+                    return nxt
+            break
+    # 3. fall back to the first non-flag, non-input/output token
+    for tok in argv:
+        if not tok.startswith("-") and tok not in classified:
+            return tok
+    return argv[0] if argv else None

ntrk-0.1.0/src/ntrk/record.py

@@ -0,0 +1,31 @@
+"""The lineage record schema (v1)."""
+
+import uuid
+from datetime import datetime, timezone
+
+SCHEMA_VERSION = 1
+
+
+def now_iso():
+    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def new_id():
+    return uuid.uuid4().hex
+
+
+def build(commit, branch_name, cwd_rel, script, command, inputs, outputs,
+          ts=None, rid=None):
+    """Assemble one run record. ``script`` is ``{path, blob, md5}`` or None;
+    ``inputs``/``outputs`` are lists of ``{path, md5, size}``."""
+    return {
+        "v": SCHEMA_VERSION,
+        "id": rid or new_id(),
+        "ts": ts or now_iso(),
+        "git": {"commit": commit, "branch": branch_name},
+        "cwd": cwd_rel,
+        "script": script,
+        "command": list(command),
+        "inputs": inputs,
+        "outputs": outputs,
+    }

ntrk-0.1.0/src/ntrk/store.py

@@ -0,0 +1,196 @@
+"""Storage: the append-only central log plus per-output sidecars.
+
+The log (``.ntrk/runs.jsonl``) is the source of truth — committable,
+greppable, rename-robust by md5. Each output also gets an invisible
+self-contained sidecar (``.NAME.ntrk``) so a file can be traced even when copied
+outside the repo.
+"""
+
+import fcntl
+import json
+import os
+from pathlib import Path
+
+from . import hashing
+
+STORE_DIR = ".ntrk"
+LOG_NAME = "runs.jsonl"
+
+
+# --- central log ----------------------------------------------------------
+
+def store_dir(root):
+    return Path(root) / STORE_DIR
+
+
+def log_path(root):
+    return store_dir(root) / LOG_NAME
+
+
+def ensure_store(root):
+    """Create ``.ntrk/`` with the log and a union-merge .gitattributes."""
+    d = store_dir(root)
+    d.mkdir(exist_ok=True)
+    log = d / LOG_NAME
+    if not log.exists():
+        log.touch()
+    ga = d / ".gitattributes"
+    if not ga.exists():
+        ga.write_text(f"{LOG_NAME} merge=union\n", encoding="utf-8")
+    return log
+
+
+def append(root, rec):
+    """Append one record as a single JSON line, serialized under an exclusive
+    lock so concurrent runs never interleave."""
+    ensure_store(root)
+    line = json.dumps(rec, sort_keys=True, ensure_ascii=False) + "\n"
+    with open(log_path(root), "a", encoding="utf-8") as f:
+        fcntl.flock(f.fileno(), fcntl.LOCK_EX)
+        try:
+            f.write(line)
+            f.flush()
+            os.fsync(f.fileno())
+        finally:
+            fcntl.flock(f.fileno(), fcntl.LOCK_UN)
+
+
+def read_records(root):
+    if root is None:
+        return []
+    log = log_path(root)
+    if not log.exists():
+        return []
+    records = []
+    with open(log, encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                records.append(json.loads(line))
+            except json.JSONDecodeError:
+                continue
+    return records
+
+
+# --- sidecars --------------------------------------------------------------
+
+def sidecar_path(output_path):
+    p = Path(output_path)
+    return p.with_name("." + p.name + ".ntrk")
+
+
+def write_sidecar(root, output_rel, rec):
+    side = sidecar_path(Path(root) / output_rel)
+    data = dict(rec)
+    data["sidecar_for"] = output_rel
+    tmp = side.with_name(side.name + ".tmp")
+    tmp.write_text(
+        json.dumps(data, sort_keys=True, ensure_ascii=False, indent=2),
+        encoding="utf-8",
+    )
+    os.replace(tmp, side)
+
+
+def read_sidecar(target):
+    side = sidecar_path(Path(target))
+    if not side.exists():
+        return None
+    try:
+        data = json.loads(side.read_text(encoding="utf-8"))
+    except (json.JSONDecodeError, OSError):
+        return None
+    return data if isinstance(data, dict) else None
+
+
+# --- resolution & lineage links -------------------------------------------
+
+def md5_for_output(rec, output_rel):
+    """The recorded md5 of ``output_rel`` within ``rec``."""
+    outs = rec.get("outputs", [])
+    for o in outs:
+        if o.get("path") == output_rel:
+            return o.get("md5")
+    # only guess the sole output when no specific path was asked for
+    if output_rel is None and len(outs) == 1:
+        return outs[0].get("md5")
+    return None
+
+
+def producer_of(records, md5):
+    """Newest record whose outputs contain ``md5`` -> (record, output_rel)."""
+    for rec in reversed(records):
+        for o in rec.get("outputs", []):
+            if o.get("md5") == md5:
+                return rec, o.get("path")
+    return None, None
+
+
+def producer_before(records, md5, before_idx, exclude_id=None):
+    """Newest record strictly before ``before_idx`` whose outputs contain
+    ``md5`` (skipping ``exclude_id``) -> (record, output_rel, index).
+
+    The trace walk uses this so an input always links to an *earlier* producer,
+    never to the consuming run itself or a later one that re-emits the same
+    bytes (which would mis-attribute byte-identical chain files)."""
+    for i in range(min(before_idx, len(records)) - 1, -1, -1):
+        rec = records[i]
+        if rec.get("id") == exclude_id:
+            continue
+        for o in rec.get("outputs", []):
+            if o.get("md5") == md5:
+                return rec, o.get("path"), i
+    return None, None, None
+
+
+def _rel_to_root(root, target):
+    if root is None:
+        return None
+    try:
+        return Path(target).resolve().relative_to(Path(root).resolve()).as_posix()
+    except ValueError:
+        return None
+
+
+def resolve(root, target):
+    """Find the run that produced ``target`` -> (record, output_rel).
+
+    Order: sidecar next to the file -> exact (same path AND same bytes) ->
+    content match (rename-robust) -> path match (file changed since produced).
+    Returns (None, None) if nothing produced it.
+    """
+    target = Path(target)
+
+    side = read_sidecar(target)
+    if side is not None:
+        return side, side.get("sidecar_for")
+
+    records = read_records(root)
+    rel = _rel_to_root(root, target)
+
+    on_disk_md5 = None
+    if target.exists():
+        try:
+            on_disk_md5, _ = hashing.md5_file(target)
+        except OSError:
+            on_disk_md5 = None
+
+    # 1. exact: same recorded path AND same content (most specific)
+    if rel is not None and on_disk_md5 is not None:
+        for rec in reversed(records):
+            for o in rec.get("outputs", []):
+                if o.get("path") == rel and o.get("md5") == on_disk_md5:
+                    return rec, rel
+    # 2. content match (survives renames of unchanged files)
+    if on_disk_md5 is not None:
+        rec, out_rel = producer_of(records, on_disk_md5)
+        if rec is not None:
+            return rec, out_rel
+    # 3. path match (file changed since it was produced)
+    if rel is not None:
+        for rec in reversed(records):
+            for o in rec.get("outputs", []):
+                if o.get("path") == rel:
+                    return rec, rel
+    return None, None