dset-cli 0.1.0__tar.gz

dset_cli-0.1.0/PKG-INFO

@@ -0,0 +1,96 @@
+Metadata-Version: 2.4
+Name: dset-cli
+Version: 0.1.0
+Summary: Version control for datasets — commit, diff, and roll back millions of files with semantic diffs.
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# dset — version control for datasets
+
+Commit, diff, and roll back millions of files the way you already do with
+code — and see what actually changed: samples, classes, label distributions.
+Not bytes.
+
+Pure Python, zero dependencies, single file.
+
+## Install
+
+```
+pip install .
+```
+
+(from this folder — or just run `python dset/cli.py ...` directly)
+
+## Quick start
+
+```
+cd my-dataset
+dset init
+dset add images/ labels.csv
+dset commit "June survey, initial labels"
+
+# ... collect more data, fix labels ...
+
+dset add .
+dset commit "July imagery + relabeling pass"
+
+dset diff v1 v2
+```
+
+Output:
+
+```
+comparing v1 → v2  (301 → 406 files)
+
+  files       +120   −15   ~1 modified
+  images      +120   −15
+  labels      ~1 modified
+  size        1.5 MB → 2.1 MB  (+568.2 KB)
+
+  class       acacia  19.7% → 38.8%  (+19.1%)
+  class       shrub   43.7% → 33.3%  (−10.3%)
+  class       bare    36.7% → 27.9%  (−8.8%)
+```
+
+Go back to any version, exactly:
+
+```
+dset checkout v1
+```
+
+## Commands
+
+| command | what it does |
+|---|---|
+| `dset init` | start tracking the current directory |
+| `dset add <paths>` | stage files or folders (incremental — unchanged files are skipped) |
+| `dset status` | staged / modified / deleted / untracked files |
+| `dset commit "msg"` | snapshot as a new version (v1, v2, ...) with dataset stats |
+| `dset log` | list versions with file counts and sizes |
+| `dset diff <a> <b>` | semantic diff: file counts by type, size, class distribution shift |
+| `dset checkout <ref>` | restore the working tree to a version (`--force` to discard changes) |
+
+Refs can be a tag (`v3`), `HEAD`, or a commit-id prefix.
+
+## How it works
+
+- **Content-addressed storage.** Every file is hashed (SHA-256) and stored
+  once under `.dset/objects/`, like git. A new version costs only what
+  changed; re-adding identical data costs nothing.
+- **Commits are manifests** — JSON maps of path → hash. Checkout rebuilds
+  the working tree from objects, restoring and deleting as needed.
+- **Semantic stats are computed at commit time** (file counts by type, total
+  size, class distribution parsed from CSV/TSV label files with a
+  `label`/`class`/`category`/`target` column), so `dset diff` is instant even
+  on huge datasets.
+- **Safe by construction.** Objects are copied (never hard-linked), written
+  atomically, and stored read-only, so editing a working file can never
+  corrupt history. On btrfs/XFS the copy is a free copy-on-write clone.
+- Uncommitted changes block `dset checkout` unless you pass `--force`.
+
+## Current limits (MVP)
+
+- Class stats read CSV/TSV label files only (COCO/YOLO parsers are next).
+- No remote yet — `dset push` / `dset pull` to a self-hosted server is the
+  next milestone.
+- No `.dsetignore` yet; hidden files and dot-directories are always skipped.

dset_cli-0.1.0/README.md

@@ -0,0 +1,89 @@
+# dset — version control for datasets
+
+Commit, diff, and roll back millions of files the way you already do with
+code — and see what actually changed: samples, classes, label distributions.
+Not bytes.
+
+Pure Python, zero dependencies, single file.
+
+## Install
+
+```
+pip install .
+```
+
+(from this folder — or just run `python dset/cli.py ...` directly)
+
+## Quick start
+
+```
+cd my-dataset
+dset init
+dset add images/ labels.csv
+dset commit "June survey, initial labels"
+
+# ... collect more data, fix labels ...
+
+dset add .
+dset commit "July imagery + relabeling pass"
+
+dset diff v1 v2
+```
+
+Output:
+
+```
+comparing v1 → v2  (301 → 406 files)
+
+  files       +120   −15   ~1 modified
+  images      +120   −15
+  labels      ~1 modified
+  size        1.5 MB → 2.1 MB  (+568.2 KB)
+
+  class       acacia  19.7% → 38.8%  (+19.1%)
+  class       shrub   43.7% → 33.3%  (−10.3%)
+  class       bare    36.7% → 27.9%  (−8.8%)
+```
+
+Go back to any version, exactly:
+
+```
+dset checkout v1
+```
+
+## Commands
+
+| command | what it does |
+|---|---|
+| `dset init` | start tracking the current directory |
+| `dset add <paths>` | stage files or folders (incremental — unchanged files are skipped) |
+| `dset status` | staged / modified / deleted / untracked files |
+| `dset commit "msg"` | snapshot as a new version (v1, v2, ...) with dataset stats |
+| `dset log` | list versions with file counts and sizes |
+| `dset diff <a> <b>` | semantic diff: file counts by type, size, class distribution shift |
+| `dset checkout <ref>` | restore the working tree to a version (`--force` to discard changes) |
+
+Refs can be a tag (`v3`), `HEAD`, or a commit-id prefix.
+
+## How it works
+
+- **Content-addressed storage.** Every file is hashed (SHA-256) and stored
+  once under `.dset/objects/`, like git. A new version costs only what
+  changed; re-adding identical data costs nothing.
+- **Commits are manifests** — JSON maps of path → hash. Checkout rebuilds
+  the working tree from objects, restoring and deleting as needed.
+- **Semantic stats are computed at commit time** (file counts by type, total
+  size, class distribution parsed from CSV/TSV label files with a
+  `label`/`class`/`category`/`target` column), so `dset diff` is instant even
+  on huge datasets.
+- **Safe by construction.** Objects are copied (never hard-linked), written
+  atomically, and stored read-only, so editing a working file can never
+  corrupt history. On btrfs/XFS the copy is a free copy-on-write clone.
+- Uncommitted changes block `dset checkout` unless you pass `--force`.
+
+## Current limits (MVP)
+
+- Class stats read CSV/TSV label files only (COCO/YOLO parsers are next).
+- No remote yet — `dset push` / `dset pull` to a self-hosted server is the
+  next milestone.
+- No `.dsetignore` yet; hidden files and dot-directories are always skipped.

dset_cli-0.1.0/dset/cli.py

@@ -0,0 +1,609 @@
+"""dset — version control for datasets.
+
+Content-addressed storage with semantic diffs. Zero dependencies.
+
+Commands:
+    dset init                 start tracking the current directory
+    dset add <paths...>       stage files or folders
+    dset status               see what changed since the last commit
+    dset commit "message"     snapshot the staged files as a new version
+    dset log                  list versions
+    dset diff <a> <b>         semantic diff between two versions
+    dset checkout <ref>       restore the working tree to a version
+"""
+
+import argparse
+import csv
+import hashlib
+import io
+import json
+import os
+import shutil
+import sys
+import time
+from pathlib import Path
+
+DSET = ".dset"
+CHUNK = 1024 * 1024
+
+IMAGE_EXT = {".jpg", ".jpeg", ".png", ".tif", ".tiff", ".bmp", ".gif",
+             ".webp", ".heic", ".raw", ".dng", ".svg"}
+LABEL_EXT = {".csv", ".tsv", ".json", ".jsonl", ".xml", ".yaml", ".yml"}
+CLASS_COLUMNS = ("label", "class", "class_name", "category", "label_name", "target")
+MAX_LABEL_BYTES = 100 * 1024 * 1024  # don't parse label files bigger than this
+
+
+# ---------------------------------------------------------------- utilities
+
+def _tty():
+    return sys.stdout.isatty() and os.environ.get("NO_COLOR") is None
+
+
+def col(s, c):
+    codes = {"g": "32", "r": "31", "y": "33", "d": "90", "b": "1", "c": "36"}
+    return f"\033[{codes[c]}m{s}\033[0m" if _tty() else str(s)
+
+
+def die(msg):
+    print(col("error: ", "r") + msg, file=sys.stderr)
+    sys.exit(1)
+
+
+def human(n):
+    for unit in ("B", "KB", "MB", "GB", "TB"):
+        if n < 1024 or unit == "TB":
+            return f"{n:.1f} {unit}" if unit != "B" else f"{n} B"
+        n /= 1024
+
+
+def category(path):
+    ext = Path(path).suffix.lower()
+    if ext in IMAGE_EXT:
+        return "images"
+    if ext in LABEL_EXT:
+        return "labels"
+    return "other"
+
+
+def clone_or_copy(src, dst):
+    """Copy src to dst, using a copy-on-write clone when the filesystem
+    supports it (btrfs/xfs/APFS) — instant and space-free, but unlike a
+    hard link, safe against in-place edits."""
+    if sys.platform == "linux":
+        try:
+            import fcntl
+            with open(src, "rb") as s, open(dst, "wb") as d:
+                fcntl.ioctl(d.fileno(), 0x40049409, s.fileno())  # FICLONE
+            return
+        except OSError:
+            pass
+    shutil.copyfile(src, dst)
+
+
+def hash_file(path):
+    h = hashlib.sha256()
+    with open(path, "rb") as f:
+        while True:
+            b = f.read(CHUNK)
+            if not b:
+                break
+            h.update(b)
+    return h.hexdigest()
+
+
+# ---------------------------------------------------------------- repository
+
+class Repo:
+    def __init__(self, root):
+        self.root = Path(root)
+        self.dir = self.root / DSET
+        self.objects = self.dir / "objects"
+        self.commits_dir = self.dir / "commits"
+
+    # -- discovery
+
+    @classmethod
+    def find(cls):
+        p = Path.cwd()
+        while True:
+            if (p / DSET).is_dir():
+                return cls(p)
+            if p.parent == p:
+                die(f"not a dset repository (run `dset init` first)")
+            p = p.parent
+
+    # -- state files
+
+    def _read_json(self, path, default):
+        try:
+            return json.loads(path.read_text())
+        except FileNotFoundError:
+            return default
+
+    @property
+    def index(self):
+        return self._read_json(self.dir / "index.json", {})
+
+    @index.setter
+    def index(self, value):
+        (self.dir / "index.json").write_text(json.dumps(value))
+
+    @property
+    def tags(self):
+        return self._read_json(self.dir / "tags.json", {})
+
+    @tags.setter
+    def tags(self, value):
+        (self.dir / "tags.json").write_text(json.dumps(value, indent=1))
+
+    @property
+    def head(self):
+        try:
+            return (self.dir / "HEAD").read_text().strip() or None
+        except FileNotFoundError:
+            return None
+
+    @head.setter
+    def head(self, value):
+        (self.dir / "HEAD").write_text(value or "")
+
+    # -- objects
+
+    def object_path(self, digest):
+        return self.objects / digest[:2] / digest[2:]
+
+    def store(self, src, digest):
+        dst = self.object_path(digest)
+        if dst.exists():
+            return False
+        dst.parent.mkdir(parents=True, exist_ok=True)
+        tmp = dst.with_suffix(".tmp")
+        clone_or_copy(src, tmp)
+        os.chmod(tmp, 0o444)  # objects are immutable
+        os.replace(tmp, dst)
+        return True
+
+    def open_object(self, digest):
+        return open(self.object_path(digest), "rb")
+
+    # -- commits
+
+    def load_commit(self, cid):
+        return self._read_json(self.commits_dir / f"{cid}.json", None)
+
+    def resolve(self, ref):
+        """Resolve v3 / HEAD / commit-id-prefix to a commit id."""
+        if ref in ("HEAD", "head"):
+            if not self.head:
+                die("no commits yet")
+            return self.head
+        tags = self.tags
+        if ref in tags:
+            return tags[ref]
+        matches = [p.stem for p in self.commits_dir.glob(f"{ref}*.json")]
+        if len(matches) == 1:
+            return matches[0]
+        if len(matches) > 1:
+            die(f"ambiguous reference '{ref}'")
+        die(f"unknown version '{ref}' (try `dset log`)")
+
+    # -- working tree
+
+    def walk_files(self, paths=None):
+        """Yield relative paths of files under the given paths (or repo root)."""
+        roots = [self.root / p for p in paths] if paths else [self.root]
+        for r in roots:
+            r = r.resolve()
+            if r.is_file():
+                yield r.relative_to(self.root).as_posix()
+                continue
+            if not r.exists():
+                die(f"path not found: {r}")
+            for base, dirs, files in os.walk(r):
+                dirs[:] = [d for d in dirs if d != DSET and not d.startswith(".")]
+                for f in files:
+                    if f.startswith("."):
+                        continue
+                    yield (Path(base) / f).relative_to(self.root).as_posix()
+
+
+# ---------------------------------------------------------------- semantics
+
+def analyze_classes(repo, manifest):
+    """Parse tabular label files in a manifest, return {class_name: count}."""
+    counts = {}
+    for rel, (digest, size) in manifest.items():
+        if Path(rel).suffix.lower() not in (".csv", ".tsv") or size > MAX_LABEL_BYTES:
+            continue
+        delim = "\t" if rel.endswith(".tsv") else ","
+        try:
+            with repo.open_object(digest) as f:
+                text = io.TextIOWrapper(f, encoding="utf-8", errors="replace")
+                reader = csv.reader(text, delimiter=delim)
+                header = next(reader, None)
+                if not header:
+                    continue
+                lower = [h.strip().lower() for h in header]
+                idx = next((lower.index(c) for c in CLASS_COLUMNS if c in lower), None)
+                if idx is None:
+                    continue
+                for row in reader:
+                    if idx < len(row) and row[idx].strip():
+                        v = row[idx].strip()
+                        counts[v] = counts.get(v, 0) + 1
+        except OSError:
+            continue
+    return counts
+
+
+def commit_stats(repo, manifest):
+    by_cat = {"images": 0, "labels": 0, "other": 0}
+    total = 0
+    for rel, (digest, size) in manifest.items():
+        by_cat[category(rel)] += 1
+        total += size
+    return {
+        "files": len(manifest),
+        "images": by_cat["images"],
+        "labels": by_cat["labels"],
+        "other": by_cat["other"],
+        "bytes": total,
+        "classes": analyze_classes(repo, manifest),
+    }
+
+
+# ---------------------------------------------------------------- commands
+
+def cmd_init(args):
+    root = Path.cwd()
+    if (root / DSET).exists():
+        die("already a dset repository")
+    for sub in ("objects", "commits"):
+        (root / DSET / sub).mkdir(parents=True)
+    Repo(root).head = ""
+    print(f"Initialized empty dset repository in {root / DSET}")
+
+
+def cmd_add(args):
+    repo = Repo.find()
+    index = repo.index
+    files = sorted(set(repo.walk_files(args.paths)))
+    if not files:
+        die("no files found under the given paths")
+
+    added = reused = skipped = 0
+    total_bytes = 0
+    t0 = time.time()
+    for i, rel in enumerate(files, 1):
+        full = repo.root / rel
+        try:
+            st = full.stat()
+        except OSError:
+            continue
+        prev = index.get(rel)
+        # unchanged since last add? (same size + mtime) -> skip rehash
+        if prev and prev[1] == st.st_size and prev[2] == st.st_mtime_ns:
+            skipped += 1
+        else:
+            digest = hash_file(full)
+            new = repo.store(full, digest)
+            index[rel] = [digest, st.st_size, st.st_mtime_ns]
+            total_bytes += st.st_size
+            added += int(new)
+            reused += int(not new)
+        if i % 2000 == 0:
+            print(f"\r  hashing {i:,}/{len(files):,} files...", end="", flush=True)
+    if len(files) >= 2000:
+        print("\r" + " " * 40 + "\r", end="")
+
+    repo.index = index
+    dt = time.time() - t0
+    parts = [f"{added:,} new"]
+    if reused:
+        parts.append(f"{reused:,} already stored")
+    if skipped:
+        parts.append(f"{skipped:,} unchanged")
+    print(f"Added {len(files):,} files ({', '.join(parts)}, "
+          f"{human(total_bytes)} in {dt:.1f}s)")
+
+
+def _tree_changes(repo):
+    """Compare disk vs index. Returns (modified, deleted, untracked)."""
+    index = repo.index
+    on_disk = set(repo.walk_files())
+    tracked = set(index)
+    untracked = sorted(on_disk - tracked)
+    deleted = sorted(tracked - on_disk)
+    modified = []
+    for rel in sorted(tracked & on_disk):
+        st = (repo.root / rel).stat()
+        digest, size, mtime = index[rel]
+        if size != st.st_size or mtime != st.st_mtime_ns:
+            if hash_file(repo.root / rel) != digest:
+                modified.append(rel)
+    return modified, deleted, untracked
+
+
+def _staged_changes(repo):
+    """Compare index vs HEAD manifest. Returns (added, removed, changed)."""
+    head_manifest = {}
+    if repo.head:
+        head_manifest = repo.load_commit(repo.head)["manifest"]
+    index = {rel: v[0] for rel, v in repo.index.items()}
+    head = {rel: v[0] for rel, v in head_manifest.items()}
+    added = sorted(set(index) - set(head))
+    removed = sorted(set(head) - set(index))
+    changed = sorted(r for r in set(index) & set(head) if index[r] != head[r])
+    return added, removed, changed
+
+
+def _preview(items, verb, limit=8):
+    for rel in items[:limit]:
+        print(f"    {verb}  {rel}")
+    if len(items) > limit:
+        print(col(f"    ... and {len(items) - limit:,} more", "d"))
+
+
+def cmd_status(args):
+    repo = Repo.find()
+    added, removed, changed = _staged_changes(repo)
+    modified, deleted, untracked = _tree_changes(repo)
+
+    if repo.head:
+        tag = next((t for t, c in repo.tags.items() if c == repo.head), None)
+        print(f"On version {col(tag or repo.head[:10], 'b')}")
+    else:
+        print("No commits yet")
+
+    if added or removed or changed:
+        print(f"\n  Staged for commit "
+              f"({col('+' + format(len(added), ','), 'g')} "
+              f"{col('-' + format(len(removed), ','), 'r')} "
+              f"{col('~' + format(len(changed), ','), 'y')}):")
+        _preview([*added[:4], *changed[:4]], "staged")
+    if modified:
+        print(f"\n  Modified since `dset add` ({len(modified):,}):")
+        _preview(modified, "modified")
+    if deleted:
+        print(f"\n  Deleted from disk but still tracked ({len(deleted):,}):")
+        _preview(deleted, "deleted")
+    if untracked:
+        print(f"\n  Untracked ({len(untracked):,}) — use `dset add` to track:")
+        _preview(untracked, "")
+    if not any([added, removed, changed, modified, deleted, untracked]):
+        print("Nothing to commit, working tree clean")
+
+
+def cmd_commit(args):
+    repo = Repo.find()
+    index = repo.index
+    if not index:
+        die("nothing staged (use `dset add <paths>` first)")
+
+    # drop staged entries whose files were deleted from disk
+    manifest = {rel: [v[0], v[1]] for rel, v in index.items()
+                if (repo.root / rel).exists()}
+
+    if repo.head:
+        prev = repo.load_commit(repo.head)["manifest"]
+        if manifest == prev:
+            die("no changes since last version")
+
+    print("  computing dataset stats...", end="", flush=True)
+    stats = commit_stats(repo, manifest)
+    print("\r" + " " * 32 + "\r", end="")
+
+    commit = {
+        "parent": repo.head or None,
+        "message": args.message,
+        "time": time.time(),
+        "manifest": manifest,
+        "stats": stats,
+    }
+    raw = json.dumps(commit, sort_keys=True).encode()
+    cid = hashlib.sha256(raw).hexdigest()
+    (repo.commits_dir / f"{cid}.json").write_text(json.dumps(commit))
+
+    tags = repo.tags
+    tag = f"v{len(tags) + 1}"
+    tags[tag] = cid
+    repo.tags = tags
+    repo.head = cid
+    repo.index = {rel: [manifest[rel][0], manifest[rel][1],
+                        (repo.root / rel).stat().st_mtime_ns] for rel in manifest}
+
+    s = stats
+    print(f"{col(tag, 'b')}  {args.message}")
+    print(f"  {s['files']:,} files · {s['images']:,} images · "
+          f"{s['labels']:,} label files · {human(s['bytes'])}")
+    if s["classes"]:
+        top = sorted(s["classes"].items(), key=lambda kv: -kv[1])[:6]
+        total = sum(s["classes"].values())
+        dist = " / ".join(f"{k} {v / total:.0%}" for k, v in top)
+        print(f"  classes: {dist}")
+
+
+def cmd_log(args):
+    repo = Repo.find()
+    tags = repo.tags
+    if not tags:
+        print("No versions yet")
+        return
+    by_commit = {c: t for t, c in tags.items()}
+    cid = repo.head
+    while cid:
+        c = repo.load_commit(cid)
+        tag = by_commit.get(cid, cid[:10])
+        when = time.strftime("%Y-%m-%d %H:%M", time.localtime(c["time"]))
+        s = c["stats"]
+        mark = col(" (HEAD)", "c") if cid == repo.head else ""
+        print(f"{col(tag, 'b')}{mark}  {when}  {c['message']}")
+        print(col(f"       {s['files']:,} files · {s['images']:,} images · "
+                  f"{human(s['bytes'])}", "d"))
+        cid = c["parent"]
+
+
+def _dist(classes):
+    total = sum(classes.values())
+    return {k: v / total for k, v in classes.items()} if total else {}
+
+
+def cmd_diff(args):
+    repo = Repo.find()
+    a_id, b_id = repo.resolve(args.a), repo.resolve(args.b)
+    a, b = repo.load_commit(a_id), repo.load_commit(b_id)
+    am, bm = a["manifest"], b["manifest"]
+
+    a_files, b_files = set(am), set(bm)
+    added = b_files - a_files
+    removed = a_files - b_files
+    modified = {r for r in a_files & b_files if am[r][0] != bm[r][0]}
+
+    def by_cat(paths):
+        out = {"images": 0, "labels": 0, "other": 0}
+        for p in paths:
+            out[category(p)] += 1
+        return out
+
+    ac, rc, mc = by_cat(added), by_cat(removed), by_cat(modified)
+
+    print(f"comparing {col(args.a, 'b')} → {col(args.b, 'b')}  "
+          + col(f"({len(am):,} → {len(bm):,} files)", "d") + "\n")
+
+    def line(name, add, rem, mod):
+        if not (add or rem or mod):
+            return
+        parts = []
+        if add:
+            parts.append(col(f"+{add:,}", "g"))
+        if rem:
+            parts.append(col(f"−{rem:,}", "r"))
+        if mod:
+            parts.append(col(f"~{mod:,} modified", "y"))
+        print(f"  {name:<10}  " + "   ".join(parts))
+
+    line("files", len(added), len(removed), len(modified))
+    line("images", ac["images"], rc["images"], mc["images"])
+    line("labels", ac["labels"], rc["labels"], mc["labels"])
+    line("other", ac["other"], rc["other"], mc["other"])
+
+    da, db = a["stats"]["bytes"], b["stats"]["bytes"]
+    if da != db:
+        sign = "+" if db > da else "−"
+        print(f"  {'size':<10}  {human(da)} → {human(db)}  "
+              + col(f"({sign}{human(abs(db - da))})", "d"))
+
+    # class distribution shift
+    pa, pb = _dist(a["stats"]["classes"]), _dist(b["stats"]["classes"])
+    names = sorted(set(pa) | set(pb),
+                   key=lambda k: -abs(pb.get(k, 0) - pa.get(k, 0)))
+    shifts = [(k, pa.get(k, 0), pb.get(k, 0)) for k in names
+              if abs(pb.get(k, 0) - pa.get(k, 0)) >= 0.005]
+    if shifts:
+        print()
+        for k, x, y in shifts[:8]:
+            delta = y - x
+            arrow = col(f"{'+' if delta > 0 else '−'}{abs(delta):.1%}",
+                        "g" if delta > 0 else "r")
+            print(f"  {'class':<10}  {k}  {x:.1%} → {y:.1%}  ({arrow})")
+        if len(shifts) > 8:
+            print(col(f"  ... and {len(shifts) - 8} more classes shifted", "d"))
+
+    if not (added or removed or modified):
+        print(col("  identical", "d"))
+
+
+def cmd_checkout(args):
+    repo = Repo.find()
+    cid = repo.resolve(args.ref)
+    target = repo.load_commit(cid)["manifest"]
+
+    if not args.force:
+        modified, deleted, untracked_ = _tree_changes(repo)
+        added, removed, changed = _staged_changes(repo)
+        if modified or added or removed or changed:
+            die("you have uncommitted changes — commit them or use --force")
+
+    current = {}
+    if repo.head:
+        current = repo.load_commit(repo.head)["manifest"]
+
+    index = repo.index
+    restored = removed_n = kept = 0
+    for rel, (digest, size) in target.items():
+        full = repo.root / rel
+        entry = index.get(rel)
+        if full.exists() and entry and entry[0] == digest:
+            kept += 1
+            continue
+        full.parent.mkdir(parents=True, exist_ok=True)
+        if full.exists():
+            full.unlink()
+        clone_or_copy(repo.object_path(digest), full)
+        os.chmod(full, 0o644)  # working copies stay editable
+        restored += 1
+
+    for rel in set(current) - set(target):
+        full = repo.root / rel
+        if full.exists():
+            full.unlink()
+            removed_n += 1
+            parent = full.parent
+            while parent != repo.root and not any(parent.iterdir()):
+                parent.rmdir()
+                parent = parent.parent
+
+    repo.head = cid
+    repo.index = {rel: [d, s, (repo.root / rel).stat().st_mtime_ns]
+                  for rel, (d, s) in target.items()}
+
+    tag = next((t for t, c in repo.tags.items() if c == cid), cid[:10])
+    print(f"Checked out {col(tag, 'b')}  "
+          + col(f"({restored:,} restored, {removed_n:,} removed, "
+                f"{kept:,} unchanged)", "d"))
+
+
+# ---------------------------------------------------------------- entrypoint
+
+def main(argv=None):
+    p = argparse.ArgumentParser(
+        prog="dset", description="Version control for datasets.")
+    sub = p.add_subparsers(dest="cmd", required=True)
+
+    sub.add_parser("init", help="start tracking the current directory")
+
+    sp = sub.add_parser("add", help="stage files or folders")
+    sp.add_argument("paths", nargs="+")
+
+    sub.add_parser("status", help="see what changed since the last version")
+
+    sp = sub.add_parser("commit", help="snapshot staged files as a new version")
+    sp.add_argument("message")
+
+    sub.add_parser("log", help="list versions")
+
+    sp = sub.add_parser("diff", help="semantic diff between two versions")
+    sp.add_argument("a")
+    sp.add_argument("b")
+
+    sp = sub.add_parser("checkout", help="restore the working tree to a version")
+    sp.add_argument("ref")
+    sp.add_argument("--force", action="store_true",
+                    help="discard uncommitted changes")
+
+    args = p.parse_args(argv)
+    try:
+        {"init": cmd_init, "add": cmd_add, "status": cmd_status,
+         "commit": cmd_commit, "log": cmd_log, "diff": cmd_diff,
+         "checkout": cmd_checkout}[args.cmd](args)
+    except BrokenPipeError:
+        os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
+        sys.exit(0)
+    except KeyboardInterrupt:
+        print("\ninterrupted (repository state is safe — objects are "
+              "written atomically)", file=sys.stderr)
+        sys.exit(130)
+
+
+if __name__ == "__main__":
+    main()

dset_cli-0.1.0/dset_cli.egg-info/PKG-INFO

@@ -0,0 +1,96 @@
+Metadata-Version: 2.4
+Name: dset-cli
+Version: 0.1.0
+Summary: Version control for datasets — commit, diff, and roll back millions of files with semantic diffs.
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# dset — version control for datasets
+
+Commit, diff, and roll back millions of files the way you already do with
+code — and see what actually changed: samples, classes, label distributions.
+Not bytes.
+
+Pure Python, zero dependencies, single file.
+
+## Install
+
+```
+pip install .
+```
+
+(from this folder — or just run `python dset/cli.py ...` directly)
+
+## Quick start
+
+```
+cd my-dataset
+dset init
+dset add images/ labels.csv
+dset commit "June survey, initial labels"
+
+# ... collect more data, fix labels ...
+
+dset add .
+dset commit "July imagery + relabeling pass"
+
+dset diff v1 v2
+```
+
+Output:
+
+```
+comparing v1 → v2  (301 → 406 files)
+
+  files       +120   −15   ~1 modified
+  images      +120   −15
+  labels      ~1 modified
+  size        1.5 MB → 2.1 MB  (+568.2 KB)
+
+  class       acacia  19.7% → 38.8%  (+19.1%)
+  class       shrub   43.7% → 33.3%  (−10.3%)
+  class       bare    36.7% → 27.9%  (−8.8%)
+```
+
+Go back to any version, exactly:
+
+```
+dset checkout v1
+```
+
+## Commands
+
+| command | what it does |
+|---|---|
+| `dset init` | start tracking the current directory |
+| `dset add <paths>` | stage files or folders (incremental — unchanged files are skipped) |
+| `dset status` | staged / modified / deleted / untracked files |
+| `dset commit "msg"` | snapshot as a new version (v1, v2, ...) with dataset stats |
+| `dset log` | list versions with file counts and sizes |
+| `dset diff <a> <b>` | semantic diff: file counts by type, size, class distribution shift |
+| `dset checkout <ref>` | restore the working tree to a version (`--force` to discard changes) |
+
+Refs can be a tag (`v3`), `HEAD`, or a commit-id prefix.
+
+## How it works
+
+- **Content-addressed storage.** Every file is hashed (SHA-256) and stored
+  once under `.dset/objects/`, like git. A new version costs only what
+  changed; re-adding identical data costs nothing.
+- **Commits are manifests** — JSON maps of path → hash. Checkout rebuilds
+  the working tree from objects, restoring and deleting as needed.
+- **Semantic stats are computed at commit time** (file counts by type, total
+  size, class distribution parsed from CSV/TSV label files with a
+  `label`/`class`/`category`/`target` column), so `dset diff` is instant even
+  on huge datasets.
+- **Safe by construction.** Objects are copied (never hard-linked), written
+  atomically, and stored read-only, so editing a working file can never
+  corrupt history. On btrfs/XFS the copy is a free copy-on-write clone.
+- Uncommitted changes block `dset checkout` unless you pass `--force`.
+
+## Current limits (MVP)
+
+- Class stats read CSV/TSV label files only (COCO/YOLO parsers are next).
+- No remote yet — `dset push` / `dset pull` to a self-hosted server is the
+  next milestone.
+- No `.dsetignore` yet; hidden files and dot-directories are always skipped.

dset_cli-0.1.0/dset_cli.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+README.md
+pyproject.toml
+dset/__init__.py
+dset/cli.py
+dset_cli.egg-info/PKG-INFO
+dset_cli.egg-info/SOURCES.txt
+dset_cli.egg-info/dependency_links.txt
+dset_cli.egg-info/entry_points.txt
+dset_cli.egg-info/top_level.txt

dset_cli-0.1.0/dset_cli.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

dset_cli-0.1.0/dset_cli.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+dset = dset.cli:main

dset_cli-0.1.0/dset_cli.egg-info/top_level.txt

@@ -0,0 +1 @@
+dset

dset_cli-0.1.0/pyproject.toml

@@ -0,0 +1,16 @@
+[project]
+name = "dset-cli"
+version = "0.1.0"
+description = "Version control for datasets — commit, diff, and roll back millions of files with semantic diffs."
+requires-python = ">=3.9"
+readme = "README.md"
+
+[project.scripts]
+dset = "dset.cli:main"
+
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+include = ["dset*"]

dset_cli-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+