itar 0.2.0__tar.gz → 0.3.0__tar.gz

{itar-0.2.0 → itar-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: itar
-Version: 0.2.0
+Version: 0.3.0
 Summary: Add your description here
 Author: Karim Abou Zeid
 Author-email: Karim Abou Zeid <contact@ka.codes>
@@ -13,14 +13,18 @@ Description-Content-Type: text/markdown
 
 # itar
 
-`itar` builds constant-time indexes for one or more TAR shards so you can seek directly to a member without extracting the archive. The project ships a small CLI (`itar`) and a Python helper (`IndexedTarFile`) plus a module-level `itar.open()` convenience that mirrors `tarfile.open()`.
+[![image](https://img.shields.io/pypi/v/itar.svg)](https://pypi.python.org/pypi/itar)
+
+`itar` builds constant‑time indexes over one or more tar file shards, enabling direct, random access to members without extracting the archives. It ships a lightweight CLI (`itar`) and a Python API.
+
+Designed for large datasets and deep‑learning pipelines, it supports single or sharded tar archives with thread‑safe access for concurrent reads.
 
 ## Quickstart (single tarball)
 
 ```bash
 echo "Hello world!" > hello.txt
 tar cf hello.tar hello.txt       # regular tarball
-itar index build hello.itar      # indexes hello.tar
+itar index create hello.itar     # indexes hello.tar
 itar index list hello.itar       # list indexed members
 ```
 
@@ -43,7 +47,7 @@ Give each shard a zero-padded suffix before building the index:
 ```bash
 tar cf photos-0.tar wedding/   # shard 0
 tar cf photos-1.tar vacation/  # shard 1
-itar index build photos.itar   # discovers photos-0.tar, photos-1.tar, ...
+itar index create photos.itar  # discovers photos-0.tar, photos-1.tar, ...
 itar index list -l photos.itar # shard index, offsets, byte sizes
 ```
 
@@ -64,9 +68,10 @@ num_shards, stored_index = itar.index.load("photos.itar")
 
 | Command | Purpose |
 | --- | --- |
-| `itar index build <archive>.itar [--single TAR | --shards shard0.tar shard1.tar ...]` | Indexes a single archive or an explicit set of shards. With no flags, shards are auto-discovered next to `<archive>.itar`. |
+| `itar index create <archive>.itar [--single TAR \| --shards shard0.tar shard1.tar ...]` | Indexes a single archive or an explicit set of shards. With no flags, shards are auto-discovered next to `<archive>.itar`. |
 | `itar index list <archive>.itar` | Lists members. Use `-l` for shard/offset info and `-H` for human-readable sizes. |
 | `itar index check <archive>.itar` | Validates recorded entries; add `--member NAME` to focus on specific files. |
+| `itar cat <archive>.itar <member>` | Streams a member’s bytes to stdout. |
 
 ## Python helpers
 

{itar-0.2.0 → itar-0.3.0}/README.md

@@ -1,13 +1,17 @@
 # itar
 
-`itar` builds constant-time indexes for one or more TAR shards so you can seek directly to a member without extracting the archive. The project ships a small CLI (`itar`) and a Python helper (`IndexedTarFile`) plus a module-level `itar.open()` convenience that mirrors `tarfile.open()`.
+[![image](https://img.shields.io/pypi/v/itar.svg)](https://pypi.python.org/pypi/itar)
+
+`itar` builds constant‑time indexes over one or more tar file shards, enabling direct, random access to members without extracting the archives. It ships a lightweight CLI (`itar`) and a Python API.
+
+Designed for large datasets and deep‑learning pipelines, it supports single or sharded tar archives with thread‑safe access for concurrent reads.
 
 ## Quickstart (single tarball)
 
 ```bash
 echo "Hello world!" > hello.txt
 tar cf hello.tar hello.txt       # regular tarball
-itar index build hello.itar      # indexes hello.tar
+itar index create hello.itar     # indexes hello.tar
 itar index list hello.itar       # list indexed members
 ```
 
@@ -30,7 +34,7 @@ Give each shard a zero-padded suffix before building the index:
 ```bash
 tar cf photos-0.tar wedding/   # shard 0
 tar cf photos-1.tar vacation/  # shard 1
-itar index build photos.itar   # discovers photos-0.tar, photos-1.tar, ...
+itar index create photos.itar  # discovers photos-0.tar, photos-1.tar, ...
 itar index list -l photos.itar # shard index, offsets, byte sizes
 ```
 
@@ -51,9 +55,10 @@ num_shards, stored_index = itar.index.load("photos.itar")
 
 | Command | Purpose |
 | --- | --- |
-| `itar index build <archive>.itar [--single TAR | --shards shard0.tar shard1.tar ...]` | Indexes a single archive or an explicit set of shards. With no flags, shards are auto-discovered next to `<archive>.itar`. |
+| `itar index create <archive>.itar [--single TAR \| --shards shard0.tar shard1.tar ...]` | Indexes a single archive or an explicit set of shards. With no flags, shards are auto-discovered next to `<archive>.itar`. |
 | `itar index list <archive>.itar` | Lists members. Use `-l` for shard/offset info and `-H` for human-readable sizes. |
 | `itar index check <archive>.itar` | Validates recorded entries; add `--member NAME` to focus on specific files. |
+| `itar cat <archive>.itar <member>` | Streams a member’s bytes to stdout. |
 
 ## Python helpers
 

{itar-0.2.0 → itar-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "itar"
-version = "0.2.0"
+version = "0.3.0"
 description = "Add your description here"
 readme = "README.md"
 authors = [

{itar-0.2.0 → itar-0.3.0}/src/itar/cli.py

@@ -1,5 +1,4 @@
 import argparse
-import re
 import sys
 from pathlib import Path
 
@@ -12,15 +11,36 @@ def main() -> None:
     subparsers = parser.add_subparsers(dest="command")
     subparsers.required = True
 
+    cat_parser = subparsers.add_parser(
+        "cat", help="Stream a member from an index to stdout"
+    )
+    cat_parser.add_argument("index", type=Path, help="Path to the .itar index file")
+    cat_parser.add_argument("member", help="Member path to stream")
+    cat_parser.set_defaults(func=_cmd_cat)
+
+    ls_parser = subparsers.add_parser(
+        "ls", aliases=["list"], help="List members recorded in an index"
+    )
+    ls_parser.add_argument("index", type=Path, help="Path to the .itar index file")
+    ls_parser.add_argument(
+        "-l", "--long", action="store_true", help="Show shard and offset details"
+    )
+    ls_parser.add_argument(
+        "--bytes",
+        action="store_true",
+        help="Render sizes as number of bytes",
+    )
+    ls_parser.set_defaults(func=_cmd_ls)
+
     index_parser = subparsers.add_parser("index", help="Manage .itar index files")
     index_subparsers = index_parser.add_subparsers(dest="index_command")
     index_subparsers.required = True
 
-    build_parser = index_subparsers.add_parser(
-        "build", help="Create an index file from one or more TAR shards"
+    create_parser = index_subparsers.add_parser(
+        "create", help="Create an index file from one or more TAR shards"
     )
-    build_parser.add_argument("index", type=Path, help="Destination .itar index file")
-    build_parser.add_argument(
+    create_parser.add_argument("index", type=Path, help="Destination .itar index file")
+    create_parser.add_argument(
         "--shards",
         type=Path,
         nargs="+",
@@ -29,20 +49,20 @@ def main() -> None:
             "Explicit shard files to index. Use when you want to bypass auto-discovery."
         ),
     )
-    build_parser.add_argument(
+    create_parser.add_argument(
         "--single",
         type=Path,
         dest="single_tar",
         metavar="TAR",
         help="Path to a single TAR archive. Ignored if --shards is provided.",
     )
-    build_parser.add_argument(
+    create_parser.add_argument(
         "--no-progress",
         dest="progress",
         action="store_false",
         help="Disable the indexing progress bar",
     )
-    build_parser.set_defaults(progress=True, func=_cmd_index_build)
+    create_parser.set_defaults(progress=True, func=_cmd_index_create)
 
     check_parser = index_subparsers.add_parser(
         "check", help="Verify every member recorded in an index file"
@@ -64,10 +84,9 @@ def main() -> None:
         "-l", "--long", action="store_true", help="Show shard and offset details"
     )
     list_parser.add_argument(
-        "-H",
-        "--human-readable",
+        "--bytes",
         action="store_true",
-        help="Render sizes in human units",
+        help="Render sizes as number of bytes",
     )
     list_parser.set_defaults(func=_cmd_index_list)
 
@@ -83,7 +102,7 @@ class CLIError(RuntimeError):
     """Raised for recoverable CLI errors."""
 
 
-def _resolve_shards_for_build(
+def _resolve_shards_for_create(
     index_path: Path,
     explicit_shards: list[Path] | None,
     single_tar: Path | None,
@@ -98,15 +117,8 @@ def _resolve_shards_for_build(
             raise CLIError(f"Single tar not found: {single_tar}")
         return single_tar, None
 
-    stem = index_path.stem
-    pattern = re.compile(rf"^{re.escape(stem)}-\d+\.tar$")
-    indexed_shards = [
-        candidate
-        for candidate in index_path.parent.glob(f"{index_path.stem}-*.tar")
-        if candidate.is_file() and pattern.match(candidate.name)
-    ]
-    indexed_shards.sort()
     layout = index.IndexLayout(index_path)
+    indexed_shards = layout.discover_shards()
     single_tar_candidate = layout.single_tar()
     has_single = single_tar_candidate.is_file()
     has_indexed = len(indexed_shards) > 0
@@ -139,9 +151,30 @@ def _resolve_shards_for_build(
     return shards, num_shards
 
 
-def _cmd_index_build(args) -> None:
+def _cmd_cat(args) -> None:
+    try:
+        with index.open(args.index) as archive:
+            try:
+                source = archive.file(args.member)
+            except KeyError as exc:  # pragma: no cover - defensive
+                raise CLIError(f"Member not found: {args.member}") from exc
+
+            try:
+                while True:
+                    chunk = source.read(64 * 1024)
+                    if not chunk:
+                        break
+                    sys.stdout.buffer.write(chunk)
+                sys.stdout.buffer.flush()
+            finally:
+                source.close()
+    except FileNotFoundError as exc:  # pragma: no cover - defensive
+        raise CLIError(str(exc)) from exc
+
+
+def _cmd_index_create(args) -> None:
     index_path = Path(args.index)
-    shards, num_shards = _resolve_shards_for_build(
+    shards, num_shards = _resolve_shards_for_create(
         index_path, args.shards, args.single_tar
     )
     index.create(index_path, shards, progress_bar=args.progress)
@@ -169,11 +202,32 @@ def _cmd_index_check(args) -> None:
         sys.exit(1)
 
 
+def _cmd_ls(args) -> None:
+    _, current_index = index.load(args.index)
+    if args.long:
+        max_size = 0
+        lines = []
+        for member, (shard_idx, (offset, offset_data, size)) in current_index.items():
+            if not args.bytes:
+                from humanize import naturalsize
+
+                size = naturalsize(size, gnu=True)
+            else:
+                size = str(size)
+            max_size = max(max_size, len(size))
+            lines.append((size, member))
+        for line in [f"{size:>{max_size}} {member}" for size, member in lines]:
+            print(line)
+    else:
+        for member in current_index:
+            print(member)
+
+
 def _cmd_index_list(args) -> None:
     _, current_index = index.load(args.index)
     if args.long:
         for member, (shard_idx, (offset, offset_data, size)) in current_index.items():
-            if args.human_readable:
+            if not args.bytes:
                 from humanize import naturalsize
 
                 size = naturalsize(size, gnu=True)
@@ -181,11 +235,6 @@ def _cmd_index_list(args) -> None:
                 f"{member:<40} {shard_idx:>5} {offset:>12} {offset_data:>12} {size:>10}"
             )
         print(f"{'NAME':<40} {'SHARD':>5} {'OFFSET':>12} {'OFF_DATA':>12} {'SIZE':>10}")
-        return
-
-    for member in current_index:
-        print(member)
-
-
-if __name__ == "__main__":
-    main()
+    else:
+        for member in current_index:
+            print(member)