labeltify 0.4.0__tar.gz → 0.4.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: labeltify
3
- Version: 0.4.0
3
+ Version: 0.4.2
4
4
  Summary: Upload frames and optional annotations into LabelTify with a pipeline token.
5
5
  Author: CleverTify
6
6
  Project-URL: Homepage, https://labeltify.com
@@ -71,9 +71,24 @@ This sends the folder the way dropping it on the dataset's **Upload** page would
71
71
  - Use the portable pixel COCO files. The raw Studio shards store boxes as fractions from 0 to 1, and the server divides by width and height again.
72
72
  - Two images with the same file name in different folders stop the upload before anything is sent; rename one.
73
73
  - Images over 20 MiB, empty files, and files that are neither images nor annotations are listed in `report["files"]` and not sent. A zip is sent as it is when it holds at most 200 files and 20 MB; unzip anything bigger.
74
- - If a request fails after its retries, the error says how many images were already stored; run the folder again and exact duplicates are reported instead of stored twice.
74
+ - If a request fails after its retries, the error says how many images were already stored; run the folder again and only what is missing is uploaded (see below).
75
75
  - `compute_dhash=True` also skips near-duplicate frames, like the Upload page. It needs Pillow: `pip install "labeltify[images]"`.
76
76
 
77
+ #### Migrating thousands of images
78
+
79
+ ```bash
80
+ export LABELTIFY_TOKEN=labeltify_…
81
+ python -m labeltify upload ./photos --dataset DATASET_ID --workers 6 --work-state done
82
+ ```
83
+
84
+ or `client.upload_directory(dataset_id, folder, workers=6, progress=print)`.
85
+
86
+ - **Several requests at once.** `workers` (default 4, at most 8; `1` sends one after another) requests run in parallel over kept connections, so there is no new TLS handshake per request. Each request holds at most `max_request_bytes` (default 24 MiB) and `max_request_images` (default 50). Files are read from disk only when their request is sent, so memory is about `workers` × 24 MiB, not the size of the folder.
87
+ - **Resume for free.** With `skip_existing=True` (default) every image is hashed (SHA-256), the dataset is asked which hashes it already holds (`client.images_exist(dataset_id, hashes)`), and only the rest is sent. Running the same folder again after an interruption takes seconds, not a second upload. Label files of images that are already there are still sent. `force=True` or `skip_existing=False` turns this off, and an older server without the lookup is handled as before.
88
+ - **Progress.** `progress=` is called after each request with `requests_done`, `requests_total`, `images_stored` and `skipped_existing`; the command line prints it.
89
+ - **Frame embeddings** follow the dataset setting and are made in the background after the upload: thousands of images take a while to get their vectors. The dataset's Images tab shows how many are done.
90
+ - If a request fails, the requests not started yet are dropped, those already stored stay stored, and the error says how many; run the command again to continue.
91
+
77
92
  `upload_bundle(dataset_id, files, work_state=…)` sends one such request from bytes you already have (`files` is a list of `{"filename", "data", "content_type"}`).
78
93
 
79
94
  A deployment behind an access proxy can take extra headers on every request: `LabelTifyClient(token, base_url=…, headers={"CF-Access-Client-Id": …, "CF-Access-Client-Secret": …})`.
@@ -94,6 +109,31 @@ Boxes are normalized `[0,1]`. Camera, pipeline, and trigger are stored on the im
94
109
 
95
110
  Local API: `LabelTifyClient("labeltify_…", base_url="http://localhost:8787")`.
96
111
 
112
+ ### Download a dataset export
113
+
114
+ The export tool on the site (or the MCP export tool) hands out a file-list URL. One command pulls every image it lists, plus the annotation bundle:
115
+
116
+ ```bash
117
+ export LABELTIFY_EXPORT_KEY="<key from the export>"
118
+ python -m labeltify download "https://labeltify.com/api/ds/EXPORT_ID/files" --out ./ofan --workers 16
119
+ ```
120
+
121
+ or `labeltify download …` with the installed console script. From Python:
122
+
123
+ ```python
124
+ from labeltify import download_export
125
+
126
+ summary = download_export("https://labeltify.com/api/ds/…/files", "./ofan", key="<key>", progress=print)
127
+ # {"images": 77769, "downloaded": 77000, "skipped": 769, "failed": 0, "bytes": …, "annotations": True, "failures": []}
128
+ ```
129
+
130
+ The export key can be passed via `--key` / `LABELTIFY_EXPORT_KEY`, as an explicit `key=` argument to `download_export`, or left in the URL as `?key=…` (still supported for browsers and old scripts). For scripts and CI, prefer the header or explicit argument so the key does not land in request URLs and logs.
131
+
132
+ - **Resume for free.** Files already on disk with the right size (and SHA-256, when the list carries one) are skipped, so re-running after an interruption only fetches what is missing. Partial files land in `name.part` and are replaced atomically.
133
+ - **Parallel.** `--workers` (default 16, at most 64) downloads at once. Network errors, `429`, and `5xx` are retried with exponential backoff (`--retries`, default 5), honouring `Retry-After`. An expired image link (`403`/`404`) is reported as failed, not retried forever.
134
+ - **Annotations** are extracted next to the images, unless `--no-annotations`. Zip entries that would land outside the output folder are refused. An images-only export has no annotations, so that step is skipped on its own.
135
+ - `--no-verify` skips the size and hash checks. An expired file-list link (`410`) says so clearly. When every file made it, the exit code is 0 and the last stdout line is the JSON summary; otherwise it is 1.
136
+
97
137
  ## Develop
98
138
 
99
139
  ```sh
@@ -50,9 +50,24 @@ This sends the folder the way dropping it on the dataset's **Upload** page would
50
50
  - Use the portable pixel COCO files. The raw Studio shards store boxes as fractions from 0 to 1, and the server divides by width and height again.
51
51
  - Two images with the same file name in different folders stop the upload before anything is sent; rename one.
52
52
  - Images over 20 MiB, empty files, and files that are neither images nor annotations are listed in `report["files"]` and not sent. A zip is sent as it is when it holds at most 200 files and 20 MB; unzip anything bigger.
53
- - If a request fails after its retries, the error says how many images were already stored; run the folder again and exact duplicates are reported instead of stored twice.
53
+ - If a request fails after its retries, the error says how many images were already stored; run the folder again and only what is missing is uploaded (see below).
54
54
  - `compute_dhash=True` also skips near-duplicate frames, like the Upload page. It needs Pillow: `pip install "labeltify[images]"`.
55
55
 
56
+ #### Migrating thousands of images
57
+
58
+ ```bash
59
+ export LABELTIFY_TOKEN=labeltify_…
60
+ python -m labeltify upload ./photos --dataset DATASET_ID --workers 6 --work-state done
61
+ ```
62
+
63
+ or `client.upload_directory(dataset_id, folder, workers=6, progress=print)`.
64
+
65
+ - **Several requests at once.** `workers` (default 4, at most 8; `1` sends one after another) requests run in parallel over kept connections, so there is no new TLS handshake per request. Each request holds at most `max_request_bytes` (default 24 MiB) and `max_request_images` (default 50). Files are read from disk only when their request is sent, so memory is about `workers` × 24 MiB, not the size of the folder.
66
+ - **Resume for free.** With `skip_existing=True` (default) every image is hashed (SHA-256), the dataset is asked which hashes it already holds (`client.images_exist(dataset_id, hashes)`), and only the rest is sent. Running the same folder again after an interruption takes seconds, not a second upload. Label files of images that are already there are still sent. `force=True` or `skip_existing=False` turns this off, and an older server without the lookup is handled as before.
67
+ - **Progress.** `progress=` is called after each request with `requests_done`, `requests_total`, `images_stored` and `skipped_existing`; the command line prints it.
68
+ - **Frame embeddings** follow the dataset setting and are made in the background after the upload: thousands of images take a while to get their vectors. The dataset's Images tab shows how many are done.
69
+ - If a request fails, the requests not started yet are dropped, those already stored stay stored, and the error says how many; run the command again to continue.
70
+
56
71
  `upload_bundle(dataset_id, files, work_state=…)` sends one such request from bytes you already have (`files` is a list of `{"filename", "data", "content_type"}`).
57
72
 
58
73
  A deployment behind an access proxy can take extra headers on every request: `LabelTifyClient(token, base_url=…, headers={"CF-Access-Client-Id": …, "CF-Access-Client-Secret": …})`.
@@ -73,6 +88,31 @@ Boxes are normalized `[0,1]`. Camera, pipeline, and trigger are stored on the im
73
88
 
74
89
  Local API: `LabelTifyClient("labeltify_…", base_url="http://localhost:8787")`.
75
90
 
91
+ ### Download a dataset export
92
+
93
+ The export tool on the site (or the MCP export tool) hands out a file-list URL. One command pulls every image it lists, plus the annotation bundle:
94
+
95
+ ```bash
96
+ export LABELTIFY_EXPORT_KEY="<key from the export>"
97
+ python -m labeltify download "https://labeltify.com/api/ds/EXPORT_ID/files" --out ./ofan --workers 16
98
+ ```
99
+
100
+ or `labeltify download …` with the installed console script. From Python:
101
+
102
+ ```python
103
+ from labeltify import download_export
104
+
105
+ summary = download_export("https://labeltify.com/api/ds/…/files", "./ofan", key="<key>", progress=print)
106
+ # {"images": 77769, "downloaded": 77000, "skipped": 769, "failed": 0, "bytes": …, "annotations": True, "failures": []}
107
+ ```
108
+
109
+ The export key can be passed via `--key` / `LABELTIFY_EXPORT_KEY`, as an explicit `key=` argument to `download_export`, or left in the URL as `?key=…` (still supported for browsers and old scripts). For scripts and CI, prefer the header or explicit argument so the key does not land in request URLs and logs.
110
+
111
+ - **Resume for free.** Files already on disk with the right size (and SHA-256, when the list carries one) are skipped, so re-running after an interruption only fetches what is missing. Partial files land in `name.part` and are replaced atomically.
112
+ - **Parallel.** `--workers` (default 16, at most 64) downloads at once. Network errors, `429`, and `5xx` are retried with exponential backoff (`--retries`, default 5), honouring `Retry-After`. An expired image link (`403`/`404`) is reported as failed, not retried forever.
113
+ - **Annotations** are extracted next to the images, unless `--no-annotations`. Zip entries that would land outside the output folder are refused. An images-only export has no annotations, so that step is skipped on its own.
114
+ - `--no-verify` skips the size and hash checks. An expired file-list link (`410`) says so clearly. When every file made it, the exit code is 0 and the last stdout line is the JSON summary; otherwise it is 1.
115
+
76
116
  ## Develop
77
117
 
78
118
  ```sh
@@ -4,6 +4,7 @@ from importlib.metadata import PackageNotFoundError, version
4
4
 
5
5
  from .client import DEFAULT_BASE_URL, LabelTifyClient, LabelTifyError, UploadFile, UploadReport, slice_coco
6
6
  from .dhash import dhash_from_gray9x8, format_dhash, parse_dhash
7
+ from .download import download_export
7
8
 
8
9
  try:
9
10
  __version__ = version("labeltify")
@@ -18,6 +19,7 @@ __all__ = [
18
19
  "UploadReport",
19
20
  "__version__",
20
21
  "dhash_from_gray9x8",
22
+ "download_export",
21
23
  "format_dhash",
22
24
  "parse_dhash",
23
25
  "slice_coco",
@@ -0,0 +1,168 @@
1
+ """Command line for bulk uploads and export downloads.
2
+
3
+ export LABELTIFY_TOKEN=labeltify_…
4
+ python -m labeltify upload ./photos --dataset <dataset id> --workers 6
5
+
6
+ Run it again after an interruption: images the dataset already holds are not sent again.
7
+
8
+ python -m labeltify download "<files url from the export tool>" --out ./ofan
9
+
10
+ Downloads every image in the export plus its annotations; finished files are skipped.
11
+ """
12
+
13
+ from __future__ import annotations
14
+
15
+ import argparse
16
+ import json
17
+ import os
18
+ import sys
19
+ import time
20
+ from collections import Counter
21
+ from typing import Any
22
+
23
+ from .client import (
24
+ DEFAULT_BASE_URL,
25
+ DEFAULT_WORKERS,
26
+ DIRECTORY_CHUNK_BYTES,
27
+ DIRECTORY_CHUNK_IMAGES,
28
+ MAX_WORKERS,
29
+ WORK_STATES,
30
+ LabelTifyClient,
31
+ LabelTifyError,
32
+ )
33
+ from .download import DEFAULT_DOWNLOAD_WORKERS, MAX_DOWNLOAD_WORKERS, download_export
34
+
35
+
36
+ def build_parser() -> argparse.ArgumentParser:
37
+ parser = argparse.ArgumentParser(prog="labeltify", description="LabelTify command line.")
38
+ sub = parser.add_subparsers(dest="command", required=True)
39
+ up = sub.add_parser("upload", help="Upload a folder of images and annotation files into a dataset.")
40
+ up.add_argument("directory", help="Folder to upload (images, and optionally COCO / YOLO / VOC / LabelMe / CSV labels).")
41
+ up.add_argument("--dataset", required=True, help="Dataset id.")
42
+ up.add_argument("--token", help="Pipeline token (default: the LABELTIFY_TOKEN environment variable).")
43
+ up.add_argument("--base-url", default=os.environ.get("LABELTIFY_URL", DEFAULT_BASE_URL), help=f"API address (default {DEFAULT_BASE_URL}).")
44
+ up.add_argument("--workers", type=int, default=DEFAULT_WORKERS, help=f"Requests at once, 1-{MAX_WORKERS} (default {DEFAULT_WORKERS}).")
45
+ up.add_argument("--work-state", choices=WORK_STATES, help="State of images that come with labels (required when the folder has label files).")
46
+ up.add_argument("--no-near-dup", action="store_true", help="Do not skip near-duplicates.")
47
+ up.add_argument("--force", action="store_true", help="Store files even when the dataset already holds the same bytes.")
48
+ up.add_argument("--no-skip-existing", action="store_true", help="Send every file even if the dataset already holds it.")
49
+ up.add_argument("--dhash", action="store_true", help="Compute near-duplicate hashes (needs: pip install 'labeltify[images]').")
50
+ up.add_argument("--camera")
51
+ up.add_argument("--pipeline")
52
+ up.add_argument("--trigger")
53
+ up.add_argument("--chunk-mb", type=float, default=DIRECTORY_CHUNK_BYTES / (1024 * 1024), help="Most megabytes per request.")
54
+ up.add_argument("--chunk-images", type=int, default=DIRECTORY_CHUNK_IMAGES, help="Most images per request (the API takes 100).")
55
+ down = sub.add_parser("download", help="Download a dataset export (images plus annotations) from its file list URL.")
56
+ down.add_argument("files_url", help="File list URL from the export tool (https://…/api/ds/<id>/files). The key can be in the URL as ?key=… or passed via --key / LABELTIFY_EXPORT_KEY.")
57
+ down.add_argument("--out", required=True, help="Folder to write the images and annotations into.")
58
+ down.add_argument("--workers", type=int, default=DEFAULT_DOWNLOAD_WORKERS, help=f"Files downloaded at once, 1-{MAX_DOWNLOAD_WORKERS} (default {DEFAULT_DOWNLOAD_WORKERS}).")
59
+ down.add_argument("--retries", type=int, default=5, help="Retries per file on network errors, 429 and 5xx (default 5).")
60
+ down.add_argument("--no-annotations", action="store_true", help="Skip the annotation bundle.")
61
+ down.add_argument("--no-verify", action="store_true", help="Do not check size and sha256.")
62
+ down.add_argument("--key", default=os.environ.get("LABELTIFY_EXPORT_KEY"), help="Export key (default: LABELTIFY_EXPORT_KEY). Can also stay in the URL as ?key=….")
63
+ return parser
64
+
65
+
66
+ def _line(text: str) -> None:
67
+ if sys.stderr.isatty():
68
+ sys.stderr.write("\r" + text.ljust(78))
69
+ else:
70
+ sys.stderr.write(text + "\n")
71
+ sys.stderr.flush()
72
+
73
+
74
+ def run_upload(args: argparse.Namespace) -> int:
75
+ client = LabelTifyClient(args.token, base_url=args.base_url)
76
+ started = time.monotonic()
77
+
78
+ def show(p: dict[str, Any]) -> None:
79
+ extra = f" · {p['skipped_existing']} already there" if p["skipped_existing"] else ""
80
+ _line(f"requests {p['requests_done']}/{p['requests_total']} · {p['images_stored']} images stored{extra}")
81
+
82
+ try:
83
+ report = client.upload_directory(
84
+ args.dataset,
85
+ args.directory,
86
+ work_state=args.work_state,
87
+ skip_near_duplicates=not args.no_near_dup,
88
+ force=args.force,
89
+ camera=args.camera,
90
+ pipeline=args.pipeline,
91
+ trigger=args.trigger,
92
+ compute_dhash=args.dhash,
93
+ max_request_bytes=int(args.chunk_mb * 1024 * 1024),
94
+ max_request_images=args.chunk_images,
95
+ workers=args.workers,
96
+ skip_existing=not args.no_skip_existing,
97
+ progress=show,
98
+ )
99
+ except (LabelTifyError, OSError) as err:
100
+ if sys.stderr.isatty():
101
+ sys.stderr.write("\n")
102
+ print(f"error: {err}", file=sys.stderr)
103
+ return 1
104
+ if sys.stderr.isatty():
105
+ sys.stderr.write("\n")
106
+ statuses = Counter(f.get("status", "?") for f in report["files"])
107
+ seconds = max(time.monotonic() - started, 0.001)
108
+ stored = len(report["items"])
109
+ print(f"stored {stored} images in {report['requests']} requests, {seconds:.1f}s ({stored / seconds:.1f} images/s)")
110
+ skipped = report.get("skipped_existing", 0)
111
+ if skipped:
112
+ print(f" {skipped} already in the dataset (not uploaded again)")
113
+ for status in ("duplicate", "near-duplicate", "failed"):
114
+ n = statuses.get(status, 0) - (skipped if status == "duplicate" else 0)
115
+ if n > 0:
116
+ print(f" {n} {status}")
117
+ for f in report["files"]:
118
+ if f.get("status") == "failed":
119
+ print(f" failed: {f.get('filename')}: {f.get('message', '')}", file=sys.stderr)
120
+ return 1 if statuses.get("failed") else 0
121
+
122
+
123
+ def run_download(args: argparse.Namespace) -> int:
124
+ started = time.monotonic()
125
+
126
+ def show(p: dict[str, Any]) -> None:
127
+ _line(f"files {p['done']}/{p['total']} · {p['downloaded']} downloaded · {p['skipped']} already there · {p['failed']} failed")
128
+
129
+ try:
130
+ summary = download_export(
131
+ args.files_url,
132
+ args.out,
133
+ key=args.key,
134
+ workers=args.workers,
135
+ retries=max(0, args.retries),
136
+ annotations=not args.no_annotations,
137
+ verify=not args.no_verify,
138
+ progress=show,
139
+ )
140
+ except (LabelTifyError, OSError) as err:
141
+ if sys.stderr.isatty():
142
+ sys.stderr.write("\n")
143
+ print(f"error: {err}", file=sys.stderr)
144
+ return 1
145
+ if sys.stderr.isatty():
146
+ sys.stderr.write("\n")
147
+ seconds = max(time.monotonic() - started, 0.001)
148
+ print(json.dumps(summary, ensure_ascii=False))
149
+ if summary["failed"]:
150
+ for failure in summary["failures"]:
151
+ print(f" failed: {failure['path']}: {failure['error']}", file=sys.stderr)
152
+ rate = summary["downloaded"] / seconds
153
+ print(f"{summary['downloaded']} files in {seconds:.1f}s ({rate:.1f} files/s)", file=sys.stderr)
154
+ return 1
155
+ return 0
156
+
157
+
158
+ def main(argv: list[str] | None = None) -> int:
159
+ args = build_parser().parse_args(argv)
160
+ if args.command == "upload":
161
+ return run_upload(args)
162
+ if args.command == "download":
163
+ return run_download(args)
164
+ return 2 # pragma: no cover
165
+
166
+
167
+ if __name__ == "__main__":
168
+ sys.exit(main())