tree-distribution-shift 0.1.0__tar.gz

tree_distribution_shift-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Aaditya Nalawade
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tree_distribution_shift-0.1.0/PKG-INFO

@@ -0,0 +1,281 @@
+Metadata-Version: 2.4
+Name: tree-distribution-shift
+Version: 0.1.0
+Summary: Tree Distribution Shift benchmark: export HF configs to COCO (train/id_test/ood_test + density buckets).
+Author: Aaditya Nalawade
+License-Expression: MIT
+Project-URL: Homepage, https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: datasets>=2.19.0
+Requires-Dist: huggingface_hub>=0.23.0
+Requires-Dist: orjson>=3.9.0
+Requires-Dist: tqdm>=4.66.0
+Dynamic: license-file
+
+# tree-distribution-shift
+
+A pip-installable benchmark for **Tree Distribution Shift** — export HuggingFace dataset configs to COCO format on disk.
+
+**No git. No git-lfs. Works on clusters.**
+
+Data lives on the HF Hub: [`aadityabuilds/tree-distribution-shift`](https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift)
+
+## Getting started
+
+```bash
+pip install -U pip
+pip install tree-distribution-shift
+
+# See all available configs (queries HF Hub live)
+tree-shift list
+
+# Export a config to COCO format
+tree-shift export --config intl_train_IN__ood_US --out ./data
+```
+
+That's it. No cloning, no LFS, no large downloads up front — data streams directly from the Hub.
+
+### Authentication (recommended)
+
+Set an HF token for higher rate limits and faster downloads:
+
+```bash
+export HF_TOKEN=hf_...   # recommended
+```
+
+Or log in once with the CLI:
+
+```bash
+huggingface-cli login
+```
+
+You can also pass a token per-command:
+
+```bash
+tree-shift list --token hf_...
+tree-shift export --config intl_train_IN__ood_US --out ./data --token hf_...
+```
+
+### Export only the splits you need
+
+```bash
+tree-shift export \
+  --config in_state_train_Karnataka__ood_Rajasthan \
+  --out ./data \
+  --splits train ood_test
+```
+
+### Pin a revision for reproducibility
+
+For papers and exact replication, pin a commit hash:
+
+```bash
+tree-shift export \
+  --config in_state_train_Karnataka__ood_Rajasthan \
+  --out ./data \
+  --revision abc123def456
+```
+
+### Python API
+
+```python
+from tree_shift import list_configs, export_coco
+
+print(list_configs())
+
+export_coco(
+    config="in_state_train_Karnataka__ood_Rajasthan",
+    out_root="./data",
+    splits=["train", "id_test", "ood_test"],
+    streaming=True,   # default; avoids OOM on low-mem machines
+    revision=None,     # or pin a commit hash
+)
+```
+
+## Output structure
+
+```
+./data/<config>/
+  train/
+    images/
+      *.png
+    annotations/
+      instances_train.json
+  id_test/
+    images/
+    annotations/
+  ood_test/
+    images/
+    annotations/
+  ood_same_density/
+    images/
+    annotations/
+  ood_diff_density/
+    images/
+    annotations/
+```
+
+Annotation files follow the [COCO format](https://cocodataset.org/#format-data).
+
+## Dataset contract
+
+Every example row in the HF dataset contains:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `image_id` | int | Unique image identifier |
+| `filename` | str | Image filename (e.g. `00001.png`) |
+| `width` | int | Image width in pixels |
+| `height` | int | Image height in pixels |
+| `image_bytes` | bytes | Raw image file contents |
+| `coco_annotations` | str | JSON string — list of COCO annotation dicts |
+| `coco_categories` | str | JSON string — list of COCO category dicts |
+| `country` | str | Country code (e.g. `IN`, `US`) |
+| `state` | str | State name |
+| `zone` | str | Geographic zone |
+| `biome` | str | Biome classification |
+| `density_category` | str | Tree density bucket |
+
+### Expected splits
+
+- **`train`** — training set
+- **`id_test`** — in-distribution test set
+- **`ood_test`** — out-of-distribution test set
+- **`ood_same_density`** — OOD test filtered to same density bucket as train
+- **`ood_diff_density`** — OOD test filtered to different density bucket
+
+> **Note:** Some older configs may not include the `ood_same_density` / `ood_diff_density` splits.
+
+## Resume support
+
+Exports are **resume-safe**. If a job is preempted or interrupted:
+
+```bash
+# Just re-run the same command — already-written images are skipped
+tree-shift export --config intl_train_IN__ood_US --out ./data
+```
+
+Images that already exist on disk are not re-downloaded. The COCO annotation JSON is always regenerated to stay consistent with the full split.
+
+## Rate limits & performance
+
+- **Streaming is on by default** — data is fetched shard-by-shard, avoiding OOM and memory-map errors.
+- **`--no-streaming`** downloads the full split into an Arrow table first. This can OOM on large splits or cause `mmap` errors on low-memory machines. Use only if you have plenty of RAM.
+- For faster downloads (when available), set the environment variable:
+  ```bash
+  export HF_HUB_ENABLE_HF_TRANSFER=1
+  ```
+- Exports call `load_dataset` once per split (not per example), keeping HF API calls minimal.
+- If you hit 429 rate-limit errors, wait a few minutes and re-run — resume will skip already-written images.
+
+## Cache location (clusters/shared machines)
+
+The exporter uses the standard HuggingFace cache. On clusters with limited home directory space, set a custom cache location on a large disk:
+
+```bash
+export HF_HOME=/path/to/big/disk/hf_cache
+```
+
+This prevents re-downloading and reduces load on the HF Hub.
+
+## Diagnostic command
+
+If exports fail or behave unexpectedly, run the doctor command and share the output:
+
+```bash
+tree-shift doctor
+```
+
+Output includes package versions, environment variables, and Python version — helpful for debugging "works on my machine" issues.
+
+## Manifest file
+
+Every export writes a `manifest.json` in the config output folder:
+
+```json
+{
+  "repo_id": "aadityabuilds/tree-distribution-shift",
+  "config": "intl_train_IN__ood_US",
+  "revision": "abc123def456",
+  "splits": ["train", "ood_test"],
+  "timestamp": "2026-02-17T04:20:00+00:00",
+  "counts": {
+    "train": {"images": 11783, "annotations": 181208},
+    "ood_test": {"images": 2345, "annotations": 36000}
+  }
+}
+```
+
+This is essential for reproducibility: it records the exact revision, splits exported, and counts per split. Include `manifest.json` when sharing or publishing data exports.
+
+## API reference
+
+### `list_configs(repo_id=..., revision=None) -> list[str]`
+
+Returns the list of available dataset configs from the Hub. No data is downloaded.
+
+### `export_coco(config, out_root, repo_id=..., splits=None, revision=None, streaming=True) -> Path`
+
+Exports a config to COCO folders on disk.
+
+| Parameter   | Default                          | Description                                    |
+|-------------|----------------------------------|------------------------------------------------|
+| `config`    | *(required)*                     | Name of the HF dataset config                  |
+| `out_root`  | *(required)*                     | Root output directory                           |
+| `splits`    | all 5 splits                     | Which splits to export                          |
+| `streaming` | `True`                           | Stream data (recommended for low-mem machines)  |
+| `revision`  | `None`                           | HF dataset revision / commit hash               |
+
+## Dependencies
+
+This package is intentionally lightweight:
+
+- `datasets` — HF dataset loading
+- `huggingface_hub` — Hub API access
+- `orjson` — fast JSON serialization
+- `tqdm` — progress display
+
+No `torch`, `pycocotools`, or `opencv` required.
+
+## Why this package?
+
+- **No git / git-lfs** — works on clusters where git-lfs is painful.
+- **Streaming by default** — avoids OOM and memory-map errors.
+- **Resume-safe** — re-run after interruption without re-downloading.
+- **Uses HF cache** — respects revisions, reproducible.
+- **Thin client** — new configs on the Hub are visible immediately via `list_configs()` (no pip update needed).
+
+## Versioning policy
+
+The pip package contains tools only. Data versions are pinned with `--revision`.
+
+- **0.1.x** — exporter stability, backward compatible CLI/API.
+- **Minor bump** — when on-disk COCO layout or CLI flags change.
+- **Major bump** — if split semantics change (unlikely).
+
+New configs on the HF Hub are visible immediately via `tree-shift list`; no pip update required to access new data.
+
+## Maintenance
+
+- **Adding new configs/data** → push to the HF dataset repo only.
+- **Package updates** → only needed for new CLI flags or exporter behavior changes.
+- **Users see new configs automatically** via `tree-shift list`.
+
+## Citation
+
+If you use this dataset in research, please cite:
+
+```bibtex
+@misc{nalawade2025tree_distribution_shift,
+  title={Tree Distribution Shift: A Benchmark for Out-of-Distribution Tree Detection},
+  author={Nalawade, Aaditya},
+  year={2025},
+  howpublished={\url{https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift}}
+}
+```
+
+## License
+
+MIT

tree_distribution_shift-0.1.0/README.md

@@ -0,0 +1,265 @@
+# tree-distribution-shift
+
+A pip-installable benchmark for **Tree Distribution Shift** — export HuggingFace dataset configs to COCO format on disk.
+
+**No git. No git-lfs. Works on clusters.**
+
+Data lives on the HF Hub: [`aadityabuilds/tree-distribution-shift`](https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift)
+
+## Getting started
+
+```bash
+pip install -U pip
+pip install tree-distribution-shift
+
+# See all available configs (queries HF Hub live)
+tree-shift list
+
+# Export a config to COCO format
+tree-shift export --config intl_train_IN__ood_US --out ./data
+```
+
+That's it. No cloning, no LFS, no large downloads up front — data streams directly from the Hub.
+
+### Authentication (recommended)
+
+Set an HF token for higher rate limits and faster downloads:
+
+```bash
+export HF_TOKEN=hf_...   # recommended
+```
+
+Or log in once with the CLI:
+
+```bash
+huggingface-cli login
+```
+
+You can also pass a token per-command:
+
+```bash
+tree-shift list --token hf_...
+tree-shift export --config intl_train_IN__ood_US --out ./data --token hf_...
+```
+
+### Export only the splits you need
+
+```bash
+tree-shift export \
+  --config in_state_train_Karnataka__ood_Rajasthan \
+  --out ./data \
+  --splits train ood_test
+```
+
+### Pin a revision for reproducibility
+
+For papers and exact replication, pin a commit hash:
+
+```bash
+tree-shift export \
+  --config in_state_train_Karnataka__ood_Rajasthan \
+  --out ./data \
+  --revision abc123def456
+```
+
+### Python API
+
+```python
+from tree_shift import list_configs, export_coco
+
+print(list_configs())
+
+export_coco(
+    config="in_state_train_Karnataka__ood_Rajasthan",
+    out_root="./data",
+    splits=["train", "id_test", "ood_test"],
+    streaming=True,   # default; avoids OOM on low-mem machines
+    revision=None,     # or pin a commit hash
+)
+```
+
+## Output structure
+
+```
+./data/<config>/
+  train/
+    images/
+      *.png
+    annotations/
+      instances_train.json
+  id_test/
+    images/
+    annotations/
+  ood_test/
+    images/
+    annotations/
+  ood_same_density/
+    images/
+    annotations/
+  ood_diff_density/
+    images/
+    annotations/
+```
+
+Annotation files follow the [COCO format](https://cocodataset.org/#format-data).
+
+## Dataset contract
+
+Every example row in the HF dataset contains:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `image_id` | int | Unique image identifier |
+| `filename` | str | Image filename (e.g. `00001.png`) |
+| `width` | int | Image width in pixels |
+| `height` | int | Image height in pixels |
+| `image_bytes` | bytes | Raw image file contents |
+| `coco_annotations` | str | JSON string — list of COCO annotation dicts |
+| `coco_categories` | str | JSON string — list of COCO category dicts |
+| `country` | str | Country code (e.g. `IN`, `US`) |
+| `state` | str | State name |
+| `zone` | str | Geographic zone |
+| `biome` | str | Biome classification |
+| `density_category` | str | Tree density bucket |
+
+### Expected splits
+
+- **`train`** — training set
+- **`id_test`** — in-distribution test set
+- **`ood_test`** — out-of-distribution test set
+- **`ood_same_density`** — OOD test filtered to same density bucket as train
+- **`ood_diff_density`** — OOD test filtered to different density bucket
+
+> **Note:** Some older configs may not include the `ood_same_density` / `ood_diff_density` splits.
+
+## Resume support
+
+Exports are **resume-safe**. If a job is preempted or interrupted:
+
+```bash
+# Just re-run the same command — already-written images are skipped
+tree-shift export --config intl_train_IN__ood_US --out ./data
+```
+
+Images that already exist on disk are not re-downloaded. The COCO annotation JSON is always regenerated to stay consistent with the full split.
+
+## Rate limits & performance
+
+- **Streaming is on by default** — data is fetched shard-by-shard, avoiding OOM and memory-map errors.
+- **`--no-streaming`** downloads the full split into an Arrow table first. This can OOM on large splits or cause `mmap` errors on low-memory machines. Use only if you have plenty of RAM.
+- For faster downloads (when available), set the environment variable:
+  ```bash
+  export HF_HUB_ENABLE_HF_TRANSFER=1
+  ```
+- Exports call `load_dataset` once per split (not per example), keeping HF API calls minimal.
+- If you hit 429 rate-limit errors, wait a few minutes and re-run — resume will skip already-written images.
+
+## Cache location (clusters/shared machines)
+
+The exporter uses the standard HuggingFace cache. On clusters with limited home directory space, set a custom cache location on a large disk:
+
+```bash
+export HF_HOME=/path/to/big/disk/hf_cache
+```
+
+This prevents re-downloading and reduces load on the HF Hub.
+
+## Diagnostic command
+
+If exports fail or behave unexpectedly, run the doctor command and share the output:
+
+```bash
+tree-shift doctor
+```
+
+Output includes package versions, environment variables, and Python version — helpful for debugging "works on my machine" issues.
+
+## Manifest file
+
+Every export writes a `manifest.json` in the config output folder:
+
+```json
+{
+  "repo_id": "aadityabuilds/tree-distribution-shift",
+  "config": "intl_train_IN__ood_US",
+  "revision": "abc123def456",
+  "splits": ["train", "ood_test"],
+  "timestamp": "2026-02-17T04:20:00+00:00",
+  "counts": {
+    "train": {"images": 11783, "annotations": 181208},
+    "ood_test": {"images": 2345, "annotations": 36000}
+  }
+}
+```
+
+This is essential for reproducibility: it records the exact revision, splits exported, and counts per split. Include `manifest.json` when sharing or publishing data exports.
+
+## API reference
+
+### `list_configs(repo_id=..., revision=None) -> list[str]`
+
+Returns the list of available dataset configs from the Hub. No data is downloaded.
+
+### `export_coco(config, out_root, repo_id=..., splits=None, revision=None, streaming=True) -> Path`
+
+Exports a config to COCO folders on disk.
+
+| Parameter   | Default                          | Description                                    |
+|-------------|----------------------------------|------------------------------------------------|
+| `config`    | *(required)*                     | Name of the HF dataset config                  |
+| `out_root`  | *(required)*                     | Root output directory                           |
+| `splits`    | all 5 splits                     | Which splits to export                          |
+| `streaming` | `True`                           | Stream data (recommended for low-mem machines)  |
+| `revision`  | `None`                           | HF dataset revision / commit hash               |
+
+## Dependencies
+
+This package is intentionally lightweight:
+
+- `datasets` — HF dataset loading
+- `huggingface_hub` — Hub API access
+- `orjson` — fast JSON serialization
+- `tqdm` — progress display
+
+No `torch`, `pycocotools`, or `opencv` required.
+
+## Why this package?
+
+- **No git / git-lfs** — works on clusters where git-lfs is painful.
+- **Streaming by default** — avoids OOM and memory-map errors.
+- **Resume-safe** — re-run after interruption without re-downloading.
+- **Uses HF cache** — respects revisions, reproducible.
+- **Thin client** — new configs on the Hub are visible immediately via `list_configs()` (no pip update needed).
+
+## Versioning policy
+
+The pip package contains tools only. Data versions are pinned with `--revision`.
+
+- **0.1.x** — exporter stability, backward compatible CLI/API.
+- **Minor bump** — when on-disk COCO layout or CLI flags change.
+- **Major bump** — if split semantics change (unlikely).
+
+New configs on the HF Hub are visible immediately via `tree-shift list`; no pip update required to access new data.
+
+## Maintenance
+
+- **Adding new configs/data** → push to the HF dataset repo only.
+- **Package updates** → only needed for new CLI flags or exporter behavior changes.
+- **Users see new configs automatically** via `tree-shift list`.
+
+## Citation
+
+If you use this dataset in research, please cite:
+
+```bibtex
+@misc{nalawade2025tree_distribution_shift,
+  title={Tree Distribution Shift: A Benchmark for Out-of-Distribution Tree Detection},
+  author={Nalawade, Aaditya},
+  year={2025},
+  howpublished={\url{https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift}}
+}
+```
+
+## License
+
+MIT

tree_distribution_shift-0.1.0/pyproject.toml

@@ -0,0 +1,30 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "tree-distribution-shift"
+version = "0.1.0"
+description = "Tree Distribution Shift benchmark: export HF configs to COCO (train/id_test/ood_test + density buckets)."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+authors = [{name = "Aaditya Nalawade"}]
+dependencies = [
+  "datasets>=2.19.0",
+  "huggingface_hub>=0.23.0",
+  "orjson>=3.9.0",
+  "tqdm>=4.66.0",
+]
+
+[project.urls]
+Homepage = "https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift"
+
+[project.scripts]
+tree-shift = "tree_shift.cli:main"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]

tree_distribution_shift-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

tree_distribution_shift-0.1.0/src/tree_distribution_shift.egg-info/PKG-INFO

@@ -0,0 +1,281 @@
+Metadata-Version: 2.4
+Name: tree-distribution-shift
+Version: 0.1.0
+Summary: Tree Distribution Shift benchmark: export HF configs to COCO (train/id_test/ood_test + density buckets).
+Author: Aaditya Nalawade
+License-Expression: MIT
+Project-URL: Homepage, https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: datasets>=2.19.0
+Requires-Dist: huggingface_hub>=0.23.0
+Requires-Dist: orjson>=3.9.0
+Requires-Dist: tqdm>=4.66.0
+Dynamic: license-file
+
+# tree-distribution-shift
+
+A pip-installable benchmark for **Tree Distribution Shift** — export HuggingFace dataset configs to COCO format on disk.
+
+**No git. No git-lfs. Works on clusters.**
+
+Data lives on the HF Hub: [`aadityabuilds/tree-distribution-shift`](https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift)
+
+## Getting started
+
+```bash
+pip install -U pip
+pip install tree-distribution-shift
+
+# See all available configs (queries HF Hub live)
+tree-shift list
+
+# Export a config to COCO format
+tree-shift export --config intl_train_IN__ood_US --out ./data
+```
+
+That's it. No cloning, no LFS, no large downloads up front — data streams directly from the Hub.
+
+### Authentication (recommended)
+
+Set an HF token for higher rate limits and faster downloads:
+
+```bash
+export HF_TOKEN=hf_...   # recommended
+```
+
+Or log in once with the CLI:
+
+```bash
+huggingface-cli login
+```
+
+You can also pass a token per-command:
+
+```bash
+tree-shift list --token hf_...
+tree-shift export --config intl_train_IN__ood_US --out ./data --token hf_...
+```
+
+### Export only the splits you need
+
+```bash
+tree-shift export \
+  --config in_state_train_Karnataka__ood_Rajasthan \
+  --out ./data \
+  --splits train ood_test
+```
+
+### Pin a revision for reproducibility
+
+For papers and exact replication, pin a commit hash:
+
+```bash
+tree-shift export \
+  --config in_state_train_Karnataka__ood_Rajasthan \
+  --out ./data \
+  --revision abc123def456
+```
+
+### Python API
+
+```python
+from tree_shift import list_configs, export_coco
+
+print(list_configs())
+
+export_coco(
+    config="in_state_train_Karnataka__ood_Rajasthan",
+    out_root="./data",
+    splits=["train", "id_test", "ood_test"],
+    streaming=True,   # default; avoids OOM on low-mem machines
+    revision=None,     # or pin a commit hash
+)
+```
+
+## Output structure
+
+```
+./data/<config>/
+  train/
+    images/
+      *.png
+    annotations/
+      instances_train.json
+  id_test/
+    images/
+    annotations/
+  ood_test/
+    images/
+    annotations/
+  ood_same_density/
+    images/
+    annotations/
+  ood_diff_density/
+    images/
+    annotations/
+```
+
+Annotation files follow the [COCO format](https://cocodataset.org/#format-data).
+
+## Dataset contract
+
+Every example row in the HF dataset contains:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `image_id` | int | Unique image identifier |
+| `filename` | str | Image filename (e.g. `00001.png`) |
+| `width` | int | Image width in pixels |
+| `height` | int | Image height in pixels |
+| `image_bytes` | bytes | Raw image file contents |
+| `coco_annotations` | str | JSON string — list of COCO annotation dicts |
+| `coco_categories` | str | JSON string — list of COCO category dicts |
+| `country` | str | Country code (e.g. `IN`, `US`) |
+| `state` | str | State name |
+| `zone` | str | Geographic zone |
+| `biome` | str | Biome classification |
+| `density_category` | str | Tree density bucket |
+
+### Expected splits
+
+- **`train`** — training set
+- **`id_test`** — in-distribution test set
+- **`ood_test`** — out-of-distribution test set
+- **`ood_same_density`** — OOD test filtered to same density bucket as train
+- **`ood_diff_density`** — OOD test filtered to different density bucket
+
+> **Note:** Some older configs may not include the `ood_same_density` / `ood_diff_density` splits.
+
+## Resume support
+
+Exports are **resume-safe**. If a job is preempted or interrupted:
+
+```bash
+# Just re-run the same command — already-written images are skipped
+tree-shift export --config intl_train_IN__ood_US --out ./data
+```
+
+Images that already exist on disk are not re-downloaded. The COCO annotation JSON is always regenerated to stay consistent with the full split.
+
+## Rate limits & performance
+
+- **Streaming is on by default** — data is fetched shard-by-shard, avoiding OOM and memory-map errors.
+- **`--no-streaming`** downloads the full split into an Arrow table first. This can OOM on large splits or cause `mmap` errors on low-memory machines. Use only if you have plenty of RAM.
+- For faster downloads (when available), set the environment variable:
+  ```bash
+  export HF_HUB_ENABLE_HF_TRANSFER=1
+  ```
+- Exports call `load_dataset` once per split (not per example), keeping HF API calls minimal.
+- If you hit 429 rate-limit errors, wait a few minutes and re-run — resume will skip already-written images.
+
+## Cache location (clusters/shared machines)
+
+The exporter uses the standard HuggingFace cache. On clusters with limited home directory space, set a custom cache location on a large disk:
+
+```bash
+export HF_HOME=/path/to/big/disk/hf_cache
+```
+
+This prevents re-downloading and reduces load on the HF Hub.
+
+## Diagnostic command
+
+If exports fail or behave unexpectedly, run the doctor command and share the output:
+
+```bash
+tree-shift doctor
+```
+
+Output includes package versions, environment variables, and Python version — helpful for debugging "works on my machine" issues.
+
+## Manifest file
+
+Every export writes a `manifest.json` in the config output folder:
+
+```json
+{
+  "repo_id": "aadityabuilds/tree-distribution-shift",
+  "config": "intl_train_IN__ood_US",
+  "revision": "abc123def456",
+  "splits": ["train", "ood_test"],
+  "timestamp": "2026-02-17T04:20:00+00:00",
+  "counts": {
+    "train": {"images": 11783, "annotations": 181208},
+    "ood_test": {"images": 2345, "annotations": 36000}
+  }
+}
+```
+
+This is essential for reproducibility: it records the exact revision, splits exported, and counts per split. Include `manifest.json` when sharing or publishing data exports.
+
+## API reference
+
+### `list_configs(repo_id=..., revision=None) -> list[str]`
+
+Returns the list of available dataset configs from the Hub. No data is downloaded.
+
+### `export_coco(config, out_root, repo_id=..., splits=None, revision=None, streaming=True) -> Path`
+
+Exports a config to COCO folders on disk.
+
+| Parameter   | Default                          | Description                                    |
+|-------------|----------------------------------|------------------------------------------------|
+| `config`    | *(required)*                     | Name of the HF dataset config                  |
+| `out_root`  | *(required)*                     | Root output directory                           |
+| `splits`    | all 5 splits                     | Which splits to export                          |
+| `streaming` | `True`                           | Stream data (recommended for low-mem machines)  |
+| `revision`  | `None`                           | HF dataset revision / commit hash               |
+
+## Dependencies
+
+This package is intentionally lightweight:
+
+- `datasets` — HF dataset loading
+- `huggingface_hub` — Hub API access
+- `orjson` — fast JSON serialization
+- `tqdm` — progress display
+
+No `torch`, `pycocotools`, or `opencv` required.
+
+## Why this package?
+
+- **No git / git-lfs** — works on clusters where git-lfs is painful.
+- **Streaming by default** — avoids OOM and memory-map errors.
+- **Resume-safe** — re-run after interruption without re-downloading.
+- **Uses HF cache** — respects revisions, reproducible.
+- **Thin client** — new configs on the Hub are visible immediately via `list_configs()` (no pip update needed).
+
+## Versioning policy
+
+The pip package contains tools only. Data versions are pinned with `--revision`.
+
+- **0.1.x** — exporter stability, backward compatible CLI/API.
+- **Minor bump** — when on-disk COCO layout or CLI flags change.
+- **Major bump** — if split semantics change (unlikely).
+
+New configs on the HF Hub are visible immediately via `tree-shift list`; no pip update required to access new data.
+
+## Maintenance
+
+- **Adding new configs/data** → push to the HF dataset repo only.
+- **Package updates** → only needed for new CLI flags or exporter behavior changes.
+- **Users see new configs automatically** via `tree-shift list`.
+
+## Citation
+
+If you use this dataset in research, please cite:
+
+```bibtex
+@misc{nalawade2025tree_distribution_shift,
+  title={Tree Distribution Shift: A Benchmark for Out-of-Distribution Tree Detection},
+  author={Nalawade, Aaditya},
+  year={2025},
+  howpublished={\url{https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift}}
+}
+```
+
+## License
+
+MIT

tree_distribution_shift-0.1.0/src/tree_distribution_shift.egg-info/SOURCES.txt

@@ -0,0 +1,14 @@
+LICENSE
+README.md
+pyproject.toml
+src/tree_distribution_shift.egg-info/PKG-INFO
+src/tree_distribution_shift.egg-info/SOURCES.txt
+src/tree_distribution_shift.egg-info/dependency_links.txt
+src/tree_distribution_shift.egg-info/entry_points.txt
+src/tree_distribution_shift.egg-info/requires.txt
+src/tree_distribution_shift.egg-info/top_level.txt
+src/tree_shift/__init__.py
+src/tree_shift/cli.py
+src/tree_shift/constants.py
+src/tree_shift/export.py
+src/tree_shift/hf.py

tree_distribution_shift-0.1.0/src/tree_distribution_shift.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

tree_distribution_shift-0.1.0/src/tree_distribution_shift.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+tree-shift = tree_shift.cli:main

tree_distribution_shift-0.1.0/src/tree_distribution_shift.egg-info/requires.txt

@@ -0,0 +1,4 @@
+datasets>=2.19.0
+huggingface_hub>=0.23.0
+orjson>=3.9.0
+tqdm>=4.66.0

tree_distribution_shift-0.1.0/src/tree_distribution_shift.egg-info/top_level.txt

@@ -0,0 +1 @@
+tree_shift

tree_distribution_shift-0.1.0/src/tree_shift/__init__.py

@@ -0,0 +1,5 @@
+from .export import export_coco
+from .hf import list_configs
+
+__all__ = ["list_configs", "export_coco"]
+__version__ = "0.1.0"

tree_distribution_shift-0.1.0/src/tree_shift/cli.py

@@ -0,0 +1,90 @@
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+
+import datasets
+import huggingface_hub
+
+from .constants import DEFAULT_SPLITS, HF_REPO_ID
+from .export import export_coco
+from .hf import list_configs
+from . import __version__
+
+
+def _setup_token(token: str | None) -> None:
+    """Authenticate with HF Hub if a token is provided, otherwise print a hint."""
+    # Suppress the noisy "unauthenticated requests" warning from huggingface_hub
+    logging.getLogger("huggingface_hub.utils._auth").setLevel(logging.ERROR)
+
+    if token:
+        huggingface_hub.login(token=token, add_to_git_credential=False)
+    elif not huggingface_hub.utils.get_token():
+        print(
+            "Tip: set HF_TOKEN for higher rate limits "
+            "(export HF_TOKEN=... or run: huggingface-cli login)",
+            file=sys.stderr,
+        )
+
+
+def _doctor() -> int:
+    """Print diagnostic information."""
+    token = huggingface_hub.utils.get_token()
+    print(f"tree-distribution-shift version: {__version__}")
+    print(f"datasets version: {datasets.__version__}")
+    print(f"huggingface_hub version: {huggingface_hub.__version__}")
+    print(f"HF_TOKEN: {'set' if token else 'not set'}")
+    print(f"HF_HUB_ENABLE_HF_TRANSFER: {os.environ.get('HF_HUB_ENABLE_HF_TRANSFER', 'not set')}")
+    print(f"HF_HOME: {os.environ.get('HF_HOME', 'not set (using default ~/.cache/huggingface)')}")
+    print(f"Default repo id: {HF_REPO_ID}")
+    print(f"Python version: {sys.version}")
+    return 0
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(prog="tree-shift")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    # ── list ──────────────────────────────────────────────────────────
+    ls = sub.add_parser("list", help="List available configs from the Hub")
+    ls.add_argument("--repo", default=HF_REPO_ID)
+    ls.add_argument("--revision", default=None)
+    ls.add_argument("--token", default=None, help="HF Hub token (or set HF_TOKEN env var)")
+
+    # ── export ────────────────────────────────────────────────────────
+    ex = sub.add_parser("export", help="Export a config to COCO on disk")
+    ex.add_argument("--repo", default=HF_REPO_ID)
+    ex.add_argument("--config", required=True)
+    ex.add_argument("--out", required=True)
+    ex.add_argument("--revision", default=None)
+    ex.add_argument("--splits", nargs="*", default=DEFAULT_SPLITS)
+    ex.add_argument("--no-streaming", action="store_true")
+    ex.add_argument("--token", default=None, help="HF Hub token (or set HF_TOKEN env var)")
+
+    # ── doctor ───────────────────────────────────────────────────────
+    sub.add_parser("doctor", help="Print diagnostic info for debugging")
+
+    args = ap.parse_args()
+
+    if args.cmd == "doctor":
+        return _doctor()
+
+    _setup_token(getattr(args, "token", None))
+
+    if args.cmd == "list":
+        for c in list_configs(repo_id=args.repo, revision=args.revision):
+            print(c)
+        return
+
+    if args.cmd == "export":
+        export_coco(
+            repo_id=args.repo,
+            config=args.config,
+            out_root=args.out,
+            splits=args.splits,
+            revision=args.revision,
+            streaming=(not args.no_streaming),
+        )
+        return

tree_distribution_shift-0.1.0/src/tree_shift/constants.py

@@ -0,0 +1,2 @@
+HF_REPO_ID = "aadityabuilds/tree-distribution-shift"
+DEFAULT_SPLITS = ["train", "id_test", "ood_test", "ood_same_density", "ood_diff_density"]

tree_distribution_shift-0.1.0/src/tree_shift/export.py

@@ -0,0 +1,139 @@
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Dict, Iterable, List, Optional, Union
+
+import orjson
+from datasets import load_dataset
+from tqdm import tqdm
+
+from .constants import DEFAULT_SPLITS, HF_REPO_ID
+
+
+class _SplitStats:
+    """Track statistics for a single split export."""
+    def __init__(self):
+        self.images = 0
+        self.skipped = 0
+        self.annotations = 0
+
+
+def _export_split_iter(ds: Iterable[dict], out_dir: Path, split_name: str) -> _SplitStats:
+    """Stream a single split to COCO format on disk.
+
+    Resume-safe: images that already exist on disk are skipped.
+    The COCO annotation JSON is always regenerated so it stays
+    consistent with the full set of images.
+    """
+    images_dir = out_dir / split_name / "images"
+    ann_dir = out_dir / split_name / "annotations"
+    images_dir.mkdir(parents=True, exist_ok=True)
+    ann_dir.mkdir(parents=True, exist_ok=True)
+
+    images: list[dict] = []
+    annotations: list[dict] = []
+    categories: list[dict] | None = None
+    ann_id = 1
+
+    stats = _SplitStats()
+
+    for ex in ds:
+        stats.images += 1
+        if stats.images % 500 == 0:
+            tqdm.write(f"[{split_name}] processed {stats.images} images ({stats.skipped} skipped)...")
+
+        image_id = int(ex["image_id"])
+        filename = ex["filename"]
+        width = int(ex["width"])
+        height = int(ex["height"])
+
+        # Resume-safe: skip writing if the image already exists on disk
+        img_path = images_dir / filename
+        if img_path.exists():
+            stats.skipped += 1
+        else:
+            img_path.write_bytes(ex["image_bytes"])
+
+        images.append(
+            {"id": image_id, "file_name": filename, "width": width, "height": height}
+        )
+
+        annos = orjson.loads(ex["coco_annotations"].encode("utf-8"))
+        if categories is None:
+            categories = orjson.loads(ex["coco_categories"].encode("utf-8"))
+
+        for a in annos:
+            a = dict(a)
+            a["id"] = ann_id
+            ann_id += 1
+            annotations.append(a)
+
+    stats.annotations = len(annotations)
+
+    coco = {
+        "images": images,
+        "annotations": annotations,
+        "categories": categories or [],
+    }
+    (ann_dir / f"instances_{split_name}.json").write_bytes(orjson.dumps(coco))
+    tqdm.write(
+        f"[{split_name}] done — {stats.images} images ({stats.skipped} skipped), "
+        f"{stats.annotations} annotations"
+    )
+
+    return stats
+
+
+def export_coco(
+    config: str,
+    out_root: Union[str, Path],
+    repo_id: str = HF_REPO_ID,
+    splits: Optional[List[str]] = None,
+    revision: Optional[str] = None,
+    streaming: bool = True,
+) -> Path:
+    """
+    Export a HF config to COCO folders.
+
+    Writes to: <out_root>/<config>/<split>/{images, annotations}
+    Also writes <out_root>/<config>/manifest.json with export metadata.
+    """
+    out_root = Path(out_root)
+    out_dir = out_root / config
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    if splits is None:
+        splits = DEFAULT_SPLITS
+
+    split_counts: Dict[str, Any] = {}
+
+    for split in splits:
+        tqdm.write(f"Exporting split '{split}' for config '{config}' ...")
+        ds = load_dataset(
+            repo_id,
+            config,
+            split=split,
+            streaming=streaming,
+            revision=revision,
+        )
+        stats = _export_split_iter(ds, out_dir, split)
+        split_counts[split] = {
+            "images": stats.images,
+            "annotations": stats.annotations,
+        }
+
+    # Write manifest.json
+    manifest = {
+        "repo_id": repo_id,
+        "config": config,
+        "revision": revision,
+        "splits": splits,
+        "timestamp": datetime.now(timezone.utc).isoformat(),
+        "counts": split_counts,
+    }
+    manifest_path = out_dir / "manifest.json"
+    manifest_path.write_bytes(orjson.dumps(manifest, option=orjson.OPT_INDENT_2))
+    tqdm.write(f"Manifest written to {manifest_path}")
+
+    return out_dir

tree_distribution_shift-0.1.0/src/tree_shift/hf.py

@@ -0,0 +1,14 @@
+from __future__ import annotations
+
+from typing import List, Optional
+
+from datasets import get_dataset_config_names
+
+from .constants import HF_REPO_ID
+
+
+def list_configs(
+    repo_id: str = HF_REPO_ID, revision: Optional[str] = None
+) -> List[str]:
+    """List available configs from the HF dataset repo (no data download)."""
+    return get_dataset_config_names(repo_id, revision=revision)