usergenerated 2.0.0__tar.gz

usergenerated-2.0.0/PKG-INFO

@@ -0,0 +1,951 @@
+Metadata-Version: 2.4
+Name: usergenerated
+Version: 2.0.0
+Summary: Utilities to prepare User Generated collections for DestinE Data Lake STAC workflows.
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pystac[orjson,validation]
+Requires-Dist: shapely
+Requires-Dist: python-magic
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+Requires-Dist: netCDF4
+Requires-Dist: boto3
+Requires-Dist: botocore
+Requires-Dist: python-dotenv
+Requires-Dist: tqdm
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: pip-tools; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Provides-Extra: collection-tests
+Requires-Dist: tqdm; extra == "collection-tests"
+Requires-Dist: destinelab; extra == "collection-tests"
+
+# User Generated Data Process
+
+Table of Contents:
+1. [Overview](#overview)
+2. [User Workflow](#user-workflow)
+3. [Required Structure](#required-structure)
+4. [Metadata Files](#metadata-files)
+   - [collection.json](#collectionjson)
+   - [collection_config.json](#collection_configjson)
+5. [Generate Item Metadata](#generate-item-metadata)
+6. [Regenerate Item Metadata in S3](#regenerate-item-metadata-in-s3)
+7. [Maintainer Notes](#maintainer-notes)
+8. [Optional S3 Upload](#optional-s3-upload)
+9. [Runtime Initialization](#runtime-initialization)
+10. [Troubleshooting](#troubleshooting)
+11. [External References](#external-references)
+
+
+
+This project helps you prepare a User Generated collection for DestinE Data Lake review.
+
+Use this guide to:
+1. Structure your collection folders and data assets.
+2. Configure metadata files.
+3. Generate STAC Item metadata.
+
+For contributor and maintenance documentation, see [docs/development.md](docs/development.md).
+
+## User Workflow
+
+1. Create your collection root folder using the provided collection id, for example `EO.XXX.YYY.ZZZ`.
+2. If needed, place the collection in a subfolder under your current working directory and pass that subfolder with `--collection-path`.
+3. Add metadata files under `metadata/`.
+4. Organize data assets under `data/` according to your configured folder level.
+5. Run [generate_item_metadata.py](generate_item_metadata.py) to validate and generate STAC Items under `metadata/items/`.
+
+If you mirror a collection from S3 before regenerating items, mirrored data folders may contain `usergenerated_s3_folder_manifest.jsonl` files instead of placeholder dummy assets. Item generation detects these manifests automatically and uses their entries as the authoritative asset list.
+
+Reference collection structure is available in [EO.XXX.YYY.ZZZ](EO.XXX.YYY.ZZZ).
+
+## Required Structure
+
+Expected top-level structure:
+
+```text
+EO.XXX.YYY.ZZZ/
+  data/
+  metadata/
+    collection.json
+    collection_config.json
+    items/
+```
+
+Notes:
+1. Collection ids are case-sensitive and typically uppercase with `.` separators.
+2. The `metadata/items/` folder is where generated Item JSON files are written.
+
+## Metadata Files
+
+### collection.json
+
+#### How to Complete `collection.json`
+
+Use `metadata/collection.json` as the authoritative STAC Collection metadata file for your dataset.
+
+---
+
+#### Required STAC Collection fields
+
+The following fields **must** be completed and valid:
+
+1. `type` — Must be set to `Collection`.
+2. `stac_version` — Use the current version (`1.1.0`).
+3. `id` — Your collection identifier. This must match:
+   - your root folder name  
+   - `collection_config.json > id`
+4. `title` — A human-readable name for your collection.
+5. `description` — A detailed description of the dataset.
+6. `dedl:short_description` — A concise version of the description. Displayed in the DestinE DataLake UI.
+7. `license` — The license under which the data is distributed.
+8. `extent` — Must include both spatial and temporal components.
+9. `links` — Must be present as an array.
+10. `assets` — Include if you have collection-level assets; otherwise use `{}`.
+
+---
+
+#### Recommended best practices
+
+- Keep the `id` consistent across:
+  - the collection root folder name  
+  - `collection_config.json > id`
+- Use a clear `title` and a meaningful `description` that explains:
+  - dataset content  
+  - data source(s)  
+  - intended use cases
+- Ensure `extent` is accurate so item-level metadata is interpreted correctly.
+- Only update necessary `links` (typically `describedby` and `cite-as`).
+- Keep `type` and `stac_version` unchanged unless upgrading STAC versions.
+
+---
+
+#### Field-specific guidance
+
+##### `links`
+- Always include at least:
+  - `license` → link to license terms
+  - `cite-as` → DOI or citation reference
+  - `describedby` → external dataset documentation
+- Ensure links are stable and publicly accessible.
+- Use meaningful `title` values so users understand what each link represents.
+
+---
+
+##### `extent`
+- **Spatial (`bbox`)**:
+  - Format: `[west, south, east, north]`
+  - Use WGS84 coordinates (EPSG:4326).
+  - Ensure the bounding box fully covers your dataset.
+
+- **Temporal (`interval`)**:
+  - Use ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`).
+  - Use:
+    - a closed interval `[start, end]` for fixed datasets
+    - `[start, null]` for ongoing datasets
+
+---
+
+##### `license`
+- Prefer SPDX identifiers (e.g. `CC-BY-4.0`, `MIT`, `Apache-2.0`).
+- Use:
+  - `proprietary` for restricted datasets
+  - `various` if multiple licenses apply
+- If using `proprietary` or `various`:
+  - explain clearly in `description`
+  - provide license links in `links`
+
+---
+
+##### `providers`
+- List all organizations involved in the dataset lifecycle.
+- Common roles:
+  - `producer` → created the data
+  - `processor` → processed or transformed the data
+  - `licensor` → owns or distributes the data rights
+  - `host` → (optional) hosting platform
+- Provide:
+  - `name` (required)
+  - `url` (recommended)
+- Include multiple providers if applicable rather than combining roles into one.
+
+---
+
+##### `sci:publications`
+- Use this field to reference scientific publications or datasets.
+- Include:
+  - `sci:doi` → DOI identifier
+  - `sci:citation` → human-readable citation
+- Add multiple entries if there are several related publications.
+- Ensure DOIs resolve correctly (test the links).
+
+---
+
+##### `keywords`
+- Provide relevant, searchable terms describing the dataset.
+- Use:
+  - domain-specific terminology (e.g. *Land Cover*, *Atmospheric Data*)
+  - common synonyms where helpful
+- Avoid overly generic terms (e.g. “data”, “analysis”).
+
+---
+
+##### `assets`
+- Use for collection-level resources (not item-level data).
+- Common examples:
+  - `thumbnail` → preview image
+  - `overview` → documentation PDF
+- Include:
+  - `href` (URL)
+  - `type` (MIME type, e.g. `image/png`)
+  - `roles` (e.g. `thumbnail`, `overview`)
+- If no assets are needed, use an empty object:
+  ```json
+  "assets": {}
+
+
+### collection_config.json
+
+This controls how Item metadata generation behaves. Base example in [EO.XXX.YYY.ZZZ/metadata/collection_config.json](EO.XXX.YYY.ZZZ/metadata/collection_config.json).
+
+## Configuration Options
+
+The following options are user-facing and supported during item generation:
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `id` | string | required | Collection id. Must match your collection root folder id. |
+| `item_folder_level` | string | `DD` | Where item folders are discovered. Allowed values: `YYYY`, `MM`, `DD`, `NONE`. |
+| `item_config_optional` | boolean | `false` | If `false`, each item folder is expected to contain `item_config.json`. |
+| `item_asset_ignore_list` | list[string] | `[]` | File names to ignore when creating STAC assets. |
+| `media_type_overrides` | object | `{}` | Optional extension to MIME type map used before default detection, e.g. `{ ".grib2": "application/x-grib2" }`. |
+| `role_overrides` | object | `{}` | Optional MIME type to role map. Overrides default `data`/`metadata` classification. Allowed roles: `data`, `metadata`, `thumbnail`, `overview`. E.g. `{ "application/x-grib2": "data" }`. File-name regex patterns (`thumbnail_regex`, `overview_regex`) still take priority. |
+| `thumbnail_regex` | string | unset | Regex used to detect thumbnail assets and assign thumbnail role. |
+| `overview_regex` | string | unset | Regex used to detect overview assets and assign overview role. |
+| `additional_property_keys` | list[string] | `[]` | Additional property suffix keys parsed from item folder names (standard mode). |
+| `item_folder_naming_convention_type` | string | `standard` | Naming convention mode. Allowed values: `standard`, `non_standard_1`. |
+| `properties` | object | `{}` | Additional properties merged into each generated Item. |
+| `bbox` | list[number] | unset | Bounding box `[minx, miny, maxx, maxy]` applied to generated items. |
+| `item_date_overide` | string | unset | Simplified mode override date in `YYYYMMDD` format. |
+
+Important: the accepted key name is `item_date_overide` (spelling as shown).
+
+## Folder Granularity Options
+
+Set `item_folder_level` in `collection_config.json`.
+
+1. `DD`: item folders are expected under `data/YYYY/MM/DD/<item_folder>`.
+2. `MM`: item folders are expected under `data/YYYY/MM/<item_folder>`.
+3. `YYYY`: item folders are expected under `data/YYYY/<item_folder>`.
+4. `NONE`: simplified mode. Any folder directly under `data/` becomes an Item.
+
+## Standard Mode Example
+
+```json
+{
+  "id": "EO.XXX.YYY.ZZZ",
+  "item_asset_ignore_list": ["item_config.json"],
+  "media_type_overrides": {
+    ".grib2": "application/x-grib2"
+  },
+  "item_config_optional": false,
+  "item_folder_level": "DD",
+  "thumbnail_regex": "^thumbnail",
+  "overview_regex": "^overview",
+  "additional_property_keys": []
+}
+```
+
+## Simplified Mode Example
+
+Simplified mode should only be used with agreement from support.
+
+```json
+{
+  "id": "EO.XXX.YYY.ZZZ",
+  "item_asset_ignore_list": ["item_config.json"],
+  "item_folder_level": "NONE",
+  "item_date_overide": "20240101",
+  "thumbnail_regex": "^thumbnail",
+  "overview_regex": "^overview",
+  "additional_property_keys": [],
+  "bbox": [-10.0, 35.0, 10.0, 60.0]
+}
+```
+
+Simplified mode datetime behavior:
+1. If `item_date_overide` is set, that date is used for all generated Items.
+2. Otherwise, if the first folder level under `data/` is a year, `YYYY-01-01` is used.
+3. Otherwise, the generation date is used.
+
+## Per-Item Configuration
+
+Each item folder can include an `item_config.json` file. Typical per-item overrides:
+1. `bbox`
+2. `properties`
+3. Asset-related regex fields when needed
+
+When present, item-level values override collection-level values.
+
+## Generate Item Metadata
+
+### CLI Usage
+
+Primary command (after installing the package):
+
+```bash
+generate-item-metadata EO.XXX.YYY.ZZZ
+```
+
+This default expects the collection at `./EO.XXX.YYY.ZZZ`.
+
+Optional collection subfolder path:
+
+```bash
+generate-item-metadata EO.XXX.YYY.ZZZ --collection-path toto
+```
+
+With this option, the collection is resolved at `./toto/EO.XXX.YYY.ZZZ`.
+For safety, `--collection-path` is currently restricted to subfolders under the current working directory.
+
+Optional bucket override:
+
+```bash
+generate-item-metadata EO.XXX.YYY.ZZZ --bucket-name usergenerated-proposal-eo.xxx.yyy.zzz
+```
+
+Compatibility command (run from the project root):
+
+```bash
+python generate_item_metadata.py EO.XXX.YYY.ZZZ
+```
+
+Compatibility command with collection subfolder:
+
+```bash
+python generate_item_metadata.py EO.XXX.YYY.ZZZ --collection-path toto
+```
+
+### Programmatic Usage
+
+You can also use `ItemGenerator` directly in your Python scripts:
+
+```python
+from usergenerated import ItemGenerator
+
+# Create a generator for your collection
+generator = ItemGenerator("EO.XXX.YYY.ZZZ")
+
+# Generate Item metadata
+generator.run()
+
+# Resolve collection from ./toto/EO.XXX.YYY.ZZZ
+generator_with_path = ItemGenerator("EO.XXX.YYY.ZZZ", collection_path="toto")
+generator_with_path.run()
+
+# With optional bucket name override
+generator_with_bucket = ItemGenerator("EO.XXX.YYY.ZZZ", override_bucket_name="custom-bucket")
+generator_with_bucket.run()
+
+# Also write a reordered/normalised copy to output/collection_reordered.json (off by default)
+generator_reordered = ItemGenerator("EO.XXX.YYY.ZZZ", save_reordered_collection=True)
+generator_reordered.run()
+```
+
+Generated output is written to:
+
+```text
+<COLLECTION_ID>/metadata/items/*.json
+```
+
+The `ItemGenerator` class is the primary API for programmatic use. The CLI command is a thin wrapper around this class.
+
+## Regenerate Item Metadata in S3
+
+Use `ItemGeneratorS3` to regenerate STAC Item metadata from bucket data while restricting updates to Item metadata only. This is useful when you need to update item generation logic or fix generated metadata without modifying data files in S3.
+
+### CLI Usage
+
+Primary command (regenerate items in S3 for your collection):
+
+```bash
+regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ
+```
+
+This command:
+1. Mirrors collection data from S3 to a local folder.
+2. Regenerates STAC Item metadata locally with your configured settings.
+3. Backs up existing S3 items by default.
+4. Replaces only the `metadata/items` prefix in S3 (never modifies data files).
+
+Optional dry-run (validate without modifying S3):
+
+```bash
+regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --dry-run
+```
+
+Optional bucket name override:
+
+```bash
+regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --bucket-name custom-bucket-name
+```
+
+Optional local mirror folder (default: `tmp_structure_mirror`):
+
+```bash
+regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --local-root "./my_mirror"
+```
+
+Disable backup:
+
+```bash
+regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --no-backup-items
+```
+
+Skip the `ready_for_transfer` safety check (see [Safety Behavior](#safety-behavior); enabled by default):
+
+```bash
+regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --no-use-ready-for-transfer-flag
+```
+
+Demonstration script usage (env-first configuration via `.env`/environment variables, see
+`script_s3_regenerate_item_metadata.py` for the full list of `S3_*` variables):
+
+```bash
+python script_s3_regenerate_item_metadata.py
+```
+
+This script is intentionally a thin example of programmatic `ItemGeneratorS3` usage.
+For routine operations, prefer the `regenerate-item-metadata-s3` console command.
+
+The demo script additionally supports `USE_READY_FOR_TRANSFER_FLAG` (or the `use_ready_for_transfer_flag`
+keyword argument to `run_regenerate_item_metadata`) to toggle the `ready_for_transfer` safety check described
+below — see [Safety Behavior](#safety-behavior). The primary console command exposes the same toggle via
+`--use-ready-for-transfer-flag`/`--no-use-ready-for-transfer-flag` (shown above).
+
+### Programmatic Usage
+
+You can also use `ItemGeneratorS3` directly in your Python scripts:
+
+```python
+from usergenerated import ItemGeneratorS3
+
+# Create a generator for your collection
+generator = ItemGeneratorS3(
+    "EO.XXX.YYY.ZZZ",
+    dry_run=True,  # Validate without modifying S3
+)
+
+# Regenerate Item metadata
+generator.run()
+
+# With explicit bucket name
+generator_with_bucket = ItemGeneratorS3(
+    "EO.XXX.YYY.ZZZ",
+    bucket_name="custom-bucket",
+    backup_items=True,
+)
+generator_with_bucket.run()
+
+# Skip the ready_for_transfer safety check (see Safety Behavior below); enabled by default
+generator_without_gate = ItemGeneratorS3(
+    "EO.XXX.YYY.ZZZ",
+    use_ready_for_transfer_flag=False,
+)
+generator_without_gate.run()
+```
+
+### Safety Behavior
+
+The regeneration workflow includes several safety measures:
+
+1. Validates credentials and bucket structure before processing.
+2. Mirrors S3 content locally and ensures local processing under `<collection_id>/`.
+3. By default (`use_ready_for_transfer_flag=True`), requires `"ready_for_transfer": true` in the mirrored
+   collection's `metadata/collection_config.json`; aborts with an error if the field is missing or `false` (this
+   check applies even with `--dry-run`). Skip this check via `--no-use-ready-for-transfer-flag` on the CLI, the
+   `use_ready_for_transfer_flag=False` constructor argument, or `USE_READY_FOR_TRANSFER_FLAG=false` for the demo
+   script.
+4. Regenerates Item metadata locally with existing generation logic.
+5. Backs up existing `<collection_id>/metadata/items` by default (can be disabled).
+6. Replaces only `<collection_id>/metadata/items` in S3 (never modifies `data/`).
+7. With `--dry-run`, performs local validation and regeneration, then prints planned S3 actions without modifying the bucket.
+
+### Mirroring Behavior
+
+The mirroring process downloads real files selectively to improve performance:
+
+1. Downloads real files only for `metadata/*` and `item_config.json` keys.
+2. Writes per-folder manifest files (`usergenerated_s3_folder_manifest.jsonl`) for discovered objects.
+3. Uses manifest entries during item generation so full non-metadata assets are not required locally.
+4. Supports optional placeholder file creation for non-fetched objects (disabled in the S3 regeneration workflow).
+5. Logs a summary with discovered, downloaded, and failed object counts, plus periodic progress logs during large scans.
+
+## Maintainer Notes
+
+Recent internal refactoring improved testability and packaging workflows.
+
+1. The primary command is now `generate-item-metadata <COLLECTION_ID>`.
+2. [generate_item_metadata.py](generate_item_metadata.py) is retained as a compatibility wrapper.
+3. Internal logic was modularized into reusable helpers under `usergenerated/` for:
+  - collection path resolution
+  - item folder discovery by configured granularity
+  - item config precedence/loading
+  - simplified and hierarchy-based datetime resolution
+4. This split enables unit-heavy testing of pure logic while keeping entry-point orchestration thin.
+
+## Optional S3 Upload
+
+Automatic upload is configured at runtime via `ItemGenerator` constructor parameters
+or CLI flags (not via module constants).
+
+Programmatic example:
+
+```python
+from usergenerated import ItemGenerator
+
+generator = ItemGenerator(
+    "EO.XXX.YYY.ZZZ",
+    is_upload_s3=True,
+    s3_endpoint_url="https://s3.central.data.destination-earth.eu",
+    s3_user_generated_bucket_prefix="usergenerated-proposal",
+)
+generator.run()
+```
+
+CLI example:
+
+```bash
+generate-item-metadata EO.XXX.YYY.ZZZ \
+  --upload-s3 \
+  --s3-endpoint-url "https://s3.central.data.destination-earth.eu" \
+  --s3-bucket-prefix "usergenerated-proposal"
+```
+
+If upload is enabled, create a `.env` file at the project root with credentials:
+
+```bash
+AWS_ACCESS_KEY_ID="<your_access_key_id>"
+AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"
+```
+
+For manual upload workflows, see [README-s3cmd.md](README-s3cmd.md).
+
+S3 utility wrappers are available for programmatic execution around `S3Tools`:
+
+```python
+from script_s3_create_bucket import run_create_bucket
+from script_s3_move_bucket_contents import run_move_bucket_contents
+from script_s3_upload_file_to_bucket import run_upload_file
+from script_s3_upload_folder_to_bucket import run_upload_folder
+
+run_create_bucket(
+  aws_access_key_id="<key>",
+  aws_secret_access_key="<secret>",
+  s3_bucket_name="my-bucket",
+)
+
+run_upload_file(
+  file_name="EO.XXX.YYY.ZZZ/metadata/collection.json",
+  object_name="EO.XXX.YYY.ZZZ/metadata/collection.json",
+  aws_access_key_id="<key>",
+  aws_secret_access_key="<secret>",
+  s3_bucket_name="my-bucket",
+)
+
+run_upload_folder(
+  folder_path="EO.XXX.YYY.ZZZ/metadata/items",
+  target_path="EO.XXX.YYY.ZZZ/metadata/items",
+  aws_access_key_id="<key>",
+  aws_secret_access_key="<secret>",
+  s3_bucket_name="my-bucket",
+)
+
+run_move_bucket_contents(
+  aws_access_key_id="<key>",
+  aws_secret_access_key="<secret>",
+  s3_bucket_name="my-bucket",
+  s3_target_prefix="archive",
+)
+```
+
+### S3 Folder Upload Progress
+
+When using `run_upload_folder(...)` or `S3Tools.upload_folder_to_s3(...)`, uploads now display progress bars:
+
+1. A global `Total files` bar showing overall folder upload progress.
+2. Per-file bars (bytes transferred) for active uploads.
+
+Current behavior:
+
+1. Uploads run concurrently with a bounded worker pool.
+2. Per-file progress bars are capped to a visible range and positions are reused for large uploads.
+3. `usergenerated_s3_folder_manifest.jsonl` files and symlinks are skipped from upload discovery.
+4. A final summary logs discovered, uploaded, and failed file counts.
+
+These wrappers accept explicit parameters first and only fall back to environment variables when a parameter is omitted.
+
+**Regenerate Item Metadata**: Use the CLI command for routine operations (see [Regenerate Item Metadata in S3](#regenerate-item-metadata-in-s3)):
+
+```bash
+regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --dry-run
+```
+
+The `script_s3_regenerate_item_metadata.py` file is kept as a small demonstrator of `ItemGeneratorS3` programmatic usage. Use the CLI command or the `ItemGeneratorS3` class directly for real workflows.
+
+### Copy a Collection Between Buckets
+
+`script_s3_copy_source_bucket_collection_to_target_bucket_collection.py` copies a collection
+folder from a source S3 bucket/account to a target S3 bucket/account, using
+`usergenerated.s3.s3_collection_copier.CollectionCopier`. It:
+
+- Detects the source bucket's `BucketLayout` (`COLLECTION_ROOT` or `FLAT_ROOT`) and normalizes
+  the target to always be `<collection_id>/...` (`COLLECTION_ROOT`), fixing up `FLAT_ROOT`
+  sources in the process.
+- Never modifies the source bucket.
+- Is safely re-runnable: objects already present in the target (by key) are skipped, so an
+  interrupted transfer can just be re-run to continue where it left off.
+- Streams each object directly from source to target (no local disk buffering), since source and
+  target buckets are assumed to use different credentials/accounts.
+
+Configure both buckets in `.env` (the source variables are the same ones used elsewhere in this
+project; target variables are `TARGET_`-prefixed):
+
+```bash
+# Source bucket (existing variables, unchanged)
+S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
+S3_BUCKET_NAME="my-source-bucket"
+AWS_ACCESS_KEY_ID="<source_access_key_id>"
+AWS_SECRET_ACCESS_KEY="<source_secret_access_key>"
+
+# Target bucket
+TARGET_S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
+TARGET_S3_BUCKET_NAME="my-target-bucket"
+TARGET_AWS_ACCESS_KEY_ID="<target_access_key_id>"
+TARGET_AWS_SECRET_ACCESS_KEY="<target_secret_access_key>"
+
+S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
+S3_FRESH_START="false"  # set true to delete the target's existing '<collection_id>/' prefix first
+S3_DRY_RUN="false"      # set true to preview the copy plan without modifying the target
+```
+
+Then run:
+
+```bash
+python script_s3_copy_source_bucket_collection_to_target_bucket_collection.py
+```
+
+`S3_FRESH_START` is the only way to trigger the destructive "clean the target path first" step
+(e.g. for a first run against a target that already has stray/partial contents); it is never
+triggered automatically. Omit it (or leave it `false`) for the default incremental,
+resume-friendly sync.
+
+Folder uploads use object keys relative to the selected local folder. For example,
+uploading `./EO.XXX.YYY.ZZZ/metadata/items` with no `target_path` uploads files like
+`a.json` and `nested/b.json` at the bucket root. To place files under a collection or
+metadata prefix in the bucket, pass that prefix explicitly as `target_path`.
+
+### Verify a Collection Copy
+
+`script_s3_verify_collection_copy.py` is a read-only smoke test to run right after the copy
+script above completes: it uses `usergenerated.s3.s3_collection_copy_verifier.CollectionCopyVerifier`
+to list the collection's objects in both buckets and confirms the target has exactly the same
+relative keys as the source, with matching sizes for every key. Neither bucket is modified.
+
+It reuses the same `.env` variables as the copy script above (no new variables needed):
+
+```bash
+python script_s3_verify_collection_copy.py
+```
+
+The script exits `0` and prints `Collection copy verification passed: True` when the target
+matches the source. On a mismatch `CollectionCopyVerifier.run()` raises `S3OperationError`
+whose message summarises the missing, unexpected, and/or size-mismatched keys (sampled, up to
+20 per category); the script reports that one line and exits `1`.
+
+Similarly, `CollectionCopier.run()` (the copy script above) raises `S3OperationError` — naming
+the failed-object count and a sample of failed keys — if any object fails to copy. Objects
+that did copy are left in place, so re-running resumes from where it stopped.
+
+S3 helper code lives under `usergenerated/s3/`.
+
+### Promote a Collection (regenerate → copy → verify → read, in one call)
+
+`usergenerated.s3.collection_promoter.promote_collection` (also
+`from usergenerated import promote_collection`) runs the whole "a collection has been staged
+in a source bucket; validate it, copy it to a target bucket, and hand me its items" workflow
+as a single call, in strict order:
+
+1. `ItemGeneratorS3(...).run()` — regenerate STAC item metadata **in the source bucket**
+   (source credentials; the `ready_for_transfer` safety gate stays on; `data/` is untouched).
+2. `CollectionCopier(...).run()` — copy the collection source bucket → target bucket.
+3. `CollectionCopyVerifier(...).run()` — confirm the target matches the source key-for-key.
+4. Build a `PromotedCollection` read accessor over the **target** copy and call
+   `on_ready(handle)` **exactly once**.
+
+`on_ready` is a callback you provide. Its argument exposes just two methods:
+
+- `handle.read_collection() -> dict` — the target copy's `collection.json`
+  (wraps `get_collection_from_s3`).
+- `handle.iter_items() -> Iterator[dict]` — the target copy's STAC Items, streamed one at a
+  time (wraps `iter_collection_items_from_s3`; lazy — consume it inside the callback).
+
+Any domain-specific downstream work (database load, re-publish, indexing) stays entirely in
+your `on_ready`; `promote_collection` owns no datastore and never returns a status — a normal
+return means every stage passed and `on_ready` ran.
+
+- `on_ready` runs **only after copy and verify have both succeeded**. Any earlier failure
+  raises (`S3OperationError` / `ConfigError` / `CredentialsError` / `FileNotFoundError`, and
+  `FileExistsError` from regeneration) and `on_ready` is never called.
+- `dry_run=True` runs regeneration/validation only (via `ItemGeneratorS3(dry_run=True)`); the
+  copier and verifier are never constructed and `on_ready` is never called.
+- `fresh_start=True` passes through to `CollectionCopier`, deleting the target
+  `<collection_id>/` prefix before copying. The source bucket is never modified.
+- Idempotent / resumable: re-running after a failure repeats the skip-aware copy, re-verifies,
+  and re-invokes `on_ready`. Side effects are only on the source (item regen, plus a
+  timestamped `items` backup when `backup_items=True`) and the target (the copy).
+
+```python
+from usergenerated import PromotedCollection, promote_collection
+
+
+def on_ready(collection: PromotedCollection) -> None:
+    stac = collection.read_collection()
+    item_count = sum(1 for _ in collection.iter_items())
+    print(f"{stac['id']}: {item_count} item(s) ready in the target bucket")
+
+
+promote_collection(
+    collection_id="EO.XXX.YYY.ZZZ",
+    source_endpoint_url="https://s3.central.data.destination-earth.eu",
+    source_bucket_name="my-source-bucket",
+    source_aws_access_key_id="<source_access_key_id>",
+    source_aws_secret_access_key="<source_secret_access_key>",
+    target_endpoint_url="https://s3.central.data.destination-earth.eu",
+    target_bucket_name="my-target-bucket",
+    target_aws_access_key_id="<target_access_key_id>",
+    target_aws_secret_access_key="<target_secret_access_key>",
+    on_ready=on_ready,
+)
+```
+
+`script_s3_promote_collection.py` is a runnable `.env`-driven demo whose `on_ready` logs the
+promoted collection's id and item count. It reuses the same variables as the copy and
+regenerate scripts (`S3_COLLECTION_ID`, `S3_ENDPOINT_URL`, `S3_BUCKET_NAME`,
+`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `TARGET_S3_ENDPOINT_URL`,
+`TARGET_S3_BUCKET_NAME`, `TARGET_AWS_ACCESS_KEY_ID`, `TARGET_AWS_SECRET_ACCESS_KEY`,
+`S3_DRY_RUN`, `S3_FRESH_START`), plus two optional keys read only by this script:
+`S3_LOCAL_ROOT` (default `tmp_structure_mirror`) and `S3_BACKUP_ITEMS` (default `true`).
+
+```bash
+python script_s3_promote_collection.py
+```
+
+### Manipulate Collection Config in S3
+
+`script_s3_collection_config_manipulation.py` demonstrates the helpers in
+`usergenerated.s3.s3_collection_config_tools` for reading and updating a collection's
+`collection_config.json` directly in S3, without a local mirror. Unlike the verify script above, it
+**mutates** the live bucket. For a given collection it:
+
+1. Resolves the config key with `resolve_collection_config_key` and prints the full `collection_config.json`.
+2. Reads the `ready_for_transfer` flag via `get_ready_for_transfer`.
+3. Sets `ready_for_transfer` to `false` via `set_ready_for_transfer`.
+4. Adds an ad-hoc `last_transferred` field (current UTC time, ISO 8601) via `update_collection_config`.
+
+Setting `ready_for_transfer` to `false` engages the `regenerate-item-metadata-s3` safety gate described
+in [Safety Behavior](#safety-behavior) — a subsequent regeneration run will refuse to proceed until the
+flag is set back to `true` (or the check is explicitly skipped).
+
+For programmatic use the same module exposes two reusable building blocks so a caller does not have to
+chain layout detection and a hand-written read-modify-write:
+
+- `resolve_collection_config_key(s3_client, bucket_name, collection_id)` — locate `collection_config.json`
+  in the bucket without handling `BucketLayout` yourself.
+- `update_collection_config(s3_client, bucket_name, key, changes)` — one read-modify-write for an
+  arbitrary set of fields. `changes` is either a mapping merged into the config (use the exported
+  `REMOVE` sentinel as a value to delete a key) or a callback that mutates the dict in place. Keys you
+  do not touch keep their value, order, and the file's `indent=4` formatting; it raises `ConfigError`
+  if the object is missing or not a JSON object and `S3OperationError` on an S3 read/write failure.
+  `set_ready_for_transfer` is a one-line wrapper over it.
+
+Configure the target bucket in `.env` (the same single-bucket variables used by the regeneration demo
+script):
+
+```bash
+S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
+S3_BUCKET_NAME="my-bucket"
+AWS_ACCESS_KEY_ID="<access_key_id>"
+AWS_SECRET_ACCESS_KEY="<secret_access_key>"
+S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
+```
+
+Then run:
+
+```bash
+python script_s3_collection_config_manipulation.py
+```
+
+### Report Bucket Information
+
+`script_s3_bucket_information.py` demonstrates `usergenerated.s3.s3_bucket_info.get_bucket_information`,
+which makes a single paginated sweep of a bucket (or a prefix within it) and prints a JSON report:
+file count, total and average size, the 5 largest files, oldest/most-recent modification times, a
+count+size breakdown by top-level folder, a `generated_at` timestamp, and a sample file listing
+capped at 50 entries (with a `sample_truncated` flag) so you never get thousands of rows back.
+Every size in the report (`total_size_bytes`, `average_file_size_bytes`, and each `size_bytes` in
+`largest_files`, `top_level_prefixes`, and `sample_files`) has a `..._human`/`size_human`
+counterpart, e.g. `"total_size_human": "94.2 MB"`. It is **read-only** — the bucket is never
+modified.
+
+Writing the report to `output/<YYYYMMDDTHHMMSS>_Bucket_Information.json` (the timestamp matches
+`generated_at` in the JSON) is **off by default** — the report is only printed to stdout. Turn it
+on with either `S3_WRITE_OUTPUT_FILE="true"` in `.env`, or `write_output_file=True` when calling
+`run_get_bucket_information` programmatically; the `output/` directory is created if needed.
+
+Configure the bucket in `.env` (the same single-bucket variables used by the config manipulation
+script above); `S3_PREFIX`, `S3_BUCKET_INFO_SAMPLE_LIMIT`, `S3_WRITE_OUTPUT_FILE`, and
+`S3_BUCKET_INFO_OUTPUT_DIR` are optional:
+
+```bash
+S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
+S3_BUCKET_NAME="my-bucket"
+AWS_ACCESS_KEY_ID="<access_key_id>"
+AWS_SECRET_ACCESS_KEY="<secret_access_key>"
+S3_PREFIX=""                        # optional: scope the report to one folder/collection
+S3_BUCKET_INFO_SAMPLE_LIMIT="50"    # optional: cap on the sample file listing
+S3_WRITE_OUTPUT_FILE="false"        # optional: set true/1/yes/on to also write the JSON report to disk
+S3_BUCKET_INFO_OUTPUT_DIR="output"  # optional: where the timestamped JSON report is written
+```
+
+Then run:
+
+```bash
+python script_s3_bucket_information.py
+```
+
+### Copy Collection Metadata Locally
+
+`script_s3_get_metadata.py` demonstrates
+`usergenerated.s3.s3_collection_metadata_downloader.download_collection_metadata`, which copies a
+collection's `metadata/` folder from S3 to a local destination. The source bucket's layout
+(whether it holds `<collection_id>/metadata/` or a top-level `metadata/`) is detected
+automatically, so the caller never needs to know or check it beforehand — the result always lands
+under a folder named for the collection ID, e.g. `<destination_root>/EO.XXX.YYY.ZZZ/metadata`. It
+is **read-only** with respect to S3 — nothing in the source bucket is modified.
+
+The destination root is configurable via `S3_METADATA_DESTINATION_ROOT` (or the
+`destination_root` argument when calling `run_get_metadata` programmatically); when unset, it
+defaults to a fresh timestamped folder under `/tmp/usergenerated_tmp/`, e.g.
+`/tmp/usergenerated_tmp/20260824T101500/EO.XXX.YYY.ZZZ/metadata`.
+
+Objects are downloaded concurrently — one worker thread per object, bounded by the
+`max_workers` argument (default `MAX_METADATA_DOWNLOAD_WORKERS` = 16, capped at 32). The single
+`s3_client` is shared across those threads, so build it with a matching connection pool:
+`create_s3_client(..., max_pool_connections=MAX_METADATA_DOWNLOAD_WORKERS)` (botocore defaults
+to 10). If any object fails to download the whole call raises `S3OperationError` (a
+`RuntimeError` subclass) rather than returning a partial folder; once a systemic problem has
+failed ~20 objects it stops submitting
+new downloads instead of grinding through every remaining key.
+
+For callers that only need the files for the duration of a block — parse `collection.json`,
+stream `items/` somewhere, done — use the `downloaded_collection_metadata` context manager
+instead of calling `download_collection_metadata` directly; on exit it removes the
+`<collection_id>/` subtree it downloaded, and if it also generated the timestamped destination
+root (no explicit `destination_root` given) it removes that too, so nothing is left under
+`/tmp`. An explicit `destination_root` you pass in is otherwise left in place:
+
+```python
+with downloaded_collection_metadata(s3_client, bucket, collection_id) as metadata_folder:
+    collection = get_stac_collection(metadata_folder)
+    for item in iter_items(metadata_folder):
+        ...
+# the timestamped /tmp/usergenerated_tmp/<...> directory is gone here
+```
+
+The same module also provides small supporting functions for reading the downloaded metadata:
+`get_stac_collection`/`get_collection_config` load `collection.json`/`collection_config.json` as
+dicts, and `iter_items`/`iter_item_batches` stream STAC Items out of the downloaded `items/`
+folder — one item at a time, or in fixed-size batches — rather than loading every item into
+memory at once. This matters for collections with very large item counts (hundreds of thousands),
+where materializing every item as a single list would be memory-intensive. The batch size defaults
+to `METADATA_DOWNLOADER_DEFAULT_ITEM_BATCH_SIZE` (500) and is configurable via
+`S3_METADATA_ITEM_BATCH_SIZE` (or the `item_batch_size` argument to `run_get_metadata`).
+
+If you only need to walk the items once and don't need them to persist,
+`iter_collection_items_from_s3` / `iter_collection_item_batches_from_s3` (and
+`get_collection_from_s3` for `collection.json`) stream STAC Items straight from the bucket with no
+local copy written and no cleanup to do — a drop-in for `iter_items` / `iter_item_batches` /
+`get_stac_collection`. The bucket layout is detected automatically, item bodies are fetched with a
+bounded concurrent prefetch (`max_workers` defaults to `MAX_METADATA_DOWNLOAD_WORKERS`, capped at
+32), and any fetch or parse failure raises `S3OperationError` rather than yielding a partial
+result. Keep `download_collection_metadata` for when the files must stay on disk or be read more
+than once. Setting `S3_METADATA_STREAM_ITEMS` to a truthy value (or passing `stream_items=True` to
+`run_get_metadata`) runs `script_s3_get_metadata.py` on this no-download path.
+
+Configure the collection and bucket in `.env` (the same single-bucket variables used by the config
+manipulation script above); `S3_METADATA_DESTINATION_ROOT`, `S3_METADATA_ITEM_BATCH_SIZE`, and
+`S3_METADATA_STREAM_ITEMS` are optional:
+
+```bash
+S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
+S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
+S3_BUCKET_NAME="my-bucket"
+AWS_ACCESS_KEY_ID="<access_key_id>"
+AWS_SECRET_ACCESS_KEY="<secret_access_key>"
+S3_METADATA_DESTINATION_ROOT=""  # optional: local folder to copy metadata/ into (default: /tmp/usergenerated_tmp/<timestamp>)
+S3_METADATA_ITEM_BATCH_SIZE=""  # optional: items per batch when streaming items/ (default: 500)
+S3_METADATA_STREAM_ITEMS=""  # optional: truthy = skip the local download, stream items straight from S3
+```
+
+Then run:
+
+```bash
+python script_s3_get_metadata.py
+```
+
+## Runtime Initialization
+
+Logging initialization is now explicit and idempotent.
+
+1. Entry-point scripts call `setup_logging()` from `usergenerated.logging_config` at startup.
+2. Environment loading (`load_dotenv`) is also called explicitly at startup.
+3. Importing modules should not configure logging implicitly.
+
+This keeps behavior stable regardless of import order and avoids duplicate log handlers.
+
+### Exit codes and failure output
+
+Every console command (`generate-item-metadata`, `regenerate-item-metadata-s3`) and root
+`script_s3_*.py` shares one convention:
+
+| Exit code | Meaning |
+|---|---|
+| `0` | success |
+| `1` | an expected failure (bad config, missing credentials, a missing file or folder you named, invalid `collection.json`, a failed S3 operation) — reported as a single `ERROR` log line, no traceback; or an unexpected internal error, which keeps its traceback |
+| `2` | command-line usage error (bad or missing arguments) |
+
+## Troubleshooting
+
+1. Generation fails with folder-level errors:
+Set `item_folder_level` to match your actual `data/` layout.
+
+2. Missing `item_config.json` errors:
+Set `item_config_optional` to `true` if item config files are intentionally omitted.
+
+3. Collection id validation errors:
+Ensure root folder name and `collection_config.json > id` represent the same collection id.
+
+4. Upload credential errors:
+Ensure `.env` contains valid `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` when upload is enabled.
+
+## External References
+
+1. [DestinE Data Lake - Promote User Data to become DestinE data](https://destine-data-lake-docs.data.destination-earth.eu/en/latest/dedl-discovery-and-data-access/User-Generated-Data/Promote-user-data-to-become-DestinE-data/Promote-user-data-to-become-DestinE-data.html)
+2. [STAC Collection specification](https://stacspec.org/en/about/stac-spec/)
+3. [STAC Item specification](https://stacspec.org/en/about/stac-spec/)