sfeos-tools 0.1.1__tar.gz → 0.2.0__tar.gz

{sfeos_tools-0.1.1/sfeos_tools.egg-info → sfeos_tools-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sfeos-tools
-Version: 0.1.1
+Version: 0.2.0
 Summary: CLI tools for managing stac-fastapi-elasticsearch-opensearch deployments
 Author: CloudFerro S.A.
 Author-email: Jonathan Healy <jon@healy-hypersaptial.dev>
@@ -48,6 +48,8 @@ Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: click>=8.0.0
+Requires-Dist: httpx>=0.24.0
+Requires-Dist: orjson>=3.0.0
 Provides-Extra: elasticsearch
 Requires-Dist: stac-fastapi-elasticsearch>=6.6.0; extra == "elasticsearch"
 Provides-Extra: opensearch
@@ -94,6 +96,7 @@ CLI tools for managing [stac-fastapi-elasticsearch-opensearch](https://github.co
 - [Commands](#commands)
   - [add-bbox-shape](#add-bbox-shape)
   - [reindex](#reindex)
+  - [load-data](#load-data)
 - [Development](#development)
 - [License](#license)
 
@@ -183,63 +186,56 @@ Options:
 - `--password`: Database password (default: ES_PASS env var)
 - `--yes`: Skip confirmation prompt
 
-Example:
+Examples:
 ```bash
 # Reindex Elasticsearch with custom host and no SSL
-sfeos-tools reindex --backend elasticsearch --host localhost --port 9200 --no-ssl
+sfeos-tools reindex --backend elasticsearch --host localhost --port 9200 --no-ssl --yes
 
 # Reindex OpenSearch with default settings
-sfeos-tools reindex --backend opensearch
+sfeos-tools reindex --backend opensearch --yes
 ```
 
-# Get help for a specific command
-sfeos-tools add-bbox-shape --help
-```
-
-## Commands
-
-### add-bbox-shape
-
-Add `bbox_shape` field to existing collections for spatial search support.
+### load-data
 
-**Basic usage:**
+Load STAC collections and items from local JSON files into a STAC API instance. This command is useful for:
+- Populating a new STAC API deployment with test data
+- Migrating data between STAC API instances
+- Bulk loading STAC collections and items
 
 ```bash
-# Elasticsearch
-sfeos-tools add-bbox-shape --backend elasticsearch
-
-# OpenSearch
-sfeos-tools add-bbox-shape --backend opensearch
+sfeos-tools load-data --base-url <stac-api-url> [options]
 ```
 
-**Connection options:**
+Options:
+- `--base-url`: Base URL of the STAC API (required)
+- `--collection-id`: ID of the collection to create/update (default: test-collection)
+- `--data-dir`: Directory containing collection.json and feature collection files (default: sample_data/)
+- `--use-bulk`: Use bulk insert method for items (faster for large datasets)
+
+**Data Directory Structure:**
+
+Your data directory should contain:
+- `collection.json`: STAC collection definition
+- One or more `.json` files: Feature collections with STAC items
 
+Examples:
 ```bash
-# Local Docker Compose (no SSL)
-sfeos-tools add-bbox-shape --backend elasticsearch --no-ssl
-
-# Remote server with SSL
-sfeos-tools add-bbox-shape \
-  --backend elasticsearch \
-  --host db.example.com \
-  --port 9200 \
-  --user admin \
-  --password secret
-
-# Using environment variables
-ES_HOST=my-cluster.cloud.com ES_PORT=9243 ES_USER=elastic ES_PASS=changeme \
-  sfeos-tools add-bbox-shape --backend elasticsearch
+# Load data from default directory
+sfeos-tools load-data --base-url http://localhost:8080
+
+# Load with custom collection ID and bulk insert
+sfeos-tools load-data \
+  --base-url http://localhost:8080 \
+  --collection-id my-collection \
+  --use-bulk
+
+# Load from custom directory
+sfeos-tools load-data \
+  --base-url http://localhost:8080 \
+  --data-dir /path/to/stac/data \
+  --collection-id production-data
 ```
 
-**Available options:**
-
-- `--backend`: Database backend (elasticsearch or opensearch) - **required**
-- `--host`: Database host (default: localhost or ES_HOST env var)
-- `--port`: Database port (default: 9200 or ES_PORT env var)
-- `--use-ssl / --no-ssl`: Use SSL connection (default: true or ES_USE_SSL env var)
-- `--user`: Database username (default: ES_USER env var)
-- `--password`: Database password (default: ES_PASS env var)
-
 ## Development
 
 To develop sfeos-tools locally:
@@ -259,3 +255,6 @@ pre-commit install
 pre-commit run --all-files
 ```
 
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

{sfeos_tools-0.1.1 → sfeos_tools-0.2.0}/README.md

@@ -28,6 +28,7 @@ CLI tools for managing [stac-fastapi-elasticsearch-opensearch](https://github.co
 - [Commands](#commands)
   - [add-bbox-shape](#add-bbox-shape)
   - [reindex](#reindex)
+  - [load-data](#load-data)
 - [Development](#development)
 - [License](#license)
 
@@ -117,63 +118,56 @@ Options:
 - `--password`: Database password (default: ES_PASS env var)
 - `--yes`: Skip confirmation prompt
 
-Example:
+Examples:
 ```bash
 # Reindex Elasticsearch with custom host and no SSL
-sfeos-tools reindex --backend elasticsearch --host localhost --port 9200 --no-ssl
+sfeos-tools reindex --backend elasticsearch --host localhost --port 9200 --no-ssl --yes
 
 # Reindex OpenSearch with default settings
-sfeos-tools reindex --backend opensearch
+sfeos-tools reindex --backend opensearch --yes
 ```
 
-# Get help for a specific command
-sfeos-tools add-bbox-shape --help
-```
-
-## Commands
-
-### add-bbox-shape
-
-Add `bbox_shape` field to existing collections for spatial search support.
+### load-data
 
-**Basic usage:**
+Load STAC collections and items from local JSON files into a STAC API instance. This command is useful for:
+- Populating a new STAC API deployment with test data
+- Migrating data between STAC API instances
+- Bulk loading STAC collections and items
 
 ```bash
-# Elasticsearch
-sfeos-tools add-bbox-shape --backend elasticsearch
-
-# OpenSearch
-sfeos-tools add-bbox-shape --backend opensearch
+sfeos-tools load-data --base-url <stac-api-url> [options]
 ```
 
-**Connection options:**
+Options:
+- `--base-url`: Base URL of the STAC API (required)
+- `--collection-id`: ID of the collection to create/update (default: test-collection)
+- `--data-dir`: Directory containing collection.json and feature collection files (default: sample_data/)
+- `--use-bulk`: Use bulk insert method for items (faster for large datasets)
+
+**Data Directory Structure:**
+
+Your data directory should contain:
+- `collection.json`: STAC collection definition
+- One or more `.json` files: Feature collections with STAC items
 
+Examples:
 ```bash
-# Local Docker Compose (no SSL)
-sfeos-tools add-bbox-shape --backend elasticsearch --no-ssl
-
-# Remote server with SSL
-sfeos-tools add-bbox-shape \
-  --backend elasticsearch \
-  --host db.example.com \
-  --port 9200 \
-  --user admin \
-  --password secret
-
-# Using environment variables
-ES_HOST=my-cluster.cloud.com ES_PORT=9243 ES_USER=elastic ES_PASS=changeme \
-  sfeos-tools add-bbox-shape --backend elasticsearch
+# Load data from default directory
+sfeos-tools load-data --base-url http://localhost:8080
+
+# Load with custom collection ID and bulk insert
+sfeos-tools load-data \
+  --base-url http://localhost:8080 \
+  --collection-id my-collection \
+  --use-bulk
+
+# Load from custom directory
+sfeos-tools load-data \
+  --base-url http://localhost:8080 \
+  --data-dir /path/to/stac/data \
+  --collection-id production-data
 ```
 
-**Available options:**
-
-- `--backend`: Database backend (elasticsearch or opensearch) - **required**
-- `--host`: Database host (default: localhost or ES_HOST env var)
-- `--port`: Database port (default: 9200 or ES_PORT env var)
-- `--use-ssl / --no-ssl`: Use SSL connection (default: true or ES_USE_SSL env var)
-- `--user`: Database username (default: ES_USER env var)
-- `--password`: Database password (default: ES_PASS env var)
-
 ## Development
 
 To develop sfeos-tools locally:
@@ -193,3 +187,6 @@ pre-commit install
 pre-commit run --all-files
 ```
 
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

{sfeos_tools-0.1.1 → sfeos_tools-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "sfeos-tools"
-version = "0.1.1"
+version = "0.2.0"
 description = "CLI tools for managing stac-fastapi-elasticsearch-opensearch deployments"
 readme = "README.md"
 authors = [
@@ -33,6 +33,8 @@ classifiers = [
 ]
 dependencies = [
     "click>=8.0.0",
+    "httpx>=0.24.0",
+    "orjson>=3.0.0",
 ]
 
 [project.urls]
@@ -85,6 +87,8 @@ addopts = "-v --cov=sfeos_tools --cov-report=term-missing"
 asyncio_mode = "auto"
 filterwarnings = [
     "ignore::DeprecationWarning:pydantic._internal._config",
+    "ignore::pydantic.warnings.PydanticDeprecatedSince20",
+    "ignore:coroutine.*was never awaited:RuntimeWarning",
 ]
 
 [tool.coverage.report]

{sfeos_tools-0.1.1 → sfeos_tools-0.2.0}/sfeos_tools/cli.py

@@ -14,7 +14,15 @@ import sys
 
 import click
 
+try:
+    from importlib.metadata import version as _get_version
+except ImportError:
+    from importlib_metadata import version as _get_version  # type: ignore[no-redef]
+
+__version__ = _get_version("sfeos-tools")
+
 from .bbox_shape import run_add_bbox_shape
+from .data_loader import load_items
 from .reindex import run as unified_reindex_run
 
 logging.basicConfig(level=logging.INFO)
@@ -22,7 +30,7 @@ logger = logging.getLogger(__name__)
 
 
 @click.group()
-@click.version_option(version="0.1.0", prog_name="sfeos-tools")
+@click.version_option(version=__version__, prog_name="sfeos-tools")
 def cli():
     """SFEOS Tools - Utilities for managing stac-fastapi-elasticsearch-opensearch deployments."""
     pass
@@ -225,5 +233,47 @@ def reindex(backend, host, port, use_ssl, user, password, yes):
         sys.exit(1)
 
 
+@cli.command("load-data")
+@click.option("--base-url", required=True, help="Base URL of the STAC API")
+@click.option(
+    "--collection-id",
+    default="test-collection",
+    help="ID of the collection to which items are added",
+)
+@click.option("--use-bulk", is_flag=True, help="Use bulk insert method for items")
+@click.option(
+    "--data-dir",
+    type=click.Path(exists=True),
+    default="sample_data/",
+    help="Directory containing collection.json and feature collection file",
+)
+def load_data(base_url: str, collection_id: str, use_bulk: bool, data_dir: str) -> None:
+    """Load STAC items into the database via STAC API.
+
+    This command loads a STAC collection and its items from local JSON files
+    into a STAC API instance. It expects a directory containing:
+    - collection.json: The STAC collection definition
+    - One or more feature collection JSON files with STAC items
+
+    Examples:
+        sfeos-tools load-data --base-url http://localhost:8080
+        sfeos-tools load-data --base-url http://localhost:8080 --collection-id my-collection --use-bulk
+        sfeos-tools load-data --base-url http://localhost:8080 --data-dir /path/to/data
+    """
+    from httpx import Client
+
+    try:
+        with Client(base_url=base_url) as client:
+            load_items(client, collection_id, use_bulk, data_dir)
+        click.echo(click.style("✓ Data loading completed successfully", fg="green"))
+    except KeyboardInterrupt:
+        click.echo(click.style("\n✗ Data loading interrupted by user", fg="yellow"))
+        sys.exit(1)
+    except Exception as e:
+        error_msg = str(e)
+        click.echo(click.style(f"✗ Data loading failed: {error_msg}", fg="red"))
+        sys.exit(1)
+
+
 if __name__ == "__main__":
     cli()

sfeos_tools-0.2.0/sfeos_tools/data_loader.py

@@ -0,0 +1,102 @@
+"""Data Loader CLI STAC_API Ingestion Tool."""
+
+import os
+from typing import Any
+
+import click
+import orjson
+from httpx import Client
+
+
+def load_data(filepath: str) -> dict[str, Any]:
+    """Load json data from a file within the specified data directory."""
+    try:
+        with open(filepath, "rb") as file:
+            return orjson.loads(file.read())
+    except FileNotFoundError as e:
+        click.secho(f"File not found: {filepath}", fg="red", err=True)
+        raise click.Abort() from e
+
+
+def load_collection(client: Client, collection_id: str, data_dir: str) -> None:
+    """Load a STAC collection into the database."""
+    collection = load_data(os.path.join(data_dir, "collection.json"))
+    collection["id"] = collection_id
+    resp = client.post("/collections", json=collection)
+    if resp.status_code == 200 or resp.status_code == 201:
+        click.echo(f"Status code: {resp.status_code}")
+        click.echo(f"Added collection: {collection['id']}")
+    elif resp.status_code == 409:
+        click.echo(f"Status code: {resp.status_code}")
+        click.echo(f"Collection: {collection['id']} already exists")
+    else:
+        click.echo(f"Status code: {resp.status_code}")
+        click.echo(f"Error writing {collection['id']} collection. Message: {resp.text}")
+
+
+def load_items(
+    client: Client, collection_id: str, use_bulk: bool, data_dir: str
+) -> None:
+    """Load STAC items into the database based on the method selected."""
+    with os.scandir(data_dir) as entries:
+        # Attempt to dynamically find a suitable feature collection file
+        # Use the first found feature collection file
+        feature_file = next(
+            (
+                entry.path
+                for entry in entries
+                if entry.is_file()
+                and entry.name.endswith(".json")
+                and entry.name != "collection.json"
+            ),
+            None,
+        )
+
+    if feature_file is None:
+        click.secho(
+            "No feature collection files found in the specified directory.",
+            fg="red",
+            err=True,
+        )
+        raise click.Abort()
+
+    feature_collection = load_data(feature_file)
+
+    load_collection(client, collection_id, data_dir)
+    if use_bulk:
+        load_items_bulk_insert(client, collection_id, feature_collection)
+    else:
+        load_items_one_by_one(client, collection_id, feature_collection)
+
+
+def load_items_one_by_one(
+    client: Client, collection_id: str, feature_collection: dict[str, Any]
+) -> None:
+    """Load STAC items into the database one by one."""
+    for feature in feature_collection["features"]:
+        feature["collection"] = collection_id
+        resp = client.post(f"/collections/{collection_id}/items", json=feature)
+        if resp.status_code == 200:
+            click.echo(f"Status code: {resp.status_code}")
+            click.echo(f"Added item: {feature['id']}")
+        elif resp.status_code == 409:
+            click.echo(f"Status code: {resp.status_code}")
+            click.echo(f"Item: {feature['id']} already exists")
+
+
+def load_items_bulk_insert(
+    client: Client, collection_id: str, feature_collection: dict[str, Any]
+) -> None:
+    """Load STAC items into the database via bulk insert."""
+    for feature in feature_collection["features"]:
+        feature["collection"] = collection_id
+    resp = client.post(f"/collections/{collection_id}/items", json=feature_collection)
+    if resp.status_code == 200:
+        click.echo(f"Status code: {resp.status_code}")
+        click.echo("Bulk inserted items successfully.")
+    elif resp.status_code == 204:
+        click.echo(f"Status code: {resp.status_code}")
+        click.echo("Bulk update successful, no content returned.")
+    elif resp.status_code == 409:
+        click.echo(f"Status code: {resp.status_code}")
+        click.echo("Conflict detected, some items might already exist.")

{sfeos_tools-0.1.1 → sfeos_tools-0.2.0/sfeos_tools.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sfeos-tools
-Version: 0.1.1
+Version: 0.2.0
 Summary: CLI tools for managing stac-fastapi-elasticsearch-opensearch deployments
 Author: CloudFerro S.A.
 Author-email: Jonathan Healy <jon@healy-hypersaptial.dev>
@@ -48,6 +48,8 @@ Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: click>=8.0.0
+Requires-Dist: httpx>=0.24.0
+Requires-Dist: orjson>=3.0.0
 Provides-Extra: elasticsearch
 Requires-Dist: stac-fastapi-elasticsearch>=6.6.0; extra == "elasticsearch"
 Provides-Extra: opensearch
@@ -94,6 +96,7 @@ CLI tools for managing [stac-fastapi-elasticsearch-opensearch](https://github.co
 - [Commands](#commands)
   - [add-bbox-shape](#add-bbox-shape)
   - [reindex](#reindex)
+  - [load-data](#load-data)
 - [Development](#development)
 - [License](#license)
 
@@ -183,63 +186,56 @@ Options:
 - `--password`: Database password (default: ES_PASS env var)
 - `--yes`: Skip confirmation prompt
 
-Example:
+Examples:
 ```bash
 # Reindex Elasticsearch with custom host and no SSL
-sfeos-tools reindex --backend elasticsearch --host localhost --port 9200 --no-ssl
+sfeos-tools reindex --backend elasticsearch --host localhost --port 9200 --no-ssl --yes
 
 # Reindex OpenSearch with default settings
-sfeos-tools reindex --backend opensearch
+sfeos-tools reindex --backend opensearch --yes
 ```
 
-# Get help for a specific command
-sfeos-tools add-bbox-shape --help
-```
-
-## Commands
-
-### add-bbox-shape
-
-Add `bbox_shape` field to existing collections for spatial search support.
+### load-data
 
-**Basic usage:**
+Load STAC collections and items from local JSON files into a STAC API instance. This command is useful for:
+- Populating a new STAC API deployment with test data
+- Migrating data between STAC API instances
+- Bulk loading STAC collections and items
 
 ```bash
-# Elasticsearch
-sfeos-tools add-bbox-shape --backend elasticsearch
-
-# OpenSearch
-sfeos-tools add-bbox-shape --backend opensearch
+sfeos-tools load-data --base-url <stac-api-url> [options]
 ```
 
-**Connection options:**
+Options:
+- `--base-url`: Base URL of the STAC API (required)
+- `--collection-id`: ID of the collection to create/update (default: test-collection)
+- `--data-dir`: Directory containing collection.json and feature collection files (default: sample_data/)
+- `--use-bulk`: Use bulk insert method for items (faster for large datasets)
+
+**Data Directory Structure:**
+
+Your data directory should contain:
+- `collection.json`: STAC collection definition
+- One or more `.json` files: Feature collections with STAC items
 
+Examples:
 ```bash
-# Local Docker Compose (no SSL)
-sfeos-tools add-bbox-shape --backend elasticsearch --no-ssl
-
-# Remote server with SSL
-sfeos-tools add-bbox-shape \
-  --backend elasticsearch \
-  --host db.example.com \
-  --port 9200 \
-  --user admin \
-  --password secret
-
-# Using environment variables
-ES_HOST=my-cluster.cloud.com ES_PORT=9243 ES_USER=elastic ES_PASS=changeme \
-  sfeos-tools add-bbox-shape --backend elasticsearch
+# Load data from default directory
+sfeos-tools load-data --base-url http://localhost:8080
+
+# Load with custom collection ID and bulk insert
+sfeos-tools load-data \
+  --base-url http://localhost:8080 \
+  --collection-id my-collection \
+  --use-bulk
+
+# Load from custom directory
+sfeos-tools load-data \
+  --base-url http://localhost:8080 \
+  --data-dir /path/to/stac/data \
+  --collection-id production-data
 ```
 
-**Available options:**
-
-- `--backend`: Database backend (elasticsearch or opensearch) - **required**
-- `--host`: Database host (default: localhost or ES_HOST env var)
-- `--port`: Database port (default: 9200 or ES_PORT env var)
-- `--use-ssl / --no-ssl`: Use SSL connection (default: true or ES_USE_SSL env var)
-- `--user`: Database username (default: ES_USER env var)
-- `--password`: Database password (default: ES_PASS env var)
-
 ## Development
 
 To develop sfeos-tools locally:
@@ -259,3 +255,6 @@ pre-commit install
 pre-commit run --all-files
 ```
 
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

{sfeos_tools-0.1.1 → sfeos_tools-0.2.0}/sfeos_tools.egg-info/SOURCES.txt

@@ -4,6 +4,7 @@ pyproject.toml
 sfeos_tools/__init__.py
 sfeos_tools/bbox_shape.py
 sfeos_tools/cli.py
+sfeos_tools/data_loader.py
 sfeos_tools/reindex.py
 sfeos_tools.egg-info/PKG-INFO
 sfeos_tools.egg-info/SOURCES.txt
@@ -11,4 +12,6 @@ sfeos_tools.egg-info/dependency_links.txt
 sfeos_tools.egg-info/entry_points.txt
 sfeos_tools.egg-info/requires.txt
 sfeos_tools.egg-info/top_level.txt
-tests/test_bbox_shape.py
+tests/test_bbox_shape.py
+tests/test_data_loader.py
+tests/test_reindex.py

{sfeos_tools-0.1.1 → sfeos_tools-0.2.0}/sfeos_tools.egg-info/requires.txt

@@ -1,4 +1,6 @@
 click>=8.0.0
+httpx>=0.24.0
+orjson>=3.0.0
 
 [dev]
 stac-fastapi-elasticsearch>=6.6.0