portolan-cli 0.4.2__tar.gz → 0.4.4__tar.gz

{portolan_cli-0.4.2 → portolan_cli-0.4.4}/.pre-commit-config.yaml

@@ -43,6 +43,16 @@ repos:
 
   - repo: local
     hooks:
+      # === SAFETY HOOKS ===
+      # Prevent accidental commits of temporary/generated content
+
+      - id: forbid-worktrees
+        name: forbid worktree files
+        entry: "bash -c 'echo ERROR - Worktree files should not be committed. Remove .claude/worktrees/ from staging. && exit 1'"
+        language: system
+        files: ^\.claude/worktrees/
+        pass_filenames: false
+
       # === AUTO-UPDATE HOOKS ===
       # These regenerate content from code - keeps docs in sync automatically
 

{portolan_cli-0.4.2 → portolan_cli-0.4.4}/CHANGELOG.md

@@ -1,3 +1,44 @@
+## v0.4.4 (2026-03-07)
+
+### Feat
+
+- **scan**: batch and summarize repeated warnings (#194)
+- **add**: accept multiple paths like git add (#189)
+- **add**: support tabular parquet as auxiliary assets (#190)
+
+### Fix
+
+- **list**: show all assets grouped by item (#196) (#204)
+- **add**: batch repetitive failures by error message (#199) (#205)
+- **add**: accept PMTiles as cloud-native primary asset (#201)
+- **status**: treat FileGDB directories as single assets (#187)
+- **status**: detect and skip symlink cycles (#188)
+- **add**: continue on errors and report all failures at end (#191)
+- **scan**: list specific unrecognized files instead of just count (#185)
+- **scan**: make PATH argument optional, default to current directory (#184)
+- **scan**: remove non-existent --bundle flag reference from suggestion (#183)
+
+## v0.4.3 (2026-03-07)
+
+### BREAKING CHANGE
+
+- Removes --fix-metadata flag; use --metadata --fix instead.
+
+### Feat
+
+- **cli**: add portolan clean command to remove metadata (#172)
+- **cli**: add --item-id flag to portolan add command (#171)
+- **check**: redesign --fix to work orthogonally with --metadata/--geo-assets (#164)
+
+### Fix
+
+- **dataset**: make add_dataset atomic and track files in-place (#170)
+- **catalog**: unify catalog root detection (#162) (#169)
+- **status**: detect untracked files in uninitialized collections (#167)
+- **dry-run**: prevent network calls in dry-run mode (#168)
+- **add**: support recursive add at catalog root (#166)
+- **list**: add guidance when no items found (#165)
+
 ## v0.4.2 (2026-03-06)
 
 ### Feat

{portolan_cli-0.4.2 → portolan_cli-0.4.4}/CLAUDE.md

@@ -100,6 +100,7 @@ AI agents will write most of the code. Human review does not scale to match AI o
 | [0026](context/shared/adr/0026-conversion-config-design.md) | Conversion config: extension/path overrides, precedence rules |
 | [0027](context/shared/adr/0027-unified-config-yaml-sentinel.md) | Unified config.yaml as sentinel and user config (eliminates config.json) |
 | [0028](context/shared/adr/0028-all-files-as-assets.md) | Track ALL files in item directories as assets |
+| [0029](context/shared/adr/0029-unified-catalog-root-detection.md) | Unified catalog root detection via .portolan/config.yaml |
 
 ## Common Commands
 
@@ -375,7 +376,7 @@ error("No geometry column (required)")     # ✗ Red X
 detail("Processing chunk 3/10...")         # Dimmed text
 ```
 
-<!-- freshness: last-verified: 2026-03-06 -->
+<!-- freshness: last-verified: 2026-03-07 -->
 ## Design Principles
 
 | Principle | Meaning | ADR |

{portolan_cli-0.4.2 → portolan_cli-0.4.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: portolan-cli
-Version: 0.4.2
+Version: 0.4.4
 Summary: A CLI tool for managing cloud-native geospatial data
 Project-URL: Homepage, https://github.com/portolan-sdi/portolan-cli
 Project-URL: Bug Tracker, https://github.com/portolan-sdi/portolan-cli/issues

{portolan_cli-0.4.2 → portolan_cli-0.4.4}/SKILL.md

@@ -23,6 +23,8 @@ Portolan is a CLI for publishing and managing **cloud-native geospatial data cat
 Track files in the catalog.
 
 ```bash
+portolan add demographics/census.parquet
+portolan add file1.geojson file2.geojson   # Add multiple files
 portolan add imagery/                      # Add all files in directory
 portolan add .                             # Add all files in catalog
 ```
@@ -34,7 +36,15 @@ Validate a Portolan catalog or check files for cloud-native status.
 portolan check                        # Validate all (metadata + geo-assets)
 portolan check --metadata             # Validate metadata only
 portolan check --geo-assets           # Check geo-assets only
-portolan check /data --fix            # Convert files to cloud-native
+portolan check --fix                  # Fix both metadata and geo-assets
+```
+
+### `portolan clean`
+Remove all Portolan metadata while preserving data files.
+
+```bash
+portolan clean           # Remove all metadata
+portolan clean --dry-run # Preview what would be removed
 ```
 
 ### `portolan clone`
@@ -123,10 +133,10 @@ portolan rm -f vectors/                     # Force remove entire directory
 Scan a directory for geospatial files and potential issues.
 
 ```bash
+portolan scan                         # Scan current directory
+portolan scan --json                  # JSON output in current directory
 portolan scan /data/geospatial
-portolan scan . --json
 portolan scan /large/tree --max-depth=2
-portolan scan /data --no-recursive
 ```
 
 ### `portolan status`

portolan_cli-0.4.4/context/shared/adr/0029-unified-catalog-root-detection.md

@@ -0,0 +1,125 @@
+# ADR-0029: Unified Catalog Root Detection
+
+## Status
+
+Accepted
+
+## Context
+
+Portolan CLI has inconsistent catalog root detection across commands, causing confusing behavior where users can add files successfully but then can't see them with `list`.
+
+### Current State (Pre-fix)
+
+Two competing functions detect catalog roots using different criteria:
+
+| Function | Location | Sentinel | Used By |
+|----------|----------|----------|---------|
+| `find_catalog_root()` | `dataset.py` | `catalog.json` | `status`, `add`, `rm` |
+| `_find_catalog_root()` | `cli.py` | `.portolan/` directory | `config get/set/list/unset` |
+
+Additionally, `check` only looks at the current directory without walking up.
+
+### Problems
+
+1. **Shadow catalogs:** `add` auto-creates structure, but `list` looks for parent catalog, so added items are invisible
+2. **Inconsistent behavior:** `status` shows sibling directories from parent catalog when user is in subdirectory
+3. **Trust erosion:** Commands report contradictory states
+
+### Related Issues
+
+- GitHub Issue #162: Inconsistent catalog root detection across CLI commands
+- GitHub Issue #137: Multiple related bugs that may be symptoms of this root cause
+
+## Decision
+
+**Use `.portolan/config.yaml` as the single sentinel for catalog root detection.**
+
+All commands will use a unified `find_catalog_root()` function in `catalog.py` that:
+
+1. Walks up from the current directory (or specified path)
+2. Looks for `.portolan/config.yaml` (not `catalog.json`, not bare `.portolan/`)
+3. Returns the catalog root path, or `None` if not found
+4. Enforces a depth limit (`MAX_CATALOG_SEARCH_DEPTH = 20`) for security
+
+This aligns with ADR-0027, which established `config.yaml` as the unified sentinel file for managed catalogs.
+
+### Behavior Changes
+
+| Command | Old Behavior | New Behavior |
+|---------|--------------|--------------|
+| `add` | Auto-creates catalog structure if missing | Fails with "run `portolan init`" message |
+| `list` | Uses explicit `--catalog` argument (default `.`) | *Unchanged* - uses `--catalog` arg, not `find_catalog_root()` |
+| `status` | Finds parent catalog via `catalog.json` | Finds parent catalog via `.portolan/config.yaml` |
+| `rm` | Finds parent catalog via `catalog.json` | Finds parent catalog via `.portolan/config.yaml` |
+| `config *` | Finds catalog via `.portolan/` directory | Finds catalog via `.portolan/config.yaml` |
+
+> **Note:** `list` is excluded from the unified `find_catalog_root()` treatment because it takes `--catalog` as an explicit argument. This is a separate issue to address in a future PR.
+
+### UNMANAGED_STAC Impact
+
+Directories with only `catalog.json` (no `.portolan/`) will no longer be detected as catalogs. Users must run `portolan init` or `portolan adopt` to manage existing STAC catalogs.
+
+This is acceptable because:
+- UNMANAGED_STAC support was always incomplete (some commands worked, others didn't)
+- Clear error messages guide users to the fix
+- Pre-v1.0, so breaking changes are acceptable
+
+## Implementation
+
+### Single Function Location
+
+The unified `find_catalog_root()` lives in `portolan_cli/catalog.py` alongside `detect_state()`, since both deal with catalog detection logic.
+
+```python
+def find_catalog_root(start_path: Path | None = None) -> Path | None:
+    """Find catalog root by walking up to find .portolan/config.yaml.
+
+    Per ADR-0029, uses .portolan/config.yaml as the single sentinel,
+    unifying detection across all CLI commands.
+
+    Security: Limited to MAX_CATALOG_SEARCH_DEPTH (20) levels.
+    """
+```
+
+### Deleted Code
+
+- `find_catalog_root()` from `dataset.py` (looked for `catalog.json`)
+- `_find_catalog_root()` from `cli.py` (looked for `.portolan/` directory)
+
+## Consequences
+
+### Benefits
+
+- **Consistent behavior:** All commands agree on catalog root
+- **No shadow catalogs:** `add` requires `init` first, eliminating invisible items
+- **Simpler mental model:** One sentinel, one function, one behavior
+- **Aligns with ADR-0027:** Uses the established `config.yaml` sentinel
+
+### Trade-offs
+
+- **UNMANAGED_STAC no longer auto-detected:** Users must explicitly init/adopt
+- **`add` no longer auto-creates:** Requires explicit `init` first (git-style)
+
+### Migration
+
+None required. Pre-v1.0 with minimal users. Existing managed catalogs already have `.portolan/config.yaml`.
+
+## Alternatives Considered
+
+### Option A: Use `catalog.json` as sentinel everywhere
+
+**Rejected:** Would still allow "unmanaged" catalogs that only partially work. The root cause is supporting two modes (managed/unmanaged) inconsistently.
+
+### Option B: Current-directory-only (no walking up)
+
+**Rejected:** Breaks git-style UX that users expect. Commands should work from subdirectories.
+
+### Option C: Distinguish managed/unmanaged with two-tier behavior
+
+**Rejected:** More complex, doesn't eliminate the inconsistency problem, and UNMANAGED_STAC was never fully supported anyway.
+
+## References
+
+- [ADR-0027: Unified config.yaml as Sentinel](0027-unified-config-yaml-sentinel.md)
+- [ADR-0023: STAC Structure Separation](0023-stac-structure-separation.md)
+- [GitHub Issue #162](https://github.com/portolan-sdi/portolan-cli/issues/162)

{portolan_cli-0.4.2 → portolan_cli-0.4.4}/portolan_cli/catalog.py

@@ -95,6 +95,87 @@ def detect_state(path: Path) -> CatalogState:
     return CatalogState.FRESH
 
 
+def find_catalog_root(
+    start_path: Path | None = None,
+    *,
+    require_operational: bool = True,
+) -> Path | None:
+    """Find the catalog root by walking up from the given path.
+
+    Searches for a managed Portolan catalog starting from start_path (or cwd if None)
+    and walking up parent directories. This provides git-style behavior
+    where commands work from any subdirectory within a catalog.
+
+    Per ADR-0029, this uses .portolan/config.yaml as the primary sentinel,
+    unifying detection across all CLI commands. By default (require_operational=True),
+    it also requires at least one operational file (catalog.json or state.json)
+    to avoid detecting half-initialized repos.
+
+    Security: Limited to MAX_CATALOG_SEARCH_DEPTH levels to prevent
+    traversing to filesystem root where a malicious .portolan might exist.
+
+    Args:
+        start_path: Starting directory for search (defaults to cwd).
+        require_operational: If True (default), require .portolan/config.yaml
+            AND at least one of (catalog.json, .portolan/state.json) to exist.
+            Set to False during init_catalog() when creating a new catalog
+            where config.yaml is written before catalog.json.
+
+    Returns:
+        Path to catalog root if found, None otherwise.
+
+    Examples:
+        >>> find_catalog_root(Path("/my-catalog/collection/item"))
+        PosixPath('/my-catalog')
+
+        >>> find_catalog_root(Path("/no-catalog-here"))
+        None
+
+        >>> find_catalog_root()  # Uses current working directory
+        PosixPath('/my-catalog')
+
+        >>> # During init, check for config.yaml only (catalog.json not yet written)
+        >>> find_catalog_root(start_path, require_operational=False)
+    """
+    from portolan_cli.constants import MAX_CATALOG_SEARCH_DEPTH
+
+    def _is_catalog_root(path: Path) -> bool:
+        """Check if path is a valid catalog root."""
+        config_yaml = path / ".portolan" / "config.yaml"
+        if not config_yaml.exists():
+            return False
+
+        if not require_operational:
+            # During init, config.yaml alone is sufficient
+            return True
+
+        # Require operational file: catalog.json at root OR state.json in .portolan
+        catalog_json = path / "catalog.json"
+        state_json = path / ".portolan" / "state.json"
+        return catalog_json.exists() or state_json.exists()
+
+    # Handle non-existent paths gracefully
+    if start_path is not None and not start_path.exists():
+        return None
+
+    current = (start_path or Path.cwd()).resolve()
+    depth = 0
+
+    # Walk up until we find a valid catalog root, hit filesystem root, or exceed depth
+    while current != current.parent and depth < MAX_CATALOG_SEARCH_DEPTH:
+        if _is_catalog_root(current):
+            return current
+        current = current.parent
+        depth += 1
+
+    # Check the root directory itself (only if within depth limit)
+    if depth < MAX_CATALOG_SEARCH_DEPTH:
+        if _is_catalog_root(current):
+            return current
+
+    return None
+
+
 # Keep legacy exception for backward compatibility
 class CatalogExistsError(Exception):
     """Raised when attempting to initialize a catalog that already exists.

portolan_cli-0.4.4/portolan_cli/clean.py

@@ -0,0 +1,210 @@
+"""Clean command for removing Portolan metadata while preserving data files.
+
+This module provides functionality to remove all Portolan-generated metadata
+from a catalog while preserving user data files. Essential for testing
+workflows where you want to reset a catalog to a "clean slate" state.
+
+Files removed (metadata only):
+- .portolan/ directory (entire thing)
+- catalog.json (with "type": "Catalog")
+- */collection.json (with "type": "Collection")
+- */*/item.json or similar with "type": "Feature" (STAC items)
+- */versions.json (Portolan-specific schema)
+
+Files preserved (all data):
+- .parquet, .tif, .gpkg, .geojson, .shp, etc.
+- ANY file that isn't STAC metadata
+- Non-STAC JSON files (checked by "type" field)
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+from pathlib import Path
+
+from portolan_cli.constants import PORTOLAN_DIR
+
+
+def is_stac_metadata(path: Path) -> bool:
+    """Check if a JSON file is STAC metadata (safe to remove).
+
+    STAC metadata files have a "type" field with one of:
+    - "Catalog" (root catalog.json)
+    - "Collection" (collection.json)
+    - "Feature" (item.json - STAC items are GeoJSON Features)
+
+    Args:
+        path: Path to the file to check.
+
+    Returns:
+        True if the file is STAC metadata, False otherwise.
+
+    Examples:
+        >>> is_stac_metadata(Path("catalog.json"))  # has type=Catalog
+        True
+        >>> is_stac_metadata(Path("style.json"))  # non-STAC JSON
+        False
+        >>> is_stac_metadata(Path("data.parquet"))
+        False
+    """
+    # Only check JSON files
+    if path.suffix.lower() != ".json":
+        return False
+
+    try:
+        data = json.loads(path.read_text())
+        if not isinstance(data, dict):
+            return False
+        stac_type = data.get("type")
+        return stac_type in ("Catalog", "Collection", "Feature")
+    except (json.JSONDecodeError, OSError):
+        return False
+
+
+def is_versions_json(path: Path) -> bool:
+    """Check if a file is versions.json (Portolan-specific, safe to remove).
+
+    Args:
+        path: Path to the file to check.
+
+    Returns:
+        True if the file is named versions.json, False otherwise.
+    """
+    return path.name == "versions.json"
+
+
+def collect_files_to_remove(catalog_root: Path) -> tuple[list[Path], list[Path]]:
+    """Collect all metadata files and directories to remove from a catalog.
+
+    Walks the catalog structure and identifies:
+    1. .portolan/ directory (always removed entirely)
+    2. STAC metadata files (catalog.json, collection.json, item.json with proper type)
+    3. versions.json files (Portolan-specific)
+
+    Args:
+        catalog_root: Path to the catalog root directory.
+
+    Returns:
+        Tuple of (files_to_remove, directories_to_remove).
+        Files are sorted for deterministic output.
+    """
+    files_to_remove: list[Path] = []
+    directories_to_remove: list[Path] = []
+
+    # Always remove .portolan directory if it exists
+    portolan_dir = catalog_root / PORTOLAN_DIR
+    if portolan_dir.exists() and portolan_dir.is_dir():
+        directories_to_remove.append(portolan_dir)
+
+    # Walk the catalog and collect files
+    for path in catalog_root.rglob("*"):
+        # Skip .portolan directory contents (we're removing the whole dir)
+        if PORTOLAN_DIR in path.parts:
+            continue
+
+        if path.is_file():
+            # Check for STAC metadata
+            if is_stac_metadata(path):
+                files_to_remove.append(path)
+            # Check for versions.json
+            elif is_versions_json(path):
+                files_to_remove.append(path)
+
+    # Sort files for deterministic output
+    files_to_remove.sort()
+
+    return files_to_remove, directories_to_remove
+
+
+def remove_empty_directories(catalog_root: Path, removed_files: list[Path]) -> list[Path]:
+    """Remove directories that became empty after file removal.
+
+    Walks bottom-up from the deepest directories and removes any that are empty.
+    Stops at the catalog root.
+
+    Args:
+        catalog_root: Path to the catalog root directory.
+        removed_files: List of files that were removed.
+
+    Returns:
+        List of directories that were removed.
+    """
+    directories_removed: list[Path] = []
+
+    # Get unique parent directories of removed files, sorted deepest first
+    parent_dirs = set()
+    for file_path in removed_files:
+        parent = file_path.parent
+        while parent != catalog_root and parent != catalog_root.parent:
+            parent_dirs.add(parent)
+            parent = parent.parent
+
+    # Sort by depth (deepest first)
+    sorted_dirs = sorted(parent_dirs, key=lambda p: len(p.parts), reverse=True)
+
+    for dir_path in sorted_dirs:
+        if dir_path.exists() and dir_path.is_dir():
+            # Check if directory is empty using lazy iteration (no list allocation)
+            if next(dir_path.iterdir(), None) is None:
+                try:
+                    dir_path.rmdir()
+                except OSError:
+                    # On Windows, rmdir fails if the directory is the process CWD.
+                    # This is a best-effort cleanup, so skip and continue.
+                    continue
+                directories_removed.append(dir_path)
+
+    return directories_removed
+
+
+def clean_catalog(
+    catalog_root: Path,
+    *,
+    dry_run: bool = False,
+) -> tuple[list[Path], list[Path], int]:
+    """Remove all Portolan metadata from a catalog while preserving data.
+
+    Args:
+        catalog_root: Path to the catalog root directory.
+        dry_run: If True, only report what would be removed without deleting.
+
+    Returns:
+        Tuple of (files_removed, directories_removed, data_files_preserved).
+
+    Raises:
+        OSError: If file deletion fails (permission errors, etc.).
+    """
+    files_to_remove, directories_to_remove = collect_files_to_remove(catalog_root)
+
+    # Count data files (files that are NOT being removed)
+    # Use a set for O(1) membership checks instead of O(n) list scan
+    files_to_remove_set = set(files_to_remove)
+    data_files = 0
+    for path in catalog_root.rglob("*"):
+        if path.is_file() and PORTOLAN_DIR not in path.parts:
+            if path not in files_to_remove_set:
+                data_files += 1
+
+    if dry_run:
+        return files_to_remove, directories_to_remove, data_files
+
+    # Actually remove files
+    files_removed: list[Path] = []
+    for file_path in files_to_remove:
+        if file_path.exists():
+            file_path.unlink()
+            files_removed.append(file_path)
+
+    # Remove .portolan directory
+    directories_removed: list[Path] = []
+    for dir_path in directories_to_remove:
+        if dir_path.exists():
+            shutil.rmtree(dir_path)
+            directories_removed.append(dir_path)
+
+    # Clean up empty directories
+    empty_dirs_removed = remove_empty_directories(catalog_root, files_removed)
+    directories_removed.extend(empty_dirs_removed)
+
+    return files_removed, directories_removed, data_files