cloudcat 0.4.0__tar.gz → 0.6.0__tar.gz

{cloudcat-0.4.0 → cloudcat-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudcat
-Version: 0.4.0
+Version: 0.6.0
 Summary: Preview and analyze data files in Google Cloud Storage, AWS S3, and Azure Data Lake Storage Gen2 from your terminal
 Author: Jonathan Sudhakar
 License: MIT
@@ -38,6 +38,7 @@ Requires-Dist: boto3>=1.18.0
 Requires-Dist: azure-storage-file-datalake>=12.0.0
 Requires-Dist: azure-storage-blob>=12.0.0
 Requires-Dist: azure-identity>=1.0.0
+Requires-Dist: tomli>=1.1.0; python_version < "3.11"
 Provides-Extra: parquet
 Requires-Dist: pyarrow>=5.0.0; extra == "parquet"
 Provides-Extra: avro
@@ -54,6 +55,13 @@ Provides-Extra: compression
 Requires-Dist: zstandard>=0.15.0; extra == "compression"
 Requires-Dist: lz4>=3.0.0; extra == "compression"
 Requires-Dist: python-snappy>=0.6.0; extra == "compression"
+Provides-Extra: delta
+Requires-Dist: deltalake>=1.0.0; extra == "delta"
+Provides-Extra: iceberg
+Requires-Dist: pyiceberg[pyarrow]>=0.9.0; extra == "iceberg"
+Provides-Extra: tables
+Requires-Dist: deltalake>=1.0.0; extra == "tables"
+Requires-Dist: pyiceberg[pyarrow]>=0.9.0; extra == "tables"
 Provides-Extra: all
 Requires-Dist: pyarrow>=5.0.0; extra == "all"
 Requires-Dist: fastavro>=1.4.0; extra == "all"
@@ -63,6 +71,7 @@ Requires-Dist: python-snappy>=0.6.0; extra == "all"
 Provides-Extra: test
 Requires-Dist: pytest>=6.0.0; extra == "test"
 Requires-Dist: pytest-mock>=3.6.0; extra == "test"
+Requires-Dist: pyiceberg[sql-sqlite]>=0.9.0; extra == "test"
 Provides-Extra: dev
 Requires-Dist: pytest>=6.0.0; extra == "dev"
 Requires-Dist: pytest-mock>=3.6.0; extra == "dev"
@@ -156,6 +165,9 @@ pip install 'cloudcat[orc]'
 
 # With compression support (zstd, lz4, snappy)
 pip install 'cloudcat[compression]'
+
+# With lakehouse table support (Delta Lake / Apache Iceberg)
+pip install 'cloudcat[tables]'
 ```
 
 > **Note:** If using zsh (default on macOS), quotes around extras are required to prevent shell interpretation of brackets.
@@ -169,14 +181,17 @@ pip install --upgrade 'cloudcat[all]'
 ### Requirements
 
 - **Homebrew**: macOS (Apple Silicon only). Intel Mac users should use pip.
-- **pip**: Python 3.7+ (all platforms)
+- **pip**: Python 3.9+ (all platforms)
 - Cloud provider credentials configured (see [Authentication](#authentication))
 
 ## Quick Start
 
 ```bash
-# Preview a CSV file from GCS
-cloudcat -p gcs://my-bucket/data.csv
+# Preview a CSV file from GCS (PATH is positional; -p/--path also works)
+cloudcat gcs://my-bucket/data.csv
+
+# Preview a local file — no cloud credentials needed
+cloudcat ./data.parquet
 
 # Preview a Parquet file from S3
 cloudcat -p s3://my-bucket/analytics/events.parquet
@@ -196,11 +211,14 @@ cloudcat -p abfss://logs@account.dfs.core.windows.net/app.log -i text
 # Read from a Spark output directory
 cloudcat -p s3://my-bucket/spark-output/ -i parquet
 
+# Read a Delta Lake or Iceberg table (auto-detected, current snapshot only)
+cloudcat s3://lake/orders_delta/ -w "status=refunded" -n 10
+
 # Read compressed files (auto-detected)
 cloudcat -p gcs://my-bucket/data.csv.gz
 
-# Filter rows with WHERE clause
-cloudcat -p s3://bucket/users.parquet --where "status=active"
+# Filter rows with WHERE (streams; stops at -n matches; AND/OR supported)
+cloudcat s3://bucket/users.parquet --where "status=active AND age>30"
 
 # Skip first 100 rows (pagination)
 cloudcat -p gcs://bucket/data.csv --offset 100 -n 10
@@ -215,6 +233,7 @@ cloudcat -p gcs://bucket/data.csv --offset 100 -n 10
 | Google Cloud Storage | `gcs://` or `gs://` | ✅ Supported |
 | Amazon S3 | `s3://` | ✅ Supported |
 | Azure Data Lake Gen2 | `abfss://` | ✅ Supported |
+| Local files | `file://` or a plain path | ✅ Supported (no credentials needed) |
 
 ### File Format Support
 
@@ -228,6 +247,8 @@ cloudcat -p gcs://bucket/data.csv --offset 100 -n 10
 | ORC | ✅ | Hive, Hadoop ecosystem |
 | Text | ✅ | Log files, plain text |
 | TSV | Via `--delimiter` | Tab-separated data |
+| Delta Lake | ✅ table dirs | Lakehouse tables (`cloudcat[delta]`) |
+| Apache Iceberg | ✅ table dirs | Lakehouse tables (`cloudcat[iceberg]`) |
 
 ### Streaming Efficiency
 
@@ -237,8 +258,8 @@ CloudCat uses intelligent streaming to minimize data transfer and egress costs:
 |--------|-------------|---------|-------------------|----------------|
 | Parquet | None/Internal | ✅ | ✅ Range requests | ✅ |
 | Parquet | External (.gz) | ❌ | ❌ | ❌ |
-| ORC | None/Internal | ❌ | ❌ | ❌ |
-| ORC | External (.gz) | ❌ | ❌ | ❌ |
+| ORC | None/Internal | ✅ Stripe-by-stripe | ✅ | ✅ |
+| ORC | External (.gz) | ❌ | ✅ | ✅ |
 | CSV | None | ✅ | ❌ | ✅ |
 | CSV | gzip/zstd/lz4/bz2 | ✅ | ❌ | ✅ |
 | CSV | snappy | ❌ | ❌ | ❌ |
@@ -267,7 +288,7 @@ CloudCat automatically detects and decompresses files based on extension (e.g.,
 
 | Format | Flag | Description |
 |--------|------|-------------|
-| Table | `-o table` | Beautiful ASCII table with colored headers (default) |
+| Table | `-o table` | Rounded table with bold headers and type-aware coloring (default) |
 | JSON | `-o json` | Standard JSON Lines output |
 | Pretty JSON | `-o jsonp` | Syntax-highlighted, indented JSON |
 | CSV | `-o csv` | Comma-separated values |
@@ -324,8 +345,8 @@ cloudcat -p gcs://bucket/events.jsonl
 ### Filtering and Pagination
 
 ```bash
-# Filter rows with WHERE clause
-cloudcat -p s3://bucket/users.parquet --where "status=active"
+# Filter rows with WHERE (streams; stops at -n matches; AND/OR supported)
+cloudcat s3://bucket/users.parquet --where "status=active AND age>30"
 cloudcat -p gcs://bucket/events.json --where "age>30"
 cloudcat -p s3://bucket/logs.csv --where "level=ERROR"
 
@@ -474,7 +495,9 @@ cloudcat -p s3://api-dumps/response.json -o csv > data.csv
 Usage: cloudcat [OPTIONS]
 
 Options:
-  -p, --path TEXT              Cloud storage path (required)
+  PATH                         Positional path: gs://, gcs://, s3://, abfss://,
+                               file:// URL, or a plain local path
+                               (-p/--path remains as a compatible alias)
                                Format: gcs://bucket/path, s3://bucket/path,
                                or abfss://container@account.dfs.core.windows.net/path
 
@@ -483,7 +506,8 @@ Options:
 
   -O, --output-file PATH       Write rendered data to a file instead of stdout
 
-  -i, --input-format TEXT      Input format: csv, json, parquet, avro, orc, text
+  -i, --input-format TEXT      Input format: csv, json, parquet, avro, orc,
+                               text, delta, iceberg
                                [default: auto-detect from extension]
 
   -c, --columns TEXT           Comma-separated list of columns to display
@@ -495,10 +519,12 @@ Options:
   --offset INTEGER             Skip first N rows
                                [default: 0]
 
-  -w, --where TEXT             Filter rows with SQL-like conditions. Scans the
-                               file and returns up to --num-rows matching rows.
-                               Examples: "status=active", "age>30",
-                               "name contains john", "email endswith @gmail.com"
+  -w, --where TEXT             Filter rows with SQL-like conditions; combine
+                               with AND/OR. Streams the file and stops at
+                               --num-rows matches; Parquet skips row groups
+                               via column statistics.
+                               Examples: "status=active", "age>30 AND age<65",
+                               "level=ERROR or level=FATAL"
 
   -s, --schema TEXT            Schema display: show, dont_show, schema_only
                                [default: show]
@@ -523,6 +549,13 @@ Options:
 
   --credentials TEXT           Path to GCP service account JSON file
 
+  --stats                      Show per-column statistics (nulls, distinct,
+                               min/max) over the retrieved rows
+
+  --config-profile TEXT        Named profile from ~/.config/cloudcat/config.toml
+
+  --completion [bash|zsh|fish] Print the shell completion script and exit
+
   --az-access-key TEXT         Azure storage account access key
 
   -y, --yes                    Skip confirmation prompts (for scripting)
@@ -699,6 +732,50 @@ pip install -e ".[all]"
 pytest
 ```
 
+## Use with AI Agents
+
+CloudCat is optimized for AI coding agents: **stdout carries only data**
+(diagnostics go to stderr, color auto-disables when piped), `-o json` emits
+parseable NDJSON, `-y` makes every command non-interactive, filtering streams
+with Parquet row-group pushdown, and local files need no credentials.
+
+The repo ships an [Agent Skill](https://agentskills.io) at
+[`skills/cloudcat/SKILL.md`](skills/cloudcat/SKILL.md) — a compact reference
+that teaches agents the optimal scanning recipes (schema discovery, filtered
+sampling, metadata-fast counts, column profiling) with zero trial-and-error.
+
+Install the skill (pick one):
+
+```bash
+# 1. Built-in installer — the skill ships inside the package (works offline)
+cloudcat --install-skill claude          # Claude Code, all projects (~/.claude/skills)
+cloudcat --install-skill claude-project  # Claude Code, this project (./.claude/skills)
+cloudcat --install-skill codex           # OpenAI Codex CLI (~/.codex/skills)
+cloudcat --install-skill print           # stdout — pipe into any other agent's instructions
+
+# 2. Claude Code plugin marketplace
+/plugin marketplace add jonathansudhakar1/cloudcat
+/plugin install cloudcat@cloudcat
+
+# 3. Or fetch it directly (no cloudcat install needed)
+curl -fsSL --create-dirs -o ~/.claude/skills/cloudcat/SKILL.md \
+  https://raw.githubusercontent.com/jonathansudhakar1/cloudcat/main/skills/cloudcat/SKILL.md
+```
+
+The recipes agents learn:
+
+```bash
+cloudcat s3://bucket/events/ -s schema_only -y                 # structure, no data read
+cloudcat s3://bucket/events/ --count -s schema_only -y         # exact rows (Parquet/ORC: metadata-only)
+cloudcat s3://bucket/events/ -w "type=purchase AND amount>250" \
+         -n 5 -o json -s dont_show -y                          # filtered NDJSON sample
+cloudcat s3://bucket/events/ --stats -n 0 -s dont_show -y      # nulls/distinct/min/max per column
+```
+
+Works with any agent framework that can run shell commands and load the
+SKILL.md (or plain markdown) as instructions. See the
+[AI Agents docs](https://cloudcatcli.com/#agents) for details.
+
 ## Roadmap
 
 - [x] Azure Blob Storage support
@@ -709,8 +786,16 @@ pytest
 - [x] Compression support (gzip, zstd, lz4, snappy, bz2)
 - [x] Row offset/pagination (`--offset`)
 - [x] Output to file with `--output-file`
+- [x] Compound WHERE conditions (`AND`/`OR`)
+- [x] Streaming WHERE with Parquet row-group pushdown
+- [x] Local file support (`file://` or plain paths)
+- [x] Configuration file with named profiles
+- [x] Column statistics (`--stats`)
+- [x] Shell completion (bash/zsh/fish)
+- [x] Delta Lake support (current snapshot; `cloudcat[delta]`)
+- [x] Apache Iceberg support (current snapshot, catalog-less; `cloudcat[iceberg]`)
 - [ ] Interactive mode with pagination
-- [ ] Configuration file support
+- [ ] Table time travel (`--at-version` / `--at-snapshot`)
 
 ## Related Projects
 

{cloudcat-0.4.0 → cloudcat-0.6.0}/README.md

@@ -83,6 +83,9 @@ pip install 'cloudcat[orc]'
 
 # With compression support (zstd, lz4, snappy)
 pip install 'cloudcat[compression]'
+
+# With lakehouse table support (Delta Lake / Apache Iceberg)
+pip install 'cloudcat[tables]'
 ```
 
 > **Note:** If using zsh (default on macOS), quotes around extras are required to prevent shell interpretation of brackets.
@@ -96,14 +99,17 @@ pip install --upgrade 'cloudcat[all]'
 ### Requirements
 
 - **Homebrew**: macOS (Apple Silicon only). Intel Mac users should use pip.
-- **pip**: Python 3.7+ (all platforms)
+- **pip**: Python 3.9+ (all platforms)
 - Cloud provider credentials configured (see [Authentication](#authentication))
 
 ## Quick Start
 
 ```bash
-# Preview a CSV file from GCS
-cloudcat -p gcs://my-bucket/data.csv
+# Preview a CSV file from GCS (PATH is positional; -p/--path also works)
+cloudcat gcs://my-bucket/data.csv
+
+# Preview a local file — no cloud credentials needed
+cloudcat ./data.parquet
 
 # Preview a Parquet file from S3
 cloudcat -p s3://my-bucket/analytics/events.parquet
@@ -123,11 +129,14 @@ cloudcat -p abfss://logs@account.dfs.core.windows.net/app.log -i text
 # Read from a Spark output directory
 cloudcat -p s3://my-bucket/spark-output/ -i parquet
 
+# Read a Delta Lake or Iceberg table (auto-detected, current snapshot only)
+cloudcat s3://lake/orders_delta/ -w "status=refunded" -n 10
+
 # Read compressed files (auto-detected)
 cloudcat -p gcs://my-bucket/data.csv.gz
 
-# Filter rows with WHERE clause
-cloudcat -p s3://bucket/users.parquet --where "status=active"
+# Filter rows with WHERE (streams; stops at -n matches; AND/OR supported)
+cloudcat s3://bucket/users.parquet --where "status=active AND age>30"
 
 # Skip first 100 rows (pagination)
 cloudcat -p gcs://bucket/data.csv --offset 100 -n 10
@@ -142,6 +151,7 @@ cloudcat -p gcs://bucket/data.csv --offset 100 -n 10
 | Google Cloud Storage | `gcs://` or `gs://` | ✅ Supported |
 | Amazon S3 | `s3://` | ✅ Supported |
 | Azure Data Lake Gen2 | `abfss://` | ✅ Supported |
+| Local files | `file://` or a plain path | ✅ Supported (no credentials needed) |
 
 ### File Format Support
 
@@ -155,6 +165,8 @@ cloudcat -p gcs://bucket/data.csv --offset 100 -n 10
 | ORC | ✅ | Hive, Hadoop ecosystem |
 | Text | ✅ | Log files, plain text |
 | TSV | Via `--delimiter` | Tab-separated data |
+| Delta Lake | ✅ table dirs | Lakehouse tables (`cloudcat[delta]`) |
+| Apache Iceberg | ✅ table dirs | Lakehouse tables (`cloudcat[iceberg]`) |
 
 ### Streaming Efficiency
 
@@ -164,8 +176,8 @@ CloudCat uses intelligent streaming to minimize data transfer and egress costs:
 |--------|-------------|---------|-------------------|----------------|
 | Parquet | None/Internal | ✅ | ✅ Range requests | ✅ |
 | Parquet | External (.gz) | ❌ | ❌ | ❌ |
-| ORC | None/Internal | ❌ | ❌ | ❌ |
-| ORC | External (.gz) | ❌ | ❌ | ❌ |
+| ORC | None/Internal | ✅ Stripe-by-stripe | ✅ | ✅ |
+| ORC | External (.gz) | ❌ | ✅ | ✅ |
 | CSV | None | ✅ | ❌ | ✅ |
 | CSV | gzip/zstd/lz4/bz2 | ✅ | ❌ | ✅ |
 | CSV | snappy | ❌ | ❌ | ❌ |
@@ -194,7 +206,7 @@ CloudCat automatically detects and decompresses files based on extension (e.g.,
 
 | Format | Flag | Description |
 |--------|------|-------------|
-| Table | `-o table` | Beautiful ASCII table with colored headers (default) |
+| Table | `-o table` | Rounded table with bold headers and type-aware coloring (default) |
 | JSON | `-o json` | Standard JSON Lines output |
 | Pretty JSON | `-o jsonp` | Syntax-highlighted, indented JSON |
 | CSV | `-o csv` | Comma-separated values |
@@ -251,8 +263,8 @@ cloudcat -p gcs://bucket/events.jsonl
 ### Filtering and Pagination
 
 ```bash
-# Filter rows with WHERE clause
-cloudcat -p s3://bucket/users.parquet --where "status=active"
+# Filter rows with WHERE (streams; stops at -n matches; AND/OR supported)
+cloudcat s3://bucket/users.parquet --where "status=active AND age>30"
 cloudcat -p gcs://bucket/events.json --where "age>30"
 cloudcat -p s3://bucket/logs.csv --where "level=ERROR"
 
@@ -401,7 +413,9 @@ cloudcat -p s3://api-dumps/response.json -o csv > data.csv
 Usage: cloudcat [OPTIONS]
 
 Options:
-  -p, --path TEXT              Cloud storage path (required)
+  PATH                         Positional path: gs://, gcs://, s3://, abfss://,
+                               file:// URL, or a plain local path
+                               (-p/--path remains as a compatible alias)
                                Format: gcs://bucket/path, s3://bucket/path,
                                or abfss://container@account.dfs.core.windows.net/path
 
@@ -410,7 +424,8 @@ Options:
 
   -O, --output-file PATH       Write rendered data to a file instead of stdout
 
-  -i, --input-format TEXT      Input format: csv, json, parquet, avro, orc, text
+  -i, --input-format TEXT      Input format: csv, json, parquet, avro, orc,
+                               text, delta, iceberg
                                [default: auto-detect from extension]
 
   -c, --columns TEXT           Comma-separated list of columns to display
@@ -422,10 +437,12 @@ Options:
   --offset INTEGER             Skip first N rows
                                [default: 0]
 
-  -w, --where TEXT             Filter rows with SQL-like conditions. Scans the
-                               file and returns up to --num-rows matching rows.
-                               Examples: "status=active", "age>30",
-                               "name contains john", "email endswith @gmail.com"
+  -w, --where TEXT             Filter rows with SQL-like conditions; combine
+                               with AND/OR. Streams the file and stops at
+                               --num-rows matches; Parquet skips row groups
+                               via column statistics.
+                               Examples: "status=active", "age>30 AND age<65",
+                               "level=ERROR or level=FATAL"
 
   -s, --schema TEXT            Schema display: show, dont_show, schema_only
                                [default: show]
@@ -450,6 +467,13 @@ Options:
 
   --credentials TEXT           Path to GCP service account JSON file
 
+  --stats                      Show per-column statistics (nulls, distinct,
+                               min/max) over the retrieved rows
+
+  --config-profile TEXT        Named profile from ~/.config/cloudcat/config.toml
+
+  --completion [bash|zsh|fish] Print the shell completion script and exit
+
   --az-access-key TEXT         Azure storage account access key
 
   -y, --yes                    Skip confirmation prompts (for scripting)
@@ -626,6 +650,50 @@ pip install -e ".[all]"
 pytest
 ```
 
+## Use with AI Agents
+
+CloudCat is optimized for AI coding agents: **stdout carries only data**
+(diagnostics go to stderr, color auto-disables when piped), `-o json` emits
+parseable NDJSON, `-y` makes every command non-interactive, filtering streams
+with Parquet row-group pushdown, and local files need no credentials.
+
+The repo ships an [Agent Skill](https://agentskills.io) at
+[`skills/cloudcat/SKILL.md`](skills/cloudcat/SKILL.md) — a compact reference
+that teaches agents the optimal scanning recipes (schema discovery, filtered
+sampling, metadata-fast counts, column profiling) with zero trial-and-error.
+
+Install the skill (pick one):
+
+```bash
+# 1. Built-in installer — the skill ships inside the package (works offline)
+cloudcat --install-skill claude          # Claude Code, all projects (~/.claude/skills)
+cloudcat --install-skill claude-project  # Claude Code, this project (./.claude/skills)
+cloudcat --install-skill codex           # OpenAI Codex CLI (~/.codex/skills)
+cloudcat --install-skill print           # stdout — pipe into any other agent's instructions
+
+# 2. Claude Code plugin marketplace
+/plugin marketplace add jonathansudhakar1/cloudcat
+/plugin install cloudcat@cloudcat
+
+# 3. Or fetch it directly (no cloudcat install needed)
+curl -fsSL --create-dirs -o ~/.claude/skills/cloudcat/SKILL.md \
+  https://raw.githubusercontent.com/jonathansudhakar1/cloudcat/main/skills/cloudcat/SKILL.md
+```
+
+The recipes agents learn:
+
+```bash
+cloudcat s3://bucket/events/ -s schema_only -y                 # structure, no data read
+cloudcat s3://bucket/events/ --count -s schema_only -y         # exact rows (Parquet/ORC: metadata-only)
+cloudcat s3://bucket/events/ -w "type=purchase AND amount>250" \
+         -n 5 -o json -s dont_show -y                          # filtered NDJSON sample
+cloudcat s3://bucket/events/ --stats -n 0 -s dont_show -y      # nulls/distinct/min/max per column
+```
+
+Works with any agent framework that can run shell commands and load the
+SKILL.md (or plain markdown) as instructions. See the
+[AI Agents docs](https://cloudcatcli.com/#agents) for details.
+
 ## Roadmap
 
 - [x] Azure Blob Storage support
@@ -636,8 +704,16 @@ pytest
 - [x] Compression support (gzip, zstd, lz4, snappy, bz2)
 - [x] Row offset/pagination (`--offset`)
 - [x] Output to file with `--output-file`
+- [x] Compound WHERE conditions (`AND`/`OR`)
+- [x] Streaming WHERE with Parquet row-group pushdown
+- [x] Local file support (`file://` or plain paths)
+- [x] Configuration file with named profiles
+- [x] Column statistics (`--stats`)
+- [x] Shell completion (bash/zsh/fish)
+- [x] Delta Lake support (current snapshot; `cloudcat[delta]`)
+- [x] Apache Iceberg support (current snapshot, catalog-less; `cloudcat[iceberg]`)
 - [ ] Interactive mode with pagination
-- [ ] Configuration file support
+- [ ] Table time travel (`--at-version` / `--at-snapshot`)
 
 ## Related Projects
 

{cloudcat-0.4.0 → cloudcat-0.6.0}/cloudcat/__init__.py

@@ -2,4 +2,4 @@
 #!/usr/bin/env python
 """cloudcat - A CLI utility to read and display files from cloud storage."""
 
-__version__ = "0.4.0"
+__version__ = "0.6.0"