tab-cli 0.1.4__tar.gz → 0.1.6__tar.gz

tab_cli-0.1.6/AGENTS.md

@@ -0,0 +1,158 @@
+# AGENTS.md
+
+## Purpose
+
+This file gives coding agents the repository-specific commands and conventions for `tab-cli`.
+Use it as the default operating guide when changing code in this repo.
+
+## Project Snapshot
+
+- Language: Python.
+- Package manager and task runner: `uv`.
+- Build backend: Hatchling.
+- CLI framework: Typer.
+- Data engine: Polars.
+- Terminal rendering: Rich.
+- Logging: Loguru.
+- Main package path: `src/tab_cli`.
+- Tests path: `tests`.
+- Docs path: `docs` with `mkdocs.yml`.
+
+## Setup
+
+- Sync the development environment with `uv sync --dev`.
+- Install the CLI tool locally with `uv tool install . --force`.
+- Python requirement is `>=3.10`.
+- Optional cloud extras exist for `s3`, `gcs`, and `azure`.
+- `uv.lock` is committed; prefer `uv` commands over raw `pip` when possible.
+
+## Makefile Notes
+
+- `make dev` runs `uv sync --dev`.
+- `make test` runs `uv run pytest`.
+- `make build` runs `uv build` after `clean`.
+- `make docs` runs the MkDocs build target.
+- The current `Makefile` uses `tab_cli/` for `lint`, `format`, and `typecheck`.
+- The actual package lives under `src/tab_cli`, so prefer direct `uv run ... src/tab_cli` commands unless the Makefile is updated.
+
+## Test Guidance
+
+- Prefer targeted pytest node IDs while iterating.
+- Use `-q` for concise output when running a single test.
+- Use `-k <expr>` when the exact node ID is inconvenient.
+- The CLI tests rely on `typer.testing.CliRunner`.
+- Test data is stored under `tests/assets`.
+- Existing tests emphasize user-visible CLI output, not internal implementation details.
+- When adding CLI behavior, extend `tests/test_cli.py` unless the change clearly deserves a new file.
+- Assert both `exit_code` and key output fragments.
+- For stdin support, pass `"-"` as the path and provide `input=` to `runner.invoke(...)`.
+
+## Repository Structure
+
+- `src/tab_cli/cli.py`: Typer entrypoint and command definitions.
+- `src/tab_cli/handlers/`: reader/writer composition and CLI table output.
+- `src/tab_cli/formats/`: per-format adapters for CSV, TSV, JSONL, Avro, and Parquet.
+- `src/tab_cli/storage/`: local, cloud, and fsspec-backed storage backends.
+- `src/tab_cli/url_parser.py`: parsing for local and cloud URLs.
+- `src/tab_cli/config.py`: mutable global config for CLI flags.
+- `tests/test_cli.py`: current CLI test suite.
+
+## Architecture Conventions
+
+- Keep the CLI layer thin.
+- Put command-line option parsing in `cli.py`.
+- Put reusable IO behavior in handlers, format adapters, or storage backends.
+- Use `TableReader` to combine a storage backend with a format handler.
+- Use `TableWriter` or `FormatWriter` for output behavior.
+- Keep format-specific logic inside `src/tab_cli/formats/`.
+- Keep backend-specific auth and path normalization inside `src/tab_cli/storage/`.
+- Use `polars.LazyFrame` as the default data pipeline type.
+- Only collect eagerly when necessary for schema inspection, batch writing, counts, or summary output.
+
+## Imports
+
+- Follow the repo’s existing import style: standard library, third-party, then local imports.
+- Avoid unused imports; Ruff is already catching several in the repo.
+- Prefer explicit imports from sibling modules over wildcard imports.
+- `__init__.py` files may re-export public types via `__all__`.
+- Keep imports at module scope unless a local import is needed to avoid optional dependency failures or circular imports.
+- Local imports are already used when backend selection depends on optional cloud packages.
+
+## Formatting
+
+- Use `ruff format` for code formatting.
+- Match the existing style: double quotes, compact functions, minimal vertical whitespace.
+- Keep lines readable rather than aggressively compressed.
+- Prefer small helper functions when a command body becomes dense.
+- Do not introduce decorative comments or banner blocks.
+- Keep docstrings short and factual.
+
+## Types
+
+- Type hints are used widely and should be preserved.
+- Prefer modern built-in generics like `list[str]` and `dict[str, Any]`.
+- Use `X | None` instead of `Optional[X]` in new code unless matching nearby style requires otherwise.
+- Use `TypeAlias` and `Annotated` where CLI option reuse materially improves readability.
+- Keep abstract method signatures precise in base classes.
+- Return `pl.LazyFrame` for lazy operations and `pl.DataFrame` for eager stream reads.
+- Be careful with optional third-party imports; they currently create type-check noise.
+- When adding types around cloud storage options, prefer signatures broad enough for mixed value types if the implementation returns bools or nested dicts.
+
+## Naming
+
+- Use `snake_case` for functions, variables, and module names.
+- Use `CamelCase` for classes and dataclasses.
+- Use clear suffixes like `Format`, `Backend`, `Reader`, `Writer`, `Summary`, and `Schema`.
+- For CLI option aliases in `cli.py`, the repo uses `PathArg`, `InputOpt`, `SqlOpt`, and similar names.
+- Keep names explicit about data shape or role, for example `storage_options`, `table_summary`, or `parsed`.
+
+## Error Handling
+
+- Raise `ValueError` for invalid user input, unknown formats, unsupported schemes, or inconsistent data layout.
+- Raise `ImportError` when an optional backend dependency is required but missing.
+- Prefer helpful, user-actionable error messages.
+- Preserve current behavior where backend auth methods fail over to the next strategy.
+- Log fallback behavior with `logger.debug(...)` or `logger.warning(...)` when it aids diagnosis.
+- Do not silently swallow errors unless the code is intentionally probing multiple auth mechanisms.
+
+## Logging And Output
+
+- The CLI configures Loguru with `RichHandler` in the Typer callback.
+- User-facing table and summary output is rendered with Rich.
+- Streaming command output usually writes bytes to `sys.stdout.buffer`.
+- Keep stderr/stdout behavior consistent with the existing command design.
+- Avoid printing directly unless a component is explicitly designed for it.
+
+## Data And Polars Usage
+
+- Prefer lazy scans (`pl.scan_csv`, `pl.scan_parquet`, `pl.scan_ndjson`) for file input.
+- Use eager reads only for streams or when a library requires it.
+- For row counts, the repo commonly uses `.select(pl.len()).collect().item()`.
+- For directory reads, preserve the existing split between native glob support and manual concatenation.
+- When reading directories, keep metadata and hidden-file filtering behavior intact.
+- If adding a new format, implement the `FormatHandler` contract fully.
+
+## CLI and UX Expectations
+
+- Keep Typer command help concise.
+- Reuse the existing type aliases for command arguments and options when practical.
+- Maintain stdin support via `"-"` consistently across commands.
+- When input comes from stdin, require explicit format selection if extension inference is impossible.
+- Preserve output format inference rules unless intentionally changing UX.
+
+## Documentation
+
+- Update `docs/` and CLI help text when changing user-facing behavior.
+- Update `CHANGELOG.md` with a clear description of user-facing changes and bug fixes.
+- Keep examples aligned with the actual command names and flags.
+- If you change build, test, or auth flows, reflect that in this file too.
+
+## Agent Workflow Recommendations
+
+- Start with targeted tests for the touched command or module.
+- Run `uv run ruff check src tests` on touched files before finishing.
+- Run `uv run ty check src/tab_cli` when changing signatures or backend option types.
+- Run `uv run pytest` for broader validation before finalizing cross-cutting changes.
+- Mention pre-existing lint or type-check failures separately from regressions you introduce.
+- Update CHANGELOG.md when necessary.
+- 

tab_cli-0.1.6/CHANGELOG.md

@@ -0,0 +1,18 @@
+  - 0.1.6:
+    - Fixed bug in pyarrow loading of Parquet files.
+    - Added global config file support: settings can be persisted in `~/.config/tab/config.json`. Config file values serve as defaults that CLI flags override.
+  - 0.1.5:
+    - Added stdin support: use `-` as the file path to read from stdin (e.g. `cat data.csv | tab view -i csv -`). Requires `-i`/`--input-format` since format cannot be inferred. Works with `view`, `schema`, `summary`, `convert`, and `cat`.
+    - Added row-wise JMESPath queries via `--jmespath` / `--jp` on `view`, `convert`, and `cat`. Object results become columns; non-object results go into a `value` column. `--sql` and `--jp` are mutually exclusive.
+      - Implemented `--jp` with `LazyFrame.map_batches(...)` so row reshaping stays batch-oriented instead of materializing the full transformed dataset up front.
+ - 0.1.4:
+   - Removed `tab sql` subcommand; SQL is now a `--sql` option on `tab view`, `tab convert`, and `tab cat`.
+   - Automatic PyArrow fallback for Parquet files that fail to read with Polars' native reader.
+ - 0.1.3:
+   - Separate `tab view` from `tab cat`: `tab view` does not convert formats, `tab cat` does.
+   - Added `--max-cell-len` option to `tab view` to truncate long cell contents.
+ - 0.1.2:
+   - Bugfix on reading directories.
+ - 0.1.1:
+   - Better credential handling for Azure Blob Storage and Google Cloud Storage.
+ - 0.1.0: Initial release

{tab_cli-0.1.4 → tab_cli-0.1.6}/PKG-INFO

@@ -1,12 +1,13 @@
 Metadata-Version: 2.4
 Name: tab-cli
-Version: 0.1.4
+Version: 0.1.6
 Summary: A CLI tool for tabular data
 Author-email: Tongfei Chen <tongfei@pm.me>
 License-File: LICENSE
 Requires-Python: >=3.10
 Requires-Dist: blobfile>=3.0
 Requires-Dist: fsspec>=2025.1.0
+Requires-Dist: jmespath>=1.0
 Requires-Dist: loguru>=0.7.3
 Requires-Dist: polars-fastavro>=0.5.1
 Requires-Dist: polars>=1.0

{tab_cli-0.1.4 → tab_cli-0.1.6}/docs/cli-ref.md

@@ -13,7 +13,8 @@ Options:
 | Option                  | Description                                                                                               |
 |-------------------------|-----------------------------------------------------------------------------------------------------------|
 | `-i` / `--input-format` | Input format (`parquet`, `csv`, `tsv`, `jsonl`, `avro`). Auto-detected from extension if omitted.         |
-| `--sql`                 | SQL query to apply before displaying. The table is available as `t`.                                      |
+| `--sql`                 | SQL query to apply before displaying. The table is available as `t`. Mutually exclusive with `--jmespath`. |
+| `--jmespath` / `--jp`   | JMESPath expression to apply to each row as JSON. Object outputs become columns; non-object outputs go to a `value` column. The result shape must stay consistent across rows. |
 | `--limit`               | Maximum number of rows to display.                                                                        |
 | `--skip`                | Number of rows to skip from the beginning.                                                                |
 | `--max-cell-len`        | Truncate cell contents longer than this.                                                                 |
@@ -62,7 +63,8 @@ Options:
 |-------------------------|---------------------------------------------------------------------------------------------------------|
 | `-i` / `--input-format` | Input format (`parquet`, `csv`, `tsv`, `jsonl`, `avro`). Auto-detected from extension if omitted.       |
 | `-o` / `--output-format` | Output format (`parquet`, `csv`, `tsv`, `jsonl`, `avro`). If not specified, inherits from input format. |
-| `--sql`                 | SQL query to apply before writing. The table is available as `t`.                                       |
+| `--sql`                 | SQL query to apply before writing. The table is available as `t`. Mutually exclusive with `--jmespath`. |
+| `--jmespath` / `--jp`   | JMESPath expression to apply to each row as JSON. Object outputs become columns; non-object outputs go to a `value` column. The result shape must stay consistent across rows. |
 | `-n` / `--num-partitions` | Number of output partitions. Creates a directory with partition files.                                  |
 
 
@@ -80,7 +82,8 @@ Options:
 |-------------------------|-----------------------------------------------------------------------------------------------------------|
 | `-i` / `--input-format` | Input format (`parquet`, `csv`, `tsv`, `jsonl`, `avro`). Auto-detected from extension if omitted.         |
 | `-o` / `--output-format` | Output format (`parquet`, `csv`, `tsv`, `jsonl`, `avro`). If not specified, print Rich table in terminal. |
-| `--sql`                 | SQL query to apply after concatenation. The table is available as `t`.                                    |
+| `--sql`                 | SQL query to apply after concatenation. The table is available as `t`. Mutually exclusive with `--jmespath`. |
+| `--jmespath` / `--jp`   | JMESPath expression to apply to each row as JSON. Object outputs become columns; non-object outputs go to a `value` column. The result shape must stay consistent across rows. |
 
 
 ## Global options

tab_cli-0.1.6/docs/configuration.md

@@ -0,0 +1,34 @@
+# Configuration
+
+`tab` supports a global configuration file at `~/.config/tab/config.json`. Settings in this file serve as defaults and are overridden by CLI flags.
+
+## Setup
+
+Create the config file:
+
+```bash
+mkdir -p ~/.config/tab
+cat > ~/.config/tab/config.json << 'EOF'
+{
+  "az_url_authority_is_account": false,
+  "sampling_size_for_schema_inference": 32
+}
+EOF
+```
+
+## Available settings
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `az_url_authority_is_account` | `bool` | `false` | Interpret `az://` URL authority as storage account name instead of container name. |
+| `sampling_size_for_schema_inference` | `int` | `32` | Number of rows sampled for schema inference (e.g. when using `--jp`). |
+
+## Precedence
+
+Settings are applied in this order (last wins):
+
+1. Built-in defaults
+2. Config file (`~/.config/tab/config.json`)
+3. CLI flags (e.g. `--az-url-authority-is-account`)
+
+If the config file does not exist, built-in defaults are used. Unknown keys in the file are ignored with a warning.

tab_cli-0.1.6/docs/gen_assets.sh

@@ -0,0 +1,3 @@
+mkdir -p docs/assets
+uv run tab view tests/assets/test.csv --table-svg 2> docs/assets/test.svg
+uv run tab view --sql 'SELECT * FROM t WHERE Metric_A_Value > 80' tests/assets/test.csv --table-svg 2> docs/assets/test-where.svg

{tab_cli-0.1.4 → tab_cli-0.1.6}/docs/index.md

@@ -44,10 +44,25 @@ tab summary data.parquet
 Run SQL queries on your data. The table is referenced as `t`:
 
 ```bash
-tab sql 'SELECT * FROM t WHERE Metric_A_Value > 80' test.csv
+tab view --sql 'SELECT * FROM t WHERE Metric_A_Value > 80' test.csv
 ```
 ![tab sql](assets/test-where.svg)
 
+### JMESPath row reshaping
+
+Run a JMESPath expression against each row as a JSON object:
+
+```bash
+tab view ugly.parquet --jp '{id: participant.id, city: profile.address.city}'
+tab cat ugly.parquet --jp 'profile.address.city' -o jsonl
+tab cat ugly.parquet --jp '[participant.id, profile.address.city]' -o jsonl
+```
+
+- Object results become output columns.
+- Non-object results are written to a single `value` column.
+- The expression should return a consistent shape across the dataset.
+- Use either `--sql` or `--jp`, but not both on the same command.
+
 ### Convert
 
 Convert between formats:
@@ -85,4 +100,3 @@ tab cat data1.csv data2.csv data3.csv -o jsonl > output.jsonl
 | Option | Description |
 |--------|-------------|
 | `-n`   | Number of output partitions. Creates a directory with part files. |
-

{tab_cli-0.1.4 → tab_cli-0.1.6}/mkdocs.yml

@@ -31,6 +31,7 @@ repo_url: 'https://github.com/ctongfei/tab'
 
 nav:
   - 'Introduction': 'index.md'
+  - 'Configuration': 'configuration.md'
   - 'Cloud support': 'cloud.md'
   - 'CLI reference': 'cli-ref.md'
 

{tab_cli-0.1.4 → tab_cli-0.1.6}/pyproject.toml

@@ -1,12 +1,13 @@
 [project]
 name = "tab-cli"
-version = "0.1.4"
+version = "0.1.6"
 description = "A CLI tool for tabular data"
 authors = [{name = "Tongfei Chen", email = "tongfei@pm.me"}]
 readme = "README.md"
 repository = "https://github.com/tongfei/tab"
 requires-python = ">=3.10"
 dependencies = [
+    "jmespath>=1.0",
     "typer>=0.21.1",
     "rich>=13.0",
     "polars>=1.0",