@treasuredata/tdx 0.1.0

package/README.md

@@ -0,0 +1,706 @@
+# tdx - AI-Native CLI for Treasure Data
+
+> **Status:** 🚧 Under Development (Phase 1)
+
+A modern, AI-native command-line interface for Treasure Data, optimized for both human engineers and AI coding assistants.
+
+## Features
+
+- 🤖 **AI-Native Design**: Consistent JSON output, predictable error codes, structured responses
+- 🌍 **Multi-Site Support**: Easy switching between US, JP, EU, and AP regions
+- 🚀 **Modern Distribution**: Install via `npx` or `npm` without prior setup
+- ⚡ **Fast & Efficient**: Built with TypeScript on Node.js 24+
+
+## Installation
+
+```bash
+# Run directly with npx (no installation required)
+npx @treasuredata/tdx databases
+
+# Or install globally
+npm install -g @treasuredata/tdx
+
+# Alternative: Use with Bun
+bun x @treasuredata/tdx databases
+```
+
+## Quick Start
+
+### 1. Configure API Key
+
+Create `~/.config/tdx/.env`:
+
+```bash
+# Single API key (recommended for most users)
+TD_API_KEY=your-api-key-here/...
+
+# Or use site-specific keys if working with multiple sites
+# TD_API_KEY_US01=your-us-key-here/...
+# TD_API_KEY_JP01=your-jp-key-here/...
+```
+
+Alternatively, set as environment variable:
+```bash
+# Export for all commands in session
+export TD_API_KEY=your-api-key-here/...
+
+# Or set inline for a single command
+TD_API_KEY=your-api-key-here/... tdx databases
+```
+
+### 2. Run Commands
+
+```bash
+# List databases
+tdx databases
+
+# List databases in Tokyo region
+tdx databases --site jp01
+
+# Show table contents
+tdx show sample_datasets.www_access --limit 10
+
+# Run a query
+tdx query "SELECT * FROM mytable LIMIT 10"
+
+# Get JSON output (two ways)
+tdx databases --json
+tdx databases --format json
+
+# Get JSON Lines output
+tdx query "SELECT * FROM users" --jsonl
+
+# Get TSV output
+tdx query "SELECT * FROM users" --tsv
+
+# Get human-readable table output (default)
+tdx databases --format table
+```
+
+## Available Commands
+
+```bash
+tdx <command> [options] [arguments]
+```
+
+### Global Options
+
+Available for all commands:
+
+- `--site <site>`: TD site/region (us01, jp01, eu01, ap02) - default: us01
+- `--format <format>`: Output format (table, json, jsonl, tsv) - default: table
+- `--json`: Output in JSON format (shorthand for --format json)
+- `--jsonl`: Output in JSON Lines format (shorthand for --format jsonl)
+- `--tsv`: Output in TSV format (shorthand for --format tsv)
+- `--output <file>`: Save output to file
+- `--limit <rows>`: Maximum rows to display in table format - default: 40
+- `--color`: Force ANSI color output (overrides TTY detection)
+- `--no-color`: Disable ANSI color output (also respects NO_COLOR env var)
+- `--verbose`: Enable verbose logging
+- `--timeout <seconds>`: Set operation timeout - default: 30
+- `--dry-run`: Preview operation without executing
+- `-y, --yes`: Skip confirmation prompts
+
+**Note:** Shorthand flags (--json, --jsonl, --tsv) are only applied when --format is not explicitly specified. If both are provided, --format is used.
+
+### Interactive Table Navigation
+
+When viewing table output in an interactive terminal, `tdx` automatically pipes the output through `less` for easy navigation of large datasets. This allows you to:
+
+- View all rows without truncation (unlimited data display)
+- Scroll through wide tables without column truncation
+- Search and navigate efficiently
+
+**Keyboard shortcuts in less:**
+
+| Key | Action |
+|-----|--------|
+| `q` | Quit and return to terminal |
+| `↑` / `↓` | Scroll up/down one line |
+| `Space` / `Page Down` | Scroll down one page |
+| `b` / `Page Up` | Scroll up one page |
+| `←` / `→` | Scroll left/right (for wide tables) |
+| `/pattern` | Search forward for pattern |
+| `?pattern` | Search backward for pattern |
+| `n` | Repeat previous search forward |
+| `N` | Repeat previous search backward |
+| `g` | Go to first line |
+| `G` | Go to last line |
+| `h` | Show help |
+
+**Note:** Less pagination is automatically enabled for table format output in interactive terminals. It's disabled when:
+- Output is piped to another command
+- Output is saved to a file with `--output`
+- Using non-table formats (JSON, JSONL, TSV)
+
+
+### Database Commands
+
+| Command | Description | Example |
+|---------|-------------|---------|
+| `databases` | List all databases | `tdx databases` |
+| | Filter by pattern (glob/wildcards) | `tdx databases "prod_*"` or `tdx databases "*_analytics"` |
+
+### Table Commands
+
+All table commands use dot-separated patterns: `(database).(table)`
+
+| Command | Description | Example |
+|---------|-------------|---------|
+| `tables` | List all tables from all databases | `tdx tables` |
+| | List all tables from specific database | `tdx tables "mydb.*"` or `tdx tables --in mydb` |
+| | List tables from database (--database flag) | `tdx tables --database mydb` |
+| | Filter tables with pattern | `tdx tables "mydb.user_*"` |
+| | Database pattern with table | `tdx tables "prod_*.access_log"` |
+| | Wildcard database and table | `tdx tables "*.user*"` |
+| | All tables from all databases | `tdx tables "*.*"` |
+| `describe <table>` (alias: `desc`) | Show table schema | `tdx describe mydb.users` |
+| | Using --in or --database flag | `tdx describe users --in mydb` or `tdx describe users -d mydb` |
+| `show <table>` | Show table contents (SELECT * with limit) | `tdx show mydb.users` |
+| | With custom row limit | `tdx show mydb.users --limit 10` |
+| | With explicit catalog | `tdx show td.mydb.users --limit 5` |
+| | Using --in or --database flag | `tdx show users --in mydb` or `tdx show users -d mydb` |
+
+**Pattern syntax:**
+- Database wildcard: `"mydb.*"` → all tables from mydb (use this or `--database mydb`)
+- Database.table: `mydb.users` → specific table
+- Wildcards: `"*.users"`, `"prod_*.user*"` → pattern matching
+- Catalog: `"td.mydb.users"` → with catalog prefix
+
+**Table-specific options:**
+- `-d, --database <database>`: Specify database name when not included in pattern
+- `--in <database>`: Alias for `--database` (natural language style)
+
+**Alternative syntax for specifying database:**
+
+There are multiple ways to specify the database for table commands:
+
+```bash
+# Using dot notation (recommended for patterns)
+tdx tables "mydb.*"
+
+# Using --in flag (natural language style, no quotes needed)
+tdx tables --in mydb
+
+# Using --database flag (or -d shorthand)
+tdx tables --database mydb
+tdx tables -d mydb
+
+# All three are equivalent for listing all tables from a database
+
+# Works with describe and show commands too
+tdx describe users --in mydb
+tdx show users --in mydb
+
+# List tables matching pattern from "mydb"
+tdx tables "user*" --in mydb
+```
+
+**Note:** If the pattern already includes a database (e.g., `mydb.users`), the `--database` option is ignored:
+
+```bash
+# Uses "mydb" from pattern, ignores --database option
+tdx tables mydb.users --database other
+# Result: Lists users table from mydb, not other
+```
+
+### Query Commands
+
+Usage: `tdx query [options] [sql]`
+
+| Option/Usage | Description | Example |
+|--------------|-------------|---------|
+| `<sql>` | Execute SQL query | `tdx query "SELECT * FROM table"` |
+| `-f, --file <path>` | Execute SQL from file | `tdx query -f query.sql` |
+| Multi-statement | Execute multiple statements (CLI or file) | `tdx query "SELECT 1; SELECT 2"` |
+| | Multi-statement from file | `tdx query -f setup-and-query.sql` |
+
+**Query-specific options**:
+- `-f, --file <path>`: Read SQL from file
+- `-d, --database <db>`: Database to query (default: information_schema)
+- `--in <db>`: Alias for `--database` (natural language style)
+- `--catalog <catalog>`: Trino catalog (default: td)
+
+**Examples**:
+```bash
+# Query with --in flag
+tdx query "SELECT * FROM users" --in mydb
+
+# Equivalent to --database
+tdx query "SELECT * FROM users" --database mydb
+tdx query "SELECT * FROM users" -d mydb
+```
+
+### Job Commands
+
+Usage: `tdx job <subcommand>`
+
+| Subcommand | Description | Status |
+|------------|-------------|--------|
+| - | Job management | Coming soon |
+
+### Segment Commands (CDP)
+
+Usage: `tdx segment <subcommand> [options] [arguments]`
+
+CDP (Customer Data Platform) segment management for audiences and activations.
+
+**ID Format Conventions:**
+- Parent segment: `parent_id` (e.g., `987900`)
+- Child segment: `parent_id.child_id` (e.g., `987900.1463552`) - uses dot notation
+- Folder ID: Direct folder ID (e.g., `1453821`)
+- Pattern: Commands accept `(parent_id)(.child_id)?` format
+
+| Command | Description | Example |
+|---------|-------------|---------|
+| `segments` | List all parent segments | `tdx segments` |
+| `segments <parent_id>` | List child segments under parent | `tdx segments 987900` |
+| `segment describe <id>` (alias: `desc`) | Show parent or child segment details | `tdx segment describe 987900` or `tdx segment desc 987900.1463552` |
+| `segment show <id>` | Execute segment SQL query and show results | `tdx segment show 987900` or `tdx segment show 987900.1463552 --limit 100` |
+| `segment sql <id>` | Get SQL query for segment | `tdx segment sql 987900` or `tdx segment sql 987900.1463552` |
+| `segment folders <parent_id>` | List folders under parent | `tdx segment folders 987900` |
+| `segment folder show <folder_id>` | Show folder details | `tdx segment folder show 1453821` |
+
+**Examples**:
+```bash
+# List all parent segments
+tdx segments --format json
+
+# List child segments under parent
+tdx segments 987900
+
+# Show parent segment metadata
+tdx segment describe 987900
+# or use the alias
+tdx segment desc 987900
+
+# Show child segment metadata (dot notation)
+tdx segment describe 987900.1463552
+
+# Execute parent segment query and show results (default: 40 rows)
+tdx segment show 987900
+
+# Execute child segment query with custom limit (dot notation)
+tdx segment show 987900.1463552 --limit 100
+
+# Execute segment query and save as JSON
+tdx segment show 987900 --format json --output results.json
+
+# Get SQL for parent segment
+tdx segment sql 987900 > parent-query.sql
+
+# Get SQL for child segment with filtering (dot notation)
+tdx segment sql 987900.1463552 > child-query.sql
+
+# List folders under parent
+tdx segment folders 987900
+
+# Show folder details (direct folder ID)
+tdx segment folder show 1453821
+```
+
+### Activation Commands (CDP)
+
+Manage CDP activations (syndications) for segments.
+
+| Command | Description | Example |
+|---------|-------------|---------|
+| `activations <segment_id>` | List activations for a child segment | `tdx activations 987900.1463552` |
+
+**Examples**:
+```bash
+# List activations for a child segment (dot notation)
+tdx activations 987900.1463552
+
+# Get activations as JSON
+tdx activations 987900.1463552 --format json
+
+# Using Tokyo region
+tdx activations 987900.1463552 --site jp01
+```
+
+### Workflow Commands
+
+Usage: `tdx workflow <subcommand>` (alias: `tdx wf`)
+
+Manage Treasure Data workflows powered by Digdag. Monitor execution sessions, view logs, and control workflow attempts.
+
+| Subcommand | Description | Example |
+|------------|-------------|---------|
+| `projects [pattern]` | List workflow projects (supports glob patterns) | `tdx wf projects` or `tdx wf projects "cdp_*"` |
+| `workflows [project]` (alias: `ls`) | List workflows (filter by project, supports globs) | `tdx wf workflows` or `tdx wf workflows "lda*"` |
+| `sessions [project]` | List workflow execution sessions | `tdx wf sessions` or `tdx wf sessions myproject` or `tdx wf sessions myproject.daily_etl` |
+| `attempts [project]` | List workflow attempts | `tdx wf attempts` or `tdx wf attempts myproject` or `tdx wf attempts myproject.workflow` |
+| `attempt <attempt-id>` | Show specific attempt details | `tdx wf attempt 67890` |
+| `tasks <attempt-id>` | Show tasks for an attempt | `tdx wf tasks 67890` |
+| `logs <attempt-id> <task-name>` | Show logs for a specific task | `tdx wf logs 67890 +step1` |
+| `kill <attempt-id>` | Kill a running attempt | `tdx wf kill 67890` |
+| `retry <session-id\|attempt-id>` | Retry a session or attempt | `tdx wf retry session:12345` or `tdx wf retry attempt:67890` |
+
+**Workflow-specific options:**
+
+For `sessions` command:
+- `--status <status>`: Filter by status (running, success, error, blocked, all)
+- `--from <timestamp>`: Start time filter (ISO 8601 format)
+- `--to <timestamp>`: End time filter (ISO 8601 format)
+
+For `attempts` command:
+- `--include-retried`: Include retried attempts in the output
+
+For `tasks` command:
+- `--include-subtasks`: Include subtasks in the output
+
+For `logs` command:
+- `--offset <number>`: Starting offset for logs (default: 0)
+
+For `kill` command:
+- `--reason <text>`: Reason for killing the attempt
+- `-y, --yes`: Skip confirmation prompt
+
+For `retry` command:
+- `--from-task <task>`: Resume from specific task (session retry only)
+- `--resume-from <task>`: Resume from specific task (attempt retry only)
+- `--params <json>`: Override parameters (JSON string or @file.json)
+- `--force`: Force retry even if not failed (attempt retry only)
+- `-y, --yes`: Skip confirmation prompt
+
+**Examples**:
+```bash
+# List all workflow projects
+tdx wf projects
+
+# Filter projects by pattern
+tdx wf projects "cdp_segment_*"
+
+# List all workflows
+tdx wf workflows
+
+# List workflows for a specific project
+tdx wf workflows myproject
+
+# Filter workflows by project pattern
+tdx wf workflows "lda*"
+
+# List all workflow sessions
+tdx wf sessions
+
+# List sessions for a specific project
+tdx wf sessions myproject
+
+# List sessions for specific workflow
+tdx wf sessions myproject.daily_etl
+
+# Filter sessions by status
+tdx wf sessions --status error
+
+# Filter sessions by time range
+tdx wf sessions --from "2025-01-01" --to "2025-01-31"
+
+# List all attempts
+tdx wf attempts
+
+# List attempts for a project
+tdx wf attempts myproject
+
+# List attempts for a workflow
+tdx wf attempts myproject.daily_etl
+
+# Include retried attempts
+tdx wf attempts --include-retried
+
+# Show specific attempt details
+tdx wf attempt 67890
+
+# Show tasks for an attempt
+tdx wf tasks 67890
+
+# Show tasks including subtasks
+tdx wf tasks 67890 --include-subtasks
+
+# View logs for a specific task
+tdx wf logs 67890 +step1
+
+# View logs with pagination
+tdx wf logs 67890 +step1 --offset 100 --limit 500
+
+# Kill a running attempt (with confirmation)
+tdx wf kill 67890
+
+# Kill with reason and skip confirmation
+tdx wf kill 67890 --reason "manual stop" -y
+
+# Retry entire session
+tdx wf retry session:12345
+
+# Retry session from specific task
+tdx wf retry session:12345 --from-task +step2
+
+# Retry attempt
+tdx wf retry attempt:67890
+
+# Retry attempt from specific task
+tdx wf retry attempt:67890 --resume-from +step2
+
+# Retry with parameter override
+tdx wf retry attempt:67890 --params '{"key":"value"}'
+
+# Retry with params from file
+tdx wf retry attempt:67890 --params @params.json
+
+# Force retry without confirmation
+tdx wf retry attempt:67890 --force -y
+
+# Get workflow data as JSON
+tdx wf sessions --format json --output sessions.json
+```
+
+## Advanced Query Features
+
+### File Input
+
+Read SQL queries from files using the `-f` or `--file` option:
+
+```bash
+# Execute SQL from file
+tdx query --file query.sql
+
+# Short form
+tdx query -f query.sql
+
+# With options
+tdx query -f query.sql --database mydb --format json --output results.json
+```
+
+### Multi-Statement Execution
+
+Execute multiple SQL statements sequentially by separating them with semicolons. This works for both command-line input and file input:
+
+**From command line**:
+```bash
+tdx query "SELECT COUNT(*) FROM users; SELECT COUNT(*) FROM orders"
+```
+
+**From file** (`setup-and-query.sql`):
+```sql
+CREATE TABLE temp_users AS SELECT * FROM users WHERE active = true;
+CREATE TABLE temp_orders AS SELECT * FROM orders WHERE user_id IN (SELECT id FROM temp_users);
+SELECT COUNT(*) as order_count FROM temp_orders;
+```
+
+```bash
+tdx query -f setup-and-query.sql
+```
+
+**Output behavior**:
+- Each statement executes sequentially with progress indication
+- **Console output**: All results displayed as they complete
+- **File output** (`--output results.txt`): All results appended to single file
+- **Fail-fast**: Execution stops on first error
+
+**Example output**:
+```
+✔ Statement 1/3 - Query completed: Processed 1,000 rows in 2.3s
+┌────────────┐
+│ user_count │
+├────────────┤
+│ 1000       │
+└────────────┘
+✔ Statement 2/3 - Query completed: Processed 5,000 rows in 3.1s
+┌─────────────┐
+│ order_count │
+├─────────────┤
+│ 5000        │
+└─────────────┘
+✔ Statement 3/3 - Query completed: Processed 1 rows in 0.5s
+┌─────────────┐
+│ order_count │
+├─────────────┤
+│ 5000        │
+└─────────────┘
+```
+
+**With `--output`** (file mode):
+```bash
+tdx query -f statements.sql --output results.txt
+```
+- Console shows only progress messages
+- All results written to `results.txt` (appended sequentially)
+- Message: "All results saved to results.txt"
+
+## Output Formats
+
+tdx supports multiple output formats for query results and command outputs:
+
+### Table Format (default)
+Human-readable ASCII table with column types and row counts:
+```bash
+tdx query "SELECT name, age FROM users LIMIT 2"
+```
+```
+┌───────┬─────┐
+│ name  │ age │
+│ string│ int │
+├───────┼─────┤
+│ Alice │  25 │
+│ Bob   │  30 │
+├───────┴─────┤
+│ 2 rows      │
+└─────────────┘
+```
+
+### JSON Format
+Readable JSON array with newlines (no metadata):
+```bash
+tdx query "SELECT name, age FROM users LIMIT 2" --format json
+```
+```json
+[
+  {"name":"Alice","age":25},
+  {"name":"Bob","age":30}
+]
+```
+
+Each object is compact (single line) but separated by newlines for readability.
+
+Perfect for piping to `jq`:
+```bash
+tdx query "SELECT * FROM users" --format json | jq '.[0]'
+tdx query "SELECT * FROM users" --format json | jq 'length'
+```
+
+### JSON Lines Format (jsonl)
+One JSON object per line - ideal for streaming and line-by-line processing:
+```bash
+tdx query "SELECT name, age FROM users LIMIT 2" --format jsonl
+```
+```
+{"name":"Alice","age":25}
+{"name":"Bob","age":30}
+```
+
+Process line by line:
+```bash
+tdx query "SELECT * FROM users" --format jsonl | while read line; do
+  echo "$line" | jq '.name'
+done
+```
+
+### TSV Format
+Tab-separated values with header row:
+```bash
+tdx query "SELECT name, age FROM users LIMIT 2" --format tsv
+```
+```
+name	age
+Alice	25
+Bob	30
+```
+
+### Color Output
+
+`tdx` automatically adds ANSI colors to output in interactive terminals to improve readability:
+
+**Table format:**
+- Table borders: Dark gray (dim)
+- Column headers: Plain text (no color)
+- Type row: Dim gray
+- Data values: Plain text (no color)
+
+**JSON format:**
+- Keys: Blue
+- Strings: Green
+- Numbers: Cyan
+- Booleans: Yellow
+- Null values: Dim
+
+**JSONL format:**
+- Same color scheme as JSON format
+- Compact one-line-per-object output with colors in interactive terminals
+
+**Automatic detection:**
+- Colors are enabled automatically when output is to an interactive terminal (TTY)
+- Colors are disabled automatically when:
+  - Output is piped to another command
+  - Output is saved to a file with `--output`
+- Respects the `NO_COLOR` environment variable (see https://no-color.org/)
+
+**Manual control:**
+```bash
+# Force colors on (even when piping or saving to file)
+tdx databases --color
+tdx databases --color --output results.txt
+
+# Disable colors (even in interactive terminal)
+tdx databases --no-color
+
+# Disable colors via environment variable
+NO_COLOR=1 tdx databases
+```
+
+**Priority order:**
+1. `--no-color` flag (highest priority)
+2. `--color` flag
+3. `NO_COLOR` environment variable
+4. TTY detection (automatic)
+
+
+## Development
+
+```bash
+# Install dependencies
+npm install
+
+# Run in development mode
+npm run dev
+
+# Build
+npm run build
+
+# Run tests
+npm test
+
+# Type checking
+npm run typecheck
+
+# Linting
+npm run lint
+
+# Format code
+npm run format
+```
+
+## Requirements
+
+- **Node.js 22.0.0 or higher** (Node.js 24 Active LTS recommended for development)
+- TD API key with appropriate permissions
+
+## Architecture
+
+- **Language**: TypeScript 5.x
+- **Runtime**: Node.js 24+
+- **CLI Framework**: Commander.js
+- **Testing**: Vitest
+- **Distribution**: NPM package
+
+## Security
+
+- API keys stored in `~/.config/tdx/.env`
+- All authorization enforced server-side via API key permissions
+- Safe mode with confirmation prompts for destructive operations
+
+## License
+
+Apache-2.0
+
+## Contributing
+
+This project is maintained by the Treasure Data DevAI Unit.
+
+For issues and feature requests, please visit:
+https://github.com/treasure-data/tdx/issues
+
+## Related Documentation
+
+- **ERD**: [New TD CLI (tdx) for AI Compatibility](https://system-docs.treasuredata.com/design-notes/2025/10/31/ERD/tdx-cli) (TD internal-only)
+- **ADD**: [New TD CLI for AI Compatibility](https://system-docs.treasuredata.com/design-notes/2025/10/29/ADD/treasure-cli-ai-compatibility) (TD internal-only)
+- **JIRA**: [DEVAI-374](https://treasure-data.atlassian.net/browse/DEVAI-374)