carto-cli 0.1.0-rc.1

package/.nvmrc

@@ -0,0 +1 @@
+20.19

package/ARCHITECTURE.md

@@ -0,0 +1,497 @@
+# CARTO CLI - Engineering Architecture Document
+
+**Version:** 0.1.0
+**Last Updated:** 2025-11-06
+
+## 1. Project Overview
+
+CARTO CLI is a command-line interface for managing CARTO Geospatial Cloud Native platform resources. Built for system administrators and power users to manage maps, workflows, connections, users, and other resources via the CARTO API.
+
+**Key Goals:**
+- Provide programmatic access to CARTO platform resources
+- Enable automation and scripting of administrative tasks
+- Support multi-tenant/multi-profile authentication
+- Work as a standalone executable with no runtime dependencies
+
+## 2. Core Architecture Principles
+
+### Minimal Dependencies
+- **Zero external runtime dependencies**: Bundles into a single executable file
+- **Native Node.js modules only**: Uses built-in `https`, `http`, `fs`, `path`, `os`
+- **Strategic exceptions**: Only 1 runtime dependency:
+  - `js-yaml`: YAML parsing for tenant config fetching
+- **Optional user-installed dependencies**:
+  - `duckdb`: SQL querying of activity data (installed separately by users who need it)
+
+### Single Executable Distribution
+- TypeScript compiled to JavaScript (`dist/`)
+- Bundled with `@vercel/ncc` into single file (`bundle/carto`)
+- Optional: Platform-specific binaries via `pkg` (macOS, Linux, Windows)
+
+### Custom HTTP Implementation
+- No dependencies on `axios`, `node-fetch`, or similar HTTP libraries
+- Direct usage of Node.js native `http`/`https` modules for full control
+- Custom streaming support for NDJSON responses (AI features)
+
+## 3. Technical Stack
+
+### Language & Runtime
+- **Language**: TypeScript 5.9+
+- **Target**: CommonJS modules
+- **Runtime**: Node.js 18+
+- **Bundler**: Vercel ncc (0.38+)
+
+### Dependencies Rationale
+
+**Production:**
+- `js-yaml` (4.1.0): Parse YAML config from tenant domains
+
+**Optional (User-Installed):**
+- `duckdb` (1.1.3+): SQL querying for `carto activity query` command
+  - Not included in package.json
+  - Users install separately: `npm install duckdb`
+  - Dynamic import with helpful error if missing
+  - Only needed for activity data SQL queries (~1% of users)
+
+**Development:**
+- `typescript`: Type safety and modern JavaScript features
+- `@vercel/ncc`: Bundle TypeScript output into single file
+- `pkg`: Create platform-specific native executables
+
+**Why no axios/fetch?**
+- Smaller bundle size
+- Full control over request/response handling
+- Native streaming support for NDJSON
+- No external attack surface
+
+## 4. Project Structure
+
+```
+src/
+├── index.ts              # CLI entry point, command routing
+├── api.ts                # ApiClient class, HTTP abstraction layer
+├── http.ts               # Native Node.js HTTP/HTTPS wrapper
+├── config.ts             # Configuration & credential management
+├── auth-oauth.ts         # OAuth authentication & token validation
+├── colors.ts             # ANSI color codes for terminal output
+├── logo.ts               # ASCII CARTO logo
+├── help.ts               # Help text and documentation
+└── commands/
+    ├── auth.ts           # Personal authentication (OAuth flow)
+    ├── credentials.ts    # Application credentials (API tokens, OAuth clients)
+    ├── maps.ts           # Maps CRUD operations
+    ├── workflows.ts      # Workflows CRUD operations
+    ├── connections.ts    # Connections CRUD operations
+    ├── imports.ts        # Data import operations
+    ├── users.ts          # User management operations
+    └── admin.ts          # Superadmin operations
+```
+
+**Module Responsibilities:**
+
+- **`index.ts`**: Command-line argument parsing, routing to command handlers
+- **`api.ts`**: Centralized HTTP client with authentication, multi-endpoint support
+- **`http.ts`**: Low-level HTTP request wrapper (request, streaming)
+- **`config.ts`**: Profile management, credential storage, tenant config fetching
+- **`auth-oauth.ts`**: JWT token parsing, expiration validation, OAuth helpers
+- **`commands/*`**: Feature-specific command implementations
+
+## 5. Key Design Decisions
+
+### 5.1 Custom HTTP Client (`http.ts`)
+
+**Decision**: Implement custom HTTP wrapper instead of using libraries.
+
+**Rationale:**
+- Reduce bundle size (critical for single-file distribution)
+- Full control over request/response lifecycle
+- Native NDJSON streaming support for AI features
+- Eliminate third-party dependency risks
+
+**Implementation:**
+- `request()`: Standard JSON request/response
+- `streamRequest()`: NDJSON streaming with chunk callbacks
+- Automatic protocol detection (http/https)
+- Promise-based API
+
+### 5.2 Multi-Endpoint API Architecture (`api.ts`)
+
+**Decision**: Single ApiClient class supporting multiple CARTO API endpoints.
+
+**Endpoints:**
+- `baseUrl`: Main API (v3 endpoints) - `https://{tenant}.api.carto.com`
+- `workspaceUrl`: Workspace API - `https://workspace-{tenant}.app.carto.com`
+- `accountsUrl`: Accounts API - `https://accounts.app.carto.com`
+- `aiApiUrl`: AI Features API - `https://ai-api-{tenant}.api.carto.com`
+- `litellmUrl`: LiteLLM Proxy - `https://litellm-{tenant}.api.carto.com`
+
+**URL Resolution:**
+- Dynamic: Fetch from `https://{tenant_domain}/config.yaml` at runtime
+- Static: LiteLLM URL constructed from tenant ID (not in config.yaml yet)
+- Cached: In-memory cache per process to avoid repeated fetches
+
+**Benefits:**
+- Single client instance manages all endpoints
+- Unified authentication across all APIs
+- Consistent error handling and debugging
+
+### 5.3 Authentication & Profile Management
+
+**Token Precedence** (highest to lowest):
+1. `--token` CLI flag
+2. `CARTO_API_TOKEN` environment variable
+3. Profile credentials file (`~/.carto_credentials.json`)
+4. Legacy config file (`~/.carto/config.json`)
+
+**Profile System:**
+```json
+{
+  "current_profile": "production",
+  "profiles": {
+    "production": {
+      "token": "eyJhbG...",
+      "tenant_id": "gcp-us-east1",
+      "tenant_domain": "myorg.app.carto.com",
+      "organization_id": "ac_xxxxx",
+      "organization_name": "MyOrg",
+      "user_email": "user@example.com",
+      "auth_environment": "production"
+    }
+  }
+}
+```
+
+**Design Decisions:**
+- Multi-profile support: Switch between environments/organizations
+- Suggested profile names: `{tenant}/{org}/{email}` or `{env}/{tenant}/{org}/{email}`
+- Profile switching via `CARTO_PROFILE` env var or `--profile` flag
+- OAuth-based authentication (browser flow) with JWT tokens
+- Token expiration validation and warnings (< 10% lifetime remaining)
+
+**Token Storage:**
+- **IMPORTANT**: Tokens stored WITHOUT `Bearer` prefix in credentials file
+- `Bearer` prefix added only when creating `Authorization` header
+- Prevents duplicate prefix bugs
+
+### 5.4 Configuration Management
+
+**Dynamic Tenant Configuration:**
+- Fetch API URLs from `https://{tenant_domain}/config.yaml`
+- Enables multi-tenant support without hardcoding URLs
+- Per-tenant caching to reduce network calls
+
+**YAML Config Structure:**
+```yaml
+apis:
+  accountsUrl: https://accounts.app.carto.com
+  workspaceUrl: https://workspace-{tenant}.app.carto.com
+  baseUrl: https://{tenant}.api.carto.com
+  aiApi: https://ai-api-{tenant}.api.carto.com
+```
+
+**Migration Support:**
+- Automatic migration from old credentials format to new multi-profile format
+- Preserves backward compatibility with legacy config file
+
+### 5.5 Build & Distribution Pipeline
+
+**Three-Stage Build:**
+
+1. **TypeScript Compilation** (`npm run build`)
+   - Input: `src/*.ts`
+   - Output: `dist/*.js` (CommonJS)
+   - Tool: `tsc`
+
+2. **Bundling** (`npm run bundle`)
+   - Input: `dist/index.js`
+   - Output: `bundle/carto` (single executable file)
+   - Tool: `@vercel/ncc`
+   - Includes: All dependencies, Node.js modules
+
+3. **Platform Executables** (`npm run pkg`)
+   - Input: `bundle/carto`
+   - Output: `bin/carto-{platform}`
+   - Tool: `pkg`
+   - Platforms: macOS (x64), Linux (x64), Windows (x64)
+
+**Distribution Strategy:**
+- Primary: Single bundled JavaScript file (`bundle/carto`)
+- Optional: Native executables for platforms without Node.js
+
+### 5.6 Request Tracking Header
+
+**Header Convention**: `clientId: carto-cli/{version}`
+
+**Rationale:**
+- Platform-wide convention (used across CARTO services)
+- Enables usage analytics and tracking
+- Version tracking for API compatibility monitoring
+- Support and debugging assistance
+
+**Implementation:**
+- Added to all API requests automatically
+- Version sourced from `package.json` dynamically
+- Single source of truth for version number
+
+## 6. Code Conventions
+
+### Async/Await Pattern
+```typescript
+// All I/O operations use async/await
+async function listMaps(): Promise<void> {
+  const client = await ApiClient.create();
+  const maps = await client.getWorkspace('/workspace-api/maps');
+  // ...
+}
+```
+
+### Error Handling
+```typescript
+// Consistent try/catch with clear error messages
+try {
+  const result = await apiCall();
+} catch (error: any) {
+  if (jsonOutput) {
+    console.log(JSON.stringify({ error: error.message }, null, 2));
+  } else {
+    console.error(`${red('✗')} Error: ${error.message}`);
+  }
+  process.exit(1);
+}
+```
+
+### Output Formatting
+
+**Dual Output Mode:**
+- Human-readable: Formatted tables, colors, icons (✓, ✗, →)
+- JSON mode: `--json` flag for machine parsing
+
+```typescript
+if (jsonOutput) {
+  console.log(JSON.stringify(data, null, 2));
+} else {
+  console.log(`${green('✓')} Success: ${message}`);
+}
+```
+
+**Color Usage:**
+- Success: Green (`✓`)
+- Error: Red (`✗`)
+- Info: Cyan
+- Warning: Dim
+- Highlights: Bold
+
+### Pagination Pattern
+
+**Consistent Pagination:**
+- `--page <number>`: Page number (1-indexed)
+- `--page-size <number>`: Items per page
+- `--all`: Fetch all pages automatically
+
+```typescript
+async getAllPaginated(
+  basePath: string,
+  baseParams: Record<string, string> = {},
+  pageParamName: string = 'page',
+  pageSizeParamName: string = 'page_size',
+  defaultPageSize: number = 100
+): Promise<any>
+```
+
+**User Feedback:**
+- Single page: No message
+- Multiple pages: "Page X of Y (use --all to fetch all pages)"
+- With `--all`: "Fetched all N items"
+
+### Command Function Signature
+```typescript
+async function commandHandler(
+  args: string[],
+  token?: string,
+  baseUrl?: string,
+  jsonOutput: boolean = false,
+  debug: boolean = false,
+  profile?: string
+): Promise<void>
+```
+
+**Parameters:**
+- `args`: Command-specific arguments
+- `token`: Optional auth token override
+- `baseUrl`: Optional API base URL override
+- `jsonOutput`: Enable JSON output mode
+- `debug`: Enable HTTP request debugging
+- `profile`: Profile name for multi-tenant support
+
+## 7. Testing Strategy
+
+### Manual Testing Approach
+
+**No automated unit tests.** Testing is performed via comprehensive manual test scenarios documented in `TEST_GUIDE.md`.
+
+**Rationale:**
+- CLI tools require integration testing with real APIs
+- Command-line interactions difficult to unit test effectively
+- Manual testing provides better coverage of user workflows
+- Faster development iteration without test maintenance overhead
+
+**Test Categories:**
+1. **Smoke Tests**: Version, help, auth status (no API calls)
+2. **Command Tests**: Each command with all parameter combinations
+3. **Integration Tests**: Real API calls with valid credentials
+4. **Error Handling**: Invalid inputs, missing parameters, API errors
+5. **Profile Tests**: Multi-profile authentication and switching
+6. **Pagination Tests**: Single page, multiple pages, --all flag
+7. **Output Tests**: Human-readable vs JSON format
+
+**Test Execution:**
+```bash
+# Quick smoke test (always run before commit)
+npm run build && npm run bundle && \
+./bundle/carto --version && \
+./bundle/carto --help && \
+./bundle/carto auth status
+```
+
+**Test Documentation:**
+- Comprehensive test scenarios in `TEST_GUIDE.md`
+- Expected outputs for each command
+- Error case validation
+- Test checklist for pre-commit verification
+
+### Quality Assurance
+- **Pre-commit**: Smoke tests mandatory
+- **Pre-PR**: Full test suite execution
+- **Manual review**: User-facing changes tested with real workflows
+
+## 8. Security Considerations
+
+### Credential Storage
+- Tokens stored in `~/.carto_credentials.json` (file permissions: 600)
+- Sensitive data (tokens) masked in debug output
+- No credentials logged or printed accidentally
+
+### Token Handling
+- Never log full tokens (mask to first 20 chars)
+- Environment variables take precedence (avoid file storage)
+- Token expiration validation prevents usage of expired credentials
+
+### Debug Mode Safety
+```typescript
+// Mask Authorization header in debug output
+const displayValue = key === 'Authorization' && value.startsWith('Bearer ')
+  ? `Bearer ${value.substring(7, 27)}...`
+  : value;
+```
+
+## 9. API Integration
+
+### Resource Endpoints
+
+**V3 API** (`https://{tenant}.api.carto.com/v3`):
+- `/v3/tokens`: API Access Tokens
+- `/v3/connections`: Data connections
+- `/v3/imports`: Data imports
+
+**Workspace API** (`https://workspace-{tenant}.app.carto.com`):
+- `/workspace-api/maps`: Maps management
+- `/workspace-api/workflows`: Workflows management
+
+**Accounts API** (`https://accounts.app.carto.com`):
+- `/oauth-clients`: OAuth client management (SPA, M2M)
+- `/accounts/users`: User management
+- `/accounts/invite`: User invitations
+- `/accounts/resources`: Superadmin resource operations
+
+**AI API** (`https://ai-api-{tenant}.api.carto.com`):
+- `/feature/{featureName}`: AI features (NDJSON streaming)
+
+**LiteLLM Proxy** (`https://litellm-{tenant}.api.carto.com`):
+- `/v1/chat/completions`: OpenAI-compatible chat
+- `/v1/models`: List available models
+
+### Authentication Flow
+
+**OAuth Browser Flow:**
+1. CLI starts local HTTP server (port 8080)
+2. Opens browser to CARTO Auth0 login page
+3. User authenticates in browser
+4. Auth0 redirects to `http://localhost:8080/callback?code=...`
+5. CLI exchanges code for JWT token
+6. Token and user info saved to credentials file
+7. Server shuts down, CLI exits
+
+**Token Validation:**
+- Parse JWT to extract expiration (`exp` claim)
+- Warn when < 10% of lifetime remaining
+- Error and suggest re-authentication when expired
+- Display user-friendly time remaining (e.g., "2 hours 30 minutes")
+
+## 10. Future Considerations
+
+### Potential Enhancements
+- **Automated Testing**: Consider integration test framework for critical paths
+- **LiteLLM URL**: Add to `config.yaml` instead of constructing from tenant ID
+- **Extended DuckDB Usage**: Additional data analysis and transformation commands
+- **Shell Completions**: Generate bash/zsh completion scripts
+- **Configuration Profiles**: Support for additional per-profile settings
+- **Batch Operations**: Parallel execution for bulk resource operations
+
+### Known Limitations
+- No progress bars for long-running operations (imports show polling status)
+- File upload limited to 1GB (GCS signed URL constraint)
+- LiteLLM URL not in config.yaml (hardcoded construction)
+- No retry logic for transient API failures
+- Debug mode shows full request/response (can be verbose)
+- DuckDB requires separate installation for SQL query features
+
+## 11. Development Workflow
+
+### Adding New Commands
+
+1. Create command module in `src/commands/`
+2. Export async function with standard signature
+3. Import and route in `src/index.ts`
+4. Add help text in `src/help.ts`
+5. Update `README.md` with command documentation
+6. Add test scenarios to `TEST_GUIDE.md`
+7. Rebuild: `npm run build && npm run bundle`
+8. Test with smoke tests and feature-specific tests
+
+### Adding New API Endpoints
+
+1. Add method to `ApiClient` class in `src/api.ts`
+2. Use appropriate endpoint (get, post, patch, delete, getWorkspace, etc.)
+3. Handle both JSON and formatted output modes
+4. Follow error handling pattern
+5. Update documentation and test guide
+
+### Version Updates
+
+**Single source of truth**: `package.json`
+
+Version automatically used in:
+- `--version` flag output
+- `clientId` request header
+- Help text display
+
+Update process:
+```bash
+# 1. Update version in package.json
+npm version patch|minor|major
+
+# 2. Rebuild
+npm run build && npm run bundle
+
+# 3. Verify version
+./bundle/carto --version
+```
+
+## 12. References
+
+- **README.md**: User-facing documentation and command reference
+- **TEST_GUIDE.md**: Comprehensive manual testing scenarios
+- **CLAUDE.md**: Project guidance for AI assistants
+- **carto-api.postman_collection.json**: API reference and examples
+- **package.json**: Dependencies and build scripts
+- **tsconfig.json**: TypeScript compiler configuration

package/CHANGELOG.md

@@ -0,0 +1,28 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2025-12-31
+
+### Added
+- Initial release
+- Authentication commands (login, logout, status, whoami)
+- M2M OAuth authentication for CI/CD (`auth login --m2m`)
+- Maps commands (list, get, create, update, delete, copy)
+- Workflows commands (list, delete)
+- Connections commands (list, get, create, update, delete)
+- Credentials commands (tokens, SPA, M2M OAuth clients)
+- Users commands (list, get, invite, invitations)
+- Imports commands (create with file/URL)
+- Admin commands (list, batch-delete, transfer)
+- Multi-profile support
+- JSON output mode
+- Unit and integration test suite
+- E2E tests against live CARTO API
+- CI/CD pipeline for NPM publishing and GitHub releases
+- Nightly test workflow with Slack notifications

package/LICENSE

@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2025 CARTO
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+PERFORMANCE OF THIS SOFTWARE.