@trillioncore/cli 1.0.0-next.1 → 1.0.0-next.11

package/README.md

@@ -1,8 +1,8 @@
 # @trillioncore/cli — v1 preview
 
-The `tc` CLI signs you into your own Trillioncore organization and reads the integration accounts you authorize. No repository checkout, provider secrets, `.env` file, or manually entered organization ID is required.
+The `tc` CLI signs you into Trillioncore and reads the organizations and integration accounts you explicitly authorize. No repository checkout, provider secrets, or `.env` file is required.
 
-**Requirements:** Node.js 22+, a browser that can reach the CLI computer's loopback listener, and a Trillioncore deployment with CLI login enabled. This is a fresh v1 preview, not v0 command parity; v0 configuration is not imported.
+**Requirements:** Node.js 22+, a browser on any device, and a Trillioncore deployment with CLI device login enabled. This is a fresh v1 preview, not v0 command parity; v0 configuration is not imported.
 
 ## Get started
 
@@ -18,7 +18,9 @@ npm exec --yes --package=@trillioncore/cli@next -- tc sql '<PostgreSQL account I
 npm exec --yes --package=@trillioncore/cli@next -- tc logout
 ```
 
-Browser sign-in lets you choose an organization and explicitly select its enabled accounts. An organization administrator must approve access. For a brand-new account, finish organization setup and connect your integrations in the app, then rerun `tc login` if prompted. No provider reconnection is required just to authorize the CLI.
+`tc login` registers a five-minute request and prints a browser URL plus a separate eight-character code. Open the URL on any device and enter the code shown in your terminal. Keep the code private and approve only a login you started. The CLI polls without a callback listener, tunnel, or copied token. After approval it verifies your identity, saves the private session, and acknowledges delivery.
+
+Browser sign-in lets you choose one or more organizations and explicitly select enabled accounts within each. Membership alone does not authorize a CLI connection; current organization and account permissions still apply. For a brand-new account, finish organization setup and connect your integrations in the app, then rerun `tc login` if prompted. No provider reconnection is required just to authorize the CLI.
 
 Prefer `npm exec` during evaluation. `npm install -g @trillioncore/cli@next` replaces your local `tc` executable, even though publishing `next` does not change the npm `latest` tag.
 
@@ -27,24 +29,105 @@ Prefer `npm exec` during evaluation. `npm install -g @trillioncore/cli@next` rep
 - `login [--no-browser] [--api-url <origin>]`: browser authorization, organization/account selection, and durable local session storage.
 - `whoami [--json]`: verify the signed-in identity without printing credentials.
 - `logout`: revoke the CLI connection on the server and remove its local session. `--local-only` only forgets local credentials; use the app's **Tokens** page to revoke server access if offline.
-- `index [path]`: browse authorized records; supports depth, limits and pagination.
-- `search [query]`: lexical search with source/date filters. Results include stable references for full reads and citations; search is not an exhaustive aggregation engine.
-- `get --ref <ref>` or `get --path <path>`: retrieve an authorized record's metadata or full redacted content.
-- `help [primitive] [--json]`: machine-readable `index`, `search`, `get`, `sql`, `help` discovery.
-- `sql <integrationAccountId> <statement> [--json]`: run one bounded `SELECT` or `WITH` statement against an explicitly scoped external PostgreSQL account. Quote the SQL as one shell argument. Trillioncore uses a read-only transaction, a statement timeout, a 500-row/response-size cap, and credential redaction; a dedicated database role with `SELECT` privileges limited to approved, non-secret tables and views remains required.
+- `organizations [--json]`: list only currently approved organizations, with `--limit` and `--cursor` pagination.
+- `index [path]`: an unscoped linked-session root lists organizations; pass `--organization <id>` or an organization-qualified path to browse accounts and records. Supports depth, limits and pagination.
+- `search [query]`: lexical search with source/date filters, optional `--account-id <uuid>`, and GitHub `--repository owner/repository`. Results include stable references for full reads and citations; search is not an exhaustive aggregation engine.
+- `get --ref <ref>` or `get --path <path>`: retrieve an authorized record with `--representation metadata`, `full` (default), or `part` (bounded stored redacted text).
+- `help [primitive] [--json]`: machine-readable `organizations`, `index`, `search`, `get`, `sql`, `help` discovery.
+- `sql <integrationAccountId> <statement> [--target synced|live] [--json]`: run one bounded `SELECT` or `WITH` statement against an explicitly scoped SQL-capable connection. Default target is `synced`; PostgreSQL `live` is explicit. Unavailable synced data never triggers live retry or fallback. Quote the SQL as one shell argument. Trillioncore uses a read-only transaction, a statement timeout, a 500-row/response-size cap, and credential redaction; a dedicated database role with `SELECT` privileges limited to approved, non-secret tables and views remains required.
+
+### Read large stored transcripts
+
+Use existing search to find the record ref, then request one part at a time:
+
+```sh
+tc get --ref '<record-ref>' --representation part --json
+tc get --ref '<part.nextRequest.ref>' --representation part --cursor '<part.nextRequest.cursor>' --json
+```
+
+Concatenate `item.content.verbatim` from each response in order. Follow `part.nextRequest` until `part.complete` is true and `part.nextRequest` is null. Human output prints the next command. Neither CLI nor MCP automatically fetches later parts. An empty stored value returns one terminal empty part.
+
+Parts contain minimal provenance; use `metadata` for descriptive fields. Completion means the entire stored redacted column was read, not that ingestion captured the complete provider transcript. Parts do not summarize, backfill, or apply an ingestion-size cap. Redaction runs before splitting. Every request rechecks current access. If content changes, discard the assembled text and restart without the stale cursor. Do not reuse a cursor with another record, organization, token or representation.
+
+Hosted and fixed-token MCP use `get` with `representation: "part"` and then the returned `part.nextRequest` arguments. Direct HTTP callers put `cursor` in the URL query, with `ref` and `representation` in the existing get POST body. Successful part MCP result envelopes remain below 32 KiB, including text and structured representations. Legacy `metadata` and `full` responses are unchanged and are not size-bounded. Native documents retain their existing full-read behavior; `part` applies only to integration records.
+
+### Organization targeting
+
+```sh
+tc organizations --json
+tc index --organization '<approved-organization-id>' --json
+tc search 'budget' --organization '<approved-organization-id>' --json
+tc get --ref 'org/<approved-organization-id>/account/<account-id>/record/<record-id>' --json
+tc sql '<account-id>' 'SELECT 1' --organization '<approved-organization-id>' --target synced --json
+```
+
+Use SQL only where the discovered catalog advertises that target. `index`, `search`, `get`, and `sql` accept the existing `--organization` option. Qualified index paths and get refs select their own organization; an explicit ID must agree. With one currently approved organization, data operations can omit the target. With several, untargeted search/SQL fail with `organization_required`; use the directory, not a guessed default. Each call resolves independently. There is no saved organization switch or cross-organization fan-out. Keep pagination cursors with their original organization and query.
+
+Hosted MCP uses the same `organizations` directory and per-call `organizationId` inputs. Manual tokens, environment overrides, legacy sessions, and standalone manual-token MCP remain pinned to one organization. They cannot use targeting to expand authority.
 
 CLI and MCP share four data-tool names: `index`, `search`, `get`, and `sql`. CLI session and help commands are separate. The legacy `query`/`query_integration` and MCP `people` tools are removed; `execute_sql` is now named `sql`, without a compatibility alias. Use `search` for lexical record lookup, not legacy `query`.
 
 Use `tc <command> --help` for supported flags. PostgreSQL SQL is a focused v1 command, not general v0 compatibility. There is no v0 `agent start`, chat, or write-command compatibility layer. CLI credentials cannot be used as MCP credentials or browser administration sessions. Authorize Codex or Claude Code separately through hosted MCP.
 
+## Resource discovery
+
+Run `tc index` to obtain an authorized account path, then:
+
+```sh
+tc index 'org/<organization-id>/account/<account-id>' --resources --json
+```
+
+For interactive inspection, omit `--json` to print schema columns, operation availability, warnings and the next-page cursor as text. Use `tc index --help` to discover resource and pagination options.
+
+MCP uses the same `index` tool with that `path` and `resources: true`. Resource mode returns an additive `catalog`, empty `entries`, and the normal `nextCursor` pagination field. Repeat the same resource request with `--cursor` while a cursor is returned. Changed catalog metadata invalidates old resource cursors; restart discovery. Native index and record refs are unchanged. A PostgreSQL account path opens the catalog by default, even without `--resources`. Account entries include a `resourceDiscovery` request; PostgreSQL `itemCount: null` means the source row count is unknown, not zero. Resource mode avoids enumerating every stored record.
+
+The catalog describes stored record kinds, approved **live source** PostgreSQL tables and **active synchronized** PostgreSQL relations, supported operations, schema metadata and known sync timestamps. A PostgreSQL connection can expose both a `records` resource for visible stored row projections and table resources for its relational schemas. SQL availability does not determine whether those existing stored projections support search/get. Resource keys identify catalog entries; they are **not** refs accepted by `get`. Obtain a record ref through search before reading. Search is lexical across authorized connections, not scoped to the resource just inspected; optional source-type filters do not identify a unique connection. Use `--account-id` (MCP `accountId`) to select one authorized connection.
+
+For stored content, including `postgres_row`, `recordSchemas` describes the allowed scalar fields in `get`'s `attributes`, grouped by recognized observed record kind. These field names/types come from the same definitions used to validate stored records. Fields may be absent on individual records; a schema does not prove populated values, complete history, relationship IDs, or SQL support. Native models retain their declared schemas on empty connections; PostgreSQL stored projections require visible records. Undeclared kinds are not advertised. Each logical table has one canonical `table:<encoded-schema>.<encoded-table>` key and the same source-qualified SQL name for both targets. Exact inspection labels `relation` with `relationTarget`: synced when published, otherwise live-only. When both schemas exist, `relations.synced` and `relations.live` preserve their separately approved columns and relationships; never union them. Directory entries omit all schema variants. Legacy `live:` keys remain exact-lookup aliases, not directory entries. PostgreSQL record content is a bounded search projection, not the complete source row; its attributes identify the source schema, table, key and dataset revision. Use available SQL, not lexical search results, for authoritative analytical aggregations.
+
+CLI, API and MCP accept the same provider filters: `fathom`, `fireflies`, `harvest`, `outlook`, `gmail`, `postgres`, and `github`. For example, `tc search --source-type gmail` or `tc search --source-type postgres`, then `tc get --ref '<returned-record-ref>'`. MCP uses `search` with `sourceTypes: ["postgres"]` and `get` with that returned `ref`. PostgreSQL search reads only visible stored projections from the published revision; it never falls back to a live source or executes SQL. If no such records are visible, no PostgreSQL `records` descriptor is advertised and search has no PostgreSQL results.
+
+`lifecycle` reports the persisted `activeRun` (queued/running) and `lastFinishedRun` (completed/failed), independently of the last successful sync timestamp. Null means no matching run was recorded. The stored `startedAt` can be enqueue time; a running state does not prove worker health. Progress counts are processed pages/records, not percentages or newly created records. A completed run does not prove exhaustive source coverage. Native `page-upserts` may remain visible even if a later page fails; `active-revision` describes snapshot publication, not whether production execution is configured. `sourceRemovals: explicit-feed-events` means newly received explicit removals soft-hide stored records within the configured feed (Gmail permanent deletions; Outlook Inbox exits, not whole-mailbox deletions). Re-observation of the same live source key restores visibility. `recordsRemoved` counts newly hidden stored rows, not duplicate events or unknown IDs; old runs retain their counters. This does not reconcile previously missed events or erase historical answers/exports. `sourceRemovals: not-propagated` means the provider does not propagate source removal events; `revision-scoped` describes active-snapshot visibility, not a guarantee of continuous deletion tracking. Lifecycle/counter and freshness changes alone do not invalidate resource cursors; schema/revision changes still require restarting discovery. No cursors, raw failure messages or credentials appear in lifecycle metadata.
+
+Respect `operations[].available` and `warnings`. Synced SQL is available only when its reader is configured; otherwise the operation reports `synced_sql_not_configured`. A live-only table reports synced unavailable, not an implicit live default. Both PostgreSQL targets accept the same approved source-qualified names or unambiguous bare source table names. Legacy unqualified physical aliases remain accepted but are not advertised. Ambiguous names reject; schema-qualified column references remain unsupported, so use table aliases for columns. SQL JSON includes `target`, `lastSuccessfulSyncAt` and `datasetRevision` for PostgreSQL; unknown metadata is null. Plain CLI output adds a target/revision/sync summary before the unchanged row JSON; legacy responses without target metadata retain rows-only rendering. Live PostgreSQL schemas use selected columns and explicit `target: live`; `liveSelectionRevision` is independent of `datasetRevision`. Read them with `tc sql '<account-id>' 'SELECT "<selected-column>" FROM "<schema>"."<table>" LIMIT 20' --target live --json`. An agent can construct that query from the catalog when answering a natural-language question; users need not supply SQL. Live approval does not copy rows into Trillioncore, prove source health or establish row counts. SQL rechecks current authorization and source identities. Search does not scan live rows; empty stored search results do not prove source emptiness. `get` still reads stored record refs, not live table keys. No semantic search or actions are advertised. Native connectors retain declared search/get models when empty; `recordKinds` identifies each model's declared kind. Observed-kind discovery is capped at 100 with an explicit warning. Synced manifests retain the existing approved selection bounds. A successful sync timestamp does not establish complete history or current provider state. Business definitions, currency and attribution are not inferred.
+
+All discovery uses the same organization, live token scope, connection permission and enabled-state checks as existing agent reads. PostgreSQL record-kind discovery also uses the same active-revision and removed-record visibility predicate as search/get.
+
+### Shared integration contract
+
+Discovery follows provider → verified stored identity → declared resources. Gmail and Outlook show stored mailbox emails; Fireflies shows `User ID`, and Harvest shows `Account ID`. Fathom shows `Account email unavailable`; PostgreSQL and GitHub show `Account identity unavailable`. Editable labels, generated installation IDs and connection hosts are not verified identity. Original account paths remain available in JSON for navigation; duplicate names are disambiguated by connection ID in text. New accounts require explicit existing consent; discovery never grants access automatically.
+
+Contributors must extend `INTEGRATION_PROVIDER_DEFINITIONS` in `packages/types/src/integrations.ts` for each provider. Its exhaustive typed declaration drives API identity/model discovery and CLI captions. Native models are declared even when empty; PostgreSQL retains only existing stored row projections plus approved relational manifests. Unknown providers and undeclared model kinds have no generic fallback. Index does not fetch providers to fill missing identity.
+
+A connection is one authorized provider account, a resource describes an available representation, and an operation describes how to access it. Analytical SQL and non-analytical search/get are capabilities, not mutually exclusive integration categories. Native API connectors and optional Airbyte ingestion are internal mechanisms; Airbyte is not a search provider filter. Multiple adapters may eventually support one connection, with separate credentials or action grants where needed. GitHub uses the configured self-hosted Airbyte path; existing providers retain their current ingestion paths. No provider business actions or cross-source SQL are implemented. PostgreSQL discovery distinguishes approved live SQL from synchronized datasets; a synchronized schema is not proof that its query transport is configured.
+
+### GitHub pull requests
+
+Connect a named GitHub account and repository list in the Integrations UI. Multiple connections may select the same repositories under independent credentials and permissions. Reconnecting or changing repositories retains the logical connection ID, but serving waits for a new successful publication. Removing one connection does not remove a separately authorized connection.
+
+```bash
+tc index 'org/<organization-id>/account/<connection-id>' --resources --json
+tc search --source-type github --account-id '<connection-id>' --repository 'owner/repository' --json
+tc get --ref '<returned-record-ref>' --json
+tc sql '<connection-id>' 'SELECT repository, count(*) FROM pull_requests GROUP BY repository' --target synced --json
+```
+
+GitHub SQL and search/get read the same published dataset revision, not mutable Airbyte tables. Synced SQL is bounded to 200 returned rows and response-size limits. The SQL `creator` column corresponds to the PR author; search/get attributes use `author`. Reads recheck current account/token authorization. Repository selection shrink hides the previous publication until synchronization completes. This first slice covers pull requests only; issues/comments/reviews are not yet exposed and upstream deletion completeness is unverified.
+
+Interrupted setup remains a saved, disabled connection. Use Manage to reconnect or remove it rather than adding a duplicate. Creation intents and exact remote names allow cleanup after lost responses. If an ambiguous creation cannot yet be found in Airbyte, recovery fails closed and retains the intent; an operator must reconcile the remote outcome before it can be cleared. Superseded and abandoned dataset revisions are durably tracked for bounded cleanup on the next sync, reconnect or removal.
+
+Self-hosted Airbyte and an operator-vetted PostgreSQL provisioning endpoint must be configured before connections are available. The destination can be Trillioncore's existing Neon database: GitHub uses dedicated schemas and per-revision reader roles, not another warehouse database. Use Neon's direct endpoint for the replication/provisioning URL, not its `-pooler` endpoint: lifecycle synchronization uses session advisory locks, which transaction pooling cannot preserve. This is another connection endpoint for the same database; the application's existing pooled URL stays unchanged. Its internal schema-isolation policy permits ordinary database PUBLIC CONNECT/TEMPORARY defaults, but rejects permanent CREATE rights, PUBLIC dataset grants (including column grants), unrelated data access and non-scalar/view-backed snapshots. Queries use fresh read-only sessions with pinned function lookup and bounded validated SQL. Existing strict database-isolation callers retain their original policy. The application does not change global PUBLIC privileges to make provisioning work. Actual runtime versions, destination naming, licensing and live acceptance remain deployment checks; synthetic fixtures do not establish compatibility with every Airbyte version.
+
 ## Sessions and troubleshooting
 
 Sessions refresh automatically. File-based credentials live in `~/.trillioncore/cli-v1/session.json`, with private directory/file permissions (0700/0600 on POSIX), atomic replacement and cross-process refresh locking. Never share or print this file. Keep the OS account and its private home directory secure; Windows ACL behavior has not yet been independently validated.
 
-One profile is active at a time. A new successful login replaces it and attempts to revoke the prior connection. Change account scopes or revoke access from the app's **Tokens** page. Membership changes and server revocation are enforced on subsequent requests.
+One profile is active at a time. A new successful login replaces it and attempts to revoke the prior connection. Change account scopes or revoke access from the app's **Tokens** page. Each organization's child grant is independent: editing, revoking, or removing A does not withdraw B. Logout and refresh replay revoke the whole connection. To add another organization, use a fresh login and explicit consent. Membership deletion/rejoin does not revive an old grant. Membership changes and server revocation are enforced on subsequent requests.
 
 - Default API: `https://api.trillioncore.com`. A saved session remains pinned to its issuing server. Custom origins must use HTTPS, except literal loopback development endpoints.
-- `--no-browser` prints the sign-in URL; it is **not** a remote device-code flow. SSH users need appropriate loopback forwarding or a browser on the same computer. Do not share the sign-in URL.
+- `--no-browser` prints the device sign-in URL and separate code without launching a browser. Use it over SSH or on a headless machine. Default login uses the same remote-capable flow; there is no loopback fallback flag.
+- Pending and transient polling failures retry within a fixed five-minute deadline. Rate-limit retries respect `Retry-After`. Ctrl-C or timeout makes one best-effort cancellation attempt; unreachable requests expire on the server. If registration itself is lost, rerun login and let the unknown request expire.
+- A lost completed poll can be retrieved again before expiry. Acknowledgement occurs only after private persistence. If acknowledgement fails, the saved session remains active and the temporary delivery expires automatically.
 - A disabled/not-yet-deployed CLI login returns a clear deployment-unavailable error. Signing in repeatedly cannot enable the server.
 - Advanced/manual use: `TRILLIONCORE_TOKEN` and `TRILLIONCORE_ORG_ID` must be supplied together and override the saved session. Remove both to use browser login. API overrides prefer `TRILLIONCORE_CLI_API_URL`, then `TRILLIONCORE_API_URL`; a mismatch with a saved session is refused, not silently redirected. Application API configuration is never rewritten.
 - `TRILLIONCORE_CONFIG_DIR` selects a private CLI-only profile directory (also useful for isolated testing). Legacy v0 configuration is not migrated.