primethink-cli 1.4.0__tar.gz → 1.4.1__tar.gz

{primethink_cli-1.4.0/primethink_cli.egg-info → primethink_cli-1.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: primethink-cli
-Version: 1.4.0
+Version: 1.4.1
 Summary: PrimeThink CLI - A powerful tool for interacting with PrimeThink AI API
 Author-email: PrimeThink <support@primethink.ai>
 License-Expression: MIT
@@ -66,7 +66,7 @@ pt chat send --agent AGENT_ID --message "your message"
 | Command | Description |
 |---|---|
 | `pt version` | Display the CLI version |
-| `pt whoami` | Show the authenticated user and their groups |
+| `pt whoami` | Show the authenticated user, their groups, active group, and configured providers |
 | `pt mcp` | Run PrimeThink as an MCP server (exposes the API as tools over stdio) |
 | `pt install-skill` | Install the bundled CLI agent skill for Claude Code and compatible agents |
 | `pt install-developer-skill` | Install the complete public PrimeThink developer skill, including nested resources |
@@ -127,8 +127,10 @@ pt chat send --agent AGENT_ID --message "your message"
 | `pt agent list-docs` | List documents in an agent's knowledge base |
 | `pt agent upload-docs` | Upload files to an agent's knowledge base |
 | `pt agent attach-collections` / `pt agent detach-collection` | Attach / detach collections for agent RAG |
+| `pt agent upload-image` / `pt agent delete-image` | Upload / delete an agent's image |
 | `pt capability list` / `create` / `update` / `delete` | Manage agent capabilities |
 | `pt capability archive` / `unarchive` / `duplicate` | Archive, unarchive, or duplicate a capability |
+| `pt capability resolve` | Resolve capability codes/ids to ids |
 | `pt models list` / `pt models embeddings` | Discover LLM / embedding models from the catalog |
 | `pt scheduled-job list` / `create` / `update` | Manage scheduled jobs (prompts on a schedule) in a chat |
 | `pt scheduled-job set-status` / `delete` | Pause/resume or delete a scheduled job |
@@ -141,6 +143,10 @@ pt chat send --agent AGENT_ID --message "your message"
 | `pt tag list` / `create` / `assign` | List, create, and assign tags (task/agent/capability/collection) |
 | `pt group list` / `get` / `create` / `update` / `delete` | Manage groups (organizations) |
 | `pt group members` / `remove-member` / `invite` / `add-agent` | Manage group members, invites, and agents |
+| `pt settings list` / `get` / `set` / `delete` | Manage group/user settings and provider API keys (secrets never shown) |
+| `pt user list` / `search` | Look up users you can see by name or email |
+| `pt notification list` / `unread-count` | List your notifications or count the unread ones |
+| `pt notification mark-read` / `mark-unread` / `mark-all-read` / `delete` | Mark notifications read/unread, mark all read, or delete one |
 | `pt task list` | List tasks, filterable by name/type/status and paginated |
 | `pt task actions` | List available task actions |
 | `pt task execute` | Execute a task action with a message and optional files |
@@ -158,6 +164,10 @@ pt chat send --agent AGENT_ID --message "your message"
 | `pt task import` | Create a new task from an exported JSON file (deploy across environments with `--profile`) |
 | `pt task create-version` | Snapshot a task's current state as a named version |
 | `pt task upload-image` | Upload an image for a task |
+| `pt eval list` / `add` / `update` / `delete` | Manage a task's evaluation-data set (the test plan) |
+| `pt eval settings` | Configure a task's evaluation settings (schedule, evaluator, thresholds) |
+| `pt eval run` / `runs` / `run-get` / `results` / `download` | Run a task evaluation and inspect its runs and results |
+| `pt eval simulate` / `simulations` / `delete-simulation` | Run and manage agent-vs-agent task simulations |
 | `pt search documents` | Semantic search across documents in a vector store collection |
 | `pt search chat` | Semantic search within a chat |
 | `pt search collection` | Semantic search within a collection |
@@ -171,7 +181,7 @@ Every command that calls the API also accepts `--profile` (use a specific profil
 
 `--help` works in either position: `pt chat send --help`, `pt chat --help send`, and `pt --help chat send` all print the same page.
 
-> **Tip:** `-p` is the short flag for `--profile` in the `pt task`, `pt agent`, and `pt search` groups, `pt image generate`, and `pt whoami`, but in the `pt chat` and `pt collection` groups there is no `-p` for profile — there `-p` means `--path` (a directory inside the chat/collection) on the file commands. Use the long form `--profile` when in doubt.
+> **Tip:** `-p` is the short flag for `--profile` in most groups (e.g. `pt task`, `pt agent`, `pt search`, `pt settings`, `pt user`, `pt notification`, `pt eval`), `pt image generate`, and `pt whoami`, but in the `pt chat` and `pt collection` groups there is no `-p` for profile — there `-p` means `--path` (a directory inside the chat/collection) on the file commands. Use the long form `--profile` when in doubt.
 
 ## Configuration
 

{primethink_cli-1.4.0 → primethink_cli-1.4.1}/README.md

@@ -32,7 +32,7 @@ pt chat send --agent AGENT_ID --message "your message"
 | Command | Description |
 |---|---|
 | `pt version` | Display the CLI version |
-| `pt whoami` | Show the authenticated user and their groups |
+| `pt whoami` | Show the authenticated user, their groups, active group, and configured providers |
 | `pt mcp` | Run PrimeThink as an MCP server (exposes the API as tools over stdio) |
 | `pt install-skill` | Install the bundled CLI agent skill for Claude Code and compatible agents |
 | `pt install-developer-skill` | Install the complete public PrimeThink developer skill, including nested resources |
@@ -93,8 +93,10 @@ pt chat send --agent AGENT_ID --message "your message"
 | `pt agent list-docs` | List documents in an agent's knowledge base |
 | `pt agent upload-docs` | Upload files to an agent's knowledge base |
 | `pt agent attach-collections` / `pt agent detach-collection` | Attach / detach collections for agent RAG |
+| `pt agent upload-image` / `pt agent delete-image` | Upload / delete an agent's image |
 | `pt capability list` / `create` / `update` / `delete` | Manage agent capabilities |
 | `pt capability archive` / `unarchive` / `duplicate` | Archive, unarchive, or duplicate a capability |
+| `pt capability resolve` | Resolve capability codes/ids to ids |
 | `pt models list` / `pt models embeddings` | Discover LLM / embedding models from the catalog |
 | `pt scheduled-job list` / `create` / `update` | Manage scheduled jobs (prompts on a schedule) in a chat |
 | `pt scheduled-job set-status` / `delete` | Pause/resume or delete a scheduled job |
@@ -107,6 +109,10 @@ pt chat send --agent AGENT_ID --message "your message"
 | `pt tag list` / `create` / `assign` | List, create, and assign tags (task/agent/capability/collection) |
 | `pt group list` / `get` / `create` / `update` / `delete` | Manage groups (organizations) |
 | `pt group members` / `remove-member` / `invite` / `add-agent` | Manage group members, invites, and agents |
+| `pt settings list` / `get` / `set` / `delete` | Manage group/user settings and provider API keys (secrets never shown) |
+| `pt user list` / `search` | Look up users you can see by name or email |
+| `pt notification list` / `unread-count` | List your notifications or count the unread ones |
+| `pt notification mark-read` / `mark-unread` / `mark-all-read` / `delete` | Mark notifications read/unread, mark all read, or delete one |
 | `pt task list` | List tasks, filterable by name/type/status and paginated |
 | `pt task actions` | List available task actions |
 | `pt task execute` | Execute a task action with a message and optional files |
@@ -124,6 +130,10 @@ pt chat send --agent AGENT_ID --message "your message"
 | `pt task import` | Create a new task from an exported JSON file (deploy across environments with `--profile`) |
 | `pt task create-version` | Snapshot a task's current state as a named version |
 | `pt task upload-image` | Upload an image for a task |
+| `pt eval list` / `add` / `update` / `delete` | Manage a task's evaluation-data set (the test plan) |
+| `pt eval settings` | Configure a task's evaluation settings (schedule, evaluator, thresholds) |
+| `pt eval run` / `runs` / `run-get` / `results` / `download` | Run a task evaluation and inspect its runs and results |
+| `pt eval simulate` / `simulations` / `delete-simulation` | Run and manage agent-vs-agent task simulations |
 | `pt search documents` | Semantic search across documents in a vector store collection |
 | `pt search chat` | Semantic search within a chat |
 | `pt search collection` | Semantic search within a collection |
@@ -137,7 +147,7 @@ Every command that calls the API also accepts `--profile` (use a specific profil
 
 `--help` works in either position: `pt chat send --help`, `pt chat --help send`, and `pt --help chat send` all print the same page.
 
-> **Tip:** `-p` is the short flag for `--profile` in the `pt task`, `pt agent`, and `pt search` groups, `pt image generate`, and `pt whoami`, but in the `pt chat` and `pt collection` groups there is no `-p` for profile — there `-p` means `--path` (a directory inside the chat/collection) on the file commands. Use the long form `--profile` when in doubt.
+> **Tip:** `-p` is the short flag for `--profile` in most groups (e.g. `pt task`, `pt agent`, `pt search`, `pt settings`, `pt user`, `pt notification`, `pt eval`), `pt image generate`, and `pt whoami`, but in the `pt chat` and `pt collection` groups there is no `-p` for profile — there `-p` means `--path` (a directory inside the chat/collection) on the file commands. Use the long form `--profile` when in doubt.
 
 ## Configuration
 

{primethink_cli-1.4.0 → primethink_cli-1.4.1}/SPECS.md

@@ -93,12 +93,15 @@ To obtain an API key, go to `Settings > API Keys` in the PrimeThink app and gene
 | `pt agent upload-docs` | `POST /api/v1/virtual-assistants/{va_id}/documents` (multipart) |
 | `pt agent attach-collections` | `POST /api/v1/virtual-assistants/{va_id}/collections/attach` |
 | `pt agent detach-collection` | `DELETE /api/v1/virtual-assistants/{va_id}/collections/{collection_id}` |
+| `pt agent upload-image` | `POST /api/v1/virtual-assistants/{va_id}/image` (multipart) |
+| `pt agent delete-image` | `DELETE /api/v1/virtual-assistants/{va_id}/image` |
 | `pt capability list` | `GET /api/v1/virtual-assistants/capabilities` |
 | `pt capability create` | `POST /api/v1/virtual-assistants/capabilities` |
 | `pt capability update` | `PATCH /api/v1/virtual-assistants/capabilities/{capability_id}` |
 | `pt capability delete` | `DELETE /api/v1/virtual-assistants/capabilities/{capability_id}` |
 | `pt capability archive` / `unarchive` | `PUT /api/v1/virtual-assistants/capabilities/{capability_id}/archived?archived=true\|false` |
 | `pt capability duplicate` | `POST /api/v1/virtual-assistants/capabilities/{capability_id}/duplicate` |
+| `pt capability resolve` | resolves capability codes/ids to ids via `GET /api/v1/virtual-assistants/capabilities` |
 | `pt models list` | `GET /api/v1/catalog/llm/models` |
 | `pt models embeddings` | `GET /api/v1/catalog/embeddings/models` |
 | `pt task list` | `GET /api/v1/tasks/` |
@@ -130,7 +133,7 @@ To obtain an API key, go to `Settings > API Keys` in the PrimeThink app and gene
 | `pt voice diarize` | `POST /api/v1/voice/diarize` (multipart `file`[, `speaker_file`]) |
 | `pt voice translate` | `POST /api/v1/voice/translate` (multipart `file`) |
 | `pt video analyze` | `POST /api/v1/video/analyze` (multipart `file`) |
-| `pt whoami` | `GET /api/v1/users/me` + `GET /api/v1/users/me/groups` |
+| `pt whoami` | `GET /api/v1/users/me` + `GET /api/v1/users/me/groups` (response also includes `active_group` and `configured_providers`) |
 | `pt workspace list` | `GET /api/v1/chat-workspaces` |
 | `pt workspace create` | `POST /api/v1/chat-workspaces` |
 | `pt workspace rename` | `PUT /api/v1/chat-workspaces/{id}/name` |
@@ -152,6 +155,29 @@ To obtain an API key, go to `Settings > API Keys` in the PrimeThink app and gene
 | `pt group remove-member` | `DELETE /api/v1/groups/{group_id}/members/` (`[user_ids]`) |
 | `pt group invite` | `POST /api/v1/groups/invite/?email=&role_id=` |
 | `pt group add-agent` / `remove-agent` | `POST /api/v1/groups/{group_id}/add_vas` / `DELETE .../remove_vas` (`[vas_ids]`) |
+| `pt settings list` | `GET /api/v1/groups/current/settings` + `GET /api/v1/users/me/settings` (+ dedicated properties); secret values never shown |
+| `pt settings get` | `GET` the group or user setting for KEY (`--scope group\|user`; sensitive values redacted) |
+| `pt settings set` | routed by KEY to the group/user setting endpoint; provider `*_API_KEY` keys stored as secrets |
+| `pt settings delete` | `DELETE` the KV setting for KEY (`--scope group\|user`; dedicated properties can't be deleted) |
+| `pt user list` / `search` | `GET /api/v1/users/me/visible-users[/full]` (search/limit are client-side) |
+| `pt notification list` | `GET /api/v1/notifications?page=&page_size=` (`--unread-only` is client-side) |
+| `pt notification unread-count` | `GET /api/v1/notifications/unread-count` |
+| `pt notification mark-read` / `mark-unread` | `POST /api/v1/notifications/{notification_id}/mark-as-read` / `.../mark-as-unread` |
+| `pt notification mark-all-read` | `POST /api/v1/notifications/mark-all-as-read` |
+| `pt notification delete` | `DELETE /api/v1/notifications/{notification_id}` |
+| `pt eval list` | `GET /api/v1/tasks/{task_id}/evaluation-data` |
+| `pt eval add` | `POST /api/v1/tasks/{task_id}/evaluation-data` |
+| `pt eval update` | `PATCH /api/v1/tasks/{task_id}/evaluation-data/{eval_id}` |
+| `pt eval delete` | `DELETE /api/v1/tasks/{task_id}/evaluation-data/{eval_id}` |
+| `pt eval settings` | `PATCH /api/v1/tasks/{task_id}/evaluation-settings` |
+| `pt eval run` | `POST /api/v1/tasks/{task_id}/evaluation` |
+| `pt eval runs` | `GET /api/v1/tasks/{task_id}/evaluations/runs/` |
+| `pt eval run-get` | `GET /api/v1/tasks/{task_id}/evaluations/runs/{run_id}` |
+| `pt eval results` | `GET /api/v1/tasks/{task_id}/evaluation_results` |
+| `pt eval download` | `GET /api/v1/tasks/{task_id}/evaluations/runs/{run_id}/download` |
+| `pt eval simulate` | `POST /api/v1/tasks/{task_id}/simulations` |
+| `pt eval simulations` | `GET /api/v1/tasks/{task_id}/simulations` |
+| `pt eval delete-simulation` | `DELETE /api/v1/tasks/{task_id}/simulations/{sim_id}` |
 | `pt {chat,collection,task,agent} mkdir` | `POST /api/v1/{owner}/{id}/directories?path=…` |
 | `pt {chat,collection,task,agent} rmdir` | `DELETE /api/v1/{owner}/{id}/directories/delete` (`{dir_path, delete_recursively}`) |
 | `pt {chat,collection,task,agent} move-dir` | `POST /api/v1/{owner}/{id}/directories/move` (`{src_dir, dest_parent, merge_dirs}`) |
@@ -827,7 +853,7 @@ pt search messages QUERY --collection-name NAME [--chat-id N] [--user-id N] [--a
 
 **Endpoints**: `GET /api/v1/users/me` and `GET /api/v1/users/me/groups`
 
-The CLI calls both and prints one combined JSON object: `{ "user": { ... }, "groups": [ ... ] }`.
+The CLI calls both and prints one combined JSON object: `{ "user": { ... }, "groups": [ ... ], "active_group": { "id": …, "name": … }, "configured_providers": [ ... ] }`. `active_group` is the user's currently active group; `configured_providers` lists the LLM provider slugs that have an API key set.
 
 **CLI Command**:
 ```bash

{primethink_cli-1.4.0 → primethink_cli-1.4.1}/USER_GUIDE.md

@@ -17,12 +17,16 @@ For a terse, complete listing of every command and option, see the [CLI Referenc
 9. [Semantic Search](#semantic-search)
 10. [Managing Agents](#managing-agents)
 11. [Managing Tasks](#managing-tasks)
-12. [Scaffolding Live Apps](#scaffolding-live-apps)
-13. [MCP Server](#mcp-server)
-14. [Common Use Cases](#common-use-cases)
-15. [Tips and Tricks](#tips-and-tricks)
-16. [Troubleshooting](#troubleshooting)
-17. [FAQ](#faq)
+12. [Evaluating Tasks](#evaluating-tasks)
+13. [Managing Settings](#managing-settings)
+14. [Finding Users](#finding-users)
+15. [Notifications](#notifications)
+16. [Scaffolding Live Apps](#scaffolding-live-apps)
+17. [MCP Server](#mcp-server)
+18. [Common Use Cases](#common-use-cases)
+19. [Tips and Tricks](#tips-and-tricks)
+20. [Troubleshooting](#troubleshooting)
+21. [FAQ](#faq)
 
 ## Introduction
 
@@ -107,7 +111,7 @@ Check who you're authenticated as:
 pt whoami
 ```
 
-This prints your user details and groups as JSON — if it succeeds, your token works. It also takes `--profile`, which makes it the quickest way to verify which account each profile points at:
+This prints your user details, groups, active group, and the LLM providers you have API keys for (`configured_providers`) as JSON — if it succeeds, your token works. It also takes `--profile`, which makes it the quickest way to verify which account each profile points at:
 
 ```bash
 pt whoami --profile production | jq '.user.email'
@@ -157,11 +161,11 @@ pt task execute --action summarize --message "Test" --profile custom
 
 This works on every API command, including the `chat`, `collection`, `agent`, `task`, `search`, `image`, and `whoami` commands.
 
-> **Heads-up:** in the `pt task`, `pt agent`, and `pt search` groups, `pt image generate`,
-> and `pt whoami`, `-p` is the short flag for `--profile`. In the `pt chat` and
-> `pt collection` groups there is no `-p` for profile — there `-p` is the short flag for
-> `--path` (a directory inside the chat or collection) on the file commands. Use the long
-> form `--profile` when in doubt.
+> **Heads-up:** in most groups (e.g. `pt task`, `pt agent`, `pt search`, `pt settings`,
+> `pt user`, `pt eval`), `pt image generate`, and `pt whoami`, `-p` is the short flag for
+> `--profile`. In the `pt chat` and `pt collection` groups there is no `-p` for profile —
+> there `-p` is the short flag for `--path` (a directory inside the chat or collection) on
+> the file commands. Use the long form `--profile` when in doubt.
 
 #### List All Profiles
 
@@ -1108,6 +1112,138 @@ pt scheduled-job delete 45
 
 > Because `--schedule-nl` is interpreted by an LLM server-side, `create`/`update` calls that include it use a longer (120s) timeout — don't treat the wait as a hang.
 
+## Evaluating Tasks
+
+The `pt eval` group tests a task against a set of expected question/answer cases, then scores its actual responses. The typical flow is: **build a plan** (`pt eval add`), **configure how it runs** (`pt eval settings`), **run it** (`pt eval run`), then **read the results** (`pt eval results`). Every command takes the task ID.
+
+### 1. Build the evaluation plan
+
+Each test case pairs a user query with the ideal response and a match type — `exact` (exact match), `similar` (fuzzy/semantic match), or `agent` (an evaluator agent judges the answer):
+
+```bash
+# See existing cases
+pt eval list 99
+
+# Add cases
+pt eval add 99 --user-query "What's your return window?" --ideal-response "30 days" --type similar
+pt eval add 99 --user-query "Refund a gift?" --ideal-response "Yes, store credit" --type agent --evaluator-agent-id 7
+
+# Edit or remove a case (delete is destructive; --yes to skip the prompt)
+pt eval update 99 EVAL_ID --ideal-response "30 days from delivery"
+pt eval delete 99 EVAL_ID
+```
+
+`--examples '{...}'` attaches good/bad example fields; `--chat-group` (default `1`) groups related cases.
+
+### 2. Configure how evaluations run
+
+```bash
+pt eval settings 99 --active --run-time daily --evaluator-agent-id 7 --pass-threshold 0.8
+```
+
+`--run-time` is `manual`, `daily`, `weekly`, or `monthly`; `--pass-threshold` is the minimum score to count as a pass; `--message-delay-ms` throttles messages during a run.
+
+### 3. Run it, then read the results
+
+```bash
+# Run the evaluation (optionally against a specific task version or model)
+pt eval run 99
+pt eval run 99 --version 2 --model-override gpt-test
+
+# Inspect past runs and their results
+pt eval runs 99
+pt eval run-get 99 RUN_ID
+pt eval results 99 --run-id RUN_ID
+pt eval download 99 RUN_ID -o results.json
+```
+
+### Simulations
+
+Simulations drive the task with a *simulator* agent toward a goal for up to `--max-turns`, then score the transcript:
+
+```bash
+pt eval simulate 99 --simulator-agent-id 5 --goal "Get a refund on a gift" --max-turns 10 --evaluator-agent-id 7
+pt eval simulations 99
+pt eval delete-simulation 99 SIM_ID          # destructive; --yes to skip the prompt
+```
+
+## Managing Settings
+
+The `pt settings` group reads and writes group and user settings, including provider API keys. Settings live at two **scopes** — `group` and `user` — and some keys exist at both. **Secret values are never shown**: listing/getting a setting reports only whether it `is_set` and whether it's `sensitive`, and a sensitive value is redacted on `get`.
+
+```bash
+# See what's set (optionally narrow to one scope)
+pt settings list
+pt settings list --scope group
+
+# Read one value (sensitive values come back redacted)
+pt settings get timezone --scope user
+```
+
+### Set a provider API key
+
+Any `*_API_KEY` key (e.g. `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`) is stored as a secret at group scope. After setting one, `pt whoami` lists that provider under `configured_providers`:
+
+```bash
+pt settings set ANTHROPIC_API_KEY sk-ant-… --scope group
+pt whoami | jq '.configured_providers'
+```
+
+### Set the group's default agent, or other settings
+
+```bash
+# The group default agent is just a setting
+pt settings set default_agent 7 --scope group
+
+# Other known group keys: voice, voice_provider, new_chat_logic, group_mode,
+# default_role, document_analysis_active, public_name, custom_theme_color
+# User keys: timezone, location, default_language, default_va, auto_archive_option, custom_theme_color
+pt settings set timezone "Europe/Rome" --scope user
+```
+
+Pass `--scope` explicitly whenever a key exists at both scopes. Delete a key/value setting with `pt settings delete KEY --scope … --yes` (dedicated properties can't be deleted).
+
+## Finding Users
+
+The `pt user` group looks up the users you can see, by name or email. The endpoint returns your full visible-users list, so the CLI applies `--search` and `--limit` **client-side**:
+
+```bash
+# Everyone you can see
+pt user list
+
+# Filter by name/email, cap the count, or get richer records
+pt user search ann@acme.co
+pt user list --search support --limit 20
+pt user list --full
+```
+
+This pairs with inviting people to a chat by email — `pt chat invite-members` resolves an `--email` to a user ID via the same visible-users directory:
+
+```bash
+pt chat invite-members 123 --email teammate@acme.co
+```
+
+## Notifications
+
+The `pt notification` group reads your notifications and marks them read or unread (sending a notification isn't exposed by the API). `--unread-only` filters the current page **client-side**:
+
+```bash
+# List notifications (newest first); page through with --page / --page-size
+pt notification list
+pt notification list --unread-only
+
+# How many are unread?
+pt notification unread-count
+
+# Mark one read / unread, or clear them all
+pt notification mark-read 42
+pt notification mark-unread 42
+pt notification mark-all-read
+
+# Delete one (prompts unless --yes)
+pt notification delete 42 --yes
+```
+
 ## Scaffolding Live Apps
 
 `pt live-app new` initializes a Live App locally from PrimeThink's public template catalog. It does not call the PrimeThink API, so no token or profile is needed. This makes it suitable for coding agents that need a safe, deterministic starting point.