json-object-editor 0.10.668 → 0.10.671

package/docs/JOE_AI_Overview.md

@@ -9,23 +9,25 @@ This document gives Custom GPTs, Assistants, and AI agents a single mental model
 - **Core idea**: JOE is a schema‑driven data system. AI features are layered on top to:
   - Generate or transform field values (AI autofill)
   - Run persistent assistants and chats (in‑app + widgets)
+  - Track AI job progress in real-time at the field level
   - Expose JOE data and writes via MCP tools for external agents
 - **Main components**:
-  - **AI schemas** (`ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_conversation`, `ai_widget_conversation`)
+  - **AI schemas** (`ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_widget_conversation`)
   - **Server plugins**:
-    - `chatgpt.js` – **central / preferred** entry point for OpenAI (Responses API, tool calls, MCP bridge). New integrations should target this plugin.
-    - `chatgpt-assistants.js` – legacy Assistants‑API plugin, still used by existing in‑app chat flows (`ai_conversation` / `<joe-ai-chatbox>`) but not recommended for new work.
-    - `chatgpt-tools.js` / `chatgpt-responses.js` – helpers for tools + response shaping
+    - `chatgpt.js` – **central / preferred** entry point for OpenAI (Responses API, tool calls, MCP bridge)
+    - `chatgpt-assistants.js` – legacy Assistants‑API plugin (backward compatibility only)
+    - `AiJobs.js` – job tracking and progress management
   - **Client / UI**:
-    - `joe-ai.js` – in‑app AI chat UI (web components) wired to `ai_conversation` + assistants
-    - Standard JOE editor UI with **AI autofill** buttons on fields, driven by `ai_prompt`
+    - `joe-ai.js` – in‑app AI chat UI (web components) and job polling
+    - `field-jobs-container.js` – web component for displaying job progress
+    - Standard JOE editor UI with **AI autofill** buttons on fields
   - **MCP**:
     - `/.well-known/mcp/manifest.json` – describes tools
     - `/mcp` – JSON‑RPC endpoint whose tools route into `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`
 
-When in doubt: **schemas define structure**, **plugins call OpenAI + MCP**, **MCP exposes tools**, and **UI (`joe-ai`) is just a thin chat/controls layer on top.
+When in doubt: **schemas define structure**, **plugins call OpenAI + MCP**, **MCP exposes tools**, and **UI (`joe-ai`) is just a thin chat/controls layer on top**.
 
-### 1.5 End‑to‑end mental model (for agents and humans)
+### 1.1 End‑to‑end mental model
 
 At a high level, almost every AI flow in JOE is one of these patterns:
 
@@ -43,7 +45,7 @@ At a high level, almost every AI flow in JOE is one of these patterns:
      - Resolves the active `ai_assistant` by `_id` and loads its `instructions` + MCP config.  
      - Builds `systemText` (assistant instructions + MCP tool list + optional scope hints / `understandObject` snapshot).  
      - Gathers any attached files (from scoped objects or assistant‑level files) and passes their OpenAI ids/roles into `runWithTools`.  
-  4. `runWithTools` performs a Responses+tools call (with MCP tools where enabled) and returns the assistant’s reply; `widgetMessage` appends it to `ai_widget_conversation.messages` and returns the updated history to the widget.
+  4. `runWithTools` performs a Responses+tools call (with MCP tools where enabled) and returns the assistant's reply; `widgetMessage` appends it to `ai_widget_conversation.messages` and returns the updated history to the widget.
 
 - **MCP‑only agent flow (Custom GPT / external agent)**  
   1. The agent discovers tools and named toolsets via `/.well-known/mcp/manifest.json`.  
@@ -52,33 +54,19 @@ At a high level, almost every AI flow in JOE is one of these patterns:
 
 ---
 
-## 2. AI Schemas (Mental Model)
+## 2. AI Schemas
 
 JOE uses a small set of AI‑specific schemas. A good agent should know what each represents and how they relate.
 
-### 2.0 Modern assistant & conversation identity (2026+)
-
-- **Assistant identity**:
-  - Treat `ai_assistant._id` (JOE cuid) as the **canonical id** for assistants.
-  - `ai_assistant.assistant_id` (OpenAI Assistants id) is **optional/legacy** and only present when synced to the old Assistants API.
-  - The default assistant used by widgets/AIHub is stored in `setting.DEFAULT_AI_ASSISTANT.value` as an `ai_assistant._id`.
-- **Conversations**:
-  - `ai_widget_conversation` is the **primary** conversation schema for modern chats (`<joe-ai-widget>`, AIHub cards, object chat). It links to assistants via `assistant` (JOE cuid) and may also store a legacy `assistant_id`.
-  - `ai_conversation` is **legacy** and only used by older `<joe-ai-chatbox>` flows that still rely on the Assistants API.
-- **MCP configuration**:
-  - MCP fields (`mcp_enabled`, `mcp_toolset`, `mcp_selected_tools`, `mcp_instructions_mode`) can exist both on `ai_assistant` (assistant‑level default) and `ai_prompt` / field‑level `ai` configs (surface‑level overrides).
-  - The server (`chatgpt.js`) uses a shared helper to resolve these configs into actual toolsets and instructions for all AI surfaces (prompts, autofill, widget/object chat).
-
 ### 2.1 `ai_assistant`
 
 - **What it is**: Configuration record for a single AI assistant linked to OpenAI.
-- **Key fields** (from schema + summaries):
-  - Identity & model: `name`, `info`, `ai_model`, `assistant_id`, `openai_assistant_version`
-  - Capabilities: `file_search_enabled`, `code_interpreter_enabled`
+- **Key fields**:
+  - Identity & model: `name`, `info`, `ai_model`, `assistant_id` (optional/legacy)
   - Prompting: `instructions`, `assistant_thinking_text`, `assistant_color`
-  - Tools: `tools` (JSON OpenAI tools array – often imported from MCP), `datasets`, `tags`, `status`
-  - Sync meta: `last_synced`, timestamps (`created`, `joeUpdated`)
-- **How it’s used**:
+  - Tools: `tools` (JSON OpenAI tools array – often imported from MCP)
+  - MCP config: `mcp_enabled`, `mcp_toolset`, `mcp_selected_tools`, `mcp_instructions_mode`
+- **How it's used**:
   - One `ai_assistant` usually maps 1:1 to an OpenAI Assistant.
   - A **DEFAULT** can be set via `setting.DEFAULT_AI_ASSISTANT` and is used by `joe-ai` as the default assistant.
   - The schema exposes helper methods such as:
@@ -87,7 +75,7 @@ JOE uses a small set of AI‑specific schemas. A good agent should know what eac
     - `setAsDefaultAssistant` – update the `DEFAULT_AI_ASSISTANT` setting.
 
 **Agent note**: When reasoning about which assistant is active in a chat or widget, look for:
-- `ai_conversation.assistant` or `ai_widget_conversation.assistant`
+- `ai_widget_conversation.assistant` (JOE cuid)
 - Or the instance‑level default via the `setting` schema.
 
 ### 2.2 `ai_prompt`
@@ -95,201 +83,208 @@ JOE uses a small set of AI‑specific schemas. A good agent should know what eac
 - **What it is**: Reusable AI **prompt configuration** – how to call the `chatgpt` plugin or Responses API.
 - **Key fields**:
   - Identity: `_id`, `name`, `info`, `itemtype:'ai_prompt'`
-  - Integration with plugin:
-    - `prompt_method` – name of the server method on `chatgpt.js` to call (e.g. `executeJOEAiPrompt`).
-    - `content_items` – `objectList` of `{ itemtype, reference }` describing which JOE objects to send and under what parameter name.
-  - Instructions:
-    - `functions` – helper code snippet (Node‑style `module.exports = async function(...) { ... }`) used to shape instructions/input.
-    - `instructions_format` – format hint; `instructions` – main system‑level text.
-    - `user_prompt` – optional user‑facing prompt template or per‑call input description.
+  - Integration: `prompt_method` (e.g. `executeJOEAiPrompt`), `content_items` (objectList of `{ itemtype, reference }`)
+  - Instructions: `functions` (helper code), `instructions_format`, `instructions`, `user_prompt`
   - OpenAI tuning: `ai_model`, `temperature`
-  - Meta: `status`, `tags`, `datasets`, timestamps
-- **Methods / behavior**:
-  - `methods.buildURL(prompt, items)` – builds URLs like:
-    - `/API/plugin/chatgpt/<prompt_method>?ai_prompt=<prompt._id>&<reference>=<item._id>...`
-  - `methods.listExamples(prompt)` – picks sample content objects for exploration/testing.
+  - MCP config: `mcp_enabled`, `mcp_toolset`, `mcp_selected_tools`, `mcp_instructions_mode`
 - **Typical flows**:
-  - **AI Autofill (field‑level)**:
-    - Fields in other schemas include an `ai` config (documented in `CHANGELOG` + README).
-    - The UI calls `/API/plugin/chatgpt/autofill`, which:
-      - Uses `ai_prompt` definitions and schema metadata.
-      - Sends the right JOE objects + instructions to OpenAI.
-      - Returns structured patch JSON to update fields.
-  - **Explicit prompts**:
-    - UI actions or external tools call `/API/plugin/chatgpt/<prompt_method>?ai_prompt=<id>&...`.
-    - The plugin locates the prompt, merges helper `functions`, and runs OpenAI.
-
-**Agent note**: Treat `ai_prompt` as the **source of truth** for how to talk to OpenAI for a particular workflow (summaries, planning, refactors, etc). Never invent `prompt_method` names; reuse existing ones or ask a human to add a new prompt record.
+  - **AI Autofill (field‑level)**: Fields include an `ai` config. The UI calls `/API/plugin/chatgpt/autofill`, which uses `ai_prompt` definitions and returns structured patch JSON.
+  - **Explicit prompts**: UI actions call `/API/plugin/chatgpt/<prompt_method>?ai_prompt=<id>&...`. The plugin locates the prompt, merges helper `functions`, and runs OpenAI.
+
+**Agent note**: Treat `ai_prompt` as the **source of truth** for how to talk to OpenAI for a particular workflow. Never invent `prompt_method` names; reuse existing ones or ask a human to add a new prompt record.
 
 ### 2.3 `ai_response`
 
 - **What it is**: Persistent record of a single AI response, for **auditing**, **reuse**, and **merge‑back** into JOE.
 - **Key fields**:
-  - Links: `ai_prompt` (reference), `referenced_objects` (array of JOE `_id`s used), `tags`
+  - Links: `ai_prompt` (reference), `referenced_objects` (array of JOE `_id`s), `tags`
   - Request context: `user_prompt`, `prompt_method`, `model_used`, `response_type`
-  - Response data:
-    - `response` – raw string body (often JSON or text).
-    - `response_keys` – array of keys returned (used for patch/merge).
-    - `response_id` – OpenAI response ID.
-    - `usage` – token usage object `{ input_tokens, output_tokens, total_tokens }`.
+  - Response data: `response` (raw string), `response_keys`, `response_id`, `usage` (token usage)
+  - MCP audit: `mcp_tools_used[]`, `used_openai_file_ids[]`
 - **Methods**:
-  - `compareResponseToObject(response_id, object_id, do_alert)` – validate response JSON vs object fields, optionally trigger a merge.
-  - `updateObjectFromResponse(response_id, object_id, fields)` – apply response values into a JOE object and update tags.
-  - `listResponses(obj)` – list AI responses referencing a given object.
+  - `compareResponseToObject(response_id, object_id, do_alert)` – validate response JSON vs object fields
+  - `updateObjectFromResponse(response_id, object_id, fields)` – apply response values into a JOE object
+  - `listResponses(obj)` – list AI responses referencing a given object
 
-**Agent note**: When you are updating objects via tools, favor:
-- Using `ai_response` records to compare/merge changes, especially if the response contains multiple fields.
-- Respect existing merge flows (`compareResponseToObject` / `updateObjectFromResponse`) instead of overwriting arbitrary fields.
+**Agent note**: When updating objects via tools, favor using `ai_response` records to compare/merge changes, especially if the response contains multiple fields.
 
-### 2.4 `ai_tool`
+### 2.4 `ai_widget_conversation`
 
-- **What it is**: Definition of a reusable **function‑calling tool** that OpenAI can call and JOE can execute server‑side.
+- **What it is**: Lightweight conversation record for **embeddable AI widgets** (external sites, test pages, object chat).
 - **Key fields**:
-  - Identity: `name`, `info`, `description`, `tool_id`
-  - OpenAI schema: `tool_properties` (JSON, typically `{ type:'function', function:{ name, description, parameters } }`)
-  - Execution: `server_code` – Node.js async code to run when the tool is invoked.
-  - Meta: `datasets`, `tags`, timestamps.
-- **Methods**:
-  - `updateToolProperties(propObj)` – helper to keep `tool_properties.function.name` in sync with `tool_id`.
-
-**Agent note**: For tools exposed via MCP, the canonical shape is actually in the **MCP manifest**; `ai_tool` is more for:
-- High‑level catalog and advanced OpenAI tools, not the core MCP schema/search/save tools.
-
-### 2.5 `ai_conversation`
-
-- **What it is**: Metadata record for **in‑app AI conversations** (staff/admin chat in JOE).
-- **Key fields**:
-  - Participants: `user` (JOE user), `assistant` (`ai_assistant`), `members` (external participants)
-  - OpenAI thread: `thread_id`
-  - Summary: `summary` (WYSIWYG), `status`, `tags`
-  - Meta: `_id`, `name`, `info`, timestamps
-- **Behavior**:
-  - Messages are **not stored** in JOE; they’re fetched from OpenAI at runtime (via `chatgpt-assistants.js`).
-  - Schema methods + UI:
-    - `methods.chatSpawner(object_id)` – calls `_joe.Ai.spawnContextualChat(object_id)`.
-    - List view shows created time, user cubes, and context objects used for that chat.
-  - `joe-ai.js` uses `ai_conversation` as the anchor object for chat history + context.
-
-**Agent note**: If you need to understand a chat history, you:
-- Read `ai_conversation` for metadata and context_objects.
-- Use **MCP tools** or existing chat APIs to fetch live thread content; don’t invent local persistence.
-
-### 2.6 `ai_widget_conversation`
-
-- **What it is**: Lightweight conversation record for **embeddable AI widgets** (external sites, test pages).
-- **Key fields**:
-  - Participant & assistant: `user`, `assistant`, `user_name`, `user_color`, `assistant_color`
+  - Participant & assistant: `user`, `assistant` (JOE cuid), `user_name`, `user_color`, `assistant_color`
   - OpenAI info: `model`, `assistant_id`, `system` (effective instructions)
-  - Conversation: `messages` (compact list, typically JSON array of `{ role, content, created_at }`), `last_message_at`
-  - Source: `source` (widget origin), `tags`, timestamps
+  - Conversation: `messages` (JSON array of `{ role, content, created_at }`), `last_message_at`
+  - Source: `source` (widget origin), `scope_itemtype`, `scope_id`, `tags`
 - **Behavior**:
-  - Accessed/updated via `chatgpt-assistants` widget endpoints:
-    - `widgetStart`, `widgetMessage`, `widgetHistory`, etc.
-  - Schema exposes **subsets** and **filters** for grouping by `source`, `user`, and `assistant`.
+  - Accessed/updated via `chatgpt.js` widget endpoints: `widgetStart`, `widgetMessage`, `widgetHistory`
+  - Schema exposes **subsets** and **filters** for grouping by `source`, `user`, and `assistant`
 
-**Agent note**: Treat `ai_widget_conversation` as the external/chat‑widget equivalent of `ai_conversation`. The fields are tuned for widgets and public UI, but the mental model is the same: small record pointing to assistant + messages.
+**Agent note**: `ai_widget_conversation` is the **primary** conversation schema for modern chats (`<joe-ai-widget>`, AIHub cards, object chat). The legacy `ai_conversation` schema is only used by older `<joe-ai-chatbox>` flows.
 
----
+### 2.5 `ai_tool` (optional)
 
-## 3. `chatgpt` Plugin and Related Plugins
+- **What it is**: Definition of a reusable **function‑calling tool** that OpenAI can call and JOE can execute server‑side.
+- **Key fields**: `name`, `info`, `description`, `tool_id`, `tool_properties` (JSON), `server_code` (Node.js async code)
+- **Agent note**: For tools exposed via MCP, the canonical shape is in the **MCP manifest**; `ai_tool` is more for high‑level catalog and advanced OpenAI tools, not the core MCP schema/search/save tools.
 
-### 3.0 Modern vs legacy stacks
+---
 
-- **Modern stack (preferred)**:
-  - Uses `server/plugins/chatgpt.js` with the **Responses API** and MCP tools.
-  - Conversations live in `ai_widget_conversation` and are surfaced via `<joe-ai-widget>` and `<joe-ai-assistant-picker>`.
-  - MCP configuration is respected from `ai_assistant` and `ai_prompt` (and field‑level `ai` configs).
-- **Legacy stack (Assistants API)**:
-  - Uses `server/plugins/chatgpt-assistants.js` and OpenAI Assistants/threads.
-  - Conversations live in `ai_conversation` and are surfaced via `<joe-ai-chatbox>`.
-  - Kept for backward compatibility only; **new integrations should use the modern stack above**.
+## 3. Server Plugins
 
-### 3.1 `server/plugins/chatgpt.js`
+### 3.1 `server/plugins/chatgpt.js` (Core Plugin)
 
 This is the **core server plugin** that wires JOE to OpenAI and MCP.
 
 - **Responsibilities**:
-  - Read OpenAI API key from `setting.OPENAI_API_KEY`.
-  - Provide shared helpers for building OpenAI clients (`OpenAI` SDK).
-  - Implement orchestration for:
-    - **Responses API** calls (models like `gpt-4.1`, `gpt-4.1-mini`, `gpt-5.1`).
-    - **Tool calling** and follow‑up calls via MCP tools.
+  - Read OpenAI API key from `setting.OPENAI_API_KEY`
+  - Provide shared helpers for building OpenAI clients (`OpenAI` SDK)
+  - Implement orchestration for **Responses API** calls (models like `gpt-4.1`, `gpt-4.1-mini`, `gpt-5.1`)
+  - **Tool calling** and follow‑up calls via MCP tools
   - Bridge between JOE and MCP with helpers like:
-    - `callMCPTool(toolName, params, ctx)` – call MCP tools **in‑process** (without HTTP).
-    - Response parsing helpers such as `extractToolCalls(response)`.
-    - Payload shrinking helpers like `shrinkUnderstandObjectMessagesForTokens(...)`.
-  - Expose specific routes under `/API/plugin/chatgpt/...`, including:
-    - Field autofill (`autofill`) driven by schema‑level `ai` configs.
-    - Prompt execution handlers for `ai_prompt.prompt_method` values.
+    - `callMCPTool(toolName, params, ctx)` – call MCP tools **in‑process** (without HTTP)
+    - Response parsing helpers such as `extractToolCalls(response)`
+    - Payload shrinking helpers like `shrinkUnderstandObjectMessagesForTokens(...)`
+- **Expose routes** under `/API/plugin/chatgpt/...`:
+  - Field autofill (`autofill`) driven by schema‑level `ai` configs
+  - Prompt execution handlers for `ai_prompt.prompt_method` values
+  - Widget endpoints: `widgetStart`, `widgetMessage`, `widgetHistory`
 - **Key patterns**:
-  - **Slimming payloads**: When embedding object graphs into messages (e.g. `understandObject` results), `chatgpt.js` converts them into a slim representation to avoid token limits.
-  - **Error handling**:
-    - `isTokenLimitError(err)` identifies token/TPM error shapes for better logs and fallbacks.
-
-**Agent note**: Any time you see endpoints in docs like `/API/plugin/chatgpt/<method>`, they resolve into functions in `chatgpt.js` or its siblings. Follow those conventions instead of inventing new paths.
-
-### 3.2 `chatgpt-assistants.js`, `chatgpt-responses.js`, `chatgpt-tools.js`
-
-- **`chatgpt-assistants.js`**:
-  - Manages OpenAI Assistants and threads using the **older Assistants API**.
-  - Endpoints such as:
-    - `syncAssistantToOpenAI` (used from `ai_assistant.methods`).
-    - `getThreadMessages`, `runAssistantChat`, widget endpoints like `widgetStart`, `widgetMessage`, `widgetHistory`.
-  - Used by `joe-ai.js` and widget UIs to pull/push messages for a given conversation.
-  - **Status**: legacy bridge for existing flows; **new integrations should use the Responses‑based `chatgpt` plugin instead**, matching the README.
-
-- **`chatgpt-responses.js`**:
-  - Helpers for working with Responses API results:
-    - Extracting text and tool calls.
-    - Normalizing outputs for storage in `ai_response`.
-  - Often called from `chatgpt.js` orchestration functions.
-
-- **`chatgpt-tools.js`**:
-  - Glue between AI tools and JOE:
-    - Uses `ai_tool` definitions.
-    - Executes server‑side code blocks for tools when OpenAI calls them.
-  - Can also support MCP‑style tools exported via the manifest.
-
-**Agent note**: You typically don’t call these plugins directly from a GPT. Instead, you:
-- Use documented **APIs** (`/API/plugin/chatgpt/*`, widget endpoints) and/or
-- Use **MCP tools** for reads/writes.
+  - **Slimming payloads**: When embedding object graphs into messages (e.g. `understandObject` results), `chatgpt.js` converts them into a slim representation to avoid token limits
+  - **Error handling**: `isTokenLimitError(err)` identifies token/TPM error shapes for better logs and fallbacks
+  - **Job tracking**: Registers and updates AI jobs via `AiJobs` module for progress visibility
+
+**Agent note**: Any time you see endpoints in docs like `/API/plugin/chatgpt/<method>`, they resolve into functions in `chatgpt.js`. Follow those conventions instead of inventing new paths.
+
+### 3.2 `server/modules/AiJobs.js` (Job Tracking)
+
+Manages active AI job storage and provides API endpoints for real-time progress tracking.
+
+- **Storage**: In-memory `active` object keyed by `{objectId}_{fieldName}`
+- **Job Structure**:
+  ```javascript
+  {
+    token: string,           // Full token: {objectId}|{fieldName}|{timestamp}|{userid}
+    startTime: ISO string,  // Job start timestamp
+    status: string,          // 'starting', 'running', 'complete', 'error'
+    promptId: string|null,  // Optional prompt ID
+    promptName: string|null, // Human-readable job name
+    fieldId: string|null,    // Field identifier
+    progress: number,        // Current progress (0-100 or 0-1)
+    total: number|null,      // Total steps (typically 100 or 1)
+    message: string          // Status message
+  }
+  ```
+- **Key Functions**:
+  - `createJob(token, jobData)` - Register a new job
+  - `updateJob(token, updates)` - Update job status/progress/message
+  - `removeJob(token)` - Remove job immediately
+  - `removeJobWithDelay(token, status, message, delaySeconds)` - Remove after delay (default 10s)
+  - `extractKey(token)` - Parse token to get `{objectId}_{fieldName}` lookup key
+- **API Endpoints**:
+  - `GET /API/aijobs/:objectId/:fieldName` - Returns `{ jobs: [...], objectId, fieldName, count }` for a specific field (flat array)
+  - `GET /API/aijobs/:objectId` - Returns `{ jobs: [{ lookupKey, jobs: [...] }], objectId, count }` for an object (grouped by field)
+  - `GET /API/aijobs` - Returns `{ jobs: [{ lookupKey, jobs: [...] }], count, lookupKeys }` for all active jobs (grouped, optional/admin)
+
+**Agent note**: Jobs are automatically registered by `chatgpt.js` and `MCP.js` when AI operations start. The token format uses `|` as delimiter to avoid conflicts with field names containing underscores.
+
+### 3.3 Legacy Plugins (Backward Compatibility Only)
+
+- **`chatgpt-assistants.js`**: Manages OpenAI Assistants and threads using the **older Assistants API**. Used by legacy `<joe-ai-chatbox>` flows. **New integrations should use the Responses‑based `chatgpt` plugin instead**.
+- **`chatgpt-responses.js`**: Helpers for working with Responses API results (extracting text, tool calls, normalizing outputs).
+- **`chatgpt-tools.js`**: Glue between AI tools and JOE (uses `ai_tool` definitions, executes server‑side code blocks).
+
+**Agent note**: You typically don't call these plugins directly. Instead, use documented **APIs** (`/API/plugin/chatgpt/*`, widget endpoints) and/or **MCP tools** for reads/writes.
 
 ---
 
-## 4. `joe-ai.js` – In‑App AI UI
-
-The `js/joe-ai.js` file implements JOE’s in‑app AI user interface using custom elements and an `Ai` namespace.
-
-- **Key responsibilities**:
-  - Manage open chat boxes and default assistant selection:
-    - `Ai._openChats`, `Ai.default_ai`, `Ai.getDefaultAssistant()`.
-  - Render **modern widget‑based chat** bound to `ai_widget_conversation`:
-    - `<joe-ai-widget>` – embeddable chat component that talks to `chatgpt.js` (`widgetStart`, `widgetMessage`, `widgetHistory`).
-    - `<joe-ai-assistant-picker>` – selector that applies an `ai_assistant._id` to a widget (`ai_assistant_id` attribute) and starts fresh conversations as needed.
-    - `Ai.openObjectChatLauncher(object_id, itemtype, name)` – launches a floating shell containing a `<joe-ai-widget>` for **object‑scoped chat**:
-      - Sets `source:"object_chat"`, `scope_itemtype`, `scope_id`, and passes the current user id.
-      - Uses `DEFAULT_AI_ASSISTANT` as the initial assistant, but allows per‑conversation overrides via the picker.
-  - (Legacy) Render **Assistants‑API chat** bound to an `ai_conversation`:
-    - `<joe-ai-chatbox>` – older component that loads `ai_conversation` and uses `chatgpt-assistants` endpoints (`getThreadMessages`, etc.).
-    - Still present for existing flows but not used by the new object chat or AIHub widget experiences.
-  - Provide ergonomic chat behaviors:
-    - Markdown rendering via `renderMarkdownSafe` (prefers `marked` + `DOMPurify`).
-    - Greeting messages based on `user` and `context_objects`.
-    - Keyboard handling (Enter to send).
-    - Conversation list (`<joe-ai-conversation-list>`) for selecting existing `ai_widget_conversation` records and applying them to a widget.
-- **High‑level flow** for modern widget chat:
-  1. UI renders a `<joe-ai-widget>` with optional `ai_assistant_id`, `source`, and scope attributes.
-  2. On the first user message, the widget calls `/API/plugin/chatgpt/widgetStart`, which creates an `ai_widget_conversation`.
-  3. Subsequent messages call `/API/plugin/chatgpt/widgetMessage`, which:
-     - Resolves the active `ai_assistant` (by `_id`), builds `systemText` (assistant instructions + optional MCP tool list + scope hints), and attaches any relevant files.
-     - Calls `runWithTools` in `chatgpt.js` so the model can use MCP tools and respond.
-     - Updates `ai_widget_conversation.messages` and returns the full history to the widget.
-
-**Agent note**:
-- When describing UI behavior or helping with debugging, remember that `joe-ai.js` is **UI only**; persistent data lives in:
-  - `ai_widget_conversation` (modern chats: AIHub, object chat, widgets)
-  - `ai_assistant` (config, including MCP/tool settings and assistant‑level files)
-  - `ai_conversation` (legacy Assistants‑API chats only)
+## 4. Client-Side UI (`joe-ai.js`)
+
+The `js/joe-ai.js` file implements JOE's in‑app AI user interface using custom elements and an `Ai` namespace.
+
+### 4.1 Chat Components
+
+- **`<joe-ai-widget>`** – Modern embeddable chat component that talks to `chatgpt.js` (`widgetStart`, `widgetMessage`, `widgetHistory`). Bound to `ai_widget_conversation`.
+- **`<joe-ai-assistant-picker>`** – Selector that applies an `ai_assistant._id` to a widget (`ai_assistant_id` attribute).
+- **`<joe-ai-chatbox>`** – Legacy component that loads `ai_conversation` and uses `chatgpt-assistants` endpoints. Still present for existing flows but not used by new object chat or AIHub widget experiences.
+- **`Ai.openObjectChatLauncher(object_id, itemtype, name)`** – Launches a floating shell containing a `<joe-ai-widget>` for **object‑scoped chat**:
+  - Sets `source:"object_chat"`, `scope_itemtype`, `scope_id`, and passes the current user id
+  - Uses `DEFAULT_AI_ASSISTANT` as the initial assistant, but allows per‑conversation overrides via the picker
+
+### 4.2 Job Tracking & Polling
+
+- **Global Poller**: `Ai.jobsPoller` object manages polling for all `field-jobs-container` components
+- **Polling Intervals**:
+  - **Active jobs present**: 3 seconds
+  - **No active jobs**: 10 seconds (idle)
+- **Immediate Polling**: When an AI button is clicked, `Ai.jobsPoller.poll()` is called immediately (with 200ms delay to let server register the job)
+- **Poll Logic**:
+  1. Finds all `field-jobs-container` elements on the page
+  2. For each container, polls `/API/aijobs/{objectId}/{fieldName}`
+  3. Updates container via `container.updateJobs(jobs)`
+  4. Adjusts interval based on whether any active jobs were found
+- **Token Generation**: `Ai.generateProgressToken(objectId, fieldName, userID)` creates structured tokens: `{objectId}|{fieldName}|{timestamp}|{userid}`
+- **Debug Functions** (available in browser console):
+  - `Ai.checkPoller()` - Shows poller status, interval, and container count
+  - `Ai.restartPoller()` - Restarts the global poller
+  - `Ai.debugJobs()` - Shows all active jobs in `Ai.activeJobs` Map
+
+### 4.3 UI Component (`field-jobs-container`)
+
+- **Web Component**: `<field-jobs-container>` custom element defined in `web-components/field-jobs-container.js`
+- **Attributes**:
+  - `data-object-id` - Object CUID
+  - `data-field-name` - Field name
+- **Rendering**:
+  - Title shows active job count: "N active job(s)"
+  - Token link on right side (small, bold) - links to `/API/aijobs/{objectId}/{fieldName}` endpoint page
+  - Each job row shows:
+    - Job name (from `promptName` or `promptId`)
+    - Status (capitalized, inline): " - Running", " - Complete", etc.
+    - Elapsed time: "(Xs)" in seconds
+    - Progress percentage: "XX%" or "—" if not available
+    - Message on separate line (smaller, lighter text)
+  - Completed jobs remain visible until page/field refresh
+- **Lifecycle**:
+  - `connectedCallback()` triggers immediate poll on mount
+  - `updateJobs(jobs)` updates the display
+  - Elapsed time updates every second for active jobs
+
+### 4.4 Field Integration
+
+To enable job tracking on a field:
+
+1. **Add `queryJobs: true` to field definition**:
+   ```javascript
+   {
+     name: 'select_prompt',
+     type: 'select_prompt',
+     queryJobs: true  // Enables field-jobs-container rendering
+   }
+   ```
+
+2. **Ensure button has required data attributes**:
+   - `data-object-id` - Set automatically from current object
+   - `data-field-name` - Set from field name
+   - `data-progress-token` - Generated via `Ai.generateProgressToken()`
+
+3. **Server-side job registration**:
+   - Autofill: `chatgpt.js` → `registerAiJobIfToken()` → `updateAiJobIfToken()`
+   - Prompts: `chatgpt.js` → `executeJOEAiPrompt()` → `updateAiJobIfToken()`
+   - Thoughts: `MCP.js` → `runThoughtAgent()` → `MCP.updateJob()`
+
+### 4.5 Job Lifecycle
+
+1. **Start**: Button click → Generate token → Register job on server → Trigger immediate poll
+2. **Progress**: Server calls `updateJob(token, { status, message, progress, total })` at key stages
+3. **Complete**: Server sets `status: 'complete'`, `progress: 100` → Job remains visible in UI
+4. **Cleanup**: Server calls `removeJobWithDelay(token, 'complete', 'Complete', 10)` → Job removed from server after 10 seconds, but remains in UI until refresh
+
+**Agent note**: When adding new AI operations that should show progress:
+1. Generate token using `Ai.generateProgressToken(objectId, fieldName, userID)`
+2. Register job on server via `AiJobs.createJob(token, jobData)`
+3. Update progress via `AiJobs.updateJob(token, { status, message, progress, total })`
+4. Ensure the field has `queryJobs: true` if using `field-jobs-container`
 
 ---
 
@@ -300,41 +295,28 @@ MCP (Model Context Protocol) is how external agents (including Custom GPTs) call
 ### 5.1 Endpoints and Manifest
 
 - **Endpoints**:
-  - `GET /.well-known/mcp/manifest.json` – public manifest with:
-    - Instance info `{ joe: { name, version, hostname } }`
-    - Tool list (name, description, params schema)
-    - Privacy/terms URLs (also wired into `/privacy` and `/terms` routes).
-  - `POST /mcp` – auth‑protected JSON‑RPC 2.0 endpoint for **tool calls**.
-    - Same auth rules as other APIs; if users exist, requires cookie or basic auth.
-- **Implementation note**:
-  - The concrete MCP tool implementations live in `server/modules/MCP.js`, which is wired up by `server/init.js` and exported via the manifest and `/mcp` endpoint.
-- **Tools** (from README + CHANGELOG):
+  - `GET /.well-known/mcp/manifest.json` – public manifest with instance info, tool list, privacy/terms URLs
+  - `POST /mcp` – auth‑protected JSON‑RPC 2.0 endpoint for **tool calls** (same auth rules as other APIs)
+- **Implementation**: The concrete MCP tool implementations live in `server/modules/MCP.js`, which is wired up by `server/init.js` and exported via the manifest and `/mcp` endpoint.
+- **Tools**:
   - `listSchemas(name?)`, `getSchema(name)`
-  - `getObject(_id, itemtype?)`
+  - `getObject(_id, itemtype?)` (supports optional `flatten` and `depth`)
   - `search` (unified search across cache/storage; supports `source`, `ids`, `flatten`, `depth`, `countOnly`)
   - `fuzzySearch` – typo‑tolerant search across weighted fields
+  - `findObjectsByTag`, `findObjectsByStatus` – tag/status-based queries
   - `saveObject({ object })`
   - `saveObjects({ objects, stopOnError?, concurrency? })`
   - `hydrate` – returns core fields, schemas, statuses, tags (for agents to bootstrap context)
 
-All tools map directly into JOE’s APIs: `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`, and related helpers. Sensitive fields are sanitized before returning data.
+All tools map directly into JOE's APIs: `JOE.Schemas`, `JOE.Storage`, `JOE.Cache`, and related helpers. Sensitive fields are sanitized before returning data.
 
 ### 5.2 Agent Best Practices with MCP
 
-- **Schema‑first**: Always:
-  1. Discover schemas via `listSchemas`/`getSchema`.
-  2. Use `hydrate` on startup to cache schemas/statuses/tags.
-  3. Use `search` / `fuzzySearch` with explicit `itemtype` and filters rather than arbitrary text mining.
-- **Reads vs writes**:
-  - Prefer **cache** for read operations (`search` default) unless you explicitly need DB‑level freshness (set `source:"storage"`).
-  - For writes, use:
-    - `saveObject` for single‑item updates.
-    - `saveObjects` for batched updates with clear error handling.
-- **Performance**:
-  - Use `countOnly` before large reads to avoid pulling massive datasets.
-  - When embedding objects into prompts, prefer **slimmed** representations (id, itemtype, name, info).
-
-**Agent note**: The MCP tools are your primary way to **inspect and modify JOE data** from a Custom GPT or other agent host. Direct HTTP calls to JOE’s internal `/API/*` routes should be treated as implementation detail unless explicitly allowed in your instructions.
+- **Schema‑first**: Always discover schemas via `listSchemas`/`getSchema`, use `hydrate` on startup to cache schemas/statuses/tags, use `search` / `fuzzySearch` with explicit `itemtype` and filters
+- **Reads vs writes**: Prefer **cache** for read operations (`search` default) unless you explicitly need DB‑level freshness (set `source:"storage"`). For writes, use `saveObject` for single‑item updates, `saveObjects` for batched updates
+- **Performance**: Use `countOnly` before large reads to avoid pulling massive datasets. When embedding objects into prompts, prefer **slimmed** representations (id, itemtype, name, info)
+
+**Agent note**: The MCP tools are your primary way to **inspect and modify JOE data** from a Custom GPT or other agent host. Direct HTTP calls to JOE's internal `/API/*` routes should be treated as implementation detail unless explicitly allowed in your instructions.
 
 ---
 
@@ -343,44 +325,31 @@ All tools map directly into JOE’s APIs: `JOE.Schemas`, `JOE.Storage`, `JOE.Cac
 When configuring a JOE‑aware Custom GPT or dev agent, include the following docs in its RAG/context:
 
 - **Core architecture & instance info**
-  - `README.md` – especially:
-    - “Architecture & Mental Model (Server)”
-    - MCP overview + test instructions (including references to `_www/mcp-test.html`, `_www/mcp-schemas.html`, and `_www/plugins-test.html` for live inspection)
-  - `CHANGELOG.md` – AI/MCP‑related entries (0.10.43x+, 0.10.50x+, 0.10.62x+), already summarize important AI behavior.
+  - `README.md` – especially "Architecture & Mental Model (Server)" and MCP overview + test instructions
+  - `CHANGELOG.md` – AI/MCP‑related entries (0.10.43x+, 0.10.50x+, 0.10.62x+)
 - **Agent / instruction docs (existing)**
-  - `docs/JOE_Master_Knowledge_Export.md` – master context dump for schemas and concepts.
-  - `docs/joe_agent_spec_v_2.2.md` – agent behavior/specification.
-  - `docs/joe_agent_custom_gpt_instructions_v_3.md` – current Custom GPT / MCP prompt instructions.
-  - `docs/schema_summary_guidelines.md` – how schema summaries are constructed and how agents should interpret/use them.
+  - `docs/JOE_Master_Knowledge_Export.md` – master context dump for schemas and concepts
+  - `docs/joe_agent_spec_v_2.2.md` – agent behavior/specification
+  - `docs/joe_agent_custom_gpt_instructions_v_3.md` – current Custom GPT / MCP prompt instructions
+  - `docs/schema_summary_guidelines.md` – how schema summaries are constructed
 - **This file**
-  - `docs/JOE_AI_Overview.md` – **(you are here)** high‑level map of AI schemas, plugins, UI, and MCP.
+  - `docs/JOE_AI_Overview.md` – **(you are here)** high‑level map of AI schemas, plugins, UI, and MCP
 
 Optional / advanced (for deeper RAG or power users):
-- Selected excerpts or exports of **schema summaries** for AI‑related schemas:
-  - `ai_assistant`, `ai_prompt`, `ai_tool`, `ai_response`, `ai_conversation`, `ai_widget_conversation`
-- Any **project‑specific** AI workflow docs you maintain (e.g. playbooks for content generation, planning workflows, merge policies).
+- Selected excerpts or exports of **schema summaries** for AI‑related schemas
+- Any **project‑specific** AI workflow docs you maintain
 
 ---
 
-## 7. What’s Still Missing / Future Docs to Consider
+## 7. What's Still Missing / Future Docs to Consider
 
 For a fully self‑sufficient agent, the following additional docs can be helpful:
 
-- **AI Workflows Cookbook**:
-  - Short task‑oriented examples:
-    - “Summarize a project using schemas + ai_prompt + MCP search.”
-    - “Draft a plan then write objects via `saveObjects`.”
-    - “Use ai_response to compare vs existing data and merge.”
-- **AI Safety & Guardrails**:
-  - Which schemas/fields are sensitive.
-  - Allowed vs disallowed writes.
-  - Expectations around **review vs autonomous changes**.
-- **Instance‑specific Prompts / Taxonomy**:
-  - Any custom `ai_prompt` records or `ai_tool` definitions that embody local conventions (naming standards, planning templates, scoring rubrics, etc.).
-
-If these don’t exist yet, an agent should assume **conservative** behavior:
-- Prefer read + propose changes over direct writes.
-- Tag all AI‑generated content clearly.
-- Use `ai_response` + compare/merge flows rather than overwriting objects blindly.
-
+- **AI Workflows Cookbook**: Short task‑oriented examples (summarize a project, draft a plan, merge responses)
+- **AI Safety & Guardrails**: Which schemas/fields are sensitive, allowed vs disallowed writes, expectations around review vs autonomous changes
+- **Instance‑specific Prompts / Taxonomy**: Any custom `ai_prompt` records or `ai_tool` definitions that embody local conventions
 
+If these don't exist yet, an agent should assume **conservative** behavior:
+- Prefer read + propose changes over direct writes
+- Tag all AI‑generated content clearly
+- Use `ai_response` + compare/merge flows rather than overwriting objects blindly