conversationalist 0.0.1 → 0.0.2

package/README.md

@@ -1,47 +1,123 @@
 # Conversationalist
 
-A TypeScript library for managing LLM conversation state with immutable data structures and type-safe APIs.
+A TypeScript-first library for managing LLM conversation state with immutable updates, type-safe APIs, and provider adapters.
+
+The high-level value proposition:
+
+- keep a single, model-agnostic conversation shape across UI, storage, and providers
+- append messages without mutation (safe for React, concurrent updates, replay)
+- handle tool calls, streaming responses, and hidden or internal messages
+- trim context windows and estimate token budgets
+- validate input with Zod schemas and typed errors
 
 ## Installation
 
 ```bash
-bun add conversationalist
-npm add conversationalist
+bun add conversationalist zod
+npm add conversationalist zod
+pnpm add conversationalist zod
 ```
 
-## Core Concepts
-
-### Conversation
+This package is ESM-only. Use `import` syntax.
+Zod is a peer dependency and must be installed by your app.
 
-A conversation is an immutable data structure containing messages, metadata, and timestamps:
+## Quick Start
 
-```typescript
+```ts
 import {
-  createConversation,
+  appendAssistantMessage,
   appendUserMessage,
+  createConversation,
+  deserializeConversation,
+  serializeConversation,
+} from 'conversationalist';
+import { toOpenAIMessages } from 'conversationalist/openai';
+
+let conversation = createConversation({
+  title: 'Support chat',
+  tags: ['support'],
+});
+
+conversation = appendUserMessage(conversation, 'Where is my order?');
+conversation = appendAssistantMessage(conversation, 'Let me check that for you.');
+
+const messages = toOpenAIMessages(conversation);
+// send messages to your provider...
+
+const stored = serializeConversation(conversation);
+const restored = deserializeConversation(stored);
+```
+
+## End-to-End Example (Store + Resume)
+
+```ts
+import {
   appendAssistantMessage,
+  appendSystemMessage,
+  appendUserMessage,
+  createConversation,
+  deserializeConversation,
+  serializeConversation,
 } from 'conversationalist';
+import { toOpenAIMessages } from 'conversationalist/openai';
+
+let conversation = createConversation({ title: 'Order lookup' });
+conversation = appendSystemMessage(conversation, 'You are a support agent.');
+conversation = appendUserMessage(conversation, 'Where is order 123?');
+
+const response = await openai.chat.completions.create({
+  model: 'gpt-4.1-mini',
+  messages: toOpenAIMessages(conversation),
+});
 
-let conversation = createConversation({ title: 'My Chat' });
+conversation = appendAssistantMessage(
+  conversation,
+  response.choices[0]?.message?.content ?? '',
+);
 
-conversation = appendUserMessage(conversation, 'Hello!');
-conversation = appendAssistantMessage(conversation, 'Hi there!');
+const stored = serializeConversation(conversation);
+// await db.save(stored);
+
+let restored = deserializeConversation(stored);
+restored = appendUserMessage(restored, 'Can you email me the tracking link?');
 ```
 
-### Messages
+## When to Use
 
-Messages have roles (`user`, `assistant`, `system`, `tool-use`, `tool-result`, `developer`, `snapshot`) and can contain text or multi-modal content:
+- Build multi-provider chat apps with a consistent message model.
+- Store and replay conversation history safely.
+- Drive streaming UIs without ad-hoc state machines.
+- Enforce tool call correctness and pair calls with results.
+- Manage context windows and token limits in one place.
 
-```typescript
-import { appendMessages } from 'conversationalist';
+## Core Concepts
 
-// Text message
-conversation = appendMessages(conversation, {
-  role: 'user',
-  content: 'What is in this image?',
+### Conversations
+
+A conversation is an immutable record with metadata, tags, timestamps, and ordered messages.
+
+```ts
+import { createConversation } from 'conversationalist';
+
+const conversation = createConversation({
+  title: 'My Chat',
+  status: 'active',
+  metadata: { customerId: 'cus_123' },
+  tags: ['support', 'vip'],
 });
+```
+
+### Messages
+
+Messages have roles and can contain text or multi-modal content. Optional fields include
+`metadata`, `hidden`, `tokenUsage`, `toolCall`, `toolResult`, and `goalCompleted`.
+
+Roles: `user`, `assistant`, `system`, `developer`, `tool-use`, `tool-result`, `snapshot`.
+The `snapshot` role is for internal state and is skipped by adapters.
+
+```ts
+import { appendMessages } from 'conversationalist';
 
-// Multi-modal message with image
 conversation = appendMessages(conversation, {
   role: 'user',
   content: [
@@ -51,100 +127,278 @@ conversation = appendMessages(conversation, {
 });
 ```
 
+Hidden messages remain in history but are skipped by default when querying or adapting to
+providers.
+
 ### Tool Calls
 
-Tool use is supported with linked tool calls and results:
+Tool calls are represented as paired `tool-use` and `tool-result` messages. Tool results
+are validated to ensure the referenced call exists.
 
-```typescript
+```ts
 conversation = appendMessages(
   conversation,
   {
     role: 'tool-use',
     content: '',
-    toolCall: { id: 'call_123', name: 'get_weather', arguments: '{"city":"NYC"}' },
+    toolCall: { id: 'call_123', name: 'getWeather', arguments: { city: 'NYC' } },
   },
   {
     role: 'tool-result',
     content: '',
-    toolResult: { callId: 'call_123', outcome: 'success', content: '72°F, sunny' },
+    toolResult: {
+      callId: 'call_123',
+      outcome: 'success',
+      content: { tempF: 72, condition: 'sunny' },
+    },
   },
 );
 ```
 
-## API Reference
+Use `pairToolCallsWithResults` to render tool calls alongside their results.
+
+### Streaming
+
+Streaming helpers let you append a placeholder, update it as chunks arrive, and finalize
+when done.
+
+```ts
+import {
+  appendStreamingMessage,
+  finalizeStreamingMessage,
+  updateStreamingMessage,
+} from 'conversationalist';
+
+let { conversation, messageId } = appendStreamingMessage(conversation, 'assistant');
+let content = '';
 
-### Creating Conversations
+for await (const chunk of stream) {
+  content += chunk;
+  conversation = updateStreamingMessage(conversation, messageId, content);
+}
 
-```typescript
-createConversation(options?: {
-  id?: string;
-  title?: string;
-  status?: 'active' | 'archived' | 'deleted';
-  metadata?: Record<string, unknown>;
-  tags?: string[];
-}): Conversation
+conversation = finalizeStreamingMessage(conversation, messageId, {
+  tokenUsage: { prompt: 100, completion: 50, total: 150 },
+});
 ```
 
-### Appending Messages
+### Context Window
+
+Trim history to fit token budgets or to keep only recent messages.
 
-```typescript
-appendMessages(conversation: Conversation, ...inputs: MessageInput[]): Conversation
-appendUserMessage(conversation: Conversation, content: string | MultiModalContent[]): Conversation
-appendAssistantMessage(conversation: Conversation, content: string | MultiModalContent[]): Conversation
-appendSystemMessage(conversation: Conversation, content: string): Conversation
-prependSystemMessage(conversation: Conversation, content: string): Conversation
-replaceSystemMessage(conversation: Conversation, content: string): Conversation
+```ts
+import { simpleTokenEstimator, truncateToTokenLimit } from 'conversationalist';
+
+conversation = truncateToTokenLimit(conversation, 4000, simpleTokenEstimator, {
+  preserveSystemMessages: true,
+  preserveLastN: 2,
+});
 ```
 
-### Querying Messages
+### Provider Adapters
+
+Convert the same conversation into provider-specific formats.
 
-```typescript
-getConversationMessages(conversation: Conversation, options?: { includeHidden?: boolean }): Message[]
-getMessageAtPosition(conversation: Conversation, position: number): Message | undefined
-getMessageByIdentifier(conversation: Conversation, id: string): Message | undefined
-searchConversationMessages(conversation: Conversation, predicate: (m: Message) => boolean): Message[]
+Adapters skip hidden and snapshot messages and map system or developer roles as needed.
+
+```ts
+import { toOpenAIMessages, toOpenAIMessagesGrouped } from 'conversationalist/openai';
+import { toAnthropicMessages } from 'conversationalist/anthropic';
+import { toGeminiMessages } from 'conversationalist/gemini';
 ```
 
-### System Messages
+- OpenAI: `toOpenAIMessages` and `toOpenAIMessagesGrouped` (groups consecutive tool calls)
+- Anthropic: `toAnthropicMessages`
+- Gemini: `toGeminiMessages`
+
+### Tool Call Wiring Examples
+
+#### OpenAI
+
+```ts
+import { appendAssistantMessage, appendMessages } from 'conversationalist';
+import { toOpenAIMessages } from 'conversationalist/openai';
+
+const tools = [
+  {
+    type: 'function',
+    function: {
+      name: 'getWeather',
+      description: 'Get current weather by city.',
+      parameters: {
+        type: 'object',
+        properties: { city: { type: 'string' } },
+        required: ['city'],
+      },
+    },
+  },
+];
+
+const response = await openai.chat.completions.create({
+  model: 'gpt-4.1-mini',
+  messages: toOpenAIMessages(conversation),
+  tools,
+});
 
-```typescript
-hasSystemMessage(conversation: Conversation): boolean
-getFirstSystemMessage(conversation: Conversation): Message | undefined
-getSystemMessages(conversation: Conversation): Message[]
-collapseSystemMessages(conversation: Conversation): Conversation
+const toolCalls = response.choices[0]?.message?.tool_calls ?? [];
+for (const call of toolCalls) {
+  conversation = appendMessages(conversation, {
+    role: 'tool-use',
+    content: '',
+    toolCall: {
+      id: call.id,
+      name: call.function.name,
+      arguments: call.function.arguments,
+    },
+  });
+
+  const result = await getWeather(JSON.parse(call.function.arguments));
+  conversation = appendMessages(conversation, {
+    role: 'tool-result',
+    content: '',
+    toolResult: { callId: call.id, outcome: 'success', content: result },
+  });
+}
+
+const followUp = await openai.chat.completions.create({
+  model: 'gpt-4.1-mini',
+  messages: toOpenAIMessages(conversation),
+  tools,
+});
+
+conversation = appendAssistantMessage(
+  conversation,
+  followUp.choices[0]?.message?.content ?? '',
+);
 ```
 
-### Utilities
+#### Anthropic
 
-```typescript
-computeConversationStatistics(conversation: Conversation): {
-  total: number;
-  byRole: Record<string, number>;
-  hidden: number;
-  withImages: number;
+```ts
+import { appendAssistantMessage, appendMessages } from 'conversationalist';
+import { toAnthropicMessages } from 'conversationalist/anthropic';
+
+const tools = [
+  {
+    name: 'getWeather',
+    description: 'Get current weather by city.',
+    input_schema: {
+      type: 'object',
+      properties: { city: { type: 'string' } },
+      required: ['city'],
+    },
+  },
+];
+
+const { system, messages } = toAnthropicMessages(conversation);
+const response = await anthropic.messages.create({
+  model: 'claude-3-5-sonnet-20240620',
+  system,
+  messages,
+  tools,
+});
+
+for (const block of response.content) {
+  if (block.type !== 'tool_use') continue;
+  conversation = appendMessages(conversation, {
+    role: 'tool-use',
+    content: '',
+    toolCall: { id: block.id, name: block.name, arguments: block.input },
+  });
+
+  const result = await getWeather(block.input as { city: string });
+  conversation = appendMessages(conversation, {
+    role: 'tool-result',
+    content: '',
+    toolResult: { callId: block.id, outcome: 'success', content: result },
+  });
 }
 
-redactMessageAtPosition(conversation: Conversation, position: number, placeholder?: string): Conversation
+const followUp = await anthropic.messages.create({
+  model: 'claude-3-5-sonnet-20240620',
+  ...toAnthropicMessages(conversation),
+  tools,
+});
+
+const assistantText = followUp.content
+  .filter((block) => block.type === 'text')
+  .map((block) => block.text)
+  .join('\n');
+
+conversation = appendAssistantMessage(conversation, assistantText);
 ```
 
-### Serialization
+#### Gemini
+
+Gemini does not include tool call IDs, so generate one to pair the tool result.
+
+```ts
+import { appendMessages } from 'conversationalist';
+import { toGeminiMessages } from 'conversationalist/gemini';
+
+const tools = [
+  {
+    functionDeclarations: [
+      {
+        name: 'getWeather',
+        description: 'Get current weather by city.',
+        parameters: {
+          type: 'object',
+          properties: { city: { type: 'string' } },
+          required: ['city'],
+        },
+      },
+    ],
+  },
+];
+
+const { systemInstruction, contents } = toGeminiMessages(conversation);
+const response = await model.generateContent({
+  systemInstruction,
+  contents,
+  tools,
+});
+
+const parts = response.response.candidates?.[0]?.content?.parts ?? [];
+for (const part of parts) {
+  if (!('functionCall' in part)) continue;
+  const callId = crypto.randomUUID();
+  const args = part.functionCall.args as { city: string };
+
+  conversation = appendMessages(conversation, {
+    role: 'tool-use',
+    content: '',
+    toolCall: { id: callId, name: part.functionCall.name, arguments: args },
+  });
+
+  const result = await getWeather(args);
+  conversation = appendMessages(conversation, {
+    role: 'tool-result',
+    content: '',
+    toolResult: { callId, outcome: 'success', content: result },
+  });
+}
 
-```typescript
-serializeConversation(conversation: Conversation): ConversationJSON
-deserializeConversation(json: ConversationJSON): Conversation
-toChatMessages(conversation: Conversation): ExternalMessage[]
+const followUp = await model.generateContent({
+  ...toGeminiMessages(conversation),
+  tools,
+});
 ```
 
-### Builder Pattern
+## Builder Pattern
 
-For fluent API style:
+Use the draft pattern for fluent, mutable-style updates that still return immutable
+conversations.
 
-```typescript
-import { createConversation } from 'conversationalist';
-import { withConversation, pipeConversation } from 'conversationalist';
-import { simpleTokenEstimator } from 'conversationalist';
+```ts
+import {
+  appendSystemMessage,
+  appendUserMessage,
+  createConversation,
+  pipeConversation,
+  withConversation,
+} from 'conversationalist';
 
-// Draft pattern with callback
 const conversation = withConversation(createConversation(), (draft) => {
   draft
     .appendSystemMessage('You are a helpful assistant.')
@@ -152,208 +406,167 @@ const conversation = withConversation(createConversation(), (draft) => {
     .appendAssistantMessage('Hi there!');
 });
 
-// Streaming with the draft pattern
-const streamedConversation = withConversation(createConversation(), (draft) => {
-  const { draft: d, messageId } = draft.appendStreamingMessage('assistant');
-  d.updateStreamingMessage(messageId, 'Partial response...').finalizeStreamingMessage(
-    messageId,
-    { tokenUsage: { prompt: 10, completion: 5, total: 15 } },
-  );
-});
-
-// Context window management with the draft pattern
-const truncatedConversation = withConversation(conversation, (draft) => {
-  draft.truncateToTokenLimit(4000, simpleTokenEstimator, { preserveLastN: 2 });
-});
-
-// Pipe pattern
-const conversation = pipeConversation(
+const piped = pipeConversation(
   createConversation(),
   (c) => appendSystemMessage(c, 'You are helpful.'),
   (c) => appendUserMessage(c, 'Hello!'),
 );
 ```
 
-The `ConversationDraft` provides all conversation manipulation methods:
-
-- **Message appending**: `appendMessages`, `appendUserMessage`, `appendAssistantMessage`, `appendSystemMessage`
-- **System messages**: `prependSystemMessage`, `replaceSystemMessage`, `collapseSystemMessages`
-- **Modification**: `redactMessageAtPosition`
-- **Streaming**: `appendStreamingMessage`, `updateStreamingMessage`, `finalizeStreamingMessage`, `cancelStreamingMessage`
-- **Context window**: `truncateFromPosition`, `truncateToTokenLimit`
+`ConversationDraft` includes appending, system message helpers, streaming, redaction, and
+context window utilities.
 
-### Tool Call Pairing
+## API Overview
 
-```typescript
-import { pairToolCallsWithResults } from 'conversationalist';
+### Conversation Creation and Serialization
 
-const pairs = pairToolCallsWithResults(conversation.messages);
-// Returns: [{ call: ToolCall, result?: ToolResult }, ...]
+```ts
+createConversation(options?, environment?)
+serializeConversation(conversation)
+deserializeConversation(json)
+toChatMessages(conversation)
 ```
 
-## Provider Adapters
-
-Convert conversations to provider-specific formats using subpath exports:
+### Message Appending
 
-### OpenAI
+```ts
+appendMessages(conversation, ...inputs, environment?)
+appendUserMessage(conversation, content, metadata?, environment?)
+appendAssistantMessage(conversation, content, metadata?, environment?)
+appendSystemMessage(conversation, content, metadata?, environment?)
+```
 
-```typescript
-import { toOpenAIMessages } from 'conversationalist/openai';
+### System Message Utilities
 
-const messages = toOpenAIMessages(conversation);
-const response = await openai.chat.completions.create({
-  model: 'gpt-4',
-  messages,
-});
+```ts
+prependSystemMessage(conversation, content, metadata?, environment?)
+replaceSystemMessage(conversation, content, metadata?, environment?)
+collapseSystemMessages(conversation, environment?)
+hasSystemMessage(conversation)
+getFirstSystemMessage(conversation)
+getSystemMessages(conversation)
 ```
 
-### Anthropic
+### Querying and Stats
 
-```typescript
-import { toAnthropicMessages } from 'conversationalist/anthropic';
-
-const { system, messages } = toAnthropicMessages(conversation);
-const response = await anthropic.messages.create({
-  model: 'claude-3-opus-20240229',
-  system,
-  messages,
-});
+```ts
+getConversationMessages(conversation, { includeHidden? })
+getMessageAtPosition(conversation, position)
+getMessageByIdentifier(conversation, id)
+searchConversationMessages(conversation, predicate)
+computeConversationStatistics(conversation)
 ```
 
-### Google Gemini
+### Modification
 
-```typescript
-import { toGeminiMessages } from 'conversationalist/gemini';
-
-const { systemInstruction, contents } = toGeminiMessages(conversation);
-const response = await model.generateContent({
-  systemInstruction,
-  contents,
-});
+```ts
+redactMessageAtPosition(conversation, position, placeholder?)
 ```
 
-## Streaming Support
+### Streaming
 
-Handle streaming responses with pending message utilities:
+```ts
+appendStreamingMessage(conversation, role, metadata?, environment?)
+updateStreamingMessage(conversation, messageId, content, environment?)
+finalizeStreamingMessage(conversation, messageId, { tokenUsage?, metadata? }, environment?)
+cancelStreamingMessage(conversation, messageId, environment?)
+isStreamingMessage(message)
+getStreamingMessage(conversation)
+```
 
-```typescript
-import {
-  appendStreamingMessage,
-  updateStreamingMessage,
-  finalizeStreamingMessage,
-  cancelStreamingMessage,
-  isStreamingMessage,
-  getStreamingMessage,
-} from 'conversationalist';
+### Context Window
 
-// Start a streaming message
-let { conversation, messageId } = appendStreamingMessage(conversation, 'assistant');
+```ts
+getRecentMessages(conversation, count, { includeHidden?, includeSystem? })
+truncateFromPosition(conversation, position, { preserveSystemMessages? }, environment?)
+truncateToTokenLimit(
+  conversation,
+  maxTokens,
+  estimateTokens,
+  { preserveSystemMessages?, preserveLastN? },
+  environment?,
+)
+estimateConversationTokens(conversation, estimateTokens)
+simpleTokenEstimator(message)
+```
 
-// Update content as chunks arrive
-for await (const chunk of stream) {
-  accumulatedContent += chunk;
-  conversation = updateStreamingMessage(conversation, messageId, accumulatedContent);
-}
+### Utilities
 
-// Finalize when complete
-conversation = finalizeStreamingMessage(conversation, messageId, {
-  tokenUsage: { prompt: 100, completion: 50, total: 150 },
-});
+```ts
+pairToolCallsWithResults(messages);
+normalizeContent(content);
+toMultiModalArray(content);
+createMessage(messageJSON);
+copyContent(content);
+copyMultiModalContent(item);
+```
 
-// Or cancel if needed
-conversation = cancelStreamingMessage(conversation, messageId);
+### Schemas
+
+```ts
+conversationSchema;
+conversationShape;
+messageInputSchema;
+messageJSONSchema;
+messageRoleSchema;
+multiModalContentSchema;
+tokenUsageSchema;
+toolCallSchema;
+toolResultSchema;
 ```
 
-## Context Window Utilities
+### Errors
+
+```ts
+ConversationalistError;
+createDuplicateIdError;
+createInvalidInputError;
+createInvalidPositionError;
+createInvalidToolReferenceError;
+createLockedError;
+createNotFoundError;
+createSerializationError;
+createValidationError;
+```
 
-Manage token limits and message truncation:
+## Deterministic Environments (Testing)
 
-```typescript
-import {
-  getRecentMessages,
-  truncateFromPosition,
-  truncateToTokenLimit,
-  estimateConversationTokens,
-  simpleTokenEstimator,
-} from 'conversationalist';
+Pass a custom environment to control timestamps and IDs.
 
-// Get last N messages (excluding system by default)
-const recent = getRecentMessages(conversation, 10);
+```ts
+import { appendUserMessage, createConversation } from 'conversationalist';
 
-// Truncate to messages from position onwards
-const truncated = truncateFromPosition(conversation, 5);
+const env = {
+  now: () => '2024-01-01T00:00:00.000Z',
+  randomId: () => 'fixed-id',
+};
 
-// Truncate to fit token limit
-const fitted = truncateToTokenLimit(conversation, 4000, simpleTokenEstimator, {
-  preserveSystemMessages: true,
-  preserveLastN: 2,
-});
-
-// Estimate total tokens
-const tokens = estimateConversationTokens(conversation, simpleTokenEstimator);
+let conversation = createConversation({ title: 'Test' }, env);
+conversation = appendUserMessage(conversation, 'Hello', undefined, env);
 ```
 
 ## Types
 
-```typescript
+```ts
 import type {
   Conversation,
+  ConversationEnvironment,
   ConversationJSON,
   ConversationStatus,
+  ExternalMessage,
+  ImageContent,
   Message,
   MessageInput,
   MessageJSON,
   MessageRole,
+  MultiModalContent,
+  TextContent,
   TokenUsage,
   ToolCall,
   ToolResult,
-  MultiModalContent,
-  TextContent,
-  ImageContent,
 } from 'conversationalist';
 ```
 
-## Validation Schemas
-
-Zod schemas are exported for runtime validation:
-
-```typescript
-import {
-  conversationSchema,
-  messageInputSchema,
-  messageJSONSchema,
-  messageRoleSchema,
-  multiModalContentSchema,
-  tokenUsageSchema,
-  toolCallSchema,
-  toolResultSchema,
-} from 'conversationalist';
-
-const result = messageInputSchema.safeParse(data);
-```
-
-## Error Handling
-
-Custom error types with codes:
-
-```typescript
-import {
-  ConversationalistError,
-  createInvalidInputError,
-  createInvalidPositionError,
-  createNotFoundError,
-  createValidationError,
-} from 'conversationalist';
-
-try {
-  // ...
-} catch (error) {
-  if (error instanceof ConversationalistError) {
-    console.log(error.code); // e.g., 'INVALID_POSITION'
-  }
-}
-```
-
 ## Development
 
 ```bash