mdi-llmkit 1.0.1 → 1.0.6

package/README.md

@@ -1,190 +1,170 @@
-# mdi-llmkit (TypeScript)
-
-Utilities for managing LLM chat conversations and structured JSON responses with OpenAI's Responses API.
-
-## Installation
-
-```bash
-npm install mdi-llmkit openai
-```
-
-## Quick Start
-
-### `gptSubmit`
-
-```ts
-import OpenAI from 'openai';
-import { gptSubmit } from 'mdi-llmkit';
-
-const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
-
-const reply = await gptSubmit(
-  [{ role: 'user', content: 'Say hello.' }],
-  client
-);
-
-console.log(reply);
-```
-
-### `GptConversation`
-
-```ts
-import OpenAI from 'openai';
-import { GptConversation } from 'mdi-llmkit';
-
-const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
-const conversation = new GptConversation([], { openaiClient: client });
-
-const reply = await conversation.submitUserMessage(
-  'Give me three project name ideas.'
-);
-console.log(reply);
-```
-
-### `JSONSchemaFormat`
-
-```ts
-import { JSONSchemaFormat, JSON_INTEGER, gptSubmit } from 'mdi-llmkit';
-
-const responseFormat = JSONSchemaFormat(
-  'answer_payload',
-  {
-    answer: 'The final answer',
-    confidence: ['Confidence score', [0, 100], []],
-    rank: JSON_INTEGER,
-  },
-  'Structured answer payload'
-);
-
-const result = await gptSubmit(
-  [{ role: 'user', content: 'Return answer as structured JSON.' }],
-  client,
-  { jsonResponse: responseFormat }
-);
-```
-
-## `jsonSurgery`
-
-`jsonSurgery` applies iterative, model-guided edits to a JSON-compatible object using
-structured JSON-path operations (`assign`, `append`, `insert`, `delete`, `rename`).
-
-```ts
-import { jsonSurgery } from 'mdi-llmkit/jsonSurgery';
-```
-
-- It deep-copies the input object and returns the modified copy.
-- It supports optional schema guidance and key-skipping for model-visible context.
-- It supports validation/progress callbacks and soft iteration/time limits.
-
-## `compareItemLists` (comparison)
-
-`compareItemLists` performs a semantic diff between a "before" list and an "after" list,
-including LLM-assisted rename/add/remove decisions.
-
-Types:
-
-- `SemanticallyComparableListItem`
-  - `string`
-  - `{ name: string; description?: string }`
-- `ItemComparisonResult`
-  - `Removed | Added | Renamed | Unchanged`
-- `OnComparingItemCallback`
-  - `(item, isFromBeforeList, isStarting, result, newName, error, totalProcessedSoFar, totalLeftToProcess) => void`
-
-Behavior notes:
-
-- Item matching is name-based and case-insensitive.
-- `description` provides extra model context but is not identity.
-- Names are expected to be unique within each list (case-insensitive).
-- Progress callback is fired at item start (`isStarting=true`) and finish (`isStarting=false`).
-
-Example:
-
-```ts
-import OpenAI from 'openai';
-import {
-  compareItemLists,
-  ItemComparisonResult,
-  type OnComparingItemCallback,
-} from 'mdi-llmkit/comparison';
-
-const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
-
-const onComparingItem: OnComparingItemCallback = (
-  item,
-  isFromBeforeList,
-  isStarting,
-  result,
-  newName,
-  error,
-  processed,
-  left
-) => {
-  if (error) {
-    console.warn('Comparison warning:', error);
-  }
-  if (!isStarting && result === ItemComparisonResult.Renamed) {
-    console.log('Renamed:', item, '->', newName);
-  }
-  console.log({ isFromBeforeList, isStarting, result, processed, left });
-};
-
-const comparison = await compareItemLists(
-  client,
-  [{ name: 'Widget A', description: 'Legacy widget' }, 'Widget B'],
-  [
-    { name: 'Widget Alpha', description: 'Migrated name for Widget A' },
-    'Widget B',
-  ],
-  'Widgets migrated from legacy catalog to new naming standards.',
-  onComparingItem
-);
-
-console.log(comparison);
-```
-
-## JSON Response Mode
-
-```ts
-import OpenAI from 'openai';
-import { gptSubmit } from 'mdi-llmkit';
-
-const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
-
-const result = await gptSubmit(
-  [{ role: 'user', content: 'Return JSON with keys a and b.' }],
-  client,
-  { jsonResponse: true }
-);
-
-console.log(result);
-```
-
-## Notes
-
-- Current TypeScript parity slices include `gptSubmit`, `GptConversation`, and `JSONSchemaFormat`.
-- You can import GPT API symbols via subpath imports, e.g. `import { GptConversation } from "mdi-llmkit/gptApi"`.
-- Comparison symbols are available via `mdi-llmkit/comparison`.
-- Integer schemas can be expressed with `JSON_INTEGER`; numeric (float-capable) schemas can use `JSON_NUMBER`.
-
-## Migration from Python
-
-- Function naming: Python `gpt_submit(...)` maps to TypeScript `gptSubmit(...)`.
-- Argument style: Python keyword args map to a TypeScript options object.
-- Conversation submit methods: Python `submit_user_message(...)` maps to `submitUserMessage(...)`.
-- JSON schema DSL: Python tuple metadata uses TypeScript array metadata.
-  - Python: `("Age", (0, 120), int)`
-  - TypeScript: `["Age", [0, 120], JSON_INTEGER]`
-- JSON schema type markers in TypeScript:
-  - `JSON_INTEGER` for integer-only values.
-  - `JSON_NUMBER` for float-capable numeric values.
-
-## CI and Release
-
-- CI workflow: `.github/workflows/typescript-ci.yml`
-  - Runs on push to `main` and on pull requests when TypeScript package files change.
-  - Executes `npm ci`, `npm test`, and `npm run build` in `packages/typescript-mdi-llmkit`.
-- Release workflow: `.github/workflows/typescript-release.yml`
-  - Runs on tags matching `typescript-v*` (for example: `typescript-v1.0.1`).
-  - Requires repository secret `NPM_TOKEN` with publish permission to npm.
-  - Executes tests/build before `npm publish --access public --provenance`.
+# mdi-llmkit (TypeScript)
+
+Utilities for managing LLM chat conversations and structured JSON responses with OpenAI's Responses API.
+
+## Installation
+
+```bash
+npm install mdi-llmkit openai
+```
+
+## Quick Start
+
+### `gptSubmit`
+
+```ts
+import OpenAI from 'openai';
+import { gptSubmit } from 'mdi-llmkit';
+
+const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
+
+const reply = await gptSubmit(
+  [{ role: 'user', content: 'Say hello.' }],
+  client
+);
+
+console.log(reply);
+```
+
+### `GptConversation`
+
+```ts
+import OpenAI from 'openai';
+import { GptConversation } from 'mdi-llmkit';
+
+const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
+const conversation = new GptConversation([], { openaiClient: client });
+
+const reply = await conversation.submitUserMessage(
+  'Give me three project name ideas.'
+);
+console.log(reply);
+```
+
+### `JSONSchemaFormat`
+
+```ts
+import { JSONSchemaFormat, JSON_INTEGER, gptSubmit } from 'mdi-llmkit';
+
+const responseFormat = JSONSchemaFormat(
+  'answer_payload',
+  {
+    answer: 'The final answer',
+    confidence: ['Confidence score', [0, 100], []],
+    rank: JSON_INTEGER,
+  },
+  'Structured answer payload'
+);
+
+const result = await gptSubmit(
+  [{ role: 'user', content: 'Return answer as structured JSON.' }],
+  client,
+  { jsonResponse: responseFormat }
+);
+```
+
+## `jsonSurgery`
+
+`jsonSurgery` applies iterative, model-guided edits to a JSON-compatible object using
+structured JSON-path operations (`assign`, `append`, `insert`, `delete`, `rename`).
+
+```ts
+import { jsonSurgery } from 'mdi-llmkit/jsonSurgery';
+```
+
+- It deep-copies the input object and returns the modified copy.
+- It supports optional schema guidance and key-skipping for model-visible context.
+- It supports validation/progress callbacks and soft iteration/time limits.
+
+## `compareItemLists` (comparison)
+
+`compareItemLists` performs a semantic diff between a "before" list and an "after" list,
+including LLM-assisted rename/add/remove decisions.
+
+Types:
+
+- `SemanticallyComparableListItem`
+  - `string`
+  - `{ name: string; description?: string }`
+- `ItemComparisonResult`
+  - `Removed | Added | Renamed | Unchanged`
+- `OnComparingItemCallback`
+  - `(item, isFromBeforeList, isStarting, result, newName, error, totalProcessedSoFar, totalLeftToProcess) => void`
+
+Behavior notes:
+
+- Item matching is name-based and case-insensitive.
+- `description` provides extra model context but is not identity.
+- Names are expected to be unique within each list (case-insensitive).
+- Progress callback is fired at item start (`isStarting=true`) and finish (`isStarting=false`).
+
+Example:
+
+```ts
+import OpenAI from 'openai';
+import {
+  compareItemLists,
+  ItemComparisonResult,
+  type OnComparingItemCallback,
+} from 'mdi-llmkit/comparison';
+
+const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
+
+const onComparingItem: OnComparingItemCallback = (
+  item,
+  isFromBeforeList,
+  isStarting,
+  result,
+  newName,
+  error,
+  processed,
+  left
+) => {
+  if (error) {
+    console.warn('Comparison warning:', error);
+  }
+  if (!isStarting && result === ItemComparisonResult.Renamed) {
+    console.log('Renamed:', item, '->', newName);
+  }
+  console.log({ isFromBeforeList, isStarting, result, processed, left });
+};
+
+const comparison = await compareItemLists(
+  client,
+  [{ name: 'Widget A', description: 'Legacy widget' }, 'Widget B'],
+  [
+    { name: 'Widget Alpha', description: 'Migrated name for Widget A' },
+    'Widget B',
+  ],
+  'Widgets migrated from legacy catalog to new naming standards.',
+  onComparingItem
+);
+
+console.log(comparison);
+```
+
+## JSON Response Mode
+
+```ts
+import OpenAI from 'openai';
+import { gptSubmit } from 'mdi-llmkit';
+
+const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
+
+const result = await gptSubmit(
+  [{ role: 'user', content: 'Return JSON with keys a and b.' }],
+  client,
+  { jsonResponse: true }
+);
+
+console.log(result);
+```
+
+
+## CI and Release
+
+- Unified CI + release workflow: `.github/workflows/typescript-release.yml`
+  - Runs CI on pull requests and on pushes to `main` when TypeScript package files change.
+  - Executes `npm ci`, `npm test`, and `npm run build` in `packages/typescript-mdi-llmkit`.
+  - On push to `main`, publishes to npm only if `package.json` version changed and that version is not already published.
+  - Uses repository secret `NPM_TOKEN` for npm authentication.

package/dist/src/comparison/compareLists.js

@@ -157,35 +157,35 @@ export const compareItemLists = async (openaiClient, listBefore, listAfter, expl
     // However, some of these items may be renames rather than pure additions/removals.
     // The only way to tell is with AI.
     const convo = new GptConversation([], { openaiClient });
-    convo.addSystemMessage(`
-You are a data analyst who has been hired to try to preserve the integrity of a list of
-data items that have recently undergone migration from one data system to another.
-
-You will be given two lists of items: a "before" list and an "after" list.
-(The exact nature of the items is not important. They could be names of products from
-receipts or purchase orders, for example.)
-
-In the migration from the old data system to the new, some items may have been removed,
-some items may have been added, and some items may have been renamed. We can't tell
-just by performing string comparisons on the two lists, because the renames may be subtle.
-
-We're going to go through the items in the "before" list, one by one. For each one,
-you will look for the best matching item in the "after" list. If you find a good match,
-you will consider that item to be a rename of the original item. If you don't find a
-good match, you will consider that item to have been removed.
+    convo.addSystemMessage(`
+You are a data analyst who has been hired to try to preserve the integrity of a list of
+data items that have recently undergone migration from one data system to another.
+
+You will be given two lists of items: a "before" list and an "after" list.
+(The exact nature of the items is not important. They could be names of products from
+receipts or purchase orders, for example.)
+
+In the migration from the old data system to the new, some items may have been removed,
+some items may have been added, and some items may have been renamed. We can't tell
+just by performing string comparisons on the two lists, because the renames may be subtle.
+
+We're going to go through the items in the "before" list, one by one. For each one,
+you will look for the best matching item in the "after" list. If you find a good match,
+you will consider that item to be a rename of the original item. If you don't find a
+good match, you will consider that item to have been removed.
 `);
     if (explanation) {
-        convo.addSystemMessage(`
-Here is some additional context that may help you make better decisions about which items
-have been renamed versus removed/added:
-
-${explanation}
+        convo.addSystemMessage(`
+Here is some additional context that may help you make better decisions about which items
+have been renamed versus removed/added:
+
+${explanation}
 `);
     }
-    convo.addUserMessage(`
-"BEFORE" LIST:
-
-${listBefore.map(itemToPromptString).join('\n')}
+    convo.addUserMessage(`
+"BEFORE" LIST:
+
+${listBefore.map(itemToPromptString).join('\n')}
 `);
     // Counts used for onComparingItem telemetry across both loops.
     let totalProcessedItems = 0;
@@ -198,30 +198,30 @@ ${listBefore.map(itemToPromptString).join('\n')}
             const convoIter = convo.clone();
             // We rebuild the "after" list each time, since items may get removed from it
             // as they get matched.
-            convoIter.addUserMessage(`
-"AFTER" LIST:
-
-${listAfter.map(itemToPromptString).join('\n')}
+            convoIter.addUserMessage(`
+"AFTER" LIST:
+
+${listAfter.map(itemToPromptString).join('\n')}
 `);
-            convoIter.addUserMessage(`
-For the moment, let's focus on this item from the "before" list:
-
-${itemToPromptString(itemBefore)}
-
-Look through the entire "after" list and try to find an item that might be a rename 
-or alternative version of this item.
-
-Feel free to think aloud, brainstorm, and reason through the possibilities. Later on,
-I'll ask you to formalize your decision in JSON format; but for now, just explore the options.
-
-If you find an item that seems like a good match, tell us what it is.
-!IMPORTANT: You may only pick *one* item from the "after" list as a potential rename of this item.
-
-If you don't find any good match, simply say that no good match was found. In this situation,
-we'll consider this item as having been removed/deleted.
-
-Naturally, if you have any higher-level instructions or context that apply to this item,
-please take them into account as you reason through the possibilities.
+            convoIter.addUserMessage(`
+For the moment, let's focus on this item from the "before" list:
+
+${itemToPromptString(itemBefore)}
+
+Look through the entire "after" list and try to find an item that might be a rename 
+or alternative version of this item.
+
+Feel free to think aloud, brainstorm, and reason through the possibilities. Later on,
+I'll ask you to formalize your decision in JSON format; but for now, just explore the options.
+
+If you find an item that seems like a good match, tell us what it is.
+!IMPORTANT: You may only pick *one* item from the "after" list as a potential rename of this item.
+
+If you don't find any good match, simply say that no good match was found. In this situation,
+we'll consider this item as having been removed/deleted.
+
+Naturally, if you have any higher-level instructions or context that apply to this item,
+please take them into account as you reason through the possibilities.
 `);
             await convoIter.submit();
             await convoIter.submit(undefined, undefined, {
@@ -322,22 +322,22 @@ please take them into account as you reason through the possibilities.
         onComparingItem?.(itemAfter, false, true, ItemComparisonResult.Unchanged, undefined, undefined, totalProcessedItems, listAfter.length - iItem);
         try {
             const convoIter = convo.clone();
-            convoIter.addUserMessage(`
-At the moment, let's focus on this item from the "after" list:
-
-${itemToPromptString(itemAfter)}
-
-We think that this item was newly added, because we can't find any matching item
-from the "before" list. However, it's possible that we have instructions or context
-that indicate otherwise.
-
-At this point, we don't have the option of matching this item to any item from the "before"
-list, since we've already processed all those items. However, we still have the option
-of rejecting this item from addition -- in which case, it will be considered as not having
-been added at all (or, in other words, it will be ignored in downstream processing).
-
-What do you think? Should we consider this item as truly added, or should we reject / ignore
-this item?
+            convoIter.addUserMessage(`
+At the moment, let's focus on this item from the "after" list:
+
+${itemToPromptString(itemAfter)}
+
+We think that this item was newly added, because we can't find any matching item
+from the "before" list. However, it's possible that we have instructions or context
+that indicate otherwise.
+
+At this point, we don't have the option of matching this item to any item from the "before"
+list, since we've already processed all those items. However, we still have the option
+of rejecting this item from addition -- in which case, it will be considered as not having
+been added at all (or, in other words, it will be ignored in downstream processing).
+
+What do you think? Should we consider this item as truly added, or should we reject / ignore
+this item?
 `);
             await convoIter.submit();
             await convoIter.submit(undefined, undefined, {