aimyapi 1.0.7 → 1.0.8

package/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2023 WhiteNoise
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2023 WhiteNoise
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,195 +1,195 @@
-# AI My API
-
-AIMyAPI is a typescript code sandbox for your LLM using [QuickJS](https://github.com/justjake/quickjs-emscripten). It allows you to expose a global API object and allows your agent to write functions against it without giving it access to the internet or your filesystem except through the methods you define.
-
-Requires an OpenAI compatible API
-
-AIMyAPI will handle async operations and timers in your API object.
-
-## About version 1.0 update and motivation
-
-AIMyAPI was originally written at a time (circa ChatGPT 3.5 / GPT-4) before tool calling existed, before Claude code and codex and just before OpenAI released their code interpreter. LLM models have also gotten drastically better since. Some of the problems it originally aimed to solve are better solved today with simple tool calling. However, there is still a need to safely run small programs in a sandboxed way, reliable way. Tools are good if you have a set interface but there may be times that an agent needs to do some sorting or filtering, or an algorithm or analysis, or some repeatable workflow which isn't implemented by the available tool functions. Writing code allows the agent to do things it might not otherwise be able to do with tools alone. All the AI models are thoroughly trained to write code do solve all kinds of problems. Having the ability to write code allows the tools API surface to be smaller; it could simply be a set of CRUD operations.
-
-The other advantage of writing code is privacy and security. When an agent executes code in a sandbox, it doesn't see any of the results*, API keys or other things you may want to keep hidden. When an agent uses tools, it sees all the parameters and results of the tool calls. Code can be audited for security and correctness before running it. Finally, code can encompass a series of operations, whereas tool calls need to be approved one at a time or just trusted, often without knowing what will come next.
-
-Version 1.0 updates AIMyAPI code generation to use the tool calling process for writing code. This allows the agent to check the code and run some tests or API calls before committing to a final version. This should result in better code since the previous model relied on the agent always being able to one-shot the code, which depended on it having a complete view of the problem before writing the code which wasn't always realistic. We could also update this process a bit to add "Plan" / "Build" / "Test" phases instead of just one code generation prompt.
-
-*You'll have the option to hide logs and errors from the agent - logging can allow the agent to fix it's own coding mistakes and test the code, but it also allows information to leave the sandbox which you may want to limit depending on the sensitivity the data your api is handling. Another approach would be to implement a mocked version of your API that the agent can use to write code and debug against while keeping your real data hidden and safe from potentially destructive operations.
-
-## Quickstart
-
-Install the package
-
-```bash
-npm i aimyapi --save
-```
-
-**Setup:** create a .env file in the root of your project and add your OpenAI api key:
-```bash
-OPENAI_API_KEY=XXXX
-OPENAI_API_BASE_URL=XXXX       // If using your own server, otherwise omit to use the default
-OPENAI_API_MODEL=XXXX              // defaults to "gpt-5-mini"
-```
-
-
-Simply define your strongly-typed typescript API:
-
-`sales_api.ts`
-```typescript
-// Sales data for each month, per business unit
-export interface SalesData {
-    sales: number;
-    month: number;  // 1...12
-    year: number;
-    businessUnit: "Advertising" | "Hardware" | "Software";
-}
-
-export interface APIInterface {
-    getSalesData(): SalesData[];
-    // Returns an array of sales data.
-
-    print(text: string): void;
-    // Prints the specified text to the user.
-};
-
-// Global object you will use for accessing the APIInterface. Any other globals can go here too if you declare them in the `apiGlobals` parameter.
-declare global {
-    const api: APIInterface;
-}
-
-```
-
-Implement it (`sales_api_impl.ts`):
-```typescript
-export class API implements APIInterface {
-
-    getSalesData() {
-    ...
-```
-
-Document it (`sales_api.md`):
-
-```
-Examples of how you should respond:
-
-user: How are you today?
-
-assistant:
-
-api.print("I'm great, thanks for asking!");
-```
-
-Instantiate your api and AIMyAPI. AIMyAPI will wrap your API so that it can be called from within the QuickJS sandbox.
-
-```typescript
-import { AIMyAPI } from "aimyapi"
-
-const api = new API();
-
-const aimyapi = await AIMyAPI.createWithAPI({
-    apiObject: api,
-    apiExports: APIExports,
-    apiDefFilePath: "./sales_api.ts",
-    apiDocsPath: "./sales_api.md",
-})
-```
-
-And now you can make requests against the API in natural language.
-Any code generated by the LLM will be run within the sandbox. The LLM will ONLY have access to functions provided within your API and cannot use javascript to make http requests etc.
-
-```typescript
-await aimyapi.processRequest("Which business unit had the highest sales in 2020?");
-```
-
-This would produce the following code generated by the LLM prompt, which is run in the sandbox and makes calls on your API object:
-
-```typescript
-
-let salesData = api.getSalesData();
-let maxSales = 0;
-let businessUnitWithMaxSales = "";
-for (let i = 0; i < salesData.length; i++) {
-    if (salesData[i].year === 2020 && salesData[i].sales > maxSales) {
-        maxSales = salesData[i].sales;
-        businessUnitWithMaxSales = salesData[i].businessUnit;
-    }
-}
-api.print(\`The business unit with the highest sales in 2020 was ${businessUnitWithMaxSales}.\`);
-
-```
-
-This example does not take advantage of the chat history and treats every request as a fresh chat. See `example2` for a more complex example that uses the chat history.
-
-APIs should be kept simple and understandable. If you have a complicated API you may want to write a simpler facade for it.
-
-
-## API Reference for AIMyAPI
-
-### createWithAPI(options: AIMyAPIOptions): Promise<CreateWithAPIExports>
-
-Primary function you will use to create a module for working your API API. For a simple example with no chat history, see example1. For an example with chat history see example2.
-
-#### Options:
-
-```typescript
-export interface AIMyAPIOptions {
-    apiObject: object;
-    apiExports: object;
-    apiWhitelist?: string[];
-    apiGlobals?: object;
-    apiDefFilePath: string;
-    apiGlobalName?: string;    
-    apiDocsPath?: string;
-    debug?: boolean;
-    model?: string;
-    additionalModelOptions?: object;
-    hideLogsFromAgent?: boolean; 
-};
-```
-
-- `apiObject`: An object representing the API to be used.
-- `apiExports`: An object representing the exported functions and variables from the API.
-- `apiGlobals` (optional): An object representing global variables for the API.
-- `apiDefFilePath`: A string representing the full path to the API definition file.
-- `apiGlobalName` (optional): A string representing the name of the global variable for the API.
-- `apiDocsPath` (optional): A string representing the path to the documentation for the API.
-- `debug` (optional): A boolean indicating whether to output debugging information.
-
-### The AIMyAPI module (type: CreateWithAPIExports)
-
-The module created by the `createWithAPI` function:
-
-#### Methods
-
-```typescript
-export interface AIMyAPIInstance {
-    options:AIMyAPIOptions;
-    generateCode: (queryText:string, userChatHistory:ChatCompletionMessageParam[]) => Promise<GenerateCodeResult | null>;
-    runCode: (code:string) => Promise<CodeRunResult>;
-    checkCode: (code:string) => Promise<CodeRunResult>;
-    // run a single request with no history
-    processRequest: (userQuery:string, context?: object) => Promise<GenerateCodeResult | null>;
-}
-```
-
-- `generateCode()`: A function that generates code to be run in the sandbox for the given query text and user chat history.
-- `runCode()`: A function that runs the generated code in the sandbox.
-- `checkCode()`: Typecheck / compile the code
-- `processRequest()`: A single function that generates and runs code assuming no chat history.
-
-
-## Safety notes:
-
-- If the user is giving sensitive information in a query, that information isn't private. Refer to the OpenAI policies and plan accordingly.
-- The LLM will do stupid things with your API. You should idiot proof your functions, make sure you sanity check any inputs, etc.
-- Even though we are running the code in sandbox, you should still treat anything coming back from the LLM as untrusted code.
-- You should never give the LLM access to anything mission critical through your API such as live production data or services. 
-- Give as few permissions as is necessary - read only access would be preferred. 
-- Users can trick the LLM to do things that you may not want or intend. Again, access controls are critical. You shouldn't let users access anything they wouldn't normally be able to access via your API.
-- Any 'dangerous' operations should involve some kind of human oversight and approval.
-- A way to audit and rollback anything the LLM does would be wise. The LLM can misunderstand requests due to false assumptions or incomplete information (or trickery). If you allow it to modify your data it could mess things up.
-- By using this library you acknowledge that you take full responsibility for anything that happens as a result of it's use. We cannot gaurantee what will happen based on your instructions to the LLM or how the LLM interprets them etc.
-
-
-
-
+# AI My API
+
+AIMyAPI is a typescript code sandbox for your LLM using [QuickJS](https://github.com/justjake/quickjs-emscripten). It allows you to expose a global API object and allows your agent to write functions against it without giving it access to the internet or your filesystem except through the methods you define.
+
+Requires an OpenAI compatible API
+
+AIMyAPI will handle async operations and timers in your API object.
+
+## About version 1.0 update and motivation
+
+AIMyAPI was originally written at a time (circa ChatGPT 3.5 / GPT-4) before tool calling existed, before Claude code and codex and just before OpenAI released their code interpreter. LLM models have also gotten drastically better since. Some of the problems it originally aimed to solve are better solved today with simple tool calling. However, there is still a need to safely run small programs in a sandboxed way, reliable way. Tools are good if you have a set interface but there may be times that an agent needs to do some sorting or filtering, or an algorithm or analysis, or some repeatable workflow which isn't implemented by the available tool functions. Writing code allows the agent to do things it might not otherwise be able to do with tools alone. All the AI models are thoroughly trained to write code do solve all kinds of problems. Having the ability to write code allows the tools API surface to be smaller; it could simply be a set of CRUD operations.
+
+The other advantage of writing code is privacy and security. When an agent executes code in a sandbox, it doesn't see any of the results*, API keys or other things you may want to keep hidden. When an agent uses tools, it sees all the parameters and results of the tool calls. Code can be audited for security and correctness before running it. Finally, code can encompass a series of operations, whereas tool calls need to be approved one at a time or just trusted, often without knowing what will come next.
+
+Version 1.0 updates AIMyAPI code generation to use the tool calling process for writing code. This allows the agent to check the code and run some tests or API calls before committing to a final version. This should result in better code since the previous model relied on the agent always being able to one-shot the code, which depended on it having a complete view of the problem before writing the code which wasn't always realistic. We could also update this process a bit to add "Plan" / "Build" / "Test" phases instead of just one code generation prompt.
+
+*You'll have the option to hide logs and errors from the agent - logging can allow the agent to fix it's own coding mistakes and test the code, but it also allows information to leave the sandbox which you may want to limit depending on the sensitivity the data your api is handling. Another approach would be to implement a mocked version of your API that the agent can use to write code and debug against while keeping your real data hidden and safe from potentially destructive operations.
+
+## Quickstart
+
+Install the package
+
+```bash
+npm i aimyapi --save
+```
+
+**Setup:** create a .env file in the root of your project and add your OpenAI api key:
+```bash
+OPENAI_API_KEY=XXXX
+OPENAI_API_BASE_URL=XXXX       // If using your own server, otherwise omit to use the default
+OPENAI_API_MODEL=XXXX              // defaults to "gpt-5-mini"
+```
+
+
+Simply define your strongly-typed typescript API:
+
+`sales_api.ts`
+```typescript
+// Sales data for each month, per business unit
+export interface SalesData {
+    sales: number;
+    month: number;  // 1...12
+    year: number;
+    businessUnit: "Advertising" | "Hardware" | "Software";
+}
+
+export interface APIInterface {
+    getSalesData(): SalesData[];
+    // Returns an array of sales data.
+
+    print(text: string): void;
+    // Prints the specified text to the user.
+};
+
+// Global object you will use for accessing the APIInterface. Any other globals can go here too if you declare them in the `apiGlobals` parameter.
+declare global {
+    const api: APIInterface;
+}
+
+```
+
+Implement it (`sales_api_impl.ts`):
+```typescript
+export class API implements APIInterface {
+
+    getSalesData() {
+    ...
+```
+
+Document it (`sales_api.md`):
+
+```
+Examples of how you should respond:
+
+user: How are you today?
+
+assistant:
+
+api.print("I'm great, thanks for asking!");
+```
+
+Instantiate your api and AIMyAPI. AIMyAPI will wrap your API so that it can be called from within the QuickJS sandbox.
+
+```typescript
+import { AIMyAPI } from "aimyapi"
+
+const api = new API();
+
+const aimyapi = await AIMyAPI.createWithAPI({
+    apiObject: api,
+    apiExports: APIExports,
+    apiDefFilePath: "./sales_api.ts",
+    apiDocsPath: "./sales_api.md",
+})
+```
+
+And now you can make requests against the API in natural language.
+Any code generated by the LLM will be run within the sandbox. The LLM will ONLY have access to functions provided within your API and cannot use javascript to make http requests etc.
+
+```typescript
+await aimyapi.processRequest("Which business unit had the highest sales in 2020?");
+```
+
+This would produce the following code generated by the LLM prompt, which is run in the sandbox and makes calls on your API object:
+
+```typescript
+
+let salesData = api.getSalesData();
+let maxSales = 0;
+let businessUnitWithMaxSales = "";
+for (let i = 0; i < salesData.length; i++) {
+    if (salesData[i].year === 2020 && salesData[i].sales > maxSales) {
+        maxSales = salesData[i].sales;
+        businessUnitWithMaxSales = salesData[i].businessUnit;
+    }
+}
+api.print(\`The business unit with the highest sales in 2020 was ${businessUnitWithMaxSales}.\`);
+
+```
+
+This example does not take advantage of the chat history and treats every request as a fresh chat. See `example2` for a more complex example that uses the chat history.
+
+APIs should be kept simple and understandable. If you have a complicated API you may want to write a simpler facade for it.
+
+
+## API Reference for AIMyAPI
+
+### createWithAPI(options: AIMyAPIOptions): Promise<CreateWithAPIExports>
+
+Primary function you will use to create a module for working your API API. For a simple example with no chat history, see example1. For an example with chat history see example2.
+
+#### Options:
+
+```typescript
+export interface AIMyAPIOptions {
+    apiObject: object;
+    apiExports: object;
+    apiWhitelist?: string[];
+    apiGlobals?: object;
+    apiDefFilePath: string;
+    apiGlobalName?: string;    
+    apiDocsPath?: string;
+    debug?: boolean;
+    model?: string;
+    additionalModelOptions?: object;
+    hideLogsFromAgent?: boolean; 
+};
+```
+
+- `apiObject`: An object representing the API to be used.
+- `apiExports`: An object representing the exported functions and variables from the API.
+- `apiGlobals` (optional): An object representing global variables for the API.
+- `apiDefFilePath`: A string representing the full path to the API definition file.
+- `apiGlobalName` (optional): A string representing the name of the global variable for the API.
+- `apiDocsPath` (optional): A string representing the path to the documentation for the API.
+- `debug` (optional): A boolean indicating whether to output debugging information.
+
+### The AIMyAPI module (type: CreateWithAPIExports)
+
+The module created by the `createWithAPI` function:
+
+#### Methods
+
+```typescript
+export interface AIMyAPIInstance {
+    options:AIMyAPIOptions;
+    generateCode: (queryText:string, userChatHistory:ChatCompletionMessageParam[]) => Promise<GenerateCodeResult | null>;
+    runCode: (code:string) => Promise<CodeRunResult>;
+    checkCode: (code:string) => Promise<CodeRunResult>;
+    // run a single request with no history
+    processRequest: (userQuery:string, context?: object) => Promise<GenerateCodeResult | null>;
+}
+```
+
+- `generateCode()`: A function that generates code to be run in the sandbox for the given query text and user chat history.
+- `runCode()`: A function that runs the generated code in the sandbox.
+- `checkCode()`: Typecheck / compile the code
+- `processRequest()`: A single function that generates and runs code assuming no chat history.
+
+
+## Safety notes:
+
+- If the user is giving sensitive information in a query, that information isn't private. Refer to the OpenAI policies and plan accordingly.
+- The LLM will do stupid things with your API. You should idiot proof your functions, make sure you sanity check any inputs, etc.
+- Even though we are running the code in sandbox, you should still treat anything coming back from the LLM as untrusted code.
+- You should never give the LLM access to anything mission critical through your API such as live production data or services. 
+- Give as few permissions as is necessary - read only access would be preferred. 
+- Users can trick the LLM to do things that you may not want or intend. Again, access controls are critical. You shouldn't let users access anything they wouldn't normally be able to access via your API.
+- Any 'dangerous' operations should involve some kind of human oversight and approval.
+- A way to audit and rollback anything the LLM does would be wise. The LLM can misunderstand requests due to false assumptions or incomplete information (or trickery). If you allow it to modify your data it could mess things up.
+- By using this library you acknowledge that you take full responsibility for anything that happens as a result of it's use. We cannot gaurantee what will happen based on your instructions to the LLM or how the LLM interprets them etc.
+
+
+
+

package/dist/index.cjs

@@ -438,11 +438,12 @@ const checkCodeSchema = zod.z.object({
 const submitCodeSchema = zod.z.object({
   code: zod.z.string()
 });
-const generateCode = async function(instance, queryText, userChatHistory, createTaskPrompt, debug = false, hideLogsFromAgent = false, model = "gpt-5-mini", additionalModelOptions) {
+const generateCode = async function(instance, queryText, userChatHistory, createTaskPrompt, debug = false, hideLogsFromAgent = false, model = "gpt-5-mini", additionalModelOptions, attempt = 0) {
   let finalSubmittedCode = "";
   if (debug) {
     console.log("Generating code for query:", queryText);
   }
+  const newHistoryItems = [];
   const codingTools = [
     {
       type: "function",
@@ -487,6 +488,14 @@ const generateCode = async function(instance, queryText, userChatHistory, create
         name: "submitCode",
         // @ts-ignore
         function: async ({ code }) => {
+          const checkRes = await instance.checkCode(code);
+          if (!checkRes.success) {
+            return {
+              success: false,
+              reason: "Code check failed",
+              logs: checkRes.logs
+            };
+          }
           finalSubmittedCode = code;
           return {
             success: true
@@ -509,7 +518,7 @@ const generateCode = async function(instance, queryText, userChatHistory, create
         ...userChatHistory,
         {
           role: "user",
-          content: createTaskPrompt.replace("{{USER_QUERY}}", queryText)
+          content: attempt == 0 ? createTaskPrompt.replace("{{USER_QUERY}}", queryText) : queryText
         }
       ],
       tools: codingTools,
@@ -518,6 +527,7 @@ const generateCode = async function(instance, queryText, userChatHistory, create
       if (debug) {
         console.log("Message:", JSON.stringify(message, null, 2));
       }
+      newHistoryItems.push(message);
     }).on("functionToolCall", (functionCall) => {
       if (debug) {
         console.log("Function Call:", JSON.stringify(functionCall, null, 2));
@@ -529,15 +539,37 @@ const generateCode = async function(instance, queryText, userChatHistory, create
     });
     const finalContent = await runner.finalContent();
     if (debug) {
-      console.log("Final Content:", finalContent);
+      console.log("Generate code final response:", finalContent);
+    }
+    if (!finalSubmittedCode) {
+      console.error("No code was submitted by the agent.");
+      if (attempt < 3) {
+        return generateCode(instance, `You must call submitCode with valid code before proceeding. Attempt ${attempt + 1} of 3.`, [
+          ...userChatHistory,
+          {
+            role: "user",
+            content: attempt == 0 ? createTaskPrompt.replace("{{USER_QUERY}}", queryText) : queryText
+          },
+          ...newHistoryItems
+        ], createTaskPrompt, debug, hideLogsFromAgent, model, additionalModelOptions, attempt + 1);
+      } else {
+        return {
+          success: false,
+          comments: "No code was submitted by the agent after multiple attempts. Final comment: " + finalContent
+        };
+      }
     }
     return {
+      success: true,
       code: finalSubmittedCode,
       comments: finalContent || void 0
     };
   } catch (err) {
     console.error(err);
-    return null;
+    return {
+      success: false,
+      comments: `Error generating code: ${err}`
+    };
   }
 };
 async function createWithAPI(options) {

package/dist/index.d.cts

@@ -40,12 +40,13 @@ interface AIMyAPIInstance {
     processRequest: (userQuery: string, context?: object) => Promise<GenerateCodeResult | null>;
 }
 interface GenerateCodeResult {
+    success: boolean;
     code?: string;
     comments?: string;
 }
 declare function zodParseJSON<T>(schema: ZodSchema<T>): (input: string) => T;
 declare function createBasePrompt(apiFilePath: string, apiGlobalName: string, documentationPath?: string): string;
-declare const generateCode: (instance: AIMyAPIInstance, queryText: string, userChatHistory: ChatCompletionMessageParam[], createTaskPrompt: string, debug?: boolean, hideLogsFromAgent?: boolean, model?: string, additionalModelOptions?: object) => Promise<GenerateCodeResult | null>;
+declare const generateCode: (instance: AIMyAPIInstance, queryText: string, userChatHistory: ChatCompletionMessageParam[], createTaskPrompt: string, debug?: boolean, hideLogsFromAgent?: boolean, model?: string, additionalModelOptions?: object, attempt?: number) => Promise<GenerateCodeResult | null>;
 interface AIMyAPIModuleExports {
     createWithAPI: (options: AIMyAPIOptions) => Promise<AIMyAPIInstance>;
     createBasePrompt: (apiFilePath: string, documentationPath: string) => string;

package/dist/index.d.mts

@@ -40,12 +40,13 @@ interface AIMyAPIInstance {
     processRequest: (userQuery: string, context?: object) => Promise<GenerateCodeResult | null>;
 }
 interface GenerateCodeResult {
+    success: boolean;
     code?: string;
     comments?: string;
 }
 declare function zodParseJSON<T>(schema: ZodSchema<T>): (input: string) => T;
 declare function createBasePrompt(apiFilePath: string, apiGlobalName: string, documentationPath?: string): string;
-declare const generateCode: (instance: AIMyAPIInstance, queryText: string, userChatHistory: ChatCompletionMessageParam[], createTaskPrompt: string, debug?: boolean, hideLogsFromAgent?: boolean, model?: string, additionalModelOptions?: object) => Promise<GenerateCodeResult | null>;
+declare const generateCode: (instance: AIMyAPIInstance, queryText: string, userChatHistory: ChatCompletionMessageParam[], createTaskPrompt: string, debug?: boolean, hideLogsFromAgent?: boolean, model?: string, additionalModelOptions?: object, attempt?: number) => Promise<GenerateCodeResult | null>;
 interface AIMyAPIModuleExports {
     createWithAPI: (options: AIMyAPIOptions) => Promise<AIMyAPIInstance>;
     createBasePrompt: (apiFilePath: string, documentationPath: string) => string;

package/dist/index.d.ts

@@ -40,12 +40,13 @@ interface AIMyAPIInstance {
     processRequest: (userQuery: string, context?: object) => Promise<GenerateCodeResult | null>;
 }
 interface GenerateCodeResult {
+    success: boolean;
     code?: string;
     comments?: string;
 }
 declare function zodParseJSON<T>(schema: ZodSchema<T>): (input: string) => T;
 declare function createBasePrompt(apiFilePath: string, apiGlobalName: string, documentationPath?: string): string;
-declare const generateCode: (instance: AIMyAPIInstance, queryText: string, userChatHistory: ChatCompletionMessageParam[], createTaskPrompt: string, debug?: boolean, hideLogsFromAgent?: boolean, model?: string, additionalModelOptions?: object) => Promise<GenerateCodeResult | null>;
+declare const generateCode: (instance: AIMyAPIInstance, queryText: string, userChatHistory: ChatCompletionMessageParam[], createTaskPrompt: string, debug?: boolean, hideLogsFromAgent?: boolean, model?: string, additionalModelOptions?: object, attempt?: number) => Promise<GenerateCodeResult | null>;
 interface AIMyAPIModuleExports {
     createWithAPI: (options: AIMyAPIOptions) => Promise<AIMyAPIInstance>;
     createBasePrompt: (apiFilePath: string, documentationPath: string) => string;

package/dist/index.js

@@ -438,11 +438,12 @@ const checkCodeSchema = zod.z.object({
 const submitCodeSchema = zod.z.object({
   code: zod.z.string()
 });
-const generateCode = async function(instance, queryText, userChatHistory, createTaskPrompt, debug = false, hideLogsFromAgent = false, model = "gpt-5-mini", additionalModelOptions) {
+const generateCode = async function(instance, queryText, userChatHistory, createTaskPrompt, debug = false, hideLogsFromAgent = false, model = "gpt-5-mini", additionalModelOptions, attempt = 0) {
   let finalSubmittedCode = "";
   if (debug) {
     console.log("Generating code for query:", queryText);
   }
+  const newHistoryItems = [];
   const codingTools = [
     {
       type: "function",
@@ -487,6 +488,14 @@ const generateCode = async function(instance, queryText, userChatHistory, create
         name: "submitCode",
         // @ts-ignore
         function: async ({ code }) => {
+          const checkRes = await instance.checkCode(code);
+          if (!checkRes.success) {
+            return {
+              success: false,
+              reason: "Code check failed",
+              logs: checkRes.logs
+            };
+          }
           finalSubmittedCode = code;
           return {
             success: true
@@ -509,7 +518,7 @@ const generateCode = async function(instance, queryText, userChatHistory, create
         ...userChatHistory,
         {
           role: "user",
-          content: createTaskPrompt.replace("{{USER_QUERY}}", queryText)
+          content: attempt == 0 ? createTaskPrompt.replace("{{USER_QUERY}}", queryText) : queryText
         }
       ],
       tools: codingTools,
@@ -518,6 +527,7 @@ const generateCode = async function(instance, queryText, userChatHistory, create
       if (debug) {
         console.log("Message:", JSON.stringify(message, null, 2));
       }
+      newHistoryItems.push(message);
     }).on("functionToolCall", (functionCall) => {
       if (debug) {
         console.log("Function Call:", JSON.stringify(functionCall, null, 2));
@@ -529,15 +539,37 @@ const generateCode = async function(instance, queryText, userChatHistory, create
     });
     const finalContent = await runner.finalContent();
     if (debug) {
-      console.log("Final Content:", finalContent);
+      console.log("Generate code final response:", finalContent);
+    }
+    if (!finalSubmittedCode) {
+      console.error("No code was submitted by the agent.");
+      if (attempt < 3) {
+        return generateCode(instance, `You must call submitCode with valid code before proceeding. Attempt ${attempt + 1} of 3.`, [
+          ...userChatHistory,
+          {
+            role: "user",
+            content: attempt == 0 ? createTaskPrompt.replace("{{USER_QUERY}}", queryText) : queryText
+          },
+          ...newHistoryItems
+        ], createTaskPrompt, debug, hideLogsFromAgent, model, additionalModelOptions, attempt + 1);
+      } else {
+        return {
+          success: false,
+          comments: "No code was submitted by the agent after multiple attempts. Final comment: " + finalContent
+        };
+      }
     }
     return {
+      success: true,
       code: finalSubmittedCode,
       comments: finalContent || void 0
     };
   } catch (err) {
     console.error(err);
-    return null;
+    return {
+      success: false,
+      comments: `Error generating code: ${err}`
+    };
   }
 };
 async function createWithAPI(options) {