@sqaitech/mcp 0.30.10

package/dist/api.mdx

@@ -0,0 +1,1167 @@
+# API reference
+
+> In the documentation below, you might see function calls prefixed with `agent.`. If you utilize destructuring in Playwright (e.g., `async ({ ai, aiQuery }) => { /* ... */ }`), you can call these functions without the `agent.` prefix. This is merely a syntactical difference.
+
+## Constructors
+
+Each Agent in Midscene has its own constructor.
+
+- In Puppeteer, use [PuppeteerAgent](./integrate-with-puppeteer)
+- In Bridge Mode, use [AgentOverChromeBridge](./bridge-mode-by-chrome-extension#constructor)
+- In Android, use [AndroidAgent](./integrate-with-android)
+- For GUI Agent integrating with your own interface, refer to [Integrate with any interface](./integrate-with-any-interface)
+
+These Agents share some common constructor parameters:
+
+- `generateReport: boolean`: If true, a report file will be generated. (Default: true)
+- `reportFileName: string`: The name of the report file. (Default: generated by midscene)
+- `autoPrintReportMsg: boolean`: If true, report messages will be printed. (Default: true)
+- `cacheId: string | undefined`: If provided, this cacheId will be used to save or match the cache. (Default: undefined, means cache feature is disabled)
+- `actionContext: string`: Some background knowledge that should be sent to the AI model when calling `agent.aiAction()`, like 'close the cookie consent dialog first if it exists' (Default: undefined)
+- `onTaskStartTip: (tip: string) => void | Promise<void>`: Optional hook that fires before each execution task begins with a human-readable summary of the task (Default: undefined)
+
+In Playwright and Puppeteer, there are some common parameters:
+
+- `forceSameTabNavigation: boolean`: If true, page navigation is restricted to the current tab. (Default: true)
+- `waitForNavigationTimeout: number`: The timeout for waiting for navigation finished. (Default: 5000ms, set to 0 means not waiting for navigation finished)
+
+In Puppeteer, there is also a parameter:
+
+- `waitForNetworkIdleTimeout: number`: The timeout for waiting for network idle between each action. (Default: 2000ms, set to 0 means not waiting for network idle)
+
+## Interaction methods
+
+Below are the main APIs available for the various Agents in Midscene.
+
+:::info Auto Planning v.s. Instant Action
+
+In Midscene, you can choose to use either auto planning or instant action.
+
+- `agent.ai()` is for Auto Planning: Midscene will automatically plan the steps and execute them. It's more smart and looks like more fashionable style for AI agents. But it may be slower and heavily rely on the quality of the AI model.
+- `agent.aiTap()`, `agent.aiHover()`, `agent.aiInput()`, `agent.aiKeyboardPress()`, `agent.aiScroll()`, `agent.aiDoubleClick()`, `agent.aiRightClick()` are for Instant Action: Midscene will directly perform the specified action, while the AI model is responsible for basic tasks such as locating elements. It's faster and more reliable if you are certain about the action you want to perform.
+
+:::
+
+### `agent.aiAction()` or `.ai()`
+
+This method allows you to perform a series of UI actions described in natural language. Midscene automatically plans the steps and executes them.
+
+- Type
+
+```typescript
+function aiAction(
+  prompt: string,
+  options?: {
+    cacheable?: boolean;
+  },
+): Promise<void>;
+function ai(prompt: string): Promise<void>; // shorthand form
+```
+
+- Parameters:
+
+  - `prompt: string` - A natural language description of the UI steps.
+  - `options?: Object` - Optional, a configuration object containing:
+    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
+
+- Return Value:
+
+  - Returns a Promise that resolves to void when all steps are completed; if execution fails, an error is thrown.
+
+- Examples:
+
+```typescript
+// Basic usage
+await agent.aiAction(
+  'Type "JavaScript" into the search box, then click the search button',
+);
+
+// Using the shorthand .ai form
+await agent.ai(
+  'Click the login button at the top of the page, then enter "test@example.com" in the username field',
+);
+
+// When using UI Agent models like ui-tars, you can try a more goal-driven prompt
+await agent.aiAction('Post a Tweet "Hello World"');
+```
+
+:::tip
+
+Under the hood, Midscene uses AI model to split the instruction into a series of steps (a.k.a. "Planning"). It then executes these steps sequentially. If Midscene determines that the actions cannot be performed, an error will be thrown.
+
+For optimal results, please provide clear and detailed instructions for `agent.aiAction()`. For guides about writing prompts, you may read this doc: [Tips for Writing Prompts](./prompting-tips).
+
+Related Documentation:
+
+- [Choose a model](./choose-a-model)
+
+:::
+
+### `agent.aiTap()`
+
+Tap something.
+
+- Type
+
+```typescript
+function aiTap(locate: string | Object, options?: Object): Promise<void>;
+```
+
+- Parameters:
+
+  - `locate: string | Object` - A natural language description of the element to tap, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element. False by default.
+    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
+    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
+
+- Return Value:
+
+  - Returns a `Promise<void>`
+
+- Examples:
+
+```typescript
+await agent.aiTap('The login button at the top of the page');
+
+// Use deepThink feature to precisely locate the element
+await agent.aiTap('The login button at the top of the page', {
+  deepThink: true,
+});
+```
+
+### `agent.aiHover()`
+
+> Only available in web pages, not available in Android.
+
+Move mouse over something.
+
+- Type
+
+```typescript
+function aiHover(locate: string | Object, options?: Object): Promise<void>;
+```
+
+- Parameters:
+
+  - `locate: string | Object` - A natural language description of the element to hover over, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element. False by default.
+    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
+    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
+
+- Return Value:
+
+  - Returns a `Promise<void>`
+
+- Examples:
+
+```typescript
+await agent.aiHover('The version number of the current page');
+```
+
+### `agent.aiInput()`
+
+Input text into something.
+
+- Type
+
+```typescript
+function aiInput(
+  text: string | Object,
+  locate: string,
+  options?: Object,
+): Promise<void>;
+```
+
+- Parameters:
+
+  - `text: string` - The text content to input.
+    - When `mode` is `'replace'`: The text will replace all existing content in the input field.
+    - When `mode` is `'append'`: The text will be appended to the existing content.
+    - When `mode` is `'clear'`: The text is ignored and the input field will be cleared.
+  - `locate: string | Object` - A natural language description of the element to input text into, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element. False by default.
+    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
+    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
+    - `autoDismissKeyboard?: boolean` - If true, the keyboard will be dismissed after input text, only available in Android. (Default: true)
+    - `mode?: 'replace' | 'clear' | 'append'` - Input mode. (Default: 'replace')
+      - `'replace'`: Clear the input field first, then input the text.
+      - `'append'`: Append the text to existing content without clearing.
+      - `'clear'`: Clear the input field without entering new text.
+
+- Return Value:
+
+  - Returns a `Promise<void>`
+
+- Examples:
+
+```typescript
+await agent.aiInput('Hello World', 'The search input box');
+```
+
+### `agent.aiKeyboardPress()`
+
+Press a keyboard key.
+
+- Type
+
+```typescript
+function aiKeyboardPress(
+  key: string,
+  locate?: string | Object,
+  options?: Object,
+): Promise<void>;
+```
+
+- Parameters:
+
+  - `key: string` - The web key to press, e.g. 'Enter', 'Tab', 'Escape', etc. Key Combination is not supported.
+  - `locate?: string | Object` - Optional, a natural language description of the element to press the key on, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element. False by default.
+    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
+    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
+
+- Return Value:
+
+  - Returns a `Promise<void>`
+
+- Examples:
+
+```typescript
+await agent.aiKeyboardPress('Enter', 'The search input box');
+```
+
+### `agent.aiScroll()`
+
+Scroll a page or an element.
+
+- Type
+
+```typescript
+function aiScroll(
+  scrollParam: PlanningActionParamScroll,
+  locate?: string | Object,
+  options?: Object,
+): Promise<void>;
+```
+
+- Parameters:
+
+  - `scrollParam: PlanningActionParamScroll` - The scroll parameter
+    - `direction: 'up' | 'down' | 'left' | 'right'` - The direction to scroll. Whether it is Android or Web, the scrolling direction here all refers to which direction of the page's content will appear on the screen. For example, when the scrolling direction is `down`, the hidden content at the bottom of the page will gradually reveal itself from the bottom of the screen upwards.
+    - `scrollType: 'once' | 'untilBottom' | 'untilTop' | 'untilRight' | 'untilLeft'` - Optional, the type of scroll to perform.
+    - `distance: number` - Optional, the distance to scroll in px.
+  - `locate?: string | Object` - Optional, a natural language description of the element to scroll on, or [prompting with images](#prompting-with-images). If not provided, Midscene will perform scroll on the current mouse position.
+  - `options?: Object` - Optional, a configuration object containing:
+    - `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element. False by default.
+    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
+    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
+
+- Return Value:
+
+  - Returns a `Promise<void>`
+
+- Examples:
+
+```typescript
+await agent.aiScroll(
+  { direction: 'up', distance: 100, scrollType: 'once' },
+  'The form panel',
+);
+```
+
+### `agent.aiDoubleClick()`
+
+Double-click on an element.
+
+- Type
+
+```typescript
+function aiDoubleClick(locate: string | Object, options?: Object): Promise<void>;
+```
+
+- Parameters:
+
+  - `locate: string | Object` - A natural language description of the element to double-click on, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element. False by default.
+    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
+    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
+
+- Return Value:
+
+  - Returns a `Promise<void>`
+
+- Examples:
+
+```typescript
+await agent.aiDoubleClick('The file name at the top of the page');
+
+// Use deepThink feature to precisely locate the element
+await agent.aiDoubleClick('The file name at the top of the page', {
+  deepThink: true,
+});
+```
+
+### `agent.aiRightClick()`
+
+> Only available in web pages, not available in Android.
+
+Right-click on an element. Please note that Midscene cannot interact with the native context menu in browser after right-clicking. This interface is usually used for the element that listens to the right-click event by itself.
+
+- Type
+
+```typescript
+function aiRightClick(locate: string, options?: Object): Promise<void>;
+```
+
+- Parameters:
+
+  - `locate: string | Object` - A natural language description of the element to right-click on, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element. False by default.
+    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
+    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
+
+- Return Value:
+
+  - Returns a `Promise<void>`
+
+- Examples:
+
+```typescript
+await agent.aiRightClick('The file name at the top of the page');
+
+// Use deepThink feature to precisely locate the element
+await agent.aiRightClick('The file name at the top of the page', {
+  deepThink: true,
+});
+```
+
+:::tip About the `deepThink` feature
+
+The `deepThink` feature is a powerful feature that allows Midscene to call AI model twice to precisely locate the element. False by default. It is useful when the AI model find it hard to distinguish the element from its surroundings.
+
+:::
+
+## Data extraction
+
+### `agent.aiAsk()`
+
+Ask the AI model any question about the current page. It returns the answer in string from the AI model.
+
+- Type
+
+```typescript
+function aiAsk(prompt: string | Object, options?: Object): Promise<string>;
+```
+
+- Parameters:
+
+  - `prompt: string | Object` - A natural language description of the question, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
+    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.
+
+- Return Value:
+
+  - Return a Promise. Return the answer from the AI model.
+
+- Examples:
+
+```typescript
+const result = await agent.aiAsk('What should I do to test this page?');
+console.log(result); // Output the answer from the AI model
+```
+
+Besides `aiAsk`, you can also use `aiQuery` to extract structured data from the UI.
+
+### `agent.aiQuery()`
+
+This method allows you to extract structured data from current page. Simply define the expected format (e.g., string, number, JSON, or an array) in the `dataDemand`, and Midscene will return a result that matches the format.
+
+- Type
+
+```typescript
+function aiQuery<T>(dataDemand: string | Object, options?: Object): Promise<T>;
+```
+
+- Parameters:
+
+  - `dataDemand: T`: A description of the expected data and its return format.
+  - `options?: Object` - Optional, a configuration object containing:
+    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
+    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.
+
+- Return Value:
+
+  - Returns any valid basic type, such as string, number, JSON, array, etc.
+  - Just describe the format in `dataDemand`, and Midscene will return a matching result.
+
+- Examples:
+
+```typescript
+const dataA = await agent.aiQuery({
+  time: 'The date and time displayed in the top-left corner as a string',
+  userInfo: 'User information in the format {name: string}',
+  tableFields: 'An array of table field names, string[]',
+  tableDataRecord:
+    'Table records in the format {id: string, [fieldName]: string}[]',
+});
+
+// You can also describe the expected return format using a string:
+
+// dataB will be an array of strings
+const dataB = await agent.aiQuery('string[], list of task names');
+
+// dataC will be an array of objects
+const dataC = await agent.aiQuery(
+  '{name: string, age: string}[], table data records',
+);
+
+// Use domIncluded feature to extract invisible attributes
+const dataD = await agent.aiQuery(
+  '{name: string, age: string, avatarUrl: string}[], table data records',
+  { domIncluded: true },
+);
+```
+
+### `agent.aiBoolean()`
+
+Extract a boolean value from the UI.
+
+- Type
+
+```typescript
+function aiBoolean(prompt: string | Object, options?: Object): Promise<boolean>;
+```
+
+- Parameters:
+  - `prompt: string | Object` - A natural language description of the expected value, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
+    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.
+- Return Value:
+
+  - Returns a `Promise<boolean>` when AI returns a boolean value.
+
+- Examples:
+
+```typescript
+const boolA = await agent.aiBoolean('Whether there is a login dialog');
+
+// Use domIncluded feature to extract invisible attributes
+const boolB = await agent.aiBoolean('Whether the login button has a link', {
+  domIncluded: true,
+});
+```
+
+### `agent.aiNumber()`
+
+Extract a number value from the UI.
+
+- Type
+
+```typescript
+function aiNumber(prompt: string | Object, options?: Object): Promise<number>;
+```
+
+- Parameters:
+  - `prompt: string | Object` - A natural language description of the expected value, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
+    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.
+- Return Value:
+
+  - Returns a `Promise<number>` when AI returns a number value.
+
+- Examples:
+
+```typescript
+const numberA = await agent.aiNumber('The remaining points of the account');
+
+// Use domIncluded feature to extract invisible attributes
+const numberB = await agent.aiNumber(
+  'The value of the remaining points element',
+  { domIncluded: true },
+);
+```
+
+### `agent.aiString()`
+
+Extract a string value from the UI.
+
+- Type
+
+```typescript
+function aiString(prompt: string | Object, options?: Object): Promise<string>;
+```
+
+- Parameters:
+  - `prompt: string | Object` - A natural language description of the expected value, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
+    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.
+- Return Value:
+
+  - Returns a `Promise<string>` when AI returns a string value.
+
+- Examples:
+
+```typescript
+const stringA = await agent.aiString('The first item in the list');
+
+// Use domIncluded feature to extract invisible attributes
+const stringB = await agent.aiString('The link of the first item in the list', {
+  domIncluded: true,
+});
+```
+
+## More APIs
+
+### `agent.aiAssert()`
+
+Specify an assertion in natural language, and the AI determines whether the condition is true. If the assertion fails, the SDK throws an error that includes both the optional `errorMsg` and a detailed reason generated by the AI.
+
+- Type
+
+```typescript
+function aiAssert(
+  assertion: string | Object, 
+  errorMsg?: string, 
+  options?: Object
+): Promise<void>;
+```
+
+- Parameters:
+
+  - `assertion: string | Object` - The assertion described in natural language, or [prompting with images](#prompting-with-images).
+  - `errorMsg?: string` - An optional error message to append if the assertion fails.
+  - `options?: Object` - Optional, a configuration object containing:
+    - `domIncluded?: boolean | 'visible-only'` - Whether to send simplified DOM information to the model, usually used for extracting invisible attributes like image links. If set to `'visible-only'`, only the visible elements will be sent. False by default.
+    - `screenshotIncluded?: boolean` - Whether to send screenshot to the model. True by default.
+
+- Return Value:
+
+  - Returns a Promise that resolves to void if the assertion passes; if it fails, an error is thrown with `errorMsg` and additional AI-provided information.
+
+- Example:
+
+```typescript
+await agent.aiAssert('The price of "Sauce Labs Onesie" is 7.99');
+```
+
+:::tip
+Assertions are critical in test scripts. To reduce the risk of errors due to AI hallucination (e.g., missing an error), you can also combine `.aiQuery` with standard JavaScript assertions instead of using `.aiAssert`.
+
+For example, you might replace the above code with:
+
+```typescript
+const items = await agent.aiQuery(
+  '"{name: string, price: number}[], return product names and prices',
+);
+const onesieItem = items.find((item) => item.name === 'Sauce Labs Onesie');
+expect(onesieItem).toBeTruthy();
+expect(onesieItem.price).toBe(7.99);
+```
+
+:::
+
+### `agent.aiLocate()`
+
+Locate an element using natural language.
+
+- Type
+
+```typescript
+function aiLocate(
+  locate: string | Object,
+  options?: Object,
+): Promise<{
+  rect: {
+    left: number;
+    top: number;
+    width: number;
+    height: number;
+  };
+  center: [number, number];
+  scale: number; // device pixel ratio
+}>;
+```
+
+- Parameters:
+
+  - `locate: string | Object` - A natural language description of the element to locate, or [prompting with images](#prompting-with-images).
+  - `options?: Object` - Optional, a configuration object containing:
+    - `deepThink?: boolean` - If true, Midscene will call AI model twice to precisely locate the element. False by default.
+    - `xpath?: string` - The xpath of the element to operate. If provided, Midscene will first use this xpath to locate the element before using the cache and the AI model. Empty by default.
+    - `cacheable?: boolean` - Whether cacheable when enabling [caching feature](./caching.mdx). True by default.
+
+- Return Value:
+
+  - Returns a `Promise` when the element is located parsed as an locate info object.
+
+- Examples:
+
+```typescript
+const locateInfo = await agent.aiLocate(
+  'The login button at the top of the page',
+);
+console.log(locateInfo);
+```
+
+### `agent.aiWaitFor()`
+
+Wait until a specified condition, described in natural language, becomes true. Considering the cost of AI calls, the check interval will not exceed the specified `checkIntervalMs`.
+
+- Type
+
+```typescript
+function aiWaitFor(
+  assertion: string,
+  options?: {
+    timeoutMs?: number;
+    checkIntervalMs?: number;
+  },
+): Promise<void>;
+```
+
+- Parameters:
+
+  - `assertion: string` - The condition described in natural language.
+  - `options?: object` - An optional configuration object containing:
+    - `timeoutMs?: number` - Timeout in milliseconds (default: 15000).
+    - `checkIntervalMs?: number` - Interval for checking in milliseconds (default: 3000).
+
+- Return Value:
+
+  - Returns a Promise that resolves to void if the condition is met; if not, an error is thrown when the timeout is reached.
+
+- Examples:
+
+```typescript
+// Basic usage
+await agent.aiWaitFor(
+  'There is at least one headphone information displayed on the interface',
+);
+
+// Using custom options
+await agent.aiWaitFor('The shopping cart icon shows a quantity of 2', {
+  timeoutMs: 30000, // Wait for 30 seconds
+  checkIntervalMs: 5000, // Check every 5 seconds
+});
+```
+
+:::tip
+Given the time consumption of AI services, `.aiWaitFor` might not be the most efficient method. Sometimes, using a simple sleep function may be a better alternative.
+:::
+
+### `agent.runYaml()`
+
+Execute an automation script written in YAML. Only the `tasks` part of the script is executed, and it returns the results of all `.aiQuery` calls within the script.
+
+- Type
+
+```typescript
+function runYaml(yamlScriptContent: string): Promise<{ result: any }>;
+```
+
+- Parameters:
+
+  - `yamlScriptContent: string` - The YAML-formatted script content.
+
+- Return Value:
+
+  - Returns an object with a `result` property that includes the results of all `.aiQuery` calls.
+
+- Example:
+
+```typescript
+const { result } = await agent.runYaml(`
+tasks:
+  - name: search weather
+    flow:
+      - ai: input 'weather today' in input box, click search button
+      - sleep: 3000
+
+  - name: query weather
+    flow:
+      - aiQuery: "the result shows the weather info, {description: string}"
+`);
+console.log(result);
+```
+
+:::tip
+For more information about YAML scripts, please refer to [Automate with Scripts in YAML](./automate-with-scripts-in-yaml).
+:::
+
+### `agent.setAIActionContext()`
+
+Set the background knowledge that should be sent to the AI model when calling `agent.aiAction()` or `agent.ai()`. This will override the previous setting.
+
+For instant action type APIs, like `aiTap()`, this setting will not take effect.
+
+- Type
+
+```typescript
+function setAIActionContext(actionContext: string): void;
+```
+
+- Parameters:
+
+  - `actionContext: string` - The background knowledge that should be sent to the AI model.
+
+- Example:
+
+```typescript
+await agent.setAIActionContext(
+  'Close the cookie consent dialog first if it exists',
+);
+```
+
+### `agent.evaluateJavaScript()`
+
+> Only available in web pages, not available in Android.
+
+Evaluate a JavaScript expression in the web page context.
+
+- Type
+
+```typescript
+function evaluateJavaScript(script: string): Promise<any>;
+```
+
+- Parameters:
+
+  - `script: string` - The JavaScript expression to evaluate.
+
+- Return Value:
+
+  - Returns the result of the JavaScript expression.
+
+- Example:
+
+```typescript
+const result = await agent.evaluateJavaScript('document.title');
+console.log(result);
+```
+
+### `agent.logScreenshot()`
+
+Log the current screenshot with a description in the report file.
+
+- Type
+
+```typescript
+function logScreenshot(title?: string, options?: Object): Promise<void>;
+```
+
+- Parameters:
+
+  - `title?: string` - Optional, the title of the screenshot, if not provided, the title will be 'untitled'.
+  - `options?: Object` - Optional, a configuration object containing:
+    - `content?: string` - The description of the screenshot.
+
+- Return Value:
+
+  - Returns a `Promise<void>`
+
+- Examples:
+
+```typescript
+await agent.logScreenshot('Login page', {
+  content: 'User A',
+});
+```
+
+### `agent.freezePageContext()`
+
+Freeze the current page context, allowing all subsequent operations to reuse the same page snapshot without retrieving the page state repeatedly. This significantly improves performance when executing a large number of concurrent operations. 
+
+Some notes:
+* Usually, you do not need to use this method, unless you are certain that "context retrieval" is the bottleneck of your test script.
+* You need to call `agent.unfreezePageContext()` in time to restore the real-time page state.
+* Do not call this method in interaction operations, it will make the AI model unable to perceive the latest page state, causing confusing errors.
+
+- Type
+
+```typescript
+function freezePageContext(): Promise<void>;
+```
+
+- Return Value:
+
+  - `Promise<void>`
+
+- Examples:
+
+```typescript
+// Freeze the page context
+await agent.freezePageContext();
+
+// Some queries...
+const results = await Promise.all([
+  await agent.aiQuery('Username input box value'),
+  await agent.aiQuery('Password input box value'),
+  await agent.aiLocate('Login button'),
+]);
+console.log(results);
+
+// Unfreeze the page context, subsequent operations will use real-time page state
+await agent.unfreezePageContext();
+```
+
+:::tip
+In the report, operations using frozen context will display a 🧊 icon in the Insight tab.
+:::
+
+### `agent.unfreezePageContext()`
+
+Unfreezes the page context, restoring the use of real-time page state.
+
+- Type
+
+```typescript
+function unfreezePageContext(): Promise<void>;
+```
+
+- Return Value:
+
+  - `Promise<void>`
+
+## Properties
+
+### `.reportFile`
+
+The path to the report file.
+
+## Additional configurations
+
+### Setting environment variables at runtime
+
+You can override environment variables at runtime by calling the `overrideAIConfig` method.
+
+```typescript
+import { overrideAIConfig } from '@sqai/web/puppeteer'; // or another Agent
+
+overrideAIConfig({
+  OPENAI_BASE_URL: '...',
+  OPENAI_API_KEY: '...',
+  MIDSCENE_MODEL_NAME: '...',
+});
+```
+
+### Print usage information for each AI call
+
+Set the `DEBUG=midscene:ai:profile:stats` to view the execution time and usage for each AI call.
+
+```bash
+export DEBUG=midscene:ai:profile:stats
+```
+
+### Customize the run artifact directory
+
+Set the `MIDSCENE_RUN_DIR` variable to customize the run artifact directory.
+
+```bash
+export MIDSCENE_RUN_DIR=midscene_run # The default value is the midscene_run in the current working directory, you can set it to an absolute path or a relative path
+```
+
+### Customize the replanning cycle limit
+
+Set the `MIDSCENE_REPLANNING_CYCLE_LIMIT` variable to customize the maximum number of replanning cycles allowed during action execution (`aiAction`).
+
+```bash
+export MIDSCENE_REPLANNING_CYCLE_LIMIT=10 # The default value is 10. When the AI needs to replan more than this limit, an error will be thrown suggesting to split the task into multiple steps
+```
+
+### Using LangSmith
+
+LangSmith is a platform for debugging large language models. To integrate LangSmith, follow these steps:
+
+```bash
+# Set environment variables
+
+# Enable debug mode
+export MIDSCENE_LANGSMITH_DEBUG=1
+
+# LangSmith configuration
+export LANGSMITH_TRACING_V2=true
+export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
+export LANGSMITH_API_KEY="your_key_here"
+export LANGSMITH_PROJECT="your_project_name_here"
+```
+
+After starting Midscene, you should see logs similar to:
+
+```log
+DEBUGGING MODE: langsmith wrapper enabled
+```
+
+## Advanced features
+
+### Prompting with images
+
+You can use images as supplements in the prompt to describe things that cannot be expressed in natural language.
+
+When prompting with images, the format of the prompt parameters is as follows:
+
+```javascript
+{
+  // Prompt text, in which images can be referred
+  prompt: string,
+  // The images referred in the prompt text
+  images?: {
+    // Image name, corresponding to the names referred in the prompt text
+    name: string,
+    // Image url, can be a local image path, Base64 string, or http link
+    url: string
+  }[]
+  // When convertHttpImage2Base64 is true，the image links in the http format will be converted into Base64 encoding and sent to the LLM.
+  // Which is applicable when the image links are not publicly accessible.
+  convertHttpImage2Base64?: boolean
+}
+```
+
+- Example 1: use images to inspect the tap position.
+
+```javascript
+await agent.aiTap({
+  prompt: 'The specific logo',
+  images: [
+    {
+      name: 'The specific logo',
+      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
+    },
+  ],
+});
+```
+
+- Example 2: use images to assert the page content.
+
+```javascript
+await agent.aiAssert({
+  prompt: 'Whether there is a specific logo on the page.',
+  images: [
+    {
+      name: 'The specific logo',
+      url: 'https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png',
+    },
+  ],
+});
+```
+
+**Notes on Image Size**
+
+When prompting with images, it is necessary to pay attention to the requirements of the AI model provider regarding the size and dimensions of the images. Images that are too large (such as exceeding 10M) or too small (such as being less than 10 pixels) may cause errors when the model is invoked. The specific restrictions should be based on the documentation provided by the AI model provider you are using.
+
+## Automation Report Merging
+
+When running multiple automation workflows, each agent generates its own report file. The `ReportMergingTool` provides the ability to merge multiple automation reports into a single report for unified viewing and management of automation results.
+
+### `new ReportMergingTool()`
+
+Create a report merging tool instance.
+
+```typescript
+import { ReportMergingTool } from '@sqai/core/report';
+
+const reportMergingTool = new ReportMergingTool();
+```
+
+### `.append()`
+
+Add an automation report to the list of reports to be merged.
+
+- Type
+
+```typescript
+function append(reportInfo: ReportFileWithAttributes): void;
+```
+
+- Parameters:
+
+  - `reportInfo: ReportFileWithAttributes` - Report information object containing:
+    - `reportFilePath: string` - Path to the report file
+    - `reportAttributes: object` - Report attributes
+      - `testId: string` - Unique identifier for the automation workflow
+      - `testTitle: string` - Automation workflow title
+      - `testDescription: string` - Automation workflow description
+      - `testDuration: number` - Automation execution duration (in milliseconds)
+      - `testStatus: TestStatus` - Automation status, options: `'passed' | 'failed' | 'skipped' | 'timedOut'`
+
+- Examples:
+
+```typescript
+reportMergingTool.append({
+  reportFilePath: agent.reportFile as string,
+  reportAttributes: {
+    testId: 'automation-001',
+    testTitle: 'Login Automation',
+    testDescription: 'Automated user login workflow',
+    testDuration: 5000,
+    testStatus: 'passed',
+  },
+});
+```
+
+### `.mergeReports()`
+
+Merge all added reports into a single report file.
+
+- Type
+
+```typescript
+function mergeReports(
+  reportFileName?: 'AUTO' | string,
+  opts?: {
+    rmOriginalReports?: boolean;
+    overwrite?: boolean;
+  },
+): string | null;
+```
+
+- Parameters:
+
+  - `reportFileName?: 'AUTO' | string` - Report filename (optional)
+    - If `'AUTO'` or not specified, a filename will be automatically generated
+    - If a custom name is provided, it will be used as the report filename
+  - `opts?: object` - Optional configuration object containing:
+    - `rmOriginalReports?: boolean` - Whether to delete the original report files after merging (default: false)
+    - `overwrite?: boolean` - Whether to overwrite if the report file already exists (default: false)
+
+- Return Value:
+
+  - Returns the path to the merged report file on success, or `null` if there are not enough reports to merge (less than 2 reports)
+
+- Examples:
+
+```typescript
+// Automatically generate report filename
+const reportPath = reportMergingTool.mergeReports();
+
+// Custom report filename
+const reportPath = reportMergingTool.mergeReports('my-test-report');
+
+// Merge and delete original reports
+const reportPath = reportMergingTool.mergeReports('my-test-report', {
+  rmOriginalReports: true,
+});
+```
+
+### `.clear()`
+
+Clear all added report information.
+
+- Type
+
+```typescript
+function clear(): void;
+```
+
+- Examples:
+
+```typescript
+reportMergingTool.clear();
+```
+
+### Complete Example
+
+Here's a complete example using Vitest + AndroidAgent:
+
+```typescript
+import {
+  AndroidAgent,
+  AndroidDevice,
+  getConnectedDevices,
+} from '@sqai/android';
+import type { TestStatus } from '@sqai/core';
+import { ReportMergingTool } from '@sqai/core/report';
+import { sleep } from '@sqai/core/utils';
+import type ADB from 'appium-adb';
+import {
+  afterAll,
+  afterEach,
+  beforeAll,
+  beforeEach,
+  describe,
+  it,
+} from 'vitest';
+
+describe('Android Settings Test', () => {
+  let page: AndroidDevice;
+  let adb: ADB;
+  let agent: AndroidAgent;
+  let startTime: number;
+  let itTestStatus: TestStatus = 'passed';
+  const reportMergingTool = new ReportMergingTool();
+
+  beforeAll(async () => {
+    const devices = await getConnectedDevices();
+    page = new AndroidDevice(devices[0].udid);
+    adb = await page.getAdb();
+  });
+
+  beforeEach((ctx) => {
+    startTime = performance.now();
+    agent = new AndroidAgent(page, {
+      groupName: ctx.task.name,
+    });
+  });
+
+  afterEach((ctx) => {
+    if (ctx.task.result?.state === 'pass') {
+      itTestStatus = 'passed';
+    } else if (ctx.task.result?.state === 'skip') {
+      itTestStatus = 'skipped';
+    } else if (ctx.task.result?.errors?.[0].message.includes('timed out')) {
+      itTestStatus = 'timedOut';
+    } else {
+      itTestStatus = 'failed';
+    }
+    reportMergingTool.append({
+      reportFilePath: agent.reportFile as string,
+      reportAttributes: {
+        testId: `${ctx.task.name}`,
+        testTitle: `${ctx.task.name}`,
+        testDescription: 'description',
+        testDuration: (Date.now() - ctx.task.result?.startTime!) | 0,
+        testStatus: itTestStatus,
+      },
+    });
+  });
+
+  afterAll(() => {
+    reportMergingTool.mergeReports('my-android-setting-test-report');
+  });
+
+  it('toggle wlan', async () => {
+    await adb.shell('input keyevent KEYCODE_HOME');
+    await sleep(1000);
+    await adb.shell('am start -n com.android.settings/.Settings');
+    await sleep(1000);
+    await agent.aiAction('find and enter WLAN setting');
+    await agent.aiAction(
+      'toggle WLAN status *once*, if WLAN is off pls turn it on, otherwise turn it off.',
+    );
+  });
+
+  it('toggle bluetooth', async () => {
+    await adb.shell('input keyevent KEYCODE_HOME');
+    await sleep(1000);
+    await adb.shell('am start -n com.android.settings/.Settings');
+    await sleep(1000);
+    await agent.aiAction('find and enter bluetooth setting');
+    await agent.aiAction(
+      'toggle bluetooth status *once*, if bluetooth is off pls turn it on, otherwise turn it off.',
+    );
+  });
+});
+```
+
+:::tip
+The merged report file will be saved in the `midscene_run/report` directory by default. You can customize the report directory location by setting the `MIDSCENE_RUN_DIR` environment variable.
+:::