automify 0.1.4 → 0.1.7

package/README.md

@@ -10,11 +10,11 @@
 
 Computer use surfaces:
 
-| Surface        | Factory                     | What it does                                               |
-| -------------- | --------------------------- | ---------------------------------------------------------- |
-| Browser        | `automify.browser()`        | Playwright browser automation with screenshots and actions |
-| Desktop        | `automify.localComputer()`  | Native desktop computer use on the current machine         |
-| Docker desktop | `automify.dockerComputer()` | Containerized Linux desktop automation with screenshots    |
+| Surface        | Factory                     | Controlled environment                                      |
+| -------------- | --------------------------- | ----------------------------------------------------------- |
+| Browser        | `automify.browser()`        | Playwright browser with screenshots and actions             |
+| Desktop        | `automify.localComputer()`  | Native desktop on the current macOS, Windows, or Linux host |
+| Docker desktop | `automify.dockerComputer()` | Linux desktop inside a Docker container                     |
 
 Command use surfaces:
 
@@ -198,12 +198,30 @@ try {
 
 ### Desktop Computer Use
 
-Local desktop computer use is optional. Install it with the command below; it may take a while because it compiles native desktop dependencies. When you use the local desktop adapter, your OS may ask for permission to control the desktop.
+Local desktop computer use controls the native desktop on the machine running your Node.js process. It supports macOS, Windows, and Linux through the local desktop adapter. It needs native desktop dependencies that are not installed by default, and your OS may ask for permission to control the desktop.
 
-```bash
-npx automify-install-desktop
+Before running `npx automify-install-desktop`, install the native build tools for your OS:
+
+```sh
+# Windows: Visual Studio C++ Build Tools plus CMake on PATH.
+winget install Kitware.CMake
+
+# macOS: Xcode Command Line Tools plus CMake on PATH.
+xcode-select --install
+brew install cmake
+
+# Debian/Ubuntu Linux.
+sudo apt-get install -y build-essential cmake libxtst-dev libpng++-dev
+
+# Fedora Linux.
+sudo dnf install -y gcc-c++ make cmake libXtst-devel libpng-devel
+
+# Arch Linux.
+sudo pacman -S --needed base-devel cmake libxtst libpng
 ```
 
+On headless Linux hosts, also install `xvfb` unless you manage `DISPLAY` yourself. On macOS and Windows, `cmake --version` must work in the terminal where you run `npx automify-install-desktop`. On Windows, the VS Code CMake Tools extension is not enough by itself.
+
 ```js
 import { initAutomify } from "automify";
 
@@ -215,6 +233,7 @@ const automify = initAutomify({
   }
 });
 
+// Reminder: local desktop support requires `npx automify-install-desktop` once for this project.
 const desktop = await automify.localComputer();
 
 try {
@@ -226,7 +245,7 @@ try {
 }
 ```
 
-For isolated Linux desktop computer use, use Docker:
+For isolated Linux desktop computer use, use Docker. `dockerComputer()` can run from a macOS, Windows, or Linux host with Docker, but the desktop it controls inside the container is Linux. Docker desktop does not use `automify-install-desktop`; it needs Docker and an initial app command:
 
 ```js
 import { initAutomify } from "automify";
@@ -299,6 +318,57 @@ const run = await browser.do("Create the lead from data and return the saved rec
 - `shared` and `sharedFiles` expose files inside Docker CLI or Docker desktop runs.
 - `jsonOutput()` requests structured JSON and makes parsed output available as `run.parsed`.
 
+For arrays of objects, the most ergonomic shape is usually an object with a named array property:
+
+```js
+const run = await browser.do("Extract the products.", {
+  output: jsonOutput("product_list", {
+    products: {
+      type: "array",
+      items: {
+        type: "object",
+        properties: {
+          sku: { type: "string" },
+          title: { type: "string" },
+          price: { type: "number" }
+        },
+        required: ["sku", "title", "price"],
+        additionalProperties: false
+      }
+    }
+  })
+});
+
+console.log(run.parsed.products);
+```
+
+If you need `run.parsed` itself to be an array, pass the lower-level `json_schema` output format directly:
+
+```js
+const run = await browser.do("Extract the products.", {
+  output: {
+    type: "json_schema",
+    name: "products",
+    strict: true,
+    schema: {
+      type: "array",
+      items: {
+        type: "object",
+        properties: {
+          sku: { type: "string" },
+          title: { type: "string" },
+          price: { type: "number" }
+        },
+        required: ["sku", "title", "price"],
+        additionalProperties: false
+      }
+    }
+  }
+});
+
+console.log(run.parsed[0].sku);
+```
+
 ### Optional Zod Output
 
 If your app already uses Zod 4, you can use the optional Zod adapter instead of writing compact shapes or JSON Schema by hand. Install `zod` in your app and import from the dedicated `automify/zod` subpath:
@@ -320,6 +390,26 @@ const run = await browser.do("Create the lead and return it.", {
 console.log(run.parsed.id);
 ```
 
+Zod works well for array outputs too:
+
+```js
+const ProductList = z.object({
+  products: z.array(
+    z.object({
+      sku: z.string(),
+      title: z.string(),
+      price: z.number()
+    })
+  )
+});
+
+const run = await browser.do("Extract the products.", {
+  output: zodOutput("product_list", ProductList)
+});
+
+console.log(run.parsed.products);
+```
+
 `zodOutput()` is not part of the main `automify` import on purpose. Zod is an optional peer dependency, so projects that only use `jsonOutput()` do not need to install it.
 
 At runtime, `zodOutput()` does two things:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "automify",
-  "version": "0.1.4",
+  "version": "0.1.7",
   "description": "AI computer use for browser, CLI, and desktop in Node.js.",
   "homepage": "https://aldovincenti.github.io/automify",
   "bugs": {

package/scripts/install-desktop.js

@@ -44,7 +44,7 @@ mkdirSync(buildRoot, { recursive: true });
 cloneOrPull("https://github.com/nut-tree/libnut-core.git", libnutSource, refs.libnutCore);
 cloneOrPull("https://github.com/nut-tree/nut.js.git", nutSource, refs.nut);
 if (process.platform === "darwin") {
-cloneOrPull("https://github.com/nut-tree/node-mac-permissions.git", macPermissionsSource, refs.macPermissions);
+  cloneOrPull("https://github.com/nut-tree/node-mac-permissions.git", macPermissionsSource, refs.macPermissions);
 }
 
 patchNutWorkspace();
@@ -66,11 +66,13 @@ patchPlatformLibnutDependency();
 
 runPnpm(["install"], { cwd: nutSource });
 runPnpm(["--filter", "@nut-tree/shared", "run", "compile"], { cwd: nutSource });
+patchNutJimpCompatibility();
 runPnpm(["--filter", "@nut-tree/provider-interfaces", "run", "compile"], { cwd: nutSource });
 runPnpm(["--filter", "@nut-tree/default-clipboard-provider", "run", "compile"], { cwd: nutSource });
 runPnpm(["--filter", "@nut-tree/libnut", "run", "compile"], { cwd: nutSource });
 writeLibnutImportBridge();
 runPnpm(["--filter", "@nut-tree/nut-js", "run", "compile"], { cwd: nutSource });
+patchNutJimpCompatibility();
 
 run("npm", ["install", "--no-save", ...runtimeDependencies], { cwd: root });
 
@@ -139,7 +141,7 @@ function checkBuildPrerequisites() {
 }
 
 function commandExists(command) {
-  const result = spawnSync(command, ["--version"], {
+  const result = spawnSync(resolveCommand(command), ["--version"], {
     cwd: root,
     stdio: "ignore"
   });
@@ -250,6 +252,77 @@ export { libnut };
   );
 }
 
+function patchNutJimpCompatibility() {
+  const sharedImageToJimpPath = join(
+    nutSource,
+    "core",
+    "shared",
+    "dist",
+    "lib",
+    "functions",
+    "imageToJimp.function.js"
+  );
+  if (existsSync(sharedImageToJimpPath)) {
+    writeText(
+      sharedImageToJimpPath,
+      `"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.imageToJimp = void 0;
+const jimp_1 = require("jimp");
+const colormode_enum_1 = require("../enums/colormode.enum");
+function imageToJimp(image) {
+    const jimpImage = new jimp_1.Jimp({
+        data: image.data,
+        width: image.width,
+        height: image.height
+    });
+    if (image.colorMode === colormode_enum_1.ColorMode.BGR) {
+        jimpImage.scan(0, 0, jimpImage.bitmap.width, jimpImage.bitmap.height, function (_, __, idx) {
+            const red = this.bitmap.data[idx];
+            this.bitmap.data[idx] = this.bitmap.data[idx + 2];
+            this.bitmap.data[idx + 2] = red;
+        });
+    }
+    return jimpImage;
+}
+exports.imageToJimp = imageToJimp;
+`
+    );
+  }
+
+  const jimpImageWriterPath = join(
+    nutSource,
+    "core",
+    "nut.js",
+    "dist",
+    "lib",
+    "provider",
+    "io",
+    "jimp-image-writer.class.js"
+  );
+  if (existsSync(jimpImageWriterPath)) {
+    writeText(
+      jimpImageWriterPath,
+      `"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+const shared_1 = require("@nut-tree/shared");
+class default_1 {
+    store(parameters) {
+        return new Promise((resolve, reject) => {
+            const jimpImage = (0, shared_1.imageToJimp)(parameters.image);
+            jimpImage
+                .write(parameters.path)
+                .then((_) => resolve())
+                .catch((err) => reject(err));
+        });
+    }
+}
+exports.default = default_1;
+`
+    );
+  }
+}
+
 function installWorkspacePackage(source, target) {
   rmSync(target, { recursive: true, force: true });
   mkdirSync(dirname(target), { recursive: true });
@@ -260,7 +333,7 @@ function installWorkspacePackage(source, target) {
 }
 
 function run(command, args, options = {}) {
-  const executable = process.platform === "win32" && ["npm", "npx"].includes(command) ? `${command}.cmd` : command;
+  const executable = resolveCommand(command);
   const result = spawnSync(executable, args, {
     cwd: options.cwd ?? root,
     env: options.env ?? process.env,
@@ -294,6 +367,10 @@ function run(command, args, options = {}) {
   return commandResult;
 }
 
+function resolveCommand(command) {
+  return process.platform === "win32" && ["npm", "npx"].includes(command) ? `${command}.cmd` : command;
+}
+
 function runPnpm(args, options = {}) {
   run("npx", ["--yes", "pnpm@8.15.2", ...args], options);
 }

package/src/lib/local-desktop-computer.js

@@ -83,6 +83,16 @@ const LOCAL_CALIBRATION_KEYS = new Set([
   "required"
 ]);
 const LOCAL_VIRTUAL_DISPLAY_KEYS = new Set(["display", "width", "height", "depth", "command", "args", "startupMs"]);
+const LOCAL_DESKTOP_ENVIRONMENTS = new Set(["mac", "windows", "ubuntu", "linux"]);
+const LOCAL_DESKTOP_ENVIRONMENT_ALIASES = new Map([
+  ["macos", "mac"],
+  ["darwin", "mac"],
+  ["win32", "windows"],
+  ["win", "windows"],
+  ["debian", "linux"],
+  ["fedora", "linux"],
+  ["arch", "linux"]
+]);
 
 const KEY_ALIASES = new Map([
   ["alt", "LeftAlt"],
@@ -327,6 +337,7 @@ function normalizeLocalDesktopOptions(options = {}) {
   const keyboard = options.keyboard ?? {};
   const calibration = options.calibration ?? {};
   const virtualDisplay = typeof options.virtualDisplay === "object" ? options.virtualDisplay : {};
+  const environment = normalizeLocalDesktopEnvironment(options.environment);
   return {
     ...options,
     debug: options.debug ?? false,
@@ -334,6 +345,7 @@ function normalizeLocalDesktopOptions(options = {}) {
     display: typeof options.display === "object" ? undefined : options.display,
     displayWidth: options.displayWidth ?? viewport.width ?? display.width,
     displayHeight: options.displayHeight ?? viewport.height ?? display.height,
+    environment,
     pixelScale: options.pixelScale ?? display.pixelScale ?? calibration.pixelScale,
     mouseScaleX: options.mouseScaleX ?? mouse.scaleX ?? calibration.mouseScaleX,
     mouseScaleY: options.mouseScaleY ?? mouse.scaleY ?? calibration.mouseScaleY,
@@ -366,7 +378,15 @@ async function captureLocalDesktopScreenshotToFile(nut, options = {}) {
   const filename = `automify-nut-${Date.now()}-${Math.random().toString(16).slice(2)}`;
   const directory = options.screenshotPath ? undefined : tmpdir();
   const requestedPath = options.screenshotPath;
-  const capturedPath = await nut.screen.capture(requestedPath ?? filename, nut.FileType?.PNG, directory);
+  let capturedPath;
+  try {
+    capturedPath = await nut.screen.capture(requestedPath ?? filename, nut.FileType?.PNG, directory);
+  } catch (error) {
+    throw new AutomifyError(
+      `local desktop screenshot capture failed. ${localDesktopRuntimeHelp(options.environment)}${errorMessageSuffix(error)}`,
+      { cause: error }
+    );
+  }
 
   if (isByteLike(capturedPath)) return capturedPath;
   if (isImageObject(capturedPath)) return capturedPath;
@@ -432,12 +452,57 @@ async function importNut() {
     return await import("@nut-tree/nut-js");
   } catch (error) {
     throw new AutomifyError(
-      "createLocalDesktopComputer requires the local desktop adapter dependency built from source. Install it with: npx automify-install-desktop",
+      `createLocalDesktopComputer requires the local desktop adapter dependency built from source. ${localDesktopInstallHelp()} After the OS prerequisites are available, run: npx automify-install-desktop`,
       { cause: error }
     );
   }
 }
 
+function normalizeLocalDesktopEnvironment(environment) {
+  if (environment == null) return undefined;
+  if (typeof environment !== "string" || environment.trim() === "") {
+    throw new AutomifyError(
+      'local desktop environment must be "mac", "windows", or "linux". "ubuntu" is also accepted for compatibility.'
+    );
+  }
+
+  const normalized = environment.trim().toLowerCase();
+  const canonical = LOCAL_DESKTOP_ENVIRONMENT_ALIASES.get(normalized) ?? normalized;
+  if (LOCAL_DESKTOP_ENVIRONMENTS.has(canonical)) return canonical;
+
+  throw new AutomifyError(
+    `Unsupported local desktop environment ${JSON.stringify(environment)}. Use "mac" for macOS, "windows" for Windows, or "linux" for Linux. "ubuntu" is also accepted for compatibility. Use instructions for OS-specific guidance, not a custom environment value.`
+  );
+}
+
+function localDesktopInstallHelp(platform = process.platform) {
+  if (platform === "darwin") {
+    return "On macOS, install Xcode Command Line Tools with `xcode-select --install` and make sure `cmake --version` works, for example after `brew install cmake`.";
+  }
+  if (platform === "win32") {
+    return "On Windows, install Visual Studio C++ Build Tools and make sure `cmake --version` works, for example after `winget install Kitware.CMake`.";
+  }
+  if (platform === "linux") {
+    return "On Linux, install the native build tools for your distro: Debian/Ubuntu need `build-essential cmake libxtst-dev libpng++-dev`; Fedora needs `gcc-c++ make cmake libXtst-devel libpng-devel`; Arch needs `base-devel cmake libxtst libpng`. Headless hosts also need `xvfb` unless DISPLAY is managed externally.";
+  }
+  return "Install the native build tools and CMake for this OS.";
+}
+
+function localDesktopRuntimeHelp(environment = defaultDesktopEnvironment()) {
+  if (environment === "mac") {
+    return "On macOS, grant the terminal or Node.js process Screen Recording and Accessibility permissions, then restart the process.";
+  }
+  if (environment === "windows") {
+    return "On Windows, make sure the desktop session is unlocked and the process is allowed to control the desktop.";
+  }
+  return "On Linux, make sure DISPLAY points to a running X server; on headless hosts install xvfb or pass virtualDisplay options.";
+}
+
+function errorMessageSuffix(error) {
+  const message = error?.message;
+  return message ? ` Original error: ${message}` : "";
+}
+
 async function maybeCall(fn, thisArg) {
   if (typeof fn !== "function") return undefined;
   return fn.call(thisArg);