@simular-ai/simulib-js 5.4.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +78 -0
- package/LICENSE +21 -0
- package/README.md +75 -0
- package/bin/init-claude.mjs +105 -0
- package/bin/postinstall-hint.mjs +32 -0
- package/examples/README.md +53 -0
- package/examples/accessibility.mjs +45 -0
- package/examples/chrome_google_search_button.mjs +88 -0
- package/examples/clipboard.mjs +29 -0
- package/examples/file-trait.mjs +8 -0
- package/examples/google_search.mjs +59 -0
- package/examples/keyboard.mjs +23 -0
- package/examples/log-window.mjs +57 -0
- package/examples/logger.mjs +33 -0
- package/examples/loopback.mjs +41 -0
- package/examples/open-app.mjs +68 -0
- package/examples/screenshot.mjs +15 -0
- package/examples/simular.mjs +15 -0
- package/examples/system.mjs +10 -0
- package/index.d.ts +1691 -0
- package/index.js +618 -0
- package/package.json +124 -0
package/CLAUDE.md
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# `@simular-ai/simulib-js`
|
|
2
|
+
|
|
3
|
+
Node.js bindings for the Rust `simulib-rs` crate (via napi-rs). Cross-platform
|
|
4
|
+
desktop automation: apps, windows, accessibility trees, mouse/keyboard,
|
|
5
|
+
screenshots, clipboard, audio, and VLM/STT model access.
|
|
6
|
+
|
|
7
|
+
This file ships in the npm tarball alongside `index.d.ts` and is versioned
|
|
8
|
+
with it.
|
|
9
|
+
|
|
10
|
+
## Where the API is documented
|
|
11
|
+
|
|
12
|
+
Read **`index.d.ts`** first — it is the source of truth. Every class,
|
|
13
|
+
function, and enum is fully typed (~1500 lines) and carries JSDoc covering
|
|
14
|
+
idioms, lifecycle rules, platform quirks, and inter-API trade-offs that types
|
|
15
|
+
alone can't express. The JSDoc is generated from doc comments,
|
|
16
|
+
so the per-symbol guidance there is authoritative — trust it over any
|
|
17
|
+
restatement elsewhere.
|
|
18
|
+
|
|
19
|
+
## Mental model
|
|
20
|
+
|
|
21
|
+
- Calls into the native module are **synchronous**. No Promises are returned;
|
|
22
|
+
`Result::Err` on the Rust side is translated to a thrown JS exception by
|
|
23
|
+
napi-rs, so JS callers `try`/`catch` as usual.
|
|
24
|
+
- Many objects are **handles to platform resources** (windows, audio devices,
|
|
25
|
+
accessibility trees, file/directory handles). Their lifetime matters;
|
|
26
|
+
dropping them can free the underlying resource.
|
|
27
|
+
- Coordinates are uniform across platforms: **top-left origin, global
|
|
28
|
+
physical pixels** (also called device pixels — the raw hardware pixels of
|
|
29
|
+
the display, not the logical / CSS / point units used in browsers and
|
|
30
|
+
some desktop UI frameworks). On a 2× HiDPI display, a 1920×1080-logical
|
|
31
|
+
screen is 3840×2160 in these coordinates. Image and screenshot dimensions
|
|
32
|
+
are likewise in physical pixels.
|
|
33
|
+
|
|
34
|
+
## Live log viewer (optional)
|
|
35
|
+
|
|
36
|
+
`@simular-ai/simulib-log-viewer` is a **human-facing aid**: a floating,
|
|
37
|
+
always-on-top log window that lets the person running the script watch
|
|
38
|
+
log records in real time when stdout is hidden behind the app
|
|
39
|
+
being driven. Suggest it when the user is writing an interactive
|
|
40
|
+
automation script and would benefit from live visibility; skip it for
|
|
41
|
+
headless / CI / unattended runs (no human is there to watch the window,
|
|
42
|
+
and it adds a window-spawning subprocess for no benefit). **Optional peer
|
|
43
|
+
dependency — install it explicitly.**
|
|
44
|
+
|
|
45
|
+
The window is **click-through by default**, so mouse input passes straight
|
|
46
|
+
through to whatever is underneath and never interferes with the app being
|
|
47
|
+
automated. On macOS it is also excluded from screen captures system-wide
|
|
48
|
+
(via `NSWindowSharingNone`), so `screenshotFull` / `screenshotCropped` and
|
|
49
|
+
any other capture tool won't include the viewer in the result. A global
|
|
50
|
+
hotkey (`Ctrl+Shift+Option+L` on macOS, `Ctrl+Shift+Alt+L` everywhere
|
|
51
|
+
else, also shown inside the window) toggles "grab mode": pressing it once
|
|
52
|
+
turns click-through off, pauses execution, and makes the window draggable;
|
|
53
|
+
pressing it again restores click-through and resumes.
|
|
54
|
+
|
|
55
|
+
Wire it up via `initLogger` and respect the pause inside automation loops:
|
|
56
|
+
|
|
57
|
+
```js
|
|
58
|
+
import { initLogger } from '@simular-ai/simulib-js'
|
|
59
|
+
import { LogWindow } from '@simular-ai/simulib-log-viewer'
|
|
60
|
+
|
|
61
|
+
const win = new LogWindow()
|
|
62
|
+
win.spawn()
|
|
63
|
+
initLogger((rec) => win.log(`[${rec.level}] [${rec.target}] ${rec.message}`), 'info')
|
|
64
|
+
|
|
65
|
+
// Yield while the user has paused the viewer (e.g. to drag it).
|
|
66
|
+
await win.waitIfPaused()
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
The second argument to `initLogger` is a filter spec in the `RUST_LOG`
|
|
70
|
+
environment-variable syntax used by Rust's `env_logger` / `log` crates —
|
|
71
|
+
e.g. `'info'` for a global level, `'simulib_rs=debug,warn'` for per-module
|
|
72
|
+
scoping. Omit it to inherit the value of the `RUST_LOG` env var (or fall
|
|
73
|
+
back to `'error'` if it's unset). See `initLogger`'s JSDoc in
|
|
74
|
+
`index.d.ts` for the full grammar.
|
|
75
|
+
|
|
76
|
+
Other methods: `clear()` drops displayed messages, `isPaused` getter for a
|
|
77
|
+
non-blocking check, `close()` to shut the viewer (also automatic on parent
|
|
78
|
+
exit).
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Simular AI
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# `@simular-ai/simulib-js`
|
|
2
|
+
|
|
3
|
+

|
|
4
|
+
|
|
5
|
+
Node.js bindings for [`simulib-rs`](https://github.com/simular-ai/simulib-rs), a Rust crate published by [Simular](https://simular.ai), built with [`napi-rs`](https://napi.rs/). Provides high-level primitives for desktop automation — keyboard and mouse input, screenshots, clipboard access, audio capture, accessibility-tree inspection, and more — exposed through idiomatic JavaScript. Ships TypeScript definitions auto-generated from the Rust source, so the types always match runtime behavior.
|
|
6
|
+
|
|
7
|
+
## Install
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npm install @simular-ai/simulib-js
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Prebuilt native binaries are published for:
|
|
14
|
+
|
|
15
|
+
| Platform | Architecture |
|
|
16
|
+
| -------- | ----------------------------------- |
|
|
17
|
+
| macOS | `aarch64` (Apple Silicon), `x86_64` |
|
|
18
|
+
| Windows | `x86_64`, `aarch64` |
|
|
19
|
+
| Linux | `x86_64`, `aarch64` (glibc) |
|
|
20
|
+
|
|
21
|
+
Node.js **20 or newer** is required. If your platform isn't covered, see [Building from source](#building-from-source).
|
|
22
|
+
|
|
23
|
+
### Optional — install the log viewer
|
|
24
|
+
|
|
25
|
+
The [`log-window.mjs`](./examples/log-window.mjs) example uses [`@simular-ai/simulib-log-viewer`](https://github.com/simular-ai/simulib-log-viewer), which is declared as an **optional peer dependency** and **not** installed by default. Install it only if you want to run that example or use the same pattern in your own code:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
npm install @simular-ai/simulib-log-viewer
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Usage
|
|
32
|
+
|
|
33
|
+
Try the included [`google_search.mjs`](./examples/google_search.mjs) example — it opens Google in Chrome, enables the accessibility tree, takes a screenshot, and displays it:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
# With the simulang CLI (no local install needed):
|
|
37
|
+
npm install -g @simular-ai/simulang
|
|
38
|
+
simulang run examples/google_search.mjs
|
|
39
|
+
|
|
40
|
+
# Or directly with Node.js if you have simulib-js installed locally:
|
|
41
|
+
node examples/google_search.mjs
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
## API documentation
|
|
45
|
+
|
|
46
|
+
Browse the [API reference](https://docs.simular.ai/simulib-js/api/latest/) for more details. A bleeding-edge preview of HEAD on `main` is also published to [GitHub Pages](https://simular-ai.github.io/simulib-js/) between releases.
|
|
47
|
+
|
|
48
|
+
## Using with Claude Code
|
|
49
|
+
|
|
50
|
+
This package ships a short [`CLAUDE.md`](./CLAUDE.md) inside the npm tarball that points Claude Code at `index.d.ts` as the source of truth for the API and adds a few cross-cutting notes that types alone can't express (lifetime rules, coordinate system, etc.). Wire it into your project's `CLAUDE.md` with:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
npx simulib-init-claude # appends to <project>/CLAUDE.md (creates it if missing)
|
|
54
|
+
npx simulib-init-claude --user # appends to ~/.claude/CLAUDE.md instead
|
|
55
|
+
npx simulib-init-claude --check # report status, don't write
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
The added line is a single `@./node_modules/@simular-ai/simulib-js/CLAUDE.md` import inside a sentinel-delimited block — safe to re-run, no-op when already present.
|
|
59
|
+
|
|
60
|
+
## Relationship to `simulang`
|
|
61
|
+
|
|
62
|
+
[`@simular-ai/simulang`](https://github.com/simular-ai/simulang-cli) is a CLI that runs desktop automation scripts (`.ts`, `.js`, `.simulang`) using this package. It bundles `simulib-js` and re-exports the full API to scripts, so you can write and run automation scripts without a build step:
|
|
63
|
+
|
|
64
|
+
```bash
|
|
65
|
+
npm install -g @simular-ai/simulang
|
|
66
|
+
simulang run my-script.js
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
`simulib-js` is the underlying primitive library; `simulang` is the batteries-included script runner built on top of it. If you are embedding desktop automation into your own Node.js application, depend on `simulib-js` directly. If you just want to run standalone automation scripts, `simulang` is the easier starting point.
|
|
70
|
+
|
|
71
|
+
You can also point `simulang` at a local `simulib-js` checkout during development:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
simulang run --simulib=/path/to/simulib-js my-script.js
|
|
75
|
+
```
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Wire up Claude Code to read this package's CLAUDE.md by appending one
|
|
3
|
+
// idempotent import line to <project>/CLAUDE.md (or ~/.claude/CLAUDE.md
|
|
4
|
+
// with --user). Re-running is a no-op.
|
|
5
|
+
//
|
|
6
|
+
// Usage (after `npm install @simular-ai/simulib-js`):
|
|
7
|
+
// npx simulib-init-claude # update <project>/CLAUDE.md
|
|
8
|
+
// npx simulib-init-claude --user # update ~/.claude/CLAUDE.md
|
|
9
|
+
// npx simulib-init-claude --check # report status, don't write
|
|
10
|
+
|
|
11
|
+
import { existsSync, readFileSync, writeFileSync, mkdirSync } from 'node:fs'
|
|
12
|
+
import { homedir } from 'node:os'
|
|
13
|
+
import { dirname, join, relative } from 'node:path'
|
|
14
|
+
import { fileURLToPath } from 'node:url'
|
|
15
|
+
|
|
16
|
+
const PKG = '@simular-ai/simulib-js'
|
|
17
|
+
const BEGIN = `<!-- ${PKG}:claude-import:begin -->`
|
|
18
|
+
const END = `<!-- ${PKG}:claude-import:end -->`
|
|
19
|
+
const BLOCK_RE = new RegExp(`${BEGIN}[\\s\\S]*?${END}\\n?`)
|
|
20
|
+
|
|
21
|
+
/** @type {{ user: boolean, check: boolean, help: boolean }} */
|
|
22
|
+
const flags = { user: false, check: false, help: false }
|
|
23
|
+
for (const a of process.argv.slice(2)) {
|
|
24
|
+
if (a === '--user') flags.user = true
|
|
25
|
+
else if (a === '--check') flags.check = true
|
|
26
|
+
else if (a === '--help' || a === '-h') flags.help = true
|
|
27
|
+
else {
|
|
28
|
+
process.stderr.write(`Unknown flag: ${a}\n`)
|
|
29
|
+
process.exit(2)
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
if (flags.help) {
|
|
34
|
+
process.stdout.write(
|
|
35
|
+
`Usage: simulib-init-claude [--user] [--check]\n` +
|
|
36
|
+
` Point Claude Code at ${PKG}/CLAUDE.md so it knows how to use this library.\n` +
|
|
37
|
+
` --user Target ~/.claude/CLAUDE.md instead of <project>/CLAUDE.md.\n` +
|
|
38
|
+
` --check Report status without writing.\n`,
|
|
39
|
+
)
|
|
40
|
+
process.exit(0)
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
// We live at <pkgRoot>/bin/init-claude.mjs in the package, so the doc is two
|
|
44
|
+
// levels up — no need to walk the filesystem for it.
|
|
45
|
+
const docPath = join(dirname(dirname(fileURLToPath(import.meta.url))), 'CLAUDE.md')
|
|
46
|
+
|
|
47
|
+
/**
|
|
48
|
+
* Walk up from `start` looking for the consumer's package.json (i.e. one that
|
|
49
|
+
* isn't ours). Returns the directory or null if we hit the filesystem root.
|
|
50
|
+
* @param {string} start
|
|
51
|
+
*/
|
|
52
|
+
function findProjectRoot(start) {
|
|
53
|
+
for (let dir = start; ; dir = dirname(dir)) {
|
|
54
|
+
try {
|
|
55
|
+
/** @type {{ name?: string }} */
|
|
56
|
+
const pkg = JSON.parse(readFileSync(join(dir, 'package.json'), 'utf8'))
|
|
57
|
+
if (pkg.name !== PKG) return dir
|
|
58
|
+
} catch {
|
|
59
|
+
// No readable package.json here — keep walking.
|
|
60
|
+
}
|
|
61
|
+
if (dirname(dir) === dir) return null
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
let targetMd, importLine
|
|
66
|
+
if (flags.user) {
|
|
67
|
+
targetMd = join(homedir(), '.claude', 'CLAUDE.md')
|
|
68
|
+
// Absolute path — node_modules location isn't predictable from ~/.claude.
|
|
69
|
+
importLine = `@${docPath}`
|
|
70
|
+
} else {
|
|
71
|
+
const projectRoot = findProjectRoot(process.cwd())
|
|
72
|
+
if (!projectRoot) {
|
|
73
|
+
process.stderr.write(`No project package.json found from ${process.cwd()}.\nRun inside a project, or use --user.\n`)
|
|
74
|
+
process.exit(1)
|
|
75
|
+
}
|
|
76
|
+
targetMd = join(projectRoot, 'CLAUDE.md')
|
|
77
|
+
const rel = relative(projectRoot, docPath)
|
|
78
|
+
// Project-relative path keeps the import portable across machines.
|
|
79
|
+
importLine = `@${rel.startsWith('.') ? rel : `./${rel}`}`
|
|
80
|
+
}
|
|
81
|
+
// Claude Code's @import resolver expects POSIX-style paths. On Windows,
|
|
82
|
+
// path.relative / path.join produce backslashes — normalise here so the
|
|
83
|
+
// resulting CLAUDE.md is identical on every OS.
|
|
84
|
+
importLine = importLine.replaceAll('\\', '/')
|
|
85
|
+
|
|
86
|
+
const block = `${BEGIN}\n${importLine}\n${END}\n`
|
|
87
|
+
const existing = existsSync(targetMd) ? readFileSync(targetMd, 'utf8') : ''
|
|
88
|
+
|
|
89
|
+
if (existing.includes(block)) {
|
|
90
|
+
process.stdout.write(`${PKG}: ${targetMd} already imports the doc — no change.\n`)
|
|
91
|
+
process.exit(0)
|
|
92
|
+
}
|
|
93
|
+
if (flags.check) {
|
|
94
|
+
process.stdout.write(`${PKG}: ${targetMd} would be ${existing ? 'updated' : 'created'}.\n`)
|
|
95
|
+
process.exit(0)
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
// Replace an existing block (handles import-line changes) or append a fresh
|
|
99
|
+
// one after a blank line. Trimming trailing newlines normalises the spacing.
|
|
100
|
+
const trimmed = existing.replace(/\n+$/, '')
|
|
101
|
+
const next = BLOCK_RE.test(existing) ? existing.replace(BLOCK_RE, block) : trimmed ? `${trimmed}\n\n${block}` : block
|
|
102
|
+
|
|
103
|
+
mkdirSync(dirname(targetMd), { recursive: true })
|
|
104
|
+
writeFileSync(targetMd, next)
|
|
105
|
+
process.stdout.write(`${PKG}: ${existing ? 'updated' : 'created'} ${targetMd}\n`)
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Postinstall hint for `@simular-ai/simulib-js`. Prints exactly one line if
|
|
3
|
+
// the user appears to be a Claude Code user, otherwise silent. Swallows all
|
|
4
|
+
// errors — a broken hint must never break `npm install`.
|
|
5
|
+
|
|
6
|
+
import { existsSync } from 'node:fs'
|
|
7
|
+
import { homedir } from 'node:os'
|
|
8
|
+
import { join } from 'node:path'
|
|
9
|
+
|
|
10
|
+
function shouldPrint() {
|
|
11
|
+
// Skip when the maintainer is running `npm install` in this repo itself —
|
|
12
|
+
// `import.meta.url` only contains a `node_modules` segment when this
|
|
13
|
+
// package has been installed as a dependency of another project.
|
|
14
|
+
if (!import.meta.url.includes('/node_modules/')) return false
|
|
15
|
+
if (!process.stdout.isTTY) return false
|
|
16
|
+
if (process.env.CI) return false
|
|
17
|
+
if (process.env.npm_command === 'ci') return false
|
|
18
|
+
const loglevel = (process.env.npm_config_loglevel ?? '').toLowerCase()
|
|
19
|
+
if (loglevel === 'silent' || loglevel === 'error' || loglevel === 'warn') return false
|
|
20
|
+
if (!existsSync(join(homedir(), '.claude'))) return false
|
|
21
|
+
return true
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
try {
|
|
25
|
+
if (shouldPrint()) {
|
|
26
|
+
process.stdout.write(
|
|
27
|
+
'@simular-ai/simulib-js: Claude Code detected. Run `npx @simular-ai/simulib-js init-claude` to point it at the bundled API docs.\n',
|
|
28
|
+
)
|
|
29
|
+
}
|
|
30
|
+
} catch {
|
|
31
|
+
// Never propagate failures from the hint.
|
|
32
|
+
}
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Examples
|
|
2
|
+
|
|
3
|
+
Runnable snippets demonstrating common `@simular-ai/simulib-js` APIs. All files are plain ESM JavaScript — no TypeScript or extra tooling needed.
|
|
4
|
+
|
|
5
|
+
> Install the package first if you haven't already — see the [main README](../README.md#install).
|
|
6
|
+
|
|
7
|
+
Run an example out of `node_modules`:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
node node_modules/@simular-ai/simulib-js/examples/system.mjs
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Or copy a file into your own project to tweak it:
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
cp node_modules/@simular-ai/simulib-js/examples/screenshot.mjs ./
|
|
17
|
+
node screenshot.mjs
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## What each example does
|
|
21
|
+
|
|
22
|
+
| File | Description |
|
|
23
|
+
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
24
|
+
| `system.mjs` | Opens a URL in the default browser. |
|
|
25
|
+
| `file-trait.mjs` | Writes a text file and reads it back. |
|
|
26
|
+
| `clipboard.mjs` | Reads, writes, pastes, and clears the system clipboard. |
|
|
27
|
+
| `keyboard.mjs` | Synthesizes keyboard input (types a string and sends key events). |
|
|
28
|
+
| `screenshot.mjs` | Captures the main screen, resizes and compresses the image, saves to disk. |
|
|
29
|
+
| `google_search.mjs` | End-to-end flow: launches Chrome, navigates, captures a screenshot. |
|
|
30
|
+
| `accessibility.mjs` | Lists windows, snapshots the foreground accessibility tree, queries actions. |
|
|
31
|
+
| `chrome_google_search_button.mjs` | Opens google.com in Chrome and locates the search button by _concept text_, using BoW Jaccard scoring over `overallDescription`. |
|
|
32
|
+
| `open-app.mjs` | Opens an app, lists its windows, snapshots its accessibility tree. |
|
|
33
|
+
| `loopback.mjs` | Plays an audio file while capturing system audio, then transcribes it. |
|
|
34
|
+
| `logger.mjs` | Forwards Rust `log` records to a JS callback (env_logger-style filter spec). |
|
|
35
|
+
| `log-window.mjs` | Pipes Rust `log::*!` records into a floating, always-on-top log window. Requires the optional [`@simular-ai/simulib-log-viewer`](../README.md#3-optional--install-the-log-viewer) peer dep. |
|
|
36
|
+
|
|
37
|
+
## Heads up — these examples affect your system
|
|
38
|
+
|
|
39
|
+
Unlike a sandboxed library call, most of these examples perform **real actions** on your machine:
|
|
40
|
+
|
|
41
|
+
- `keyboard.mjs` and `google_search.mjs` synthesize keystrokes — make sure the intended window is focused, or text will land somewhere unexpected.
|
|
42
|
+
- `clipboard.mjs` overwrites your current clipboard contents (it restores them at the end).
|
|
43
|
+
- `screenshot.mjs`, `google_search.mjs`, and `loopback.mjs` capture your screen or system audio.
|
|
44
|
+
- `system.mjs` and `google_search.mjs` launch a browser window.
|
|
45
|
+
|
|
46
|
+
### Required OS permissions
|
|
47
|
+
|
|
48
|
+
On **macOS**, the first run of certain examples will prompt for permissions. Grant these in **System Settings → Privacy & Security**:
|
|
49
|
+
|
|
50
|
+
- **Accessibility** — required for keyboard/mouse input and accessibility-tree reads (`keyboard.mjs`, `google_search.mjs`).
|
|
51
|
+
- **Screen Recording** — required for screen capture and audio loopback (`screenshot.mjs`, `google_search.mjs`, `loopback.mjs`).
|
|
52
|
+
|
|
53
|
+
The permission is granted to the process that launches Node (e.g. your terminal application), not to Node itself.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
// Run: node examples/accessibility.mjs
|
|
2
|
+
// Tests AccessibilityTree: list windows, snapshot foreground, query actions.
|
|
3
|
+
|
|
4
|
+
import { AccessibilityTree, AriaRole, ariaRoleToString, Window } from '@simular-ai/simulib-js'
|
|
5
|
+
|
|
6
|
+
// 1. List all windows
|
|
7
|
+
const windows = Window.all()
|
|
8
|
+
console.log(`=== All visible windows (${windows.length}) ===`)
|
|
9
|
+
windows.slice(0, 5).forEach((w) => console.log(` pid=${w.pid} "${w.title}"`))
|
|
10
|
+
console.log()
|
|
11
|
+
|
|
12
|
+
// 2. Snapshot foreground window
|
|
13
|
+
const tree = AccessibilityTree.fromForeground()
|
|
14
|
+
console.log(`Bound to: "${tree.windowTitle}"`)
|
|
15
|
+
|
|
16
|
+
const root = tree.snapshot()
|
|
17
|
+
console.log(`Role: ${ariaRoleToString(root.role)}`)
|
|
18
|
+
console.log(`Children: ${root.children.length}`)
|
|
19
|
+
|
|
20
|
+
// Print the whole tree as an indented Playwright-style aria snapshot,
|
|
21
|
+
// the same format as `WindowTrait::snapshot()` / the `print_ax_tree`
|
|
22
|
+
// example in simulib-rs.
|
|
23
|
+
/**
|
|
24
|
+
* @param {import('@simular-ai/simulib-js').AccessibilityNodeJs} node
|
|
25
|
+
* @param {number} [depth]
|
|
26
|
+
*/
|
|
27
|
+
function printSnapshot(node, depth = 0) {
|
|
28
|
+
const indent = ' '.repeat(depth)
|
|
29
|
+
let line = `${indent}- ${ariaRoleToString(node.role)}`
|
|
30
|
+
if (node.name) line += ` ${JSON.stringify(node.name)}`
|
|
31
|
+
if (node.value) line += `: ${JSON.stringify(node.value)}`
|
|
32
|
+
console.log(line)
|
|
33
|
+
for (const child of node.children) printSnapshot(child, depth + 1)
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
console.log('\n=== Snapshot ===')
|
|
37
|
+
printSnapshot(root)
|
|
38
|
+
|
|
39
|
+
// 3. Test actions on first button
|
|
40
|
+
const firstButton = root.children.find((c) => c.role === AriaRole.Button && c.refId != null)
|
|
41
|
+
if (firstButton && firstButton.refId != null) {
|
|
42
|
+
console.log(`\nFirst button: "${firstButton.name}" [ref=${firstButton.refId}]`)
|
|
43
|
+
console.log('Actions:', tree.getSupportedActions(firstButton.refId))
|
|
44
|
+
console.log('Bounds:', tree.getBounds(firstButton.refId))
|
|
45
|
+
}
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
// Run: node examples/chrome_google_search_button.mjs
|
|
2
|
+
//
|
|
3
|
+
// Open Google Chrome on https://www.google.com and locate the main search
|
|
4
|
+
// control by *concept text*, using `Instance.scoredSearch` — a thin binding
|
|
5
|
+
// over `simulib_rs::Instance::scored_search` + `BowJaccard` paired-Jaccard
|
|
6
|
+
// scoring against each node's `summary_with_context`.
|
|
7
|
+
//
|
|
8
|
+
// This is the JS analog of
|
|
9
|
+
// `simulib-rs/examples/chrome_google_search_button.rs`.
|
|
10
|
+
//
|
|
11
|
+
// Requirements:
|
|
12
|
+
// - Windows or macOS
|
|
13
|
+
// - Google Chrome installed
|
|
14
|
+
// - On macOS: Accessibility permission granted to the process running Node.
|
|
15
|
+
|
|
16
|
+
import {
|
|
17
|
+
AccessibilityTree,
|
|
18
|
+
App,
|
|
19
|
+
FocusPolicy,
|
|
20
|
+
TraversalOrder,
|
|
21
|
+
Visibility,
|
|
22
|
+
Window,
|
|
23
|
+
ariaRoleToString,
|
|
24
|
+
} from '@simular-ai/simulib-js'
|
|
25
|
+
|
|
26
|
+
/** Natural-language concept matched against each node's `overallDescription`. */
|
|
27
|
+
const CONCEPT = 'Auf gut Glück'
|
|
28
|
+
|
|
29
|
+
/** Match threshold (same default as the Rust example). */
|
|
30
|
+
const MATCH_THRESHOLD = 0.75
|
|
31
|
+
|
|
32
|
+
/** Upper bound on nodes visited so a pathological Chromium tree can't run unbounded. */
|
|
33
|
+
const MAX_NODES_VISITED = 50_000
|
|
34
|
+
|
|
35
|
+
const browser = process.platform === 'darwin' ? 'Google Chrome' : 'Chrome'
|
|
36
|
+
|
|
37
|
+
/** @param {number} timeoutMs */
|
|
38
|
+
async function waitForChromeForeground(timeoutMs) {
|
|
39
|
+
const deadline = Date.now() + timeoutMs
|
|
40
|
+
while (Date.now() < deadline) {
|
|
41
|
+
try {
|
|
42
|
+
const title = AccessibilityTree.fromForeground().windowTitle.toLowerCase()
|
|
43
|
+
if (title.includes('chrome') || title.includes('chromium') || title.includes('谷歌')) {
|
|
44
|
+
return
|
|
45
|
+
}
|
|
46
|
+
} catch {
|
|
47
|
+
// ignore – the foreground app may not be accessible yet
|
|
48
|
+
}
|
|
49
|
+
await new Promise((r) => setTimeout(r, 200))
|
|
50
|
+
}
|
|
51
|
+
throw new Error(`Chrome did not become the foreground window within ${timeoutMs} ms`)
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
const instance = App.exactName(browser).open('https://www.google.com', FocusPolicy.Steal, Visibility.Show, true)
|
|
55
|
+
console.log(`Opened ${browser} (pid=${instance.pid})`)
|
|
56
|
+
|
|
57
|
+
await new Promise((r) => setTimeout(r, 2000))
|
|
58
|
+
await waitForChromeForeground(45_000)
|
|
59
|
+
|
|
60
|
+
// Chrome ships with its accessibility tree disabled by default for perf. Turn
|
|
61
|
+
// it on for this process and give it a moment to populate.
|
|
62
|
+
instance.enableAccessibility()
|
|
63
|
+
await new Promise((r) => setTimeout(r, 3000))
|
|
64
|
+
|
|
65
|
+
const matches = instance.scoredSearch(TraversalOrder.BreadthFirst, MAX_NODES_VISITED, false, CONCEPT, MATCH_THRESHOLD)
|
|
66
|
+
|
|
67
|
+
const button = matches[0]
|
|
68
|
+
if (!button) {
|
|
69
|
+
throw new Error(
|
|
70
|
+
`No node scored above ${MATCH_THRESHOLD} on concept ${JSON.stringify(CONCEPT)} ` +
|
|
71
|
+
`within ${MAX_NODES_VISITED} BFS nodes. Try another CONCEPT for your locale, ` +
|
|
72
|
+
`close overlays, or wait for the page to finish loading.`,
|
|
73
|
+
)
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
console.log(`Found accessibility node for concept ${JSON.stringify(CONCEPT)}:`)
|
|
77
|
+
console.log(` role: ${ariaRoleToString(button.role)}`)
|
|
78
|
+
console.log(` name: ${button.name}`)
|
|
79
|
+
console.log(` overallDescription: ${button.overallDescription}`)
|
|
80
|
+
console.log(` supportedActions: ${JSON.stringify(button.supportedActions())}`)
|
|
81
|
+
// Action methods (`button.activate()`, `button.setValue(...)`, …) are
|
|
82
|
+
// available directly on the returned `AccessibilityNode`.
|
|
83
|
+
|
|
84
|
+
console.log('\nMinimizing Chrome window…')
|
|
85
|
+
const [chromeWindow] = Window.allForPid(instance.pid)
|
|
86
|
+
if (!chromeWindow) throw new Error('no windows found for instance')
|
|
87
|
+
chromeWindow.minimize()
|
|
88
|
+
console.log('Window should be minimized.')
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
// Run: node examples/clipboard.mjs
|
|
2
|
+
// This example reads and writes the clipboard.
|
|
3
|
+
|
|
4
|
+
import { Clipboard } from '@simular-ai/simulib-js'
|
|
5
|
+
|
|
6
|
+
console.log('Waiting three seconds...')
|
|
7
|
+
await new Promise((resolve) => setTimeout(resolve, 3000))
|
|
8
|
+
|
|
9
|
+
const clipboard = new Clipboard()
|
|
10
|
+
|
|
11
|
+
const previous = clipboard.setString('Hello from simulib-js')
|
|
12
|
+
const current = clipboard.getString()
|
|
13
|
+
|
|
14
|
+
console.log('Previous clipboard:', previous)
|
|
15
|
+
console.log('Current clipboard:', current)
|
|
16
|
+
|
|
17
|
+
// Paste text by simulating Cmd/Ctrl+V.
|
|
18
|
+
clipboard.pasteText('Pasted from simulib-js')
|
|
19
|
+
|
|
20
|
+
clipboard.clear()
|
|
21
|
+
|
|
22
|
+
const afterClear = clipboard.getString()
|
|
23
|
+
console.log('After clear:', afterClear)
|
|
24
|
+
|
|
25
|
+
// Restore previous clipboard content if it existed.
|
|
26
|
+
if (previous !== null) {
|
|
27
|
+
console.log('Restored previous clipboard:')
|
|
28
|
+
clipboard.setString(previous)
|
|
29
|
+
}
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
// Run: node examples/file-trait.mjs
|
|
2
|
+
// This example writes a file and reads it back.
|
|
3
|
+
|
|
4
|
+
import { readFile, writeFile } from '@simular-ai/simulib-js'
|
|
5
|
+
|
|
6
|
+
const writtenPath = writeFile('simulib-js-example.txt', 'hello from simulib-js', false)
|
|
7
|
+
console.log('Wrote file:', writtenPath)
|
|
8
|
+
console.log('Read file:', readFile(writtenPath))
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
// Run: node examples/google_search.mjs
|
|
2
|
+
// This example opens Google in Chrome.
|
|
3
|
+
|
|
4
|
+
import { tmpdir } from 'node:os'
|
|
5
|
+
import { join } from 'node:path'
|
|
6
|
+
import { pathToFileURL } from 'node:url'
|
|
7
|
+
import {
|
|
8
|
+
App,
|
|
9
|
+
KeyboardController,
|
|
10
|
+
Key,
|
|
11
|
+
Direction,
|
|
12
|
+
FocusPolicy,
|
|
13
|
+
Visibility,
|
|
14
|
+
Screen,
|
|
15
|
+
enableAccessibilityForFrontmostApp,
|
|
16
|
+
screenshotFull,
|
|
17
|
+
} from '@simular-ai/simulib-js'
|
|
18
|
+
|
|
19
|
+
const browser = process.platform === 'darwin' ? 'Google Chrome' : 'Chrome'
|
|
20
|
+
const imageViewer = process.platform === 'darwin' ? 'Preview' : 'Chrome'
|
|
21
|
+
const modifierKey = process.platform === 'darwin' ? Key.Meta : Key.Control
|
|
22
|
+
|
|
23
|
+
// Open a URL in Chrome
|
|
24
|
+
App.exactName(browser).open('https://google.com', FocusPolicy.Steal, Visibility.Show, true)
|
|
25
|
+
console.log('Launched Google Chrome')
|
|
26
|
+
|
|
27
|
+
// Wait two seconds to give Chrome time to start
|
|
28
|
+
console.log('Waiting two seconds...')
|
|
29
|
+
await new Promise((resolve) => setTimeout(resolve, 2000))
|
|
30
|
+
|
|
31
|
+
const foreground = new KeyboardController()
|
|
32
|
+
|
|
33
|
+
// Independent of keyboard layout
|
|
34
|
+
foreground.keyUnicode('z', Direction.Click)
|
|
35
|
+
await new Promise((resolve) => setTimeout(resolve, 1000))
|
|
36
|
+
|
|
37
|
+
// Open the accessibility menu
|
|
38
|
+
foreground.key(modifierKey, Direction.Press)
|
|
39
|
+
foreground.keyUnicode('l', Direction.Click)
|
|
40
|
+
foreground.key(modifierKey, Direction.Release)
|
|
41
|
+
await new Promise((resolve) => setTimeout(resolve, 500))
|
|
42
|
+
foreground.text('chrome://accessibility')
|
|
43
|
+
foreground.key(Key.Return, Direction.Click)
|
|
44
|
+
await new Promise((resolve) => setTimeout(resolve, 2000))
|
|
45
|
+
|
|
46
|
+
// Enable the accessibility tree for Chrome
|
|
47
|
+
enableAccessibilityForFrontmostApp()
|
|
48
|
+
await new Promise((resolve) => setTimeout(resolve, 5000))
|
|
49
|
+
|
|
50
|
+
// Take a screenshot of the entire screen and save it to a file
|
|
51
|
+
const screenshot = screenshotFull(true, Screen.mainScreen())
|
|
52
|
+
const path = join(tmpdir(), 'screenshot.png')
|
|
53
|
+
screenshot.save(path)
|
|
54
|
+
const fileUrl = pathToFileURL(path).toString()
|
|
55
|
+
|
|
56
|
+
// Open the screenshot in the image viewer
|
|
57
|
+
App.exactName(imageViewer).open(fileUrl, FocusPolicy.Steal, Visibility.Show, true)
|
|
58
|
+
|
|
59
|
+
console.log('Done!')
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
// Run: node examples/keyboard.mjs
|
|
2
|
+
// This example uses the keyboard controller to type text.
|
|
3
|
+
// Ensure a text field is focused (e.g. in a text editor or browser) before running.
|
|
4
|
+
|
|
5
|
+
import { KeyboardController, Key, Direction, keyFromString } from '@simular-ai/simulib-js'
|
|
6
|
+
|
|
7
|
+
const keyboard = new KeyboardController()
|
|
8
|
+
|
|
9
|
+
// Wait two seconds
|
|
10
|
+
await new Promise((resolve) => setTimeout(resolve, 2000))
|
|
11
|
+
|
|
12
|
+
// Type a string. Works regardless of keyboard layout; supports Unicode (e.g. ❤️).
|
|
13
|
+
keyboard.text('Hello from simulib-js ❤️')
|
|
14
|
+
|
|
15
|
+
// Send individual key events, e.g. press Enter.
|
|
16
|
+
keyboard.key(Key.Return, Direction.Click)
|
|
17
|
+
|
|
18
|
+
// Or parse a key from a string.
|
|
19
|
+
const key = keyFromString('Return')
|
|
20
|
+
keyboard.key(key, Direction.Click)
|
|
21
|
+
|
|
22
|
+
// Wait two seconds
|
|
23
|
+
await new Promise((resolve) => setTimeout(resolve, 2000))
|