@mirsella/browser-mcp 2.1.0 → 2.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.codex-plugin/plugin.json +1 -1
- package/CONTRIBUTING.md +63 -21
- package/package.json +1 -1
- package/vendor/browser-mcp +0 -0
package/CONTRIBUTING.md
CHANGED
|
@@ -93,29 +93,71 @@ checkout that fails its first valid submission. No money or external account
|
|
|
93
93
|
is involved. `bun test tests/model-eval.test.ts` checks the fixture's rejection
|
|
94
94
|
and retry behavior.
|
|
95
95
|
|
|
96
|
-
Use a fresh `.scratch/` directory per experiment.
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
96
|
+
Use a fresh `.scratch/` directory per experiment. Record the source commit,
|
|
97
|
+
binary checksum, browser version, provider, exact model, and reasoning
|
|
98
|
+
setting. For source comparisons, give each checkout its own Cargo build
|
|
99
|
+
directory and reuse the shared Kache cache. Keep each condition's browser
|
|
100
|
+
skill.
|
|
101
|
+
|
|
102
|
+
Start `bun scripts/browser-model-eval.ts site --root=<root> --port=0` under a
|
|
103
|
+
known PID or held lock. Its `site.json` records the listening URL and PID. The
|
|
104
|
+
browser origin exposes no scoring endpoint. The host scorer reads order state
|
|
105
|
+
from files in the experiment directory; agents must not read those files.
|
|
106
|
+
|
|
107
|
+
Run names have a condition label and seed, such as `ours-current-1`,
|
|
108
|
+
`ours-guidance-2`, or `browser-use-1`. Seeds 1 and 2 select paired fixture
|
|
109
|
+
cases. Generate temporary project configuration with `config --root=<root>
|
|
110
|
+
--labels=ours-current,ours-guidance --condition=ours-guidance --seed=1
|
|
111
|
+
--ours-current=<binary> --ours-guidance=<binary>`. Merge it without
|
|
112
|
+
overwriting existing configuration. Each run gets its own profile and an
|
|
113
|
+
allocated CDP port. Restore the configuration and collect the fixture process
|
|
114
|
+
when finished.
|
|
115
|
+
|
|
116
|
+
For browser-use 1.0.0, install `browser-use==1.0.0` in a task-local Python
|
|
117
|
+
virtual environment. Keep the interpreter's virtual-environment path; do not
|
|
118
|
+
resolve its symlink. Write a JSON argv array containing that Python
|
|
119
|
+
executable, `scripts/browser-use-eval.py`, `--profile`, a fresh profile path,
|
|
120
|
+
`--ipc`, and a short task-local IPC path. Pass its path as
|
|
121
|
+
`--command=<argv.json> --backend=browser-use` to `config` or `mcp`.
|
|
122
|
+
External-command configurations require `--seed=1` or `--seed=2` to avoid
|
|
123
|
+
sharing one profile between active trials. The wrapper runs the official MCP
|
|
124
|
+
implementation with local headless Chromium and explicitly stops its browser.
|
|
125
|
+
Use a separate command file and profile for each seed. This dependency is for
|
|
126
|
+
evaluation only and is not part of the published server.
|
|
127
|
+
|
|
128
|
+
Generate each task with `prompt --root=<root> --run=ours-guidance-1
|
|
129
|
+
--port=<fixture-port> --skill=<condition-skill>`. Omit `--skill` for a server
|
|
130
|
+
that supplies its own packaged instructions. Page JavaScript, DOM operations
|
|
131
|
+
and normal page requests through the assigned browser MCP are permitted for
|
|
132
|
+
every condition. Host file/network access, fixture implementation (including
|
|
133
|
+
inline script contents), scoring data and other tools are forbidden. A
|
|
134
|
+
code-based server may use host code to orchestrate its browser helpers and
|
|
135
|
+
process observed page data, but cannot use it as a shortcut to the fixture's
|
|
136
|
+
private state.
|
|
137
|
+
|
|
138
|
+
Use separate model conversations with the same provider, model, reasoning,
|
|
139
|
+
harness and budgets. Counterbalance condition order across seeds. Compare the
|
|
140
|
+
current and candidate guidance on the same server binary before attributing
|
|
141
|
+
any difference to a runtime change. Test multi-harness routing separately from
|
|
142
|
+
single-agent task quality.
|
|
109
143
|
|
|
110
144
|
Save final answers to `<root>/answers.json` as objects containing `run`,
|
|
111
|
-
`answer`, `elapsedSeconds`,
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
145
|
+
`answer`, `elapsedSeconds`, `thread`, optional provider `usage`, and `valid`.
|
|
146
|
+
Set `valid: true` only after inspecting the agent trace for forbidden tools,
|
|
147
|
+
source or oracle access. Run `score --root=<root>
|
|
148
|
+
--runs=ours-current-1,ours-guidance-1,browser-use-1`. A passing score requires
|
|
149
|
+
that audit, a complete trace of issued tool calls within the 50-call and
|
|
150
|
+
10-minute budgets, one correct order, all facts, and matching receipt data.
|
|
151
|
+
Setup failures and unaudited trials do not establish reliability.
|
|
152
|
+
|
|
153
|
+
The adapter records run identities independently of tool names and navigation,
|
|
154
|
+
so Python `browser_exec` calls and compact `dispatch` calls use the same
|
|
155
|
+
scorer. Reports include tool responses, client request bytes, initialization
|
|
156
|
+
and catalog response bytes, images, errors and supplied provider usage. Byte
|
|
157
|
+
counts are not tokens or billing. Include task/skill inputs and provider
|
|
158
|
+
token/cache/reasoning usage when available; do not infer total context or cost
|
|
159
|
+
from response bytes. Two paired cases are a smoke test, not a statistical
|
|
160
|
+
reliability estimate.
|
|
119
161
|
|
|
120
162
|
## Releases
|
|
121
163
|
|
package/package.json
CHANGED
package/vendor/browser-mcp
CHANGED
|
Binary file
|