onebrowser 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +36 -26
- package/package.json +1 -1
- package/src/cli.js +6 -3
- package/src/cost.js +22 -0
- package/src/index.js +105 -55
- package/src/ledger.js +71 -20
- package/src/providers/anchor.js +5 -0
- package/src/providers/firecrawl.js +2 -0
- package/src/providers/hyperbrowser.js +2 -0
- package/src/providers/index.js +28 -18
- package/src/providers/kernel.js +4 -2
- package/src/providers/lightpanda.js +6 -4
- package/src/providers/steel.js +2 -0
- package/src/providers/tinyfish.js +2 -0
- package/src/session.js +25 -36
package/README.md
CHANGED
|
@@ -15,13 +15,15 @@ const page = await session.newPage(); // Playwright; or any CDP client on sessio
|
|
|
15
15
|
await page.goto('https://example.com');
|
|
16
16
|
await session.close();
|
|
17
17
|
|
|
18
|
-
await browsers.
|
|
19
|
-
|
|
18
|
+
await browsers.readQuota('steel'); // { remaining, unbilled, unit: 'USD', renewsAt: null }
|
|
19
|
+
browsers.quota('steel'); // the same now, from that reading: sessions opened since count at once
|
|
20
|
+
browsers.maxSessionCost('steel'); // what a session with the default options costs at most, in USD
|
|
21
|
+
await browsers.syncUsage(); // stores the providers' records of ended sessions in the ledger
|
|
20
22
|
browsers.usage({ since: Date.parse('2026-11-01') }); // [{ day, provider, client, sessions, minutes, … }]
|
|
21
23
|
await browsers.close();
|
|
22
24
|
```
|
|
23
25
|
|
|
24
|
-
|
|
26
|
+
Choosing a provider, capping sessions and stopping before a quota runs out are the program's policy. onebrowser opens the sessions it is asked for and gives the program what it needs to decide: what the quota has left, what a session costs at most, and `QuotaError` when a provider refuses a session for the account's quota.
|
|
25
27
|
|
|
26
28
|
Built in: [Steel](https://steel.dev), [TinyFish](https://tinyfish.ai), [Hyperbrowser](https://hyperbrowser.ai), [Lightpanda](https://lightpanda.io), [Kernel](https://www.kernel.sh), [Anchor Browser](https://anchorbrowser.io) and [Firecrawl](https://www.firecrawl.dev); a program can add its own (see [Adding a provider](#adding-a-provider)). Node 22.13 or later.
|
|
27
29
|
|
|
@@ -33,24 +35,26 @@ Built in: [Steel](https://steel.dev), [TinyFish](https://tinyfish.ai), [Hyperbro
|
|
|
33
35
|
|---|---|
|
|
34
36
|
| `client` | The program using it, recorded with each session; required |
|
|
35
37
|
| `ledgerPath` | SQLite file of the ledger (default below; `':memory:'` for tests). Opened when first needed |
|
|
36
|
-
| `providers` | Settings per provider name, on top of its adapter's `defaults`: `apiKey`, and the adapter's own
|
|
38
|
+
| `providers` | Settings per provider name, on top of its adapter's `defaults`: `apiKey`, `price`, `billingDelayMs`, `minIdleTimeoutSec` (see [Quota](#quota)) and the adapter's own (API base URLs, plan facts such as `monthlyHours`) |
|
|
37
39
|
| `adapters` | Further adapters by provider name, or replacements of built-in ones |
|
|
38
40
|
|
|
39
41
|
| Method | |
|
|
40
42
|
|---|---|
|
|
41
43
|
| `providers` | Names of the providers it can open sessions with |
|
|
42
44
|
| `checkReady(provider)` | Throws when the provider is unknown or has no API key |
|
|
43
|
-
| `openSession({ provider, proxyCountry, idleTimeoutSec, maxDurationMs })` | A new session,
|
|
44
|
-
| `
|
|
45
|
-
| `
|
|
45
|
+
| `openSession({ provider, proxyCountry, idleTimeoutSec, maxDurationMs })` | A new session, in the ledger from when the provider has created it. Throws `QuotaError` when the provider refuses it for the account's quota |
|
|
46
|
+
| `readQuota(provider, { maxAgeMs })` | Reads the provider's quota, unless a reading in progress or started at most `maxAgeMs` ago (default 0) can be used, a failed one too; then answers as `quota()` |
|
|
47
|
+
| `quota(provider)` | What is left now, from the last reading: `{ remaining, unbilled, unit, renewsAt }` (see [Quota](#quota)). Throws until the provider has been read, and while its last reading failed |
|
|
48
|
+
| `maxSessionCost(provider, options)` | What a session with these options costs at most, in the unit of the provider's quota |
|
|
49
|
+
| `syncUsage()` | Stores the providers' records (duration, proxy traffic) of ended sessions: `{ reported, failed: [{ provider, error }] }` |
|
|
46
50
|
| `usage({ since })` | Sessions opened since `since` (ms) per UTC day, provider and client: `sessions`, `minutes` (the provider's record, else the open-to-close timing), `untimed`, `unsynced`, `proxyBytes` |
|
|
47
51
|
| `close()` | Closes the sessions still open and the ledger |
|
|
48
52
|
|
|
49
|
-
A session
|
|
53
|
+
A session has `id`, `provider`, `egress`, `cdpUrl` (it may carry the API key: never log it), `openedAt` (when it was asked for), `closedAt` (when its provider released it), and:
|
|
50
54
|
|
|
51
|
-
- `connect()`: Playwright's `Browser` over its CDP connection, made once. A keep-alive command every 15 s keeps the provider's idle timeout from ending a session
|
|
52
|
-
- `newPage()`:
|
|
53
|
-
- `close()`: disconnects and releases it;
|
|
55
|
+
- `connect()`: Playwright's `Browser` over its CDP connection, made once. A keep-alive command every 15 s keeps the provider's idle timeout from ending a session in use.
|
|
56
|
+
- `newPage()`: a new page in the default context: the page a remote browser starts with cannot always be navigated.
|
|
57
|
+
- `close()`: disconnects and releases it; every call answers as the first. Throws when the provider could not release it: the provider ends it by itself, and the ledger counts it as lasting as long as it can.
|
|
54
58
|
|
|
55
59
|
A session and a `OneBrowser` are async-disposable: `await using session = await browsers.openSession({ provider: 'steel' })` closes the session at the end of its block (Node 24 or later; on Node 22 call `close()`).
|
|
56
60
|
|
|
@@ -59,7 +63,7 @@ A session and a `OneBrowser` are async-disposable: `await using session = await
|
|
|
59
63
|
| Option | Default | Meaning |
|
|
60
64
|
|---|---|---|
|
|
61
65
|
| `proxyCountry` | `null` | A country code: the provider's proxy in that country. `null`: its default egress |
|
|
62
|
-
| `idleTimeoutSec` | 60 | A session whose client is gone (its process was killed) ends this long after at the latest. At least 30 (twice the keep-alive interval) |
|
|
66
|
+
| `idleTimeoutSec` | 60 | A session whose client is gone (its process was killed) ends this long after at the latest, or after the provider's `minIdleTimeoutSec` if longer. At least 30 (twice the keep-alive interval) |
|
|
63
67
|
| `maxDurationMs` | 15 min | The session ends this long after it was opened at the latest: this process closes it then, and the provider's own limit is set to it where its API takes one |
|
|
64
68
|
|
|
65
69
|
`sessionOptions(options)` returns them with the defaults filled in and throws when one is unknown or out of range, so that a program can check its configuration before it opens any session.
|
|
@@ -78,15 +82,21 @@ How each provider meets them:
|
|
|
78
82
|
|
|
79
83
|
### Quota
|
|
80
84
|
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
|
86
|
-
|
|
87
|
-
| `
|
|
88
|
-
| `
|
|
89
|
-
| `
|
|
85
|
+
`quota()` is the last reading less `unbilled`: what the ledger's sessions that the reading does not show cost at the provider's `price`. A reading shows the sessions that ended `billingDelayMs` before it; the others are open ones, counted as lasting as long as they can (`maxDurationMs`, then `idleTimeoutSec`, at least the provider's `minIdleTimeoutSec`), and those ended since, counted as long as they lasted. So each session counts once from when its provider has created it, whichever program on the machine opened it, and a program can check before each session that the account pays for it (`maxSessionCost`). A provider that bills a session while it runs has part of it counted twice, on the safe side. Not counted: sessions opened through another machine's ledger, until the provider bills them, and proxy traffic billed by the GB.
|
|
86
|
+
|
|
87
|
+
`price` is in the quota's unit: `perHour`, `perSession` (default 0), the session time billed in whole `stepSec`, rounded up (default 1), and at least `minSec` (default 0). Where a provider does not document its rounding, the adapter takes the minute. `renewsAt` (ms) is when the quota renews by itself; `null` when only a top-up renews it. `price` and `billingDelayMs` are settings: a program on another plan sets its own.
|
|
88
|
+
|
|
89
|
+
| Provider | Quota, unit | Read from | `price` | `billingDelayMs` |
|
|
90
|
+
|---|---|---|---|---|
|
|
91
|
+
| `steel` | balance, USD | `GET /v1/credits/balance`, the endpoint of Steel's dashboard (not in its API docs) | $0.10 an hour (Launch plan), by the minute | 5 min |
|
|
92
|
+
| `tinyfish` | wallet, USD | [`GET /v1/wallet`](https://docs.tinyfish.ai/wallet); a session refused with 402 throws `QuotaError` | $0.002 a minute, by the minute | 5 min |
|
|
93
|
+
| `hyperbrowser` | credits | `GET /api/team/credit-info`, the endpoint of Hyperbrowser's SDK (not in its API docs) | 100 an hour, by the minute | 5 min |
|
|
94
|
+
| `lightpanda` | browser hours of the calendar month (`monthlyHours`, default: the free plan's) | the ledger's record of its sessions that have ended: its console shows the month's usage, but its token cannot read it. So the reading is whole while the account is used only through ledgers of this machine | 1 per browser hour, by the second | 0 |
|
|
95
|
+
| `kernel` | browser hours of the calendar month (`monthlyHours`, default: what the free plan's credits pay for), counted from the first of the month, 00:00 UTC | the uptime of the account's sessions this month, from `GET /browsers` (Kernel has no API for what its credits leave); a session refused with 403 (spending cap) throws `QuotaError` | 1 per browser hour, by the second | 5 min |
|
|
96
|
+
| `anchor` | credits | `GET /v1/billing`; a session refused with 402 throws `QuotaError` | 0.01 a session and 0.15 an hour, by the minute: above what its sessions' records bill (its docs: 0.05 an hour) | 5 min |
|
|
97
|
+
| `firecrawl` | credits; the plan's credits renew at the end of each billing period | [`GET /v2/team/credit-usage`](https://docs.firecrawl.dev/api-reference/endpoint/credit-usage); a session refused with 402 throws `QuotaError` | 2 a minute, in 30 s steps, at least a minute | 5 min |
|
|
98
|
+
|
|
99
|
+
Every provider read from its API showed a session's cost within about a minute of its end (2026-11); `billingDelayMs` leaves room for more.
|
|
90
100
|
|
|
91
101
|
### API keys
|
|
92
102
|
|
|
@@ -94,30 +104,30 @@ How each provider meets them:
|
|
|
94
104
|
|
|
95
105
|
### The ledger
|
|
96
106
|
|
|
97
|
-
One SQLite file per machine by default: `$ONEBROWSER_DB`, else `onebrowser/sessions.db` in `$XDG_DATA_HOME` (default `~/.local/share`)
|
|
107
|
+
One SQLite file per machine by default: `$ONEBROWSER_DB`, else `onebrowser/sessions.db` in `$XDG_DATA_HOME` (default `~/.local/share`), shared by every program on the machine under its own `client`. A session is recorded when its provider has created it (it is billed from then on), with when it ends at the latest; closed when its provider has released it; and completed with the provider's own record by `syncUsage()`. A ledger made by an earlier onebrowser is brought up to date when opened; one made by a later onebrowser is refused.
|
|
98
108
|
|
|
99
109
|
## Command line
|
|
100
110
|
|
|
101
111
|
The provider accounts and the ledger belong to the machine rather than to one program, so the `onebrowser` command reads them without a program of your own. It opens no sessions.
|
|
102
112
|
|
|
103
113
|
```sh
|
|
104
|
-
npx onebrowser quota # what each provider account with an API key has left
|
|
114
|
+
npx onebrowser quota # what each provider account with an API key has left, once its sessions not billed yet are paid for
|
|
105
115
|
npx onebrowser quota steel kernel # only these; one without an API key is a failed reading
|
|
106
116
|
npx onebrowser usage --days 7 # the ledger's sessions per UTC day, provider and client, today included
|
|
107
117
|
npx onebrowser usage --json --env-file .env
|
|
108
118
|
```
|
|
109
119
|
|
|
110
|
-
`usage` first fetches the providers' records of the sessions that have ended, as `syncUsage()` does. With `--json`, `quota` prints an array of what `
|
|
120
|
+
`usage` first fetches the providers' records of the sessions that have ended, as `syncUsage()` does. With `--json`, `quota` prints an array of what `readQuota()` returns, with the provider: `{ provider, remaining, unbilled, unit, renewsAt }`, or `{ provider, error }` for a failed reading; `usage` prints `{ ledger, since, sessions, failed }`, where `sessions` is what `usage()` returns and `failed` what `syncUsage()` could not fetch. API keys are read as by the library, from the environment (`--env-file PATH` loads a file into it first). Exit status: 0 done, 1 a reading failed, 2 the command line is wrong.
|
|
111
121
|
|
|
112
122
|
## Adding a provider
|
|
113
123
|
|
|
114
124
|
Write `src/providers/<name>.js` and register it in `ADAPTERS` (`src/providers/index.js`, where the contract is):
|
|
115
125
|
|
|
116
|
-
- `defaults`: its settings (API base URLs, plan facts).
|
|
126
|
+
- `defaults`: `price`, `billingDelayMs`, `minIdleTimeoutSec` if the provider takes no idle timeout as short as asked (see [Quota](#quota)), and its own settings (API base URLs, plan facts).
|
|
117
127
|
- `createSession(settings, apiKey, { proxyCountry, idleTimeoutSec, maxDurationMs }) → { id, cdpUrl, egress }`: maps the session options onto the provider's parameters; throws `QuotaError` when the provider refuses the session for the account's quota.
|
|
118
128
|
- `releaseSession(settings, apiKey, session)`.
|
|
119
129
|
- `sessionUsage(settings, apiKey, [{ id, openedAt, closedAt }]) → [{ id, durationMs, proxyBytes }]`: the record of those sessions that have ended: the provider's own, or the ledger's timing for a provider whose records its API key cannot read.
|
|
120
|
-
- `quota(settings, apiKey, history) → { remaining, unit, renewsAt? }`: `history.
|
|
130
|
+
- `quota(settings, apiKey, history) → { remaining, unit, renewsAt? }`: for a provider that cannot report its usage, `history.endedSessionMsSince(ms)` is the session time of the ledger's sessions that the reading shows.
|
|
121
131
|
|
|
122
132
|
Requests go through `src/providers/http.js`: every status other than 2xx is an `HttpError` (with `status`), so an adapter handles only the statuses that mean something of their own, such as a quota refusal.
|
|
123
133
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "onebrowser",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.0",
|
|
4
4
|
"description": "One interface to remote browser providers: sessions by provider name, a usage ledger, account quotas",
|
|
5
5
|
"keywords": ["browser", "remote-browser", "cdp", "playwright", "steel", "hyperbrowser", "lightpanda", "kernel", "tinyfish", "anchorbrowser", "firecrawl"],
|
|
6
6
|
"license": "MIT",
|
package/src/cli.js
CHANGED
|
@@ -6,6 +6,8 @@ const DAY_MS = 864e5;
|
|
|
6
6
|
|
|
7
7
|
const day = (ms) => new Date(ms).toISOString().slice(0, 10);
|
|
8
8
|
|
|
9
|
+
const round = (x) => Number(x.toFixed(2));
|
|
10
|
+
|
|
9
11
|
/** The command line is wrong: exit status 2 */
|
|
10
12
|
class CommandLineError extends Error {}
|
|
11
13
|
|
|
@@ -28,11 +30,11 @@ const COMMANDS = {
|
|
|
28
30
|
if (unknown.length) throw new CommandLineError(`unknown provider ${unknown.join(', ')} (expected ${browsers.providers.join(', ')})`);
|
|
29
31
|
const asked = names.length ? names : browsers.providers.filter((p) => isReady(browsers, p));
|
|
30
32
|
if (!asked.length) throw new Error(`no provider has an API key: set ${browsers.providers.map(apiKeyEnv).join(', ')}`);
|
|
31
|
-
const quotas = await Promise.all(asked.map((provider) => browsers.
|
|
33
|
+
const quotas = await Promise.all(asked.map((provider) => browsers.readQuota(provider).then((q) => ({ provider, ...q }), (e) => ({ provider, error: e.message }))));
|
|
32
34
|
if (json) out.write(`${JSON.stringify(quotas, null, 2)}\n`);
|
|
33
35
|
else {
|
|
34
36
|
for (const q of quotas) {
|
|
35
|
-
const text = q.error ? `unknown (${q.error})` : `${
|
|
37
|
+
const text = q.error ? `unknown (${q.error})` : `${round(q.remaining)} ${q.unit} left${q.unbilled ? ` (after ${round(q.unbilled)} not billed yet)` : ''}${q.renewsAt ? `, renews ${day(q.renewsAt)}` : ''}`;
|
|
36
38
|
out.write(`${q.provider.padEnd(14)} ${text}\n`);
|
|
37
39
|
}
|
|
38
40
|
}
|
|
@@ -66,7 +68,8 @@ const COMMANDS = {
|
|
|
66
68
|
const help = (providers) => `onebrowser: the quotas of remote browser provider accounts, and the sessions in this machine's ledger
|
|
67
69
|
|
|
68
70
|
Usage:
|
|
69
|
-
onebrowser quota [PROVIDER...] what each provider account has left
|
|
71
|
+
onebrowser quota [PROVIDER...] what each provider account has left, less what the sessions in the ledger
|
|
72
|
+
that it has not billed yet cost; default: every provider with an API key
|
|
70
73
|
onebrowser usage [--days N] sessions per UTC day, provider and client in the last N UTC days, today
|
|
71
74
|
included (default 30), after fetching the providers' records of ended sessions
|
|
72
75
|
|
package/src/cost.js
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* A provider's price (its `price` setting), in the unit its quota is read in:
|
|
3
|
+
* perHour for the session time
|
|
4
|
+
* perSession for each session, however short (default 0)
|
|
5
|
+
* stepSec the session time is billed in whole steps of this, rounded up (default 1)
|
|
6
|
+
* minSec and at least this much of it (default 0)
|
|
7
|
+
* It bounds what sessions cost, so where a provider does not document its rounding the adapter takes the minute.
|
|
8
|
+
*/
|
|
9
|
+
|
|
10
|
+
/** The price with its defaults filled in; throws when it is not one */
|
|
11
|
+
export function fullPrice(price) {
|
|
12
|
+
const { perHour, perSession = 0, stepSec = 1, minSec = 0, ...unknown } = price ?? {};
|
|
13
|
+
const fine = [perHour, perSession, minSec].every((x) => Number.isFinite(x) && x >= 0) && Number.isFinite(stepSec) && stepSec > 0;
|
|
14
|
+
if (!fine || Object.keys(unknown).length) throw new Error(`price must be { perHour, perSession, stepSec, minSec } of numbers ≥ 0 (stepSec > 0), not ${JSON.stringify(price)}`);
|
|
15
|
+
return { perHour, perSession, stepSec, minSec };
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
/** What a session that lasts durationMs costs at `price` (fullPrice) */
|
|
19
|
+
export function sessionCost({ perHour, perSession, stepSec, minSec }, durationMs) {
|
|
20
|
+
const billedSec = Math.max(minSec, Math.ceil(durationMs / 1000 / stepSec) * stepSec);
|
|
21
|
+
return perSession + (perHour * billedSec) / 3600;
|
|
22
|
+
}
|
package/src/index.js
CHANGED
|
@@ -1,7 +1,8 @@
|
|
|
1
|
+
import { sessionCost } from './cost.js';
|
|
1
2
|
import { readApiKey } from './credentials.js';
|
|
2
3
|
import { QuotaError } from './errors.js';
|
|
3
4
|
import { defaultLedgerPath, Ledger } from './ledger.js';
|
|
4
|
-
import { ADAPTERS, checkAdapter } from './providers/index.js';
|
|
5
|
+
import { ADAPTERS, checkAdapter, fullSettings } from './providers/index.js';
|
|
5
6
|
import { KEEP_ALIVE_MS, Session } from './session.js';
|
|
6
7
|
|
|
7
8
|
export { apiKeyEnv } from './credentials.js';
|
|
@@ -9,18 +10,18 @@ export { QuotaError };
|
|
|
9
10
|
|
|
10
11
|
/** Session options when openSession() is not given them */
|
|
11
12
|
const SESSION_DEFAULTS = {
|
|
12
|
-
proxyCountry: null,
|
|
13
|
-
idleTimeoutSec: 60,
|
|
13
|
+
proxyCountry: null,
|
|
14
|
+
idleTimeoutSec: 60,
|
|
14
15
|
maxDurationMs: 15 * 60_000,
|
|
15
16
|
};
|
|
16
17
|
|
|
17
18
|
/**
|
|
18
|
-
* The session options of openSession() with the defaults filled in
|
|
19
|
-
*
|
|
19
|
+
* The session options of openSession() with the defaults filled in; throws when one is unknown or out of range,
|
|
20
|
+
* so that a program can check its configuration before it opens any session:
|
|
20
21
|
* proxyCountry country code: the provider's proxy there; null: its default egress
|
|
21
|
-
* idleTimeoutSec a session whose client is gone ends this long after at the latest
|
|
22
|
-
*
|
|
23
|
-
* maxDurationMs
|
|
22
|
+
* idleTimeoutSec a session whose client is gone ends this long after at the latest, or after the provider's
|
|
23
|
+
* minIdleTimeoutSec if longer (≥ 2 × KEEP_ALIVE_MS)
|
|
24
|
+
* maxDurationMs a session ends this long after it was opened at the latest
|
|
24
25
|
*/
|
|
25
26
|
export function sessionOptions(options = {}) {
|
|
26
27
|
const unknown = Object.keys(options).filter((k) => !Object.hasOwn(SESSION_DEFAULTS, k));
|
|
@@ -33,26 +34,24 @@ export function sessionOptions(options = {}) {
|
|
|
33
34
|
return o;
|
|
34
35
|
}
|
|
35
36
|
|
|
37
|
+
/** How long a session lasts at most, whatever becomes of its process: closed at maxDurationMs, else ended by its provider once idle */
|
|
38
|
+
const sessionMsAtMost = (o, settings) => o.maxDurationMs + Math.max(o.idleTimeoutSec, settings.minIdleTimeoutSec) * 1000;
|
|
39
|
+
|
|
36
40
|
/**
|
|
37
|
-
* Remote browsers from several providers behind one interface: a session is asked for by provider name
|
|
38
|
-
*
|
|
39
|
-
*
|
|
40
|
-
*
|
|
41
|
-
* Which provider to use, how many sessions to hold and when to stop using one are the caller's decisions:
|
|
42
|
-
* OneBrowser opens the sessions it is asked for.
|
|
41
|
+
* Remote browsers from several providers behind one interface: a session is asked for by provider name. Every
|
|
42
|
+
* session is recorded in a ledger (ledger.js) shared by the programs on the machine, so that each provider
|
|
43
|
+
* account's usage, and what its quota has left at any moment, are known. Which provider to use, and when to
|
|
44
|
+
* stop using one, are the caller's decisions.
|
|
43
45
|
*
|
|
44
|
-
*
|
|
45
|
-
*
|
|
46
|
-
*
|
|
47
|
-
* opened when first needed
|
|
48
|
-
* providers settings per provider name: apiKey (else from the environment, see credentials.js), and the
|
|
49
|
-
* adapter's own settings on top of its defaults (providers/<name>.js)
|
|
50
|
-
* adapters further adapters by provider name, or replacements of built-in ones (providers/index.js)
|
|
46
|
+
* client: the program using it, recorded with each session; ledgerPath: the ledger's SQLite file, opened when
|
|
47
|
+
* first needed; providers: settings per provider name on top of its adapter's defaults (providers/index.js),
|
|
48
|
+
* apiKey among them (else credentials.js); adapters: further or replacement adapters by provider name.
|
|
51
49
|
*/
|
|
52
50
|
export class OneBrowser {
|
|
53
|
-
#providers = new Map(); // name → { adapter, settings
|
|
51
|
+
#providers = new Map(); // name → { adapter, settings, apiKey }
|
|
54
52
|
#ledger = null;
|
|
55
53
|
#open = new Set();
|
|
54
|
+
#readings = new Map(); // provider → { latest, done }: its latest quota reading, and the latest one done
|
|
56
55
|
|
|
57
56
|
constructor({ client, ledgerPath = defaultLedgerPath(), providers = {}, adapters = {} }) {
|
|
58
57
|
if (!client?.trim()) throw new Error('onebrowser: client must name the program using it');
|
|
@@ -60,14 +59,13 @@ export class OneBrowser {
|
|
|
60
59
|
this.ledgerPath = ledgerPath;
|
|
61
60
|
for (const [name, adapter] of Object.entries({ ...ADAPTERS, ...adapters })) {
|
|
62
61
|
checkAdapter(name, adapter);
|
|
63
|
-
const { apiKey, ...
|
|
64
|
-
this.#providers.set(name, { adapter, settings: { ...adapter.defaults, ...
|
|
62
|
+
const { apiKey, ...given } = providers[name] ?? {};
|
|
63
|
+
this.#providers.set(name, { adapter, settings: fullSettings(name, { ...adapter.defaults, ...given }), apiKey });
|
|
65
64
|
}
|
|
66
65
|
const unknown = Object.keys(providers).filter((name) => !this.#providers.has(name));
|
|
67
66
|
if (unknown.length) throw new Error(`onebrowser: no adapter for provider ${unknown.join(', ')}`);
|
|
68
67
|
}
|
|
69
68
|
|
|
70
|
-
/** Names of the providers it can open sessions with */
|
|
71
69
|
get providers() {
|
|
72
70
|
return [...this.#providers.keys()];
|
|
73
71
|
}
|
|
@@ -76,10 +74,14 @@ export class OneBrowser {
|
|
|
76
74
|
return (this.#ledger ??= new Ledger(this.ledgerPath));
|
|
77
75
|
}
|
|
78
76
|
|
|
79
|
-
|
|
80
|
-
#ready(provider) {
|
|
77
|
+
#known(provider) {
|
|
81
78
|
const p = this.#providers.get(provider);
|
|
82
79
|
if (!p) throw new Error(`unknown provider ${provider} (expected ${this.providers.join(', ')})`);
|
|
80
|
+
return p;
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
#ready(provider) {
|
|
84
|
+
const p = this.#known(provider);
|
|
83
85
|
return { ...p, apiKey: readApiKey(provider, p.apiKey) };
|
|
84
86
|
}
|
|
85
87
|
|
|
@@ -89,13 +91,13 @@ export class OneBrowser {
|
|
|
89
91
|
}
|
|
90
92
|
|
|
91
93
|
/**
|
|
92
|
-
* A new session with `provider
|
|
93
|
-
*
|
|
94
|
-
* quota.
|
|
94
|
+
* A new session with `provider` (options: sessionOptions()), in the ledger from when its provider has created
|
|
95
|
+
* it. Throws when the provider creates none; QuotaError when it refuses for the account's quota.
|
|
95
96
|
*/
|
|
96
97
|
async openSession({ provider, ...options }) {
|
|
97
98
|
const { adapter, settings, apiKey } = this.#ready(provider);
|
|
98
99
|
const o = sessionOptions(options);
|
|
100
|
+
const openedAt = Date.now();
|
|
99
101
|
let remote;
|
|
100
102
|
try {
|
|
101
103
|
remote = await adapter.createSession(settings, apiKey, o);
|
|
@@ -103,40 +105,91 @@ export class OneBrowser {
|
|
|
103
105
|
if (e instanceof QuotaError) throw e;
|
|
104
106
|
throw new Error(`failed to create a ${provider} session: ${e.message}`);
|
|
105
107
|
}
|
|
106
|
-
const session = new Session({
|
|
107
|
-
provider, remote, maxDurationMs: o.maxDurationMs,
|
|
108
|
-
release: async (s) => {
|
|
109
|
-
this.#open.delete(s);
|
|
110
|
-
this.#store.closed(s);
|
|
111
|
-
await adapter.releaseSession(settings, apiKey, s);
|
|
112
|
-
},
|
|
113
|
-
});
|
|
108
|
+
const session = new Session({ provider, remote, openedAt, maxDurationMs: o.maxDurationMs, release: (s) => this.#release(adapter, settings, apiKey, s) });
|
|
114
109
|
try {
|
|
115
|
-
this.#store.opened({ provider, id: session.id, client: this.client, openedAt:
|
|
110
|
+
this.#store.opened({ provider, id: session.id, client: this.client, openedAt, endsBy: openedAt + sessionMsAtMost(o, settings) });
|
|
116
111
|
} catch (e) {
|
|
117
|
-
await session.close().catch(() => {}); // a session the ledger
|
|
112
|
+
await session.close().catch(() => {}); // a session the ledger does not know is not handed out
|
|
118
113
|
throw e;
|
|
119
114
|
}
|
|
120
115
|
this.#open.add(session);
|
|
121
116
|
return session;
|
|
122
117
|
}
|
|
123
118
|
|
|
119
|
+
/** Releases `session` at its provider and records it closed: when it was released, so its whole billed time counts */
|
|
120
|
+
async #release(adapter, settings, apiKey, session) {
|
|
121
|
+
try {
|
|
122
|
+
await adapter.releaseSession(settings, apiKey, session);
|
|
123
|
+
const closedAt = Date.now();
|
|
124
|
+
this.#store.closed({ provider: session.provider, id: session.id, closedAt });
|
|
125
|
+
return closedAt;
|
|
126
|
+
} finally {
|
|
127
|
+
this.#open.delete(session);
|
|
128
|
+
}
|
|
129
|
+
}
|
|
130
|
+
|
|
124
131
|
/**
|
|
125
|
-
*
|
|
126
|
-
*
|
|
132
|
+
* Reads the account's quota with `provider` from the provider, unless a reading in progress or started at most
|
|
133
|
+
* maxAgeMs ago can be used (a failed one throws again); then answers as quota().
|
|
127
134
|
*/
|
|
128
|
-
async
|
|
135
|
+
async readQuota(provider, { maxAgeMs = 0 } = {}) {
|
|
136
|
+
const reading = this.#reading(provider, maxAgeMs);
|
|
137
|
+
await reading.settled;
|
|
138
|
+
if (reading.error) throw reading.error;
|
|
139
|
+
return this.quota(provider);
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
/**
|
|
143
|
+
* What is left of the account's quota now, in the provider's unit: { remaining, unbilled, unit, renewsAt }. The
|
|
144
|
+
* last reading shows the sessions that ended billingDelayMs before it; `unbilled` is what the ledger's others
|
|
145
|
+
* cost at the provider's price, open ones as if they lasted as long as they can, and remaining the reading less
|
|
146
|
+
* that. renewsAt (ms): when the quota renews by itself; null: only a top-up renews it. Throws until the
|
|
147
|
+
* provider has been read, and while its last reading failed.
|
|
148
|
+
*/
|
|
149
|
+
quota(provider) {
|
|
150
|
+
const { settings } = this.#known(provider);
|
|
151
|
+
const reading = this.#readings.get(provider)?.done;
|
|
152
|
+
if (!reading) throw new Error(`the quota of ${provider} has not been read yet (readQuota)`);
|
|
153
|
+
if (reading.error) throw reading.error;
|
|
154
|
+
const unbilled = this.#store.openSince(provider, reading.billedBefore).reduce((cost, s) => cost + sessionCost(settings.price, s.durationMs), 0);
|
|
155
|
+
const { remaining, unit, renewsAt = null } = reading.value;
|
|
156
|
+
return { remaining: remaining - unbilled, unbilled, unit, renewsAt };
|
|
157
|
+
}
|
|
158
|
+
|
|
159
|
+
/** The latest reading of the provider's quota if it is in progress or at most maxAgeMs old, else a new one */
|
|
160
|
+
#reading(provider, maxAgeMs) {
|
|
161
|
+
const readings = this.#readings.get(provider) ?? { latest: null, done: null };
|
|
162
|
+
this.#readings.set(provider, readings);
|
|
163
|
+
const { latest } = readings;
|
|
164
|
+
const inProgress = latest && latest !== readings.done;
|
|
165
|
+
if (inProgress || (latest && Date.now() - latest.startedAt <= maxAgeMs)) return latest;
|
|
129
166
|
const { adapter, settings, apiKey } = this.#ready(provider);
|
|
130
|
-
const
|
|
131
|
-
|
|
132
|
-
|
|
167
|
+
const startedAt = Date.now();
|
|
168
|
+
const reading = { startedAt, billedBefore: startedAt - settings.billingDelayMs, value: null, error: null };
|
|
169
|
+
readings.latest = reading;
|
|
170
|
+
reading.settled = (async () => {
|
|
171
|
+
try {
|
|
172
|
+
const q = await adapter.quota(settings, apiKey, this.#store.history(provider, reading.billedBefore));
|
|
173
|
+
if (!(Number.isFinite(q?.remaining) && typeof q.unit === 'string')) throw new Error(`${provider} answered no quota: ${JSON.stringify(q)}`);
|
|
174
|
+
reading.value = q;
|
|
175
|
+
} catch (e) {
|
|
176
|
+
reading.error = e;
|
|
177
|
+
}
|
|
178
|
+
readings.done = reading;
|
|
179
|
+
})();
|
|
180
|
+
return reading;
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
/** What a session with these options costs at most, in the unit of the provider's quota */
|
|
184
|
+
maxSessionCost(provider, options) {
|
|
185
|
+
const { settings } = this.#known(provider);
|
|
186
|
+
return sessionCost(settings.price, sessionMsAtMost(sessionOptions(options), settings));
|
|
133
187
|
}
|
|
134
188
|
|
|
135
189
|
/**
|
|
136
|
-
* Stores
|
|
137
|
-
*
|
|
138
|
-
*
|
|
139
|
-
* to the programs that have one. Returns { reported: sessions completed, failed: [{ provider, error }] }.
|
|
190
|
+
* Stores the providers' records (duration, proxy traffic) of the ledger's ended sessions that have none yet:
|
|
191
|
+
* { reported, failed: [{ provider, error }] }. What cannot be fetched now, and the sessions of providers it has
|
|
192
|
+
* no adapter for, are left for a later call.
|
|
140
193
|
*/
|
|
141
194
|
async syncUsage() {
|
|
142
195
|
const store = this.#store;
|
|
@@ -157,10 +210,8 @@ export class OneBrowser {
|
|
|
157
210
|
|
|
158
211
|
/**
|
|
159
212
|
* Sessions opened since `since` (ms), per UTC day, provider and client:
|
|
160
|
-
* [{ day, provider, client, sessions, minutes, untimed, unsynced, proxyBytes }].
|
|
161
|
-
*
|
|
162
|
-
* session; untimed: sessions with neither (their process ended before closing them and the provider has not
|
|
163
|
-
* reported them yet); unsynced: sessions without the provider's record.
|
|
213
|
+
* [{ day, provider, client, sessions, minutes, untimed, unsynced, proxyBytes }]. minutes: the provider's record
|
|
214
|
+
* (syncUsage), else the open-to-close timing; untimed: sessions with neither; unsynced: those without the record.
|
|
164
215
|
*/
|
|
165
216
|
usage({ since }) {
|
|
166
217
|
return this.#store.usage(since);
|
|
@@ -173,7 +224,6 @@ export class OneBrowser {
|
|
|
173
224
|
this.#ledger = null;
|
|
174
225
|
}
|
|
175
226
|
|
|
176
|
-
/** close(), for `await using browsers = new OneBrowser(…)` */
|
|
177
227
|
[Symbol.asyncDispose]() {
|
|
178
228
|
return this.close();
|
|
179
229
|
}
|
package/src/ledger.js
CHANGED
|
@@ -14,41 +14,85 @@ export function defaultLedgerPath() {
|
|
|
14
14
|
return path.join(data, 'onebrowser', 'sessions.db');
|
|
15
15
|
}
|
|
16
16
|
|
|
17
|
-
/**
|
|
18
|
-
* sessions every remote browser session: recorded when its provider has created it (it is billed from then on,
|
|
19
|
-
* even if connecting fails), closed when released, and completed with the provider's own record
|
|
20
|
-
* (duration, proxy traffic) once the session has ended and syncUsage has fetched it
|
|
21
|
-
*/
|
|
17
|
+
/** Every remote browser session, from when its provider has created it: it is billed from then on, even if connecting fails */
|
|
22
18
|
const SCHEMA = `
|
|
23
|
-
CREATE TABLE
|
|
19
|
+
CREATE TABLE sessions (
|
|
24
20
|
provider TEXT NOT NULL,
|
|
25
21
|
id TEXT NOT NULL, -- the session id its provider gave it
|
|
26
22
|
client TEXT NOT NULL, -- the program that opened it (OneBrowser's client)
|
|
27
|
-
opened_at TEXT NOT NULL, -- when
|
|
28
|
-
|
|
23
|
+
opened_at TEXT NOT NULL, -- when it was asked for; its provider created it right after
|
|
24
|
+
ends_by TEXT NOT NULL, -- when it has ended at the latest, whatever becomes of its process
|
|
25
|
+
closed_at TEXT, -- when its provider released it; empty until then, and if it never was
|
|
29
26
|
duration_ms INTEGER, -- the provider's record, once the session has ended
|
|
30
27
|
proxy_bytes INTEGER, -- proxy traffic in the provider's record
|
|
31
28
|
synced_at TEXT, -- when the provider's record was stored; empty = not yet
|
|
32
29
|
PRIMARY KEY (provider, id)
|
|
33
30
|
);
|
|
34
|
-
CREATE INDEX
|
|
31
|
+
CREATE INDEX idx_sessions_opened ON sessions(provider, opened_at);
|
|
35
32
|
`;
|
|
36
33
|
|
|
37
|
-
/**
|
|
38
|
-
|
|
34
|
+
/**
|
|
35
|
+
* The version of SCHEMA (PRAGMA user_version), and the step to it from each earlier one: a ledger outlives the
|
|
36
|
+
* onebrowser that made it. onebrowser 0.1–0.3 made version 1 and left user_version 0.
|
|
37
|
+
*/
|
|
38
|
+
const VERSION = 2;
|
|
39
|
+
const UPGRADES = {
|
|
40
|
+
// 1 → 2: ends_by, as if the sessions of 1 had the default session options (15 min, then 60 s idle). Each step
|
|
41
|
+
// makes its version as that version was, whatever SCHEMA has become since.
|
|
42
|
+
1: `ALTER TABLE sessions RENAME TO sessions_1;
|
|
43
|
+
DROP INDEX idx_sessions_opened;
|
|
44
|
+
CREATE TABLE sessions (
|
|
45
|
+
provider TEXT NOT NULL, id TEXT NOT NULL, client TEXT NOT NULL, opened_at TEXT NOT NULL, ends_by TEXT NOT NULL,
|
|
46
|
+
closed_at TEXT, duration_ms INTEGER, proxy_bytes INTEGER, synced_at TEXT, PRIMARY KEY (provider, id)
|
|
47
|
+
);
|
|
48
|
+
CREATE INDEX idx_sessions_opened ON sessions(provider, opened_at);
|
|
49
|
+
INSERT INTO sessions (provider, id, client, opened_at, ends_by, closed_at, duration_ms, proxy_bytes, synced_at)
|
|
50
|
+
SELECT provider, id, client, opened_at, strftime('%Y-%m-%dT%H:%M:%fZ', opened_at, '+16 minutes'), closed_at, duration_ms, proxy_bytes, synced_at
|
|
51
|
+
FROM sessions_1;
|
|
52
|
+
DROP TABLE sessions_1;`,
|
|
53
|
+
};
|
|
54
|
+
|
|
55
|
+
/** ms from one ISO time to another */
|
|
56
|
+
const msBetween = (from, to) => `CAST(ROUND((julianday(${to}) - julianday(${from})) * 86400000) AS INTEGER)`;
|
|
57
|
+
|
|
58
|
+
/** How long a session lasted: the provider's record, else the open-to-close timing of the process that opened it */
|
|
59
|
+
const SESSION_MS = `COALESCE(duration_ms, ${msBetween('opened_at', 'closed_at')})`;
|
|
60
|
+
|
|
61
|
+
/** How long a session lasts at most: as SESSION_MS, or until it ends at the latest while it has no timing */
|
|
62
|
+
const SESSION_MS_AT_MOST = `COALESCE(${SESSION_MS}, ${msBetween('opened_at', 'ends_by')})`;
|
|
63
|
+
|
|
64
|
+
/** When a session has ended or ends at the latest */
|
|
65
|
+
const ENDED_AT = 'COALESCE(closed_at, ends_by)';
|
|
39
66
|
|
|
40
67
|
const iso = (ms) => new Date(ms).toISOString();
|
|
41
68
|
const int = (x) => (x === null || x === undefined || !Number.isFinite(Number(x)) ? null : Math.trunc(Number(x)));
|
|
42
69
|
|
|
43
70
|
export class Ledger {
|
|
71
|
+
#file;
|
|
44
72
|
#db;
|
|
45
73
|
#stmts = new Map();
|
|
46
74
|
|
|
47
75
|
constructor(file) {
|
|
48
76
|
if (file !== ':memory:') fs.mkdirSync(path.dirname(file), { recursive: true });
|
|
77
|
+
this.#file = file;
|
|
49
78
|
this.#db = new DatabaseSync(file);
|
|
50
79
|
this.#db.exec('PRAGMA journal_mode=WAL; PRAGMA busy_timeout=5000;'); // several processes share it
|
|
51
|
-
this.#
|
|
80
|
+
this.#migrate();
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
/** Makes the ledger, or brings an earlier one up to date, in one transaction: another process may be opening it too */
|
|
84
|
+
#migrate() {
|
|
85
|
+
const version = () => this.#db.prepare('PRAGMA user_version').get().user_version;
|
|
86
|
+
if (version() === VERSION) return;
|
|
87
|
+
this.tx(() => {
|
|
88
|
+
if (!this.#db.prepare("SELECT 1 FROM sqlite_schema WHERE type = 'table' AND name = 'sessions'").get()) this.#db.exec(SCHEMA);
|
|
89
|
+
else {
|
|
90
|
+
const from = version() || 1;
|
|
91
|
+
if (from > VERSION) throw new Error(`onebrowser: the ledger ${this.#file} was made by a later onebrowser (schema ${from}; this one knows ${VERSION})`);
|
|
92
|
+
for (let v = from; v < VERSION; v++) this.#db.exec(UPGRADES[v]);
|
|
93
|
+
}
|
|
94
|
+
this.#db.exec(`PRAGMA user_version = ${VERSION}`);
|
|
95
|
+
});
|
|
52
96
|
}
|
|
53
97
|
|
|
54
98
|
#q(sql) {
|
|
@@ -57,23 +101,29 @@ export class Ledger {
|
|
|
57
101
|
return s;
|
|
58
102
|
}
|
|
59
103
|
|
|
60
|
-
opened({ provider, id, client, openedAt }) {
|
|
61
|
-
this.#q('INSERT INTO sessions (provider, id, client, opened_at) VALUES (
|
|
104
|
+
opened({ provider, id, client, openedAt, endsBy }) {
|
|
105
|
+
this.#q('INSERT INTO sessions (provider, id, client, opened_at, ends_by) VALUES (?,?,?,?,?)').run(provider, id, client, iso(openedAt), iso(endsBy));
|
|
62
106
|
}
|
|
63
107
|
|
|
64
108
|
closed({ provider, id, closedAt }) {
|
|
65
109
|
this.#q('UPDATE sessions SET closed_at=? WHERE provider=? AND id=? AND closed_at IS NULL').run(iso(closedAt), provider, id);
|
|
66
110
|
}
|
|
67
111
|
|
|
68
|
-
/** The
|
|
69
|
-
history(provider) {
|
|
112
|
+
/** The provider's sessions that a quota reading shows, those that ended before `billedBefore` (ms); openSince() has the rest */
|
|
113
|
+
history(provider, billedBefore) {
|
|
70
114
|
return {
|
|
71
|
-
/**
|
|
72
|
-
|
|
115
|
+
/** Their session time (ms) since `since` (ms), each as long as it can have lasted while it has no timing */
|
|
116
|
+
endedSessionMsSince: (since) =>
|
|
117
|
+
this.#q(`SELECT COALESCE(SUM(${SESSION_MS_AT_MOST}), 0) ms FROM sessions WHERE provider=? AND opened_at >= ? AND ${ENDED_AT} < ?`).get(provider, iso(since), iso(billedBefore)).ms,
|
|
73
118
|
};
|
|
74
119
|
}
|
|
75
120
|
|
|
76
|
-
/**
|
|
121
|
+
/** The provider's sessions open at any time since `at` (ms): [{ durationMs }], as long as they can last while untimed */
|
|
122
|
+
openSince(provider, at) {
|
|
123
|
+
return this.#q(`SELECT ${SESSION_MS_AT_MOST} durationMs FROM sessions WHERE provider=? AND ${ENDED_AT} >= ?`).all(provider, iso(at));
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
/** Sessions of `providers` without their provider's record yet: [{ provider, id, openedAt, closedAt }] (ISO times) */
|
|
77
127
|
awaitingUsage(providers) {
|
|
78
128
|
return this.#q(
|
|
79
129
|
'SELECT provider, id, opened_at openedAt, closed_at closedAt FROM sessions WHERE synced_at IS NULL AND provider IN (SELECT value FROM json_each(?)) ORDER BY opened_at',
|
|
@@ -84,8 +134,9 @@ export class Ledger {
|
|
|
84
134
|
this.#q('UPDATE sessions SET duration_ms=?, proxy_bytes=?, synced_at=? WHERE provider=? AND id=?').run(int(durationMs), int(proxyBytes), iso(Date.now()), provider, id);
|
|
85
135
|
}
|
|
86
136
|
|
|
137
|
+
/** Runs fn in a transaction that takes the write lock at once: another process may be writing too */
|
|
87
138
|
tx(fn) {
|
|
88
|
-
this.#db.exec('BEGIN');
|
|
139
|
+
this.#db.exec('BEGIN IMMEDIATE');
|
|
89
140
|
try {
|
|
90
141
|
const r = fn();
|
|
91
142
|
this.#db.exec('COMMIT');
|
package/src/providers/anchor.js
CHANGED
|
@@ -12,6 +12,11 @@ import { json } from './http.js';
|
|
|
12
12
|
*/
|
|
13
13
|
export const defaults = {
|
|
14
14
|
apiBase: 'https://api.anchorbrowser.io/v1',
|
|
15
|
+
// credits ($1 each): $0.01 a browser and, by its docs, $0.05 an hour by the minute; the sessions' own records
|
|
16
|
+
// (credits_used) bill more, about $0.06 an hour and a step of $0.003 in the second minute, which this covers
|
|
17
|
+
price: { perSession: 0.01, perHour: 0.15, stepSec: 60 },
|
|
18
|
+
billingDelayMs: 5 * 60_000, // seen within 10 s
|
|
19
|
+
minIdleTimeoutSec: 60, // idle_timeout is in whole minutes, at least 1
|
|
15
20
|
};
|
|
16
21
|
|
|
17
22
|
const headers = (apiKey) => ({ 'anchor-api-key': apiKey });
|
|
@@ -13,6 +13,8 @@ import { json } from './http.js';
|
|
|
13
13
|
*/
|
|
14
14
|
export const defaults = {
|
|
15
15
|
apiBase: 'https://api.firecrawl.dev/v2',
|
|
16
|
+
price: { perHour: 120, stepSec: 30, minSec: 60 }, // credits: 2 a minute, in whole credits rounded up, at least 1 minute
|
|
17
|
+
billingDelayMs: 5 * 60_000, // seen within 10 s
|
|
16
18
|
};
|
|
17
19
|
|
|
18
20
|
const TTL_SEC = { min: 30, max: 3600 }; // what Firecrawl takes for ttl; activityTtl has the same maximum
|
|
@@ -9,6 +9,8 @@ import { json } from './http.js';
|
|
|
9
9
|
*/
|
|
10
10
|
export const defaults = {
|
|
11
11
|
apiBase: 'https://api.hyperbrowser.ai/api',
|
|
12
|
+
price: { perHour: 100, stepSec: 60 }, // credits; its rounding is not documented
|
|
13
|
+
billingDelayMs: 5 * 60_000, // billed while the session runs, seen within 10 s of its end
|
|
12
14
|
};
|
|
13
15
|
|
|
14
16
|
const headers = (apiKey) => ({ 'x-api-key': apiKey });
|
package/src/providers/index.js
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { fullPrice } from '../cost.js';
|
|
1
2
|
import * as anchor from './anchor.js';
|
|
2
3
|
import * as firecrawl from './firecrawl.js';
|
|
3
4
|
import * as hyperbrowser from './hyperbrowser.js';
|
|
@@ -7,26 +8,22 @@ import * as steel from './steel.js';
|
|
|
7
8
|
import * as tinyfish from './tinyfish.js';
|
|
8
9
|
|
|
9
10
|
/**
|
|
10
|
-
* Provider adapters
|
|
11
|
-
* defaults
|
|
12
|
-
*
|
|
13
|
-
*
|
|
14
|
-
*
|
|
15
|
-
*
|
|
16
|
-
*
|
|
17
|
-
* session
|
|
18
|
-
* releaseSession(settings, apiKey, session)
|
|
11
|
+
* Provider adapters by name: the only provider-specific code. Each exports:
|
|
12
|
+
* defaults its settings, which OneBrowser's `providers` option overrides:
|
|
13
|
+
* price what its sessions cost, in the unit of its quota (cost.js)
|
|
14
|
+
* billingDelayMs how long after a session has ended its quota reading shows it, at the latest
|
|
15
|
+
* minIdleTimeoutSec the shortest idle timeout it takes (default 0)
|
|
16
|
+
* and its own API base URLs, plan facts such as monthlyHours
|
|
17
|
+
* createSession(settings, apiKey, { proxyCountry, idleTimeoutSec, maxDurationMs }) → { id, cdpUrl, egress }
|
|
18
|
+
* throws QuotaError (errors.js) when the provider refuses the session for the account's quota
|
|
19
|
+
* releaseSession(settings, apiKey, session)
|
|
19
20
|
* sessionUsage(settings, apiKey, [{ id, openedAt, closedAt }]) → [{ id, durationMs, proxyBytes }]
|
|
20
|
-
* the record of those
|
|
21
|
-
* API key cannot read, the ledger's timing (closedAt is empty for a session no process closed); throws on
|
|
22
|
-
* failure
|
|
21
|
+
* the provider's record of those that have ended, or the ledger's timing where its API key cannot read one
|
|
23
22
|
* quota(settings, apiKey, history) → { remaining, unit, renewsAt? }
|
|
24
|
-
* what
|
|
25
|
-
*
|
|
26
|
-
*
|
|
27
|
-
*
|
|
28
|
-
* Their requests go through http.js, which turns every status other than 2xx into an HttpError: an adapter
|
|
29
|
-
* handles only the statuses that mean something of their own.
|
|
23
|
+
* what the account has left in the provider's unit, and when it renews by itself (ms); history: the
|
|
24
|
+
* ledger's sessions that the reading shows (Ledger.history), for a provider that cannot count its usage
|
|
25
|
+
* Every function throws when it fails. Requests go through http.js, which throws HttpError for a status other
|
|
26
|
+
* than 2xx.
|
|
30
27
|
*/
|
|
31
28
|
export const ADAPTERS = { tinyfish, steel, hyperbrowser, lightpanda, kernel, anchor, firecrawl };
|
|
32
29
|
|
|
@@ -37,3 +34,16 @@ export function checkAdapter(name, adapter) {
|
|
|
37
34
|
const missing = CONTRACT.filter((f) => typeof adapter?.[f] !== 'function');
|
|
38
35
|
if (missing.length) throw new Error(`provider ${name}: its adapter lacks ${missing.join(', ')}`);
|
|
39
36
|
}
|
|
37
|
+
|
|
38
|
+
/** An adapter's settings with the defaults of those every adapter has; throws when one of them is missing or wrong */
|
|
39
|
+
export function fullSettings(name, { price, billingDelayMs, minIdleTimeoutSec = 0, ...own }) {
|
|
40
|
+
const atLeast0 = (x) => Number.isFinite(x) && x >= 0;
|
|
41
|
+
try {
|
|
42
|
+
const full = fullPrice(price);
|
|
43
|
+
if (!atLeast0(billingDelayMs)) throw new Error(`billingDelayMs must be a number ≥ 0, not ${billingDelayMs}`);
|
|
44
|
+
if (!atLeast0(minIdleTimeoutSec)) throw new Error(`minIdleTimeoutSec must be a number ≥ 0, not ${minIdleTimeoutSec}`);
|
|
45
|
+
return { ...own, price: full, billingDelayMs, minIdleTimeoutSec };
|
|
46
|
+
} catch (e) {
|
|
47
|
+
throw new Error(`provider ${name}: ${e.message}`);
|
|
48
|
+
}
|
|
49
|
+
}
|
package/src/providers/kernel.js
CHANGED
|
@@ -14,6 +14,8 @@ import { utcMonth } from './month.js';
|
|
|
14
14
|
export const defaults = {
|
|
15
15
|
apiBase: 'https://api.onkernel.com',
|
|
16
16
|
monthlyHours: 80, // headless browser hours a month that the free plan's credits pay for, rounded down
|
|
17
|
+
price: { perHour: 1 }, // the quota is in headless browser hours, billed by the second
|
|
18
|
+
billingDelayMs: 5 * 60_000, // seen within 10 s
|
|
17
19
|
};
|
|
18
20
|
|
|
19
21
|
const headers = (apiKey) => ({ Authorization: `Bearer ${apiKey}` });
|
|
@@ -42,8 +44,8 @@ export async function releaseSession(settings, apiKey, session) {
|
|
|
42
44
|
/**
|
|
43
45
|
* Browser hours left of the month's monthlyHours, after the uptime of every session of the account this month:
|
|
44
46
|
* Kernel's own records, which include sessions opened by any program. The sessions are listed newest first, so
|
|
45
|
-
* the listing stops at a page that holds none of this month.
|
|
46
|
-
*
|
|
47
|
+
* the listing stops at a page that holds none of this month. Kernel counts usage from the first of each month,
|
|
48
|
+
* 00:00 UTC (https://www.kernel.sh/docs/info/spending-caps), and has no API that reads what its credits leave.
|
|
47
49
|
*/
|
|
48
50
|
export async function quota(settings, apiKey) {
|
|
49
51
|
const month = utcMonth();
|
|
@@ -13,6 +13,8 @@ const MAX_CONNECTION_MS = 15 * 60_000;
|
|
|
13
13
|
export const defaults = {
|
|
14
14
|
connectBase: 'wss://euwest.cloud.lightpanda.io/ws',
|
|
15
15
|
monthlyHours: 10, // browser hours a month on the free plan
|
|
16
|
+
price: { perHour: 1 }, // the quota is in browser hours, billed by the second
|
|
17
|
+
billingDelayMs: 0, // the quota is the ledger's record
|
|
16
18
|
};
|
|
17
19
|
|
|
18
20
|
export async function createSession(settings, apiKey, { proxyCountry }) {
|
|
@@ -30,13 +32,13 @@ export async function releaseSession() {}
|
|
|
30
32
|
|
|
31
33
|
/**
|
|
32
34
|
* Browser hours left of the month's monthlyHours. Lightpanda's console shows the month's usage, but the token
|
|
33
|
-
* cannot read it: the month's usage is the ledger's record of the sessions (history), which is
|
|
34
|
-
* the account is used through ledgers of this machine only. The hours are assumed to renew with
|
|
35
|
-
* month (UTC).
|
|
35
|
+
* cannot read it: the month's usage is the ledger's record of the sessions that have ended (history), which is
|
|
36
|
+
* all of it while the account is used through ledgers of this machine only. The hours are assumed to renew with
|
|
37
|
+
* each calendar month (UTC).
|
|
36
38
|
*/
|
|
37
39
|
export async function quota(settings, apiKey, history) {
|
|
38
40
|
const month = utcMonth();
|
|
39
|
-
const usedHours = history.
|
|
41
|
+
const usedHours = history.endedSessionMsSince(month.start) / 3_600_000;
|
|
40
42
|
return { remaining: settings.monthlyHours - usedHours, unit: 'hours', renewsAt: month.next };
|
|
41
43
|
}
|
|
42
44
|
|
package/src/providers/steel.js
CHANGED
|
@@ -8,6 +8,8 @@ import { json } from './http.js';
|
|
|
8
8
|
export const defaults = {
|
|
9
9
|
apiBase: 'https://api.steel.dev/v1',
|
|
10
10
|
connectBase: 'wss://connect.steel.dev',
|
|
11
|
+
price: { perHour: 0.1, stepSec: 60 }, // USD on the Launch plan, by the minute rounded up
|
|
12
|
+
billingDelayMs: 5 * 60_000, // seen within 11 s
|
|
11
13
|
};
|
|
12
14
|
|
|
13
15
|
const headers = (apiKey) => ({ 'steel-api-key': apiKey });
|
|
@@ -10,6 +10,8 @@ import { json } from './http.js';
|
|
|
10
10
|
export const defaults = {
|
|
11
11
|
apiBase: 'https://api.browser.tinyfish.ai',
|
|
12
12
|
walletUrl: 'https://agent.tinyfish.ai/v1/wallet',
|
|
13
|
+
price: { perHour: 0.12, stepSec: 60 }, // USD: $0.002 a minute, proxy included; its rounding is not documented
|
|
14
|
+
billingDelayMs: 5 * 60_000, // seen within 64 s
|
|
13
15
|
};
|
|
14
16
|
|
|
15
17
|
const headers = (apiKey) => ({ 'X-API-Key': apiKey });
|
package/src/session.js
CHANGED
|
@@ -1,45 +1,39 @@
|
|
|
1
|
-
/**
|
|
2
|
-
* A CDP command this often while connected keeps a provider's idle timeout from ending a session that is in use
|
|
3
|
-
* (also while a page waits long for an answer). OneBrowser requires idleTimeoutSec to be at least twice as long.
|
|
4
|
-
*/
|
|
1
|
+
/** A CDP command this often keeps a provider's idle timeout from ending a session in use; idleTimeoutSec must be at least twice as long */
|
|
5
2
|
export const KEEP_ALIVE_MS = 15_000;
|
|
6
3
|
|
|
7
4
|
/** Playwright errors carry a multi-line call log; the first line says what failed */
|
|
8
5
|
const firstLine = (e) => String(e.message).split('\n')[0];
|
|
9
6
|
|
|
10
7
|
/**
|
|
11
|
-
*
|
|
12
|
-
*
|
|
13
|
-
*
|
|
14
|
-
* its
|
|
15
|
-
*
|
|
16
|
-
*
|
|
17
|
-
* id, provider, egress the provider's session id, its name, and the egress the session uses
|
|
18
|
-
* cdpUrl CDP WebSocket URL (it may carry the API key: never log it)
|
|
19
|
-
* openedAt, closedAt ms; closedAt is null while open
|
|
8
|
+
* A remote browser session (OneBrowser.openSession), used over CDP: cdpUrl with any client (it may carry the API
|
|
9
|
+
* key: never log it), or connect() / newPage() with Playwright. This process closes it maxDurationMs after it was
|
|
10
|
+
* opened; one whose process died is ended by its provider idleTimeoutSec (at least its minIdleTimeoutSec) after
|
|
11
|
+
* its client left.
|
|
12
|
+
* openedAt: when it was asked for; closedAt: when its provider released it, null until then.
|
|
20
13
|
*/
|
|
21
14
|
export class Session {
|
|
22
15
|
#release;
|
|
23
|
-
#
|
|
16
|
+
#closing = null;
|
|
17
|
+
#browser = null; // a promise of Playwright's Browser
|
|
24
18
|
#keepAlive = null;
|
|
25
19
|
#deadline;
|
|
26
20
|
|
|
27
|
-
/** release(session)
|
|
28
|
-
constructor({ provider, remote, maxDurationMs, release }) {
|
|
21
|
+
/** release(session) → closedAt: releases it at its provider and records it closed (OneBrowser) */
|
|
22
|
+
constructor({ provider, remote, openedAt, maxDurationMs, release }) {
|
|
29
23
|
this.provider = provider;
|
|
30
24
|
this.id = remote.id;
|
|
31
25
|
this.egress = remote.egress;
|
|
32
26
|
this.cdpUrl = remote.cdpUrl;
|
|
33
|
-
this.openedAt =
|
|
27
|
+
this.openedAt = openedAt;
|
|
34
28
|
this.closedAt = null;
|
|
35
29
|
this.#release = release;
|
|
36
|
-
this.#deadline = setTimeout(() => this.close().catch(() => {}),
|
|
30
|
+
this.#deadline = setTimeout(() => this.close().catch(() => {}), openedAt + maxDurationMs - Date.now());
|
|
37
31
|
this.#deadline.unref();
|
|
38
32
|
}
|
|
39
33
|
|
|
40
|
-
/** Playwright's Browser over this session's CDP connection, made once;
|
|
34
|
+
/** Playwright's Browser over this session's CDP connection, made once; a keep-alive runs while it is connected */
|
|
41
35
|
connect() {
|
|
42
|
-
if (this
|
|
36
|
+
if (this.#closing) return Promise.reject(new Error(`${this.provider} session ${this.id} is closed`));
|
|
43
37
|
this.#browser ??= this.#connect();
|
|
44
38
|
return this.#browser;
|
|
45
39
|
}
|
|
@@ -54,7 +48,7 @@ export class Session {
|
|
|
54
48
|
}
|
|
55
49
|
try {
|
|
56
50
|
const cdp = await browser.newBrowserCDPSession();
|
|
57
|
-
if (this
|
|
51
|
+
if (this.#closing) throw new Error(`${this.provider} session ${this.id} is closed`);
|
|
58
52
|
this.#keepAlive = setInterval(() => cdp.send('Browser.getVersion').catch(() => {}), KEEP_ALIVE_MS);
|
|
59
53
|
this.#keepAlive.unref();
|
|
60
54
|
} catch (e) {
|
|
@@ -64,10 +58,7 @@ export class Session {
|
|
|
64
58
|
return browser;
|
|
65
59
|
}
|
|
66
60
|
|
|
67
|
-
/**
|
|
68
|
-
* A new page in the browser's default context. Always a new one: the page a remote browser starts with is not
|
|
69
|
-
* always one that can be navigated.
|
|
70
|
-
*/
|
|
61
|
+
/** A new page in the default context: the page a remote browser starts with cannot always be navigated */
|
|
71
62
|
async newPage() {
|
|
72
63
|
const browser = await this.connect();
|
|
73
64
|
const context = browser.contexts()[0] ?? (await browser.newContext());
|
|
@@ -75,22 +66,20 @@ export class Session {
|
|
|
75
66
|
}
|
|
76
67
|
|
|
77
68
|
/**
|
|
78
|
-
* Disconnects and releases the session
|
|
79
|
-
*
|
|
80
|
-
* idleTimeoutSec after the disconnect.
|
|
69
|
+
* Disconnects and releases the session; every call answers as the first. Throws when its provider could not
|
|
70
|
+
* release it: the ledger then counts it as lasting as long as it can, as its provider ends it by itself.
|
|
81
71
|
*/
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
72
|
+
close() {
|
|
73
|
+
return (this.#closing ??= this.#close());
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
async #close() {
|
|
85
77
|
clearTimeout(this.#deadline);
|
|
86
78
|
clearInterval(this.#keepAlive);
|
|
87
|
-
|
|
88
|
-
this
|
|
89
|
-
await browser?.then((b) => b.close()).catch(() => {});
|
|
90
|
-
await this.#release(this);
|
|
79
|
+
await this.#browser?.then((b) => b.close()).catch(() => {});
|
|
80
|
+
this.closedAt = await this.#release(this);
|
|
91
81
|
}
|
|
92
82
|
|
|
93
|
-
/** close(): `await using session = await browsers.openSession(…)` releases it at the end of the block */
|
|
94
83
|
[Symbol.asyncDispose]() {
|
|
95
84
|
return this.close();
|
|
96
85
|
}
|