onebrowser 0.5.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +37 -27
- package/package.json +2 -2
- package/src/account.js +139 -0
- package/src/cli.js +30 -25
- package/src/credentials.js +17 -12
- package/src/index.js +62 -171
- package/src/ledger.js +83 -35
- package/src/options.js +30 -0
- package/src/session.js +9 -8
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# onebrowser
|
|
2
2
|
|
|
3
|
-
One interface to remote browser providers. A
|
|
3
|
+
One interface to remote browser providers. A provider is an adapter; an account is one API key at a provider, and sessions are opened with an account. Every session is recorded in a ledger, which each account's usage and quota are read from.
|
|
4
4
|
|
|
5
5
|
```sh
|
|
6
6
|
npm install onebrowser
|
|
@@ -10,20 +10,21 @@ npm install onebrowser
|
|
|
10
10
|
import { OneBrowser, QuotaError } from 'onebrowser';
|
|
11
11
|
|
|
12
12
|
const browsers = new OneBrowser({ client: 'my-crawler' });
|
|
13
|
-
const
|
|
13
|
+
const [steel] = browsers.accounts('steel'); // one account per key in STEEL_API_KEY
|
|
14
|
+
const session = await steel.openSession({ proxyCountry: 'DE' });
|
|
14
15
|
const page = await session.newPage(); // Playwright; or any CDP client on session.cdpUrl
|
|
15
16
|
await page.goto('https://example.com');
|
|
16
17
|
await session.close();
|
|
17
18
|
|
|
18
|
-
await
|
|
19
|
-
|
|
20
|
-
|
|
19
|
+
await steel.readQuota(); // { remaining, unbilled, unit: 'USD', renewsAt: null }
|
|
20
|
+
steel.quota(); // the same now, from that reading: sessions opened since count at once
|
|
21
|
+
steel.maxSessionCost(); // what a session with the default options costs at most, in USD
|
|
21
22
|
await browsers.syncUsage(); // stores the providers' records of ended sessions in the ledger
|
|
22
|
-
browsers.usage({ since: Date.parse('2026-11-01') }); // [{ day, provider, client, sessions, minutes, … }]
|
|
23
|
+
browsers.usage({ since: Date.parse('2026-11-01') }); // [{ day, provider, account, client, sessions, minutes, … }]
|
|
23
24
|
await browsers.close();
|
|
24
25
|
```
|
|
25
26
|
|
|
26
|
-
Choosing
|
|
27
|
+
Choosing an account, capping sessions and stopping before a quota runs out are the program's policy. onebrowser opens the sessions it is asked for and gives the program what it needs to decide: the accounts it has keys for, what each quota has left, what a session costs at most, and `QuotaError` when a provider refuses a session for the account's quota.
|
|
27
28
|
|
|
28
29
|
Built in: [Steel](https://steel.dev), [TinyFish](https://tinyfish.ai), [Hyperbrowser](https://hyperbrowser.ai), [Lightpanda](https://lightpanda.io), [Kernel](https://www.kernel.sh), [Anchor Browser](https://anchorbrowser.io) and [Firecrawl](https://www.firecrawl.dev); a program can add its own (see [Adding a provider](#adding-a-provider)). Node 22.13 or later.
|
|
29
30
|
|
|
@@ -35,28 +36,35 @@ Built in: [Steel](https://steel.dev), [TinyFish](https://tinyfish.ai), [Hyperbro
|
|
|
35
36
|
|---|---|
|
|
36
37
|
| `client` | The program using it, recorded with each session; required |
|
|
37
38
|
| `ledgerPath` | SQLite file of the ledger (default below; `':memory:'` for tests). Opened when first needed |
|
|
38
|
-
| `providers` | Settings per provider name, on top of its adapter's `defaults`: `
|
|
39
|
+
| `providers` | Settings per provider name, on top of its adapter's `defaults`: `apiKeys` (see [Accounts](#accounts)), `price`, `billingDelayMs`, `minIdleTimeoutSec` (see [Quota](#quota)) and the adapter's own (API base URLs, plan facts such as `monthlyHours`) |
|
|
39
40
|
| `adapters` | Further adapters by provider name, or replacements of built-in ones |
|
|
40
41
|
|
|
41
42
|
| Method | |
|
|
42
43
|
|---|---|
|
|
43
|
-
| `providers` | Names of the providers it
|
|
44
|
-
| `
|
|
45
|
-
| `
|
|
46
|
-
| `
|
|
47
|
-
| `
|
|
48
|
-
| `
|
|
49
|
-
| `syncUsage()` | Stores the providers' records (duration, proxy traffic) of ended sessions: `{ reported, failed: [{ provider, error }] }` |
|
|
50
|
-
| `usage({ since })` | Sessions opened since `since` (ms) per UTC day, provider and client: `sessions`, `minutes` (the provider's record, else the open-to-close timing), `untimed`, `unsynced`, `proxyBytes` |
|
|
44
|
+
| `providers` | Names of the providers it has adapters for |
|
|
45
|
+
| `accounts(provider?)` | The accounts with an API key, in the providers' order and then the keys': all of them, or the provider's (throws for an unknown provider) |
|
|
46
|
+
| `syncUsage()` | Stores the providers' records (duration, proxy traffic) of ended sessions: `{ reported, failed: [{ account, error }] }` |
|
|
47
|
+
| `usage({ since })` | Sessions opened since `since` (ms) per UTC day, account and client: `sessions`, `minutes` (the provider's record, else the open-to-close timing), `untimed`, `unsynced`, `proxyBytes` |
|
|
48
|
+
| `sessions({ since })` | The ledger's sessions opened since `since` (ms), oldest first, each as recorded: `provider`, `account`, `id`, `client`, `openedAt`, `endsBy`, `closedAt`, `durationMs`, `proxyBytes`, `syncedAt` (times in ms; `closedAt` and `syncedAt` null until then, `durationMs` and `proxyBytes` null without the provider's record) |
|
|
49
|
+
| `ledgerId` | The ledger's own id, kept for its life: a program that copies the ledger's record elsewhere tells it from another ledger's by it |
|
|
51
50
|
| `close()` | Closes the sessions still open and the ledger |
|
|
52
51
|
|
|
53
|
-
|
|
52
|
+
An account has `id` (`steel#3f9a1c`: the provider's name and the key's fingerprint), `provider`, `fingerprint`, and:
|
|
53
|
+
|
|
54
|
+
| Method | |
|
|
55
|
+
|---|---|
|
|
56
|
+
| `openSession({ proxyCountry, idleTimeoutSec, maxDurationMs })` | A new session, in the ledger from when the provider has created it. Throws `QuotaError` when the provider refuses it for the account's quota |
|
|
57
|
+
| `readQuota({ maxAgeMs })` | Reads the account's quota from the provider, unless a reading in progress or started at most `maxAgeMs` ago (default 0) can be used, a failed one too; then answers as `quota()` |
|
|
58
|
+
| `quota()` | What is left now, from the last reading: `{ remaining, unbilled, unit, renewsAt }` (see [Quota](#quota)). Throws until the account has been read, and while its last reading failed |
|
|
59
|
+
| `maxSessionCost(options)` | What a session with these options costs at most, in the unit of the quota |
|
|
60
|
+
|
|
61
|
+
A session has `id`, `account`, `egress`, `cdpUrl` (it may carry the API key: never log it), `openedAt` (when it was asked for), `closedAt` (when its provider released it), and:
|
|
54
62
|
|
|
55
63
|
- `connect()`: Playwright's `Browser` over its CDP connection, made once. A keep-alive command every 15 s keeps the provider's idle timeout from ending a session in use.
|
|
56
64
|
- `newPage()`: a new page in the default context: the page a remote browser starts with cannot always be navigated.
|
|
57
65
|
- `close()`: disconnects and releases it; every call answers as the first. Throws when the provider could not release it: the provider ends it by itself, and the ledger counts it as lasting as long as it can.
|
|
58
66
|
|
|
59
|
-
A session and a `OneBrowser` are async-disposable: `await using session = await
|
|
67
|
+
A session and a `OneBrowser` are async-disposable: `await using session = await steel.openSession()` closes the session at the end of its block (Node 24 or later; on Node 22 call `close()`).
|
|
60
68
|
|
|
61
69
|
### Session options: one meaning on every provider
|
|
62
70
|
|
|
@@ -82,7 +90,7 @@ How each provider meets them:
|
|
|
82
90
|
|
|
83
91
|
### Quota
|
|
84
92
|
|
|
85
|
-
`quota()` is the last reading less `unbilled`: what the ledger's sessions that the reading does not show cost at the provider's `price`. A reading shows the sessions that ended `billingDelayMs` before it; the others are open ones, counted as lasting as long as they can (`maxDurationMs`, then `idleTimeoutSec`, at least the provider's `minIdleTimeoutSec`), and those ended since, counted as long as they lasted. So each session counts once from when its provider has created it, whichever program on the machine opened it, and a program can check before each session that the account pays for it (`maxSessionCost`). A provider that bills a session while it runs has part of it counted twice, on the safe side. Not counted: sessions opened through another machine's ledger, until the provider bills them, and proxy traffic billed by the GB.
|
|
93
|
+
`quota()` is the last reading less `unbilled`: what the ledger's sessions of the account that the reading does not show cost at the provider's `price`. A reading shows the sessions that ended `billingDelayMs` before it; the others are open ones, counted as lasting as long as they can (`maxDurationMs`, then `idleTimeoutSec`, at least the provider's `minIdleTimeoutSec`), and those ended since, counted as long as they lasted. So each session counts once from when its provider has created it, whichever program on the machine opened it, and a program can check before each session that the account pays for it (`maxSessionCost`). A provider that bills a session while it runs has part of it counted twice, on the safe side. Not counted: sessions opened through another machine's ledger, until the provider bills them, and proxy traffic billed by the GB.
|
|
86
94
|
|
|
87
95
|
`price` is in the quota's unit: `perHour`, `perSession` (default 0), the session time billed in whole `stepSec`, rounded up (default 1), and at least `minSec` (default 0). Where a provider does not document its rounding, the adapter takes the minute. `renewsAt` (ms) is when the quota renews by itself; `null` when only a top-up renews it. `price` and `billingDelayMs` are settings: a program on another plan sets its own.
|
|
88
96
|
|
|
@@ -98,26 +106,28 @@ How each provider meets them:
|
|
|
98
106
|
|
|
99
107
|
Every provider read from its API showed a session's cost within about a minute of its end (2026-11); `billingDelayMs` leaves room for more.
|
|
100
108
|
|
|
101
|
-
###
|
|
109
|
+
### Accounts
|
|
110
|
+
|
|
111
|
+
An account is one API key at a provider: the quota, the plan's limits and the bill are the key's. The keys of a provider are `apiKeys` in its settings, else those in the environment variable `<PROVIDER>_API_KEY` (`STEEL_API_KEY`, …; `apiKeyEnv(provider)` names it), else those in the file named by `<PROVIDER>_API_KEY_FILE`, separated by commas, spaces or line breaks; each key makes one account, in that order, and a key given twice makes one. A provider without a key has no account.
|
|
102
112
|
|
|
103
|
-
`
|
|
113
|
+
An account is known by its `id`, the provider's name and the first 6 hex digits of the key's SHA-256 (`steel#3f9a1c`): the same for the same key in every program and run, without telling the key. The ledger records sessions under it, so an account's usage is whole however many programs open sessions with its key.
|
|
104
114
|
|
|
105
115
|
### The ledger
|
|
106
116
|
|
|
107
|
-
One SQLite file per machine by default: `$ONEBROWSER_DB`, else `onebrowser/sessions.db` in `$XDG_DATA_HOME` (default `~/.local/share`), shared by every program on the machine under its own `client`. A session is recorded when its provider has created it (it is billed from then on), with when it ends at the latest; closed when its provider has released it; and completed with the provider's own record by `syncUsage()`. A ledger made by an earlier onebrowser is brought up to date when opened; one made by a later onebrowser is refused.
|
|
117
|
+
One SQLite file per machine by default: `$ONEBROWSER_DB`, else `onebrowser/sessions.db` in `$XDG_DATA_HOME` (default `~/.local/share`), shared by every program on the machine under its own `client`. A session is recorded when its provider has created it (it is billed from then on), with its account and when it ends at the latest; closed when its provider has released it; and completed with the provider's own record by `syncUsage()`. Nothing removes a session from the ledger, so its record is whole from the ledger's first session on. Each ledger has an id of its own (`ledgerId`), given when it is made (or first opened by onebrowser 0.7 or later) and kept for its life. A ledger made by an earlier onebrowser is brought up to date when opened; one made by a later onebrowser is refused. Sessions recorded by onebrowser 0.5 or earlier have no account: they become the provider's only account's when it has one here, and stay without one while it has several.
|
|
108
118
|
|
|
109
119
|
## Command line
|
|
110
120
|
|
|
111
|
-
The
|
|
121
|
+
The accounts and the ledger belong to the machine rather than to one program, so the `onebrowser` command reads them without a program of your own. It opens no sessions.
|
|
112
122
|
|
|
113
123
|
```sh
|
|
114
|
-
npx onebrowser quota # what each
|
|
115
|
-
npx onebrowser quota steel kernel
|
|
116
|
-
npx onebrowser usage --days 7 # the ledger's sessions per UTC day,
|
|
124
|
+
npx onebrowser quota # what each account has left, once its sessions not billed yet are paid for
|
|
125
|
+
npx onebrowser quota steel kernel#a81f2c # only these: a provider's accounts, or one account by id
|
|
126
|
+
npx onebrowser usage --days 7 # the ledger's sessions per UTC day, account and client, today included
|
|
117
127
|
npx onebrowser usage --json --env-file .env
|
|
118
128
|
```
|
|
119
129
|
|
|
120
|
-
`usage` first fetches the providers' records of the sessions that have ended, as `syncUsage()` does. With `--json`, `quota` prints an array of what `readQuota()` returns, with the
|
|
130
|
+
`usage` first fetches the providers' records of the sessions that have ended, as `syncUsage()` does. With `--json`, `quota` prints an array of what `readQuota()` returns, with the account: `{ account, provider, remaining, unbilled, unit, renewsAt }`, or `{ account, provider, error }` for a failed reading; `usage` prints `{ ledger, ledgerId, since, sessions, failed }`, where `sessions` is what `usage()` returns and `failed` what `syncUsage()` could not fetch. API keys are read as by the library, from the environment (`--env-file PATH` loads a file into it first). Exit status: 0 done, 1 a reading failed, 2 the command line is wrong.
|
|
121
131
|
|
|
122
132
|
## Adding a provider
|
|
123
133
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "onebrowser",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "One interface to remote browser providers: sessions by provider
|
|
3
|
+
"version": "0.7.0",
|
|
4
|
+
"description": "One interface to remote browser providers: sessions by account (an API key at a provider), a usage ledger, account quotas",
|
|
5
5
|
"keywords": ["browser", "remote-browser", "cdp", "playwright", "steel", "hyperbrowser", "lightpanda", "kernel", "tinyfish", "anchorbrowser", "firecrawl"],
|
|
6
6
|
"license": "MIT",
|
|
7
7
|
"author": "Richard Wang",
|
package/src/account.js
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
import { sessionCost } from './cost.js';
|
|
2
|
+
import { fingerprint } from './credentials.js';
|
|
3
|
+
import { QuotaError } from './errors.js';
|
|
4
|
+
import { sessionMsAtMost, sessionOptions } from './options.js';
|
|
5
|
+
import { Session } from './session.js';
|
|
6
|
+
|
|
7
|
+
/**
|
|
8
|
+
* One API key at one provider: the account its sessions are billed to, with its own quota and its own record in
|
|
9
|
+
* the ledger. id, the provider's name and the key's fingerprint (`steel#3f9a1c`), names it in the ledger, in
|
|
10
|
+
* logs and on the command line; the key itself stays inside.
|
|
11
|
+
*
|
|
12
|
+
* ledger(): the shared Ledger (ledger.js), opened when first needed; client: the program, recorded with each
|
|
13
|
+
* session.
|
|
14
|
+
*/
|
|
15
|
+
export class Account {
|
|
16
|
+
#adapter;
|
|
17
|
+
#settings;
|
|
18
|
+
#apiKey;
|
|
19
|
+
#client;
|
|
20
|
+
#ledger;
|
|
21
|
+
#open = new Set();
|
|
22
|
+
#latest = null; // the latest quota reading, in progress or done
|
|
23
|
+
#done = null; // the latest reading done
|
|
24
|
+
|
|
25
|
+
constructor({ provider, adapter, settings, apiKey, client, ledger }) {
|
|
26
|
+
this.provider = provider;
|
|
27
|
+
this.fingerprint = fingerprint(apiKey);
|
|
28
|
+
this.id = `${provider}#${this.fingerprint}`;
|
|
29
|
+
this.#adapter = adapter;
|
|
30
|
+
this.#settings = settings;
|
|
31
|
+
this.#apiKey = apiKey;
|
|
32
|
+
this.#client = client;
|
|
33
|
+
this.#ledger = ledger;
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
/**
|
|
37
|
+
* A new session (options: sessionOptions()), in the ledger from when the provider has created it. Throws when
|
|
38
|
+
* the provider creates none; QuotaError when it refuses for the account's quota.
|
|
39
|
+
*/
|
|
40
|
+
async openSession(options = {}) {
|
|
41
|
+
const o = sessionOptions(options);
|
|
42
|
+
const openedAt = Date.now();
|
|
43
|
+
let remote;
|
|
44
|
+
try {
|
|
45
|
+
remote = await this.#adapter.createSession(this.#settings, this.#apiKey, o);
|
|
46
|
+
} catch (e) {
|
|
47
|
+
if (e instanceof QuotaError) throw e;
|
|
48
|
+
throw new Error(`failed to create a ${this.provider} session: ${e.message}`);
|
|
49
|
+
}
|
|
50
|
+
const session = new Session({ account: this, remote, openedAt, maxDurationMs: o.maxDurationMs, release: (s) => this.#release(s) });
|
|
51
|
+
try {
|
|
52
|
+
this.#ledger().opened({ provider: this.provider, account: this.id, id: session.id, client: this.#client, openedAt, endsBy: openedAt + sessionMsAtMost(o, this.#settings) });
|
|
53
|
+
} catch (e) {
|
|
54
|
+
await session.close().catch(() => {}); // a session the ledger does not know is not handed out
|
|
55
|
+
throw e;
|
|
56
|
+
}
|
|
57
|
+
this.#open.add(session);
|
|
58
|
+
return session;
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
/** Releases `session` at the provider and records it closed: when it was released, so its whole billed time counts */
|
|
62
|
+
async #release(session) {
|
|
63
|
+
try {
|
|
64
|
+
await this.#adapter.releaseSession(this.#settings, this.#apiKey, session);
|
|
65
|
+
const closedAt = Date.now();
|
|
66
|
+
this.#ledger().closed({ account: this.id, id: session.id, closedAt });
|
|
67
|
+
return closedAt;
|
|
68
|
+
} finally {
|
|
69
|
+
this.#open.delete(session);
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
/**
|
|
74
|
+
* Reads the quota from the provider, unless a reading in progress or started at most maxAgeMs ago can be used
|
|
75
|
+
* (a failed one throws again); then answers as quota().
|
|
76
|
+
*/
|
|
77
|
+
async readQuota({ maxAgeMs = 0 } = {}) {
|
|
78
|
+
const reading = this.#reading(maxAgeMs);
|
|
79
|
+
await reading.settled;
|
|
80
|
+
if (reading.error) throw reading.error;
|
|
81
|
+
return this.quota();
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
/**
|
|
85
|
+
* What is left of the quota now, in the provider's unit: { remaining, unbilled, unit, renewsAt }. The last
|
|
86
|
+
* reading shows the sessions that ended billingDelayMs before it; `unbilled` is what the ledger's others cost
|
|
87
|
+
* at the provider's price, open ones as if they lasted as long as they can, and remaining the reading less
|
|
88
|
+
* that. renewsAt (ms): when the quota renews by itself; null: only a top-up renews it. Throws until the
|
|
89
|
+
* provider has been read, and while the last reading failed.
|
|
90
|
+
*/
|
|
91
|
+
quota() {
|
|
92
|
+
const reading = this.#done;
|
|
93
|
+
if (!reading) throw new Error(`the quota of ${this.id} has not been read yet (readQuota)`);
|
|
94
|
+
if (reading.error) throw reading.error;
|
|
95
|
+
const unbilled = this.#ledger().openSince(this.id, reading.billedBefore).reduce((cost, s) => cost + sessionCost(this.#settings.price, s.durationMs), 0);
|
|
96
|
+
const { remaining, unit, renewsAt = null } = reading.value;
|
|
97
|
+
return { remaining: remaining - unbilled, unbilled, unit, renewsAt };
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
/** The latest reading if it is in progress or at most maxAgeMs old, else a new one */
|
|
101
|
+
#reading(maxAgeMs) {
|
|
102
|
+
const latest = this.#latest;
|
|
103
|
+
if (latest && (latest !== this.#done || Date.now() - latest.startedAt <= maxAgeMs)) return latest;
|
|
104
|
+
const startedAt = Date.now();
|
|
105
|
+
const reading = { startedAt, billedBefore: startedAt - this.#settings.billingDelayMs, value: null, error: null };
|
|
106
|
+
this.#latest = reading;
|
|
107
|
+
reading.settled = (async () => {
|
|
108
|
+
try {
|
|
109
|
+
const q = await this.#adapter.quota(this.#settings, this.#apiKey, this.#ledger().history(this.id, reading.billedBefore));
|
|
110
|
+
if (!(Number.isFinite(q?.remaining) && typeof q.unit === 'string')) throw new Error(`${this.provider} answered no quota: ${JSON.stringify(q)}`);
|
|
111
|
+
reading.value = q;
|
|
112
|
+
} catch (e) {
|
|
113
|
+
reading.error = e;
|
|
114
|
+
}
|
|
115
|
+
this.#done = reading;
|
|
116
|
+
})();
|
|
117
|
+
return reading;
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
/** What a session with these options costs at most, in the unit of the quota */
|
|
121
|
+
maxSessionCost(options) {
|
|
122
|
+
return sessionCost(this.#settings.price, sessionMsAtMost(sessionOptions(options), this.#settings));
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
/** Stores the provider's records (duration, proxy traffic) of this account's ended sessions that have none yet; how many */
|
|
126
|
+
async syncUsage() {
|
|
127
|
+
const ledger = this.#ledger();
|
|
128
|
+
const rows = ledger.awaitingUsage(this.id);
|
|
129
|
+
if (!rows.length) return 0;
|
|
130
|
+
const usage = await this.#adapter.sessionUsage(this.#settings, this.#apiKey, rows);
|
|
131
|
+
ledger.tx(() => usage.forEach((u) => ledger.recordUsage(this.id, u)));
|
|
132
|
+
return usage.length;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
/** Closes its sessions still open; their release errors are ignored: the provider ends those */
|
|
136
|
+
async close() {
|
|
137
|
+
await Promise.all([...this.#open].map((s) => s.close().catch(() => {})));
|
|
138
|
+
}
|
|
139
|
+
}
|
package/src/cli.js
CHANGED
|
@@ -18,6 +18,18 @@ const GLOBAL_OPTIONS = {
|
|
|
18
18
|
help: { type: 'boolean', short: 'h' },
|
|
19
19
|
};
|
|
20
20
|
|
|
21
|
+
/** The accounts `names` ask for: each name a provider's (it must have an API key) or one account's id */
|
|
22
|
+
function askedAccounts(browsers, names) {
|
|
23
|
+
return names.flatMap((name) => {
|
|
24
|
+
const own = browsers.accounts().find((a) => a.id === name);
|
|
25
|
+
if (own) return [own];
|
|
26
|
+
if (!browsers.providers.includes(name)) throw new CommandLineError(`unknown provider or account ${name} (expected ${browsers.providers.join(', ')}, or an account id)`);
|
|
27
|
+
const accounts = browsers.accounts(name);
|
|
28
|
+
if (!accounts.length) throw new Error(`${name} has no API key: set ${apiKeyEnv(name)} or ${apiKeyEnv(name)}_FILE`);
|
|
29
|
+
return accounts;
|
|
30
|
+
});
|
|
31
|
+
}
|
|
32
|
+
|
|
21
33
|
/**
|
|
22
34
|
* The commands: their own options, whether they take arguments, and run(browsers, values, args, { out, err })
|
|
23
35
|
* → exit status
|
|
@@ -26,16 +38,16 @@ const COMMANDS = {
|
|
|
26
38
|
quota: {
|
|
27
39
|
positionals: true,
|
|
28
40
|
async run(browsers, { json }, names, { out }) {
|
|
29
|
-
const
|
|
30
|
-
if (
|
|
31
|
-
const
|
|
32
|
-
|
|
33
|
-
|
|
41
|
+
const accounts = names.length ? askedAccounts(browsers, names) : browsers.accounts();
|
|
42
|
+
if (!accounts.length) throw new Error(`no provider has an API key: set ${browsers.providers.map(apiKeyEnv).join(', ')}`);
|
|
43
|
+
const quotas = await Promise.all(
|
|
44
|
+
accounts.map((a) => a.readQuota().then((q) => ({ account: a.id, provider: a.provider, ...q }), (e) => ({ account: a.id, provider: a.provider, error: e.message }))),
|
|
45
|
+
);
|
|
34
46
|
if (json) out.write(`${JSON.stringify(quotas, null, 2)}\n`);
|
|
35
47
|
else {
|
|
36
48
|
for (const q of quotas) {
|
|
37
49
|
const text = q.error ? `unknown (${q.error})` : `${round(q.remaining)} ${q.unit} left${q.unbilled ? ` (after ${round(q.unbilled)} not billed yet)` : ''}${q.renewsAt ? `, renews ${day(q.renewsAt)}` : ''}`;
|
|
38
|
-
out.write(`${q.
|
|
50
|
+
out.write(`${q.account.padEnd(21)} ${text}\n`);
|
|
39
51
|
}
|
|
40
52
|
}
|
|
41
53
|
return quotas.some((q) => q.error) ? 1 : 0;
|
|
@@ -49,15 +61,15 @@ const COMMANDS = {
|
|
|
49
61
|
if (!(Number.isInteger(n) && n >= 1)) throw new CommandLineError(`--days must be a whole number ≥ 1, not ${days}`);
|
|
50
62
|
const since = new Date().setUTCHours(0, 0, 0, 0) - (n - 1) * DAY_MS;
|
|
51
63
|
const { failed } = await browsers.syncUsage();
|
|
52
|
-
for (const f of failed) err.write(`${f.
|
|
64
|
+
for (const f of failed) err.write(`${f.account}: the provider's records were not fetched (${f.error})\n`);
|
|
53
65
|
const sessions = browsers.usage({ since });
|
|
54
|
-
if (json) out.write(`${JSON.stringify({ ledger: browsers.ledgerPath, since: day(since), sessions, failed }, null, 2)}\n`);
|
|
66
|
+
if (json) out.write(`${JSON.stringify({ ledger: browsers.ledgerPath, ledgerId: browsers.ledgerId, since: day(since), sessions, failed }, null, 2)}\n`);
|
|
55
67
|
else {
|
|
56
68
|
out.write(`sessions since ${day(since)} in ${browsers.ledgerPath}:\n`);
|
|
57
69
|
if (!sessions.length) out.write(' none\n');
|
|
58
|
-
for (const r of sessions) out.write(usageLine(`${r.day} ${r.
|
|
59
|
-
for (const [
|
|
60
|
-
if (rows.length > 1) out.write(usageLine(`total ${
|
|
70
|
+
for (const r of sessions) out.write(usageLine(`${r.day} ${r.account} ${r.client}`, r));
|
|
71
|
+
for (const [account, rows] of Map.groupBy(sessions, (r) => r.account)) {
|
|
72
|
+
if (rows.length > 1) out.write(usageLine(`total ${account}`, sumUsage(rows)));
|
|
61
73
|
}
|
|
62
74
|
}
|
|
63
75
|
return failed.length ? 1 : 0;
|
|
@@ -68,9 +80,10 @@ const COMMANDS = {
|
|
|
68
80
|
const help = (providers) => `onebrowser: the quotas of remote browser provider accounts, and the sessions in this machine's ledger
|
|
69
81
|
|
|
70
82
|
Usage:
|
|
71
|
-
onebrowser quota [
|
|
72
|
-
|
|
73
|
-
|
|
83
|
+
onebrowser quota [NAME...] what each account has left, less what the sessions in the ledger that it has
|
|
84
|
+
not billed yet cost; NAME: a provider (every account with its API key) or an
|
|
85
|
+
account id such as steel#3f9a1c; default: every account
|
|
86
|
+
onebrowser usage [--days N] sessions per UTC day, account and client in the last N UTC days, today
|
|
74
87
|
included (default 30), after fetching the providers' records of ended sessions
|
|
75
88
|
|
|
76
89
|
Options:
|
|
@@ -79,27 +92,19 @@ Options:
|
|
|
79
92
|
-h, --help
|
|
80
93
|
|
|
81
94
|
Environment:
|
|
82
|
-
${providers.map((p) => ` ${apiKeyEnv(p).padEnd(22)} API
|
|
95
|
+
${providers.map((p) => ` ${apiKeyEnv(p).padEnd(22)} API keys of ${p}, one per account, or ${apiKeyEnv(p)}_FILE naming a file that holds them`).join('\n')}
|
|
96
|
+
(keys are separated by commas, spaces or line breaks)
|
|
83
97
|
ONEBROWSER_DB the ledger file (default: onebrowser/sessions.db in $XDG_DATA_HOME, else ~/.local/share)
|
|
84
98
|
|
|
85
99
|
Exit status: 0 done, 1 a reading failed, 2 the command line is wrong.
|
|
86
100
|
`;
|
|
87
101
|
|
|
88
|
-
function isReady(browsers, provider) {
|
|
89
|
-
try {
|
|
90
|
-
browsers.checkReady(provider);
|
|
91
|
-
return true;
|
|
92
|
-
} catch {
|
|
93
|
-
return false;
|
|
94
|
-
}
|
|
95
|
-
}
|
|
96
|
-
|
|
97
102
|
const USAGE_FIELDS = ['sessions', 'minutes', 'untimed', 'unsynced', 'proxyBytes'];
|
|
98
103
|
|
|
99
104
|
const sumUsage = (rows) => Object.fromEntries(USAGE_FIELDS.map((k) => [k, rows.reduce((t, r) => t + r[k], 0)]));
|
|
100
105
|
|
|
101
106
|
const usageLine = (label, r) =>
|
|
102
|
-
` ${label.padEnd(
|
|
107
|
+
` ${label.padEnd(40)} ${String(r.sessions).padStart(5)} sessions ${r.minutes.toFixed(1).padStart(8)} min` +
|
|
103
108
|
` ${(r.proxyBytes / 1e6).toFixed(1).padStart(7)} MB proxy` +
|
|
104
109
|
(r.untimed ? ` (${r.untimed} untimed)` : '') +
|
|
105
110
|
(r.unsynced ? ` (${r.unsynced} not reported by the provider yet)` : '') +
|
package/src/credentials.js
CHANGED
|
@@ -1,22 +1,27 @@
|
|
|
1
|
+
import crypto from 'node:crypto';
|
|
1
2
|
import fs from 'node:fs';
|
|
2
3
|
|
|
3
|
-
/** The environment variable holding a provider's API
|
|
4
|
+
/** The environment variable holding a provider's API keys: STEEL_API_KEY for `steel` */
|
|
4
5
|
export const apiKeyEnv = (provider) => `${provider.toUpperCase()}_API_KEY`;
|
|
5
6
|
|
|
7
|
+
/** What names the account an API key belongs to without telling the key: the first 6 hex digits of its SHA-256 */
|
|
8
|
+
export const fingerprint = (apiKey) => crypto.createHash('sha256').update(apiKey).digest('hex').slice(0, 6);
|
|
9
|
+
|
|
10
|
+
/** Keys written one after another, separated by commas, spaces or line breaks */
|
|
11
|
+
const split = (text) => text.split(/[\s,]+/).filter(Boolean);
|
|
12
|
+
|
|
6
13
|
/**
|
|
7
|
-
* API
|
|
8
|
-
* else the file named by that variable with _FILE appended
|
|
9
|
-
*
|
|
14
|
+
* A provider's API keys, one per account: `apiKeys` as given in its settings, else those in the environment
|
|
15
|
+
* variable apiKeyEnv(provider), else those in the file named by that variable with _FILE appended, in their
|
|
16
|
+
* order and each once. Separators: commas, spaces or line breaks. None is fine; the file must exist.
|
|
10
17
|
*/
|
|
11
|
-
export function
|
|
12
|
-
if (apiKey?.trim()) return apiKey.trim();
|
|
18
|
+
export function readApiKeys(provider, apiKeys) {
|
|
13
19
|
const env = apiKeyEnv(provider);
|
|
14
20
|
const file = process.env[`${env}_FILE`];
|
|
15
|
-
let
|
|
16
|
-
if (!
|
|
17
|
-
if (!fs.existsSync(file)) throw new Error(`${provider}
|
|
18
|
-
|
|
21
|
+
let keys = apiKeys ?? split(process.env[env] ?? '');
|
|
22
|
+
if (!keys.length && file) {
|
|
23
|
+
if (!fs.existsSync(file)) throw new Error(`${provider} keys: ${env}_FILE names no file (${file})`);
|
|
24
|
+
keys = split(fs.readFileSync(file, 'utf8'));
|
|
19
25
|
}
|
|
20
|
-
|
|
21
|
-
return key.trim();
|
|
26
|
+
return [...new Set(keys.map((k) => k.trim()).filter(Boolean))];
|
|
22
27
|
}
|
package/src/index.js
CHANGED
|
@@ -1,225 +1,116 @@
|
|
|
1
|
-
import {
|
|
2
|
-
import {
|
|
1
|
+
import { Account } from './account.js';
|
|
2
|
+
import { readApiKeys } from './credentials.js';
|
|
3
3
|
import { QuotaError } from './errors.js';
|
|
4
4
|
import { defaultLedgerPath, Ledger } from './ledger.js';
|
|
5
5
|
import { ADAPTERS, checkAdapter, fullSettings } from './providers/index.js';
|
|
6
|
-
import { KEEP_ALIVE_MS, Session } from './session.js';
|
|
7
6
|
|
|
8
7
|
export { apiKeyEnv } from './credentials.js';
|
|
8
|
+
export { sessionOptions } from './options.js';
|
|
9
9
|
export { QuotaError };
|
|
10
10
|
|
|
11
|
-
/** Session options when openSession() is not given them */
|
|
12
|
-
const SESSION_DEFAULTS = {
|
|
13
|
-
proxyCountry: null,
|
|
14
|
-
idleTimeoutSec: 60,
|
|
15
|
-
maxDurationMs: 15 * 60_000,
|
|
16
|
-
};
|
|
17
|
-
|
|
18
11
|
/**
|
|
19
|
-
*
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
*
|
|
24
|
-
* maxDurationMs a session ends this long after it was opened at the latest
|
|
25
|
-
*/
|
|
26
|
-
export function sessionOptions(options = {}) {
|
|
27
|
-
const unknown = Object.keys(options).filter((k) => !Object.hasOwn(SESSION_DEFAULTS, k));
|
|
28
|
-
if (unknown.length) throw new Error(`unknown session option ${unknown.join(', ')} (expected ${Object.keys(SESSION_DEFAULTS).join(', ')})`);
|
|
29
|
-
const o = { ...SESSION_DEFAULTS, ...options };
|
|
30
|
-
if (!(Number.isFinite(o.idleTimeoutSec) && o.idleTimeoutSec * 1000 >= 2 * KEEP_ALIVE_MS)) {
|
|
31
|
-
throw new Error(`idleTimeoutSec must be at least ${(2 * KEEP_ALIVE_MS) / 1000} (twice the keep-alive interval), not ${o.idleTimeoutSec}`);
|
|
32
|
-
}
|
|
33
|
-
if (!(Number.isFinite(o.maxDurationMs) && o.maxDurationMs > 0)) throw new Error(`maxDurationMs must be a number > 0, not ${o.maxDurationMs}`);
|
|
34
|
-
return o;
|
|
35
|
-
}
|
|
36
|
-
|
|
37
|
-
/** How long a session lasts at most, whatever becomes of its process: closed at maxDurationMs, else ended by its provider once idle */
|
|
38
|
-
const sessionMsAtMost = (o, settings) => o.maxDurationMs + Math.max(o.idleTimeoutSec, settings.minIdleTimeoutSec) * 1000;
|
|
39
|
-
|
|
40
|
-
/**
|
|
41
|
-
* Remote browsers from several providers behind one interface: a session is asked for by provider name. Every
|
|
42
|
-
* session is recorded in a ledger (ledger.js) shared by the programs on the machine, so that each provider
|
|
43
|
-
* account's usage, and what its quota has left at any moment, are known. Which provider to use, and when to
|
|
44
|
-
* stop using one, are the caller's decisions.
|
|
12
|
+
* Remote browsers from several providers behind one interface. A provider is an adapter (providers/index.js);
|
|
13
|
+
* an account is one API key at a provider (account.js), which sessions are opened with. Every session is
|
|
14
|
+
* recorded in a ledger (ledger.js) shared by the programs on the machine, so that each account's usage, and
|
|
15
|
+
* what its quota has left at any moment, are known. Which account to use, and when to stop using one, are the
|
|
16
|
+
* caller's decisions.
|
|
45
17
|
*
|
|
46
18
|
* client: the program using it, recorded with each session; ledgerPath: the ledger's SQLite file, opened when
|
|
47
|
-
* first needed; providers: settings per provider name on top of its adapter's defaults
|
|
48
|
-
*
|
|
19
|
+
* first needed; providers: settings per provider name on top of its adapter's defaults, apiKeys among them
|
|
20
|
+
* (else credentials.js); adapters: further or replacement adapters by provider name.
|
|
49
21
|
*/
|
|
50
22
|
export class OneBrowser {
|
|
51
|
-
#providers =
|
|
23
|
+
#providers = []; // names, in the adapters' order
|
|
24
|
+
#accounts = [];
|
|
52
25
|
#ledger = null;
|
|
53
|
-
#open = new Set();
|
|
54
|
-
#readings = new Map(); // provider → { latest, done }: its latest quota reading, and the latest one done
|
|
55
26
|
|
|
56
27
|
constructor({ client, ledgerPath = defaultLedgerPath(), providers = {}, adapters = {} }) {
|
|
57
28
|
if (!client?.trim()) throw new Error('onebrowser: client must name the program using it');
|
|
58
29
|
this.client = client;
|
|
59
30
|
this.ledgerPath = ledgerPath;
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
const { apiKey, ...given } = providers[name] ?? {};
|
|
63
|
-
this.#providers.set(name, { adapter, settings: fullSettings(name, { ...adapter.defaults, ...given }), apiKey });
|
|
64
|
-
}
|
|
65
|
-
const unknown = Object.keys(providers).filter((name) => !this.#providers.has(name));
|
|
31
|
+
const all = { ...ADAPTERS, ...adapters };
|
|
32
|
+
const unknown = Object.keys(providers).filter((name) => !all[name]);
|
|
66
33
|
if (unknown.length) throw new Error(`onebrowser: no adapter for provider ${unknown.join(', ')}`);
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
}
|
|
76
|
-
|
|
77
|
-
#known(provider) {
|
|
78
|
-
const p = this.#providers.get(provider);
|
|
79
|
-
if (!p) throw new Error(`unknown provider ${provider} (expected ${this.providers.join(', ')})`);
|
|
80
|
-
return p;
|
|
81
|
-
}
|
|
82
|
-
|
|
83
|
-
#ready(provider) {
|
|
84
|
-
const p = this.#known(provider);
|
|
85
|
-
return { ...p, apiKey: readApiKey(provider, p.apiKey) };
|
|
86
|
-
}
|
|
87
|
-
|
|
88
|
-
/** Throws when sessions cannot be opened with `provider`: unknown, or no API key */
|
|
89
|
-
checkReady(provider) {
|
|
90
|
-
this.#ready(provider);
|
|
91
|
-
}
|
|
92
|
-
|
|
93
|
-
/**
|
|
94
|
-
* A new session with `provider` (options: sessionOptions()), in the ledger from when its provider has created
|
|
95
|
-
* it. Throws when the provider creates none; QuotaError when it refuses for the account's quota.
|
|
96
|
-
*/
|
|
97
|
-
async openSession({ provider, ...options }) {
|
|
98
|
-
const { adapter, settings, apiKey } = this.#ready(provider);
|
|
99
|
-
const o = sessionOptions(options);
|
|
100
|
-
const openedAt = Date.now();
|
|
101
|
-
let remote;
|
|
102
|
-
try {
|
|
103
|
-
remote = await adapter.createSession(settings, apiKey, o);
|
|
104
|
-
} catch (e) {
|
|
105
|
-
if (e instanceof QuotaError) throw e;
|
|
106
|
-
throw new Error(`failed to create a ${provider} session: ${e.message}`);
|
|
107
|
-
}
|
|
108
|
-
const session = new Session({ provider, remote, openedAt, maxDurationMs: o.maxDurationMs, release: (s) => this.#release(adapter, settings, apiKey, s) });
|
|
109
|
-
try {
|
|
110
|
-
this.#store.opened({ provider, id: session.id, client: this.client, openedAt, endsBy: openedAt + sessionMsAtMost(o, settings) });
|
|
111
|
-
} catch (e) {
|
|
112
|
-
await session.close().catch(() => {}); // a session the ledger does not know is not handed out
|
|
113
|
-
throw e;
|
|
34
|
+
for (const [provider, adapter] of Object.entries(all)) {
|
|
35
|
+
checkAdapter(provider, adapter);
|
|
36
|
+
const { apiKeys, ...given } = providers[provider] ?? {};
|
|
37
|
+
const settings = fullSettings(provider, { ...adapter.defaults, ...given });
|
|
38
|
+
this.#providers.push(provider);
|
|
39
|
+
for (const apiKey of readApiKeys(provider, apiKeys)) {
|
|
40
|
+
this.#accounts.push(new Account({ provider, adapter, settings, apiKey, client, ledger: () => this.#store }));
|
|
41
|
+
}
|
|
114
42
|
}
|
|
115
|
-
this.#open.add(session);
|
|
116
|
-
return session;
|
|
117
43
|
}
|
|
118
44
|
|
|
119
|
-
/**
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
const
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
} finally {
|
|
127
|
-
this.#open.delete(session);
|
|
45
|
+
/** The ledger, opened on first use; a provider with one account here claims its sessions recorded without one */
|
|
46
|
+
get #store() {
|
|
47
|
+
if (!this.#ledger) {
|
|
48
|
+
this.#ledger = new Ledger(this.ledgerPath);
|
|
49
|
+
for (const [provider, accounts] of Map.groupBy(this.#accounts, (a) => a.provider)) {
|
|
50
|
+
if (accounts.length === 1) this.#ledger.claim(provider, accounts[0].id);
|
|
51
|
+
}
|
|
128
52
|
}
|
|
53
|
+
return this.#ledger;
|
|
129
54
|
}
|
|
130
55
|
|
|
131
|
-
/**
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
*/
|
|
135
|
-
async readQuota(provider, { maxAgeMs = 0 } = {}) {
|
|
136
|
-
const reading = this.#reading(provider, maxAgeMs);
|
|
137
|
-
await reading.settled;
|
|
138
|
-
if (reading.error) throw reading.error;
|
|
139
|
-
return this.quota(provider);
|
|
56
|
+
/** The ledger's own id, kept for the ledger's life: it tells its record from another ledger's */
|
|
57
|
+
get ledgerId() {
|
|
58
|
+
return this.#store.id;
|
|
140
59
|
}
|
|
141
60
|
|
|
142
|
-
/**
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
* cost at the provider's price, open ones as if they lasted as long as they can, and remaining the reading less
|
|
146
|
-
* that. renewsAt (ms): when the quota renews by itself; null: only a top-up renews it. Throws until the
|
|
147
|
-
* provider has been read, and while its last reading failed.
|
|
148
|
-
*/
|
|
149
|
-
quota(provider) {
|
|
150
|
-
const { settings } = this.#known(provider);
|
|
151
|
-
const reading = this.#readings.get(provider)?.done;
|
|
152
|
-
if (!reading) throw new Error(`the quota of ${provider} has not been read yet (readQuota)`);
|
|
153
|
-
if (reading.error) throw reading.error;
|
|
154
|
-
const unbilled = this.#store.openSince(provider, reading.billedBefore).reduce((cost, s) => cost + sessionCost(settings.price, s.durationMs), 0);
|
|
155
|
-
const { remaining, unit, renewsAt = null } = reading.value;
|
|
156
|
-
return { remaining: remaining - unbilled, unbilled, unit, renewsAt };
|
|
157
|
-
}
|
|
158
|
-
|
|
159
|
-
/** The latest reading of the provider's quota if it is in progress or at most maxAgeMs old, else a new one */
|
|
160
|
-
#reading(provider, maxAgeMs) {
|
|
161
|
-
const readings = this.#readings.get(provider) ?? { latest: null, done: null };
|
|
162
|
-
this.#readings.set(provider, readings);
|
|
163
|
-
const { latest } = readings;
|
|
164
|
-
const inProgress = latest && latest !== readings.done;
|
|
165
|
-
if (inProgress || (latest && Date.now() - latest.startedAt <= maxAgeMs)) return latest;
|
|
166
|
-
const { adapter, settings, apiKey } = this.#ready(provider);
|
|
167
|
-
const startedAt = Date.now();
|
|
168
|
-
const reading = { startedAt, billedBefore: startedAt - settings.billingDelayMs, value: null, error: null };
|
|
169
|
-
readings.latest = reading;
|
|
170
|
-
reading.settled = (async () => {
|
|
171
|
-
try {
|
|
172
|
-
const q = await adapter.quota(settings, apiKey, this.#store.history(provider, reading.billedBefore));
|
|
173
|
-
if (!(Number.isFinite(q?.remaining) && typeof q.unit === 'string')) throw new Error(`${provider} answered no quota: ${JSON.stringify(q)}`);
|
|
174
|
-
reading.value = q;
|
|
175
|
-
} catch (e) {
|
|
176
|
-
reading.error = e;
|
|
177
|
-
}
|
|
178
|
-
readings.done = reading;
|
|
179
|
-
})();
|
|
180
|
-
return reading;
|
|
61
|
+
/** Names of the providers it has adapters for */
|
|
62
|
+
get providers() {
|
|
63
|
+
return [...this.#providers];
|
|
181
64
|
}
|
|
182
65
|
|
|
183
|
-
/**
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
66
|
+
/** The accounts with an API key, in the providers' order and then the keys': all, or `provider`'s (throws for an unknown one) */
|
|
67
|
+
accounts(provider) {
|
|
68
|
+
if (provider === undefined) return [...this.#accounts];
|
|
69
|
+
if (!this.#providers.includes(provider)) throw new Error(`unknown provider ${provider} (expected ${this.#providers.join(', ')})`);
|
|
70
|
+
return this.#accounts.filter((a) => a.provider === provider);
|
|
187
71
|
}
|
|
188
72
|
|
|
189
73
|
/**
|
|
190
74
|
* Stores the providers' records (duration, proxy traffic) of the ledger's ended sessions that have none yet:
|
|
191
|
-
* { reported, failed: [{
|
|
192
|
-
*
|
|
75
|
+
* { reported, failed: [{ account, error }] }. What cannot be fetched now, and the sessions of accounts without
|
|
76
|
+
* a key here, are left for a later call.
|
|
193
77
|
*/
|
|
194
78
|
async syncUsage() {
|
|
195
|
-
const store = this.#store;
|
|
196
79
|
let reported = 0;
|
|
197
80
|
const failed = [];
|
|
198
|
-
for (const
|
|
81
|
+
for (const account of this.#accounts) {
|
|
199
82
|
try {
|
|
200
|
-
|
|
201
|
-
const usage = await adapter.sessionUsage(settings, apiKey, rows);
|
|
202
|
-
store.tx(() => usage.forEach((u) => store.recordUsage(provider, u)));
|
|
203
|
-
reported += usage.length;
|
|
83
|
+
reported += await account.syncUsage();
|
|
204
84
|
} catch (e) {
|
|
205
|
-
failed.push({
|
|
85
|
+
failed.push({ account: account.id, error: e.message });
|
|
206
86
|
}
|
|
207
87
|
}
|
|
208
88
|
return { reported, failed };
|
|
209
89
|
}
|
|
210
90
|
|
|
211
91
|
/**
|
|
212
|
-
* Sessions opened since `since` (ms), per UTC day,
|
|
213
|
-
* [{ day, provider, client, sessions, minutes, untimed, unsynced, proxyBytes }]. minutes: the
|
|
214
|
-
* (syncUsage), else the open-to-close timing; untimed: sessions with neither; unsynced:
|
|
92
|
+
* Sessions opened since `since` (ms), per UTC day, account and client:
|
|
93
|
+
* [{ day, provider, account, client, sessions, minutes, untimed, unsynced, proxyBytes }]. minutes: the
|
|
94
|
+
* provider's record (syncUsage), else the open-to-close timing; untimed: sessions with neither; unsynced:
|
|
95
|
+
* those without the record. account: the provider's name alone for sessions whose account is not known.
|
|
215
96
|
*/
|
|
216
97
|
usage({ since }) {
|
|
217
98
|
return this.#store.usage(since);
|
|
218
99
|
}
|
|
219
100
|
|
|
220
|
-
/**
|
|
101
|
+
/**
|
|
102
|
+
* The ledger's sessions opened since `since` (ms), oldest first, each as recorded:
|
|
103
|
+
* [{ provider, account, id, client, openedAt, endsBy, closedAt, durationMs, proxyBytes, syncedAt }], times in
|
|
104
|
+
* ms (closedAt, syncedAt null until then; durationMs and proxyBytes null without the provider's record). account:
|
|
105
|
+
* the provider's name alone for a session whose account is not known.
|
|
106
|
+
*/
|
|
107
|
+
sessions({ since }) {
|
|
108
|
+
return this.#store.sessions(since);
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
/** Closes every session still open and the ledger */
|
|
221
112
|
async close() {
|
|
222
|
-
await Promise.all(
|
|
113
|
+
await Promise.all(this.#accounts.map((a) => a.close()));
|
|
223
114
|
this.#ledger?.close();
|
|
224
115
|
this.#ledger = null;
|
|
225
116
|
}
|
package/src/ledger.js
CHANGED
|
@@ -1,12 +1,13 @@
|
|
|
1
1
|
import fs from 'node:fs';
|
|
2
2
|
import os from 'node:os';
|
|
3
3
|
import path from 'node:path';
|
|
4
|
+
import { randomUUID } from 'node:crypto';
|
|
4
5
|
import { DatabaseSync } from 'node:sqlite';
|
|
5
6
|
|
|
6
7
|
/**
|
|
7
8
|
* Where the ledger lives unless a path is given: $ONEBROWSER_DB, else onebrowser/sessions.db in the user's data
|
|
8
9
|
* directory ($XDG_DATA_HOME, default ~/.local/share). One ledger per machine is shared by every project on it, so
|
|
9
|
-
* that the record of
|
|
10
|
+
* that the record of an account's sessions is whole, whichever project opened them.
|
|
10
11
|
*/
|
|
11
12
|
export function defaultLedgerPath() {
|
|
12
13
|
if (process.env.ONEBROWSER_DB) return process.env.ONEBROWSER_DB;
|
|
@@ -18,6 +19,8 @@ export function defaultLedgerPath() {
|
|
|
18
19
|
const SCHEMA = `
|
|
19
20
|
CREATE TABLE sessions (
|
|
20
21
|
provider TEXT NOT NULL,
|
|
22
|
+
account TEXT, -- the account it is billed to (Account.id); empty when recorded before the
|
|
23
|
+
-- ledger had accounts (schema 2) and not claimed since (claim())
|
|
21
24
|
id TEXT NOT NULL, -- the session id its provider gave it
|
|
22
25
|
client TEXT NOT NULL, -- the program that opened it (OneBrowser's client)
|
|
23
26
|
opened_at TEXT NOT NULL, -- when it was asked for; its provider created it right after
|
|
@@ -28,17 +31,17 @@ CREATE TABLE sessions (
|
|
|
28
31
|
synced_at TEXT, -- when the provider's record was stored; empty = not yet
|
|
29
32
|
PRIMARY KEY (provider, id)
|
|
30
33
|
);
|
|
31
|
-
CREATE INDEX idx_sessions_opened ON sessions(
|
|
34
|
+
CREATE INDEX idx_sessions_opened ON sessions(account, opened_at);
|
|
32
35
|
`;
|
|
33
36
|
|
|
34
37
|
/**
|
|
35
38
|
* The version of SCHEMA (PRAGMA user_version), and the step to it from each earlier one: a ledger outlives the
|
|
36
|
-
* onebrowser that made it. onebrowser 0.1–0.3 made version 1 and left user_version 0.
|
|
39
|
+
* onebrowser that made it. onebrowser 0.1–0.3 made version 1 and left user_version 0; 0.4 and 0.5 made 2. Each
|
|
40
|
+
* step makes its version as that version was, whatever SCHEMA has become since.
|
|
37
41
|
*/
|
|
38
|
-
const VERSION =
|
|
42
|
+
const VERSION = 3;
|
|
39
43
|
const UPGRADES = {
|
|
40
|
-
// 1 → 2: ends_by, as if the sessions of 1 had the default session options (15 min, then 60 s idle)
|
|
41
|
-
// makes its version as that version was, whatever SCHEMA has become since.
|
|
44
|
+
// 1 → 2: ends_by, as if the sessions of 1 had the default session options (15 min, then 60 s idle)
|
|
42
45
|
1: `ALTER TABLE sessions RENAME TO sessions_1;
|
|
43
46
|
DROP INDEX idx_sessions_opened;
|
|
44
47
|
CREATE TABLE sessions (
|
|
@@ -50,8 +53,26 @@ const UPGRADES = {
|
|
|
50
53
|
SELECT provider, id, client, opened_at, strftime('%Y-%m-%dT%H:%M:%fZ', opened_at, '+16 minutes'), closed_at, duration_ms, proxy_bytes, synced_at
|
|
51
54
|
FROM sessions_1;
|
|
52
55
|
DROP TABLE sessions_1;`,
|
|
56
|
+
// 2 → 3: account, not known for the sessions so far (claim() settles those of a provider with one account)
|
|
57
|
+
2: `ALTER TABLE sessions RENAME TO sessions_2;
|
|
58
|
+
DROP INDEX idx_sessions_opened;
|
|
59
|
+
CREATE TABLE sessions (
|
|
60
|
+
provider TEXT NOT NULL, account TEXT, id TEXT NOT NULL, client TEXT NOT NULL, opened_at TEXT NOT NULL, ends_by TEXT NOT NULL,
|
|
61
|
+
closed_at TEXT, duration_ms INTEGER, proxy_bytes INTEGER, synced_at TEXT, PRIMARY KEY (provider, id)
|
|
62
|
+
);
|
|
63
|
+
CREATE INDEX idx_sessions_opened ON sessions(account, opened_at);
|
|
64
|
+
INSERT INTO sessions (provider, id, client, opened_at, ends_by, closed_at, duration_ms, proxy_bytes, synced_at)
|
|
65
|
+
SELECT provider, id, client, opened_at, ends_by, closed_at, duration_ms, proxy_bytes, synced_at FROM sessions_2;
|
|
66
|
+
DROP TABLE sessions_2;`,
|
|
53
67
|
};
|
|
54
68
|
|
|
69
|
+
/**
|
|
70
|
+
* Beside the schema, meta holds the ledger's own id (`id`): a program that copies the ledger's record elsewhere
|
|
71
|
+
* tells it from another ledger's by it. It came with onebrowser 0.7 without a schema step, made whenever it is
|
|
72
|
+
* missing, so that a ledger a 0.7 has opened still opens in 0.6, which knows schema 3 too and ignores the table.
|
|
73
|
+
*/
|
|
74
|
+
const META = 'CREATE TABLE IF NOT EXISTS meta (key TEXT PRIMARY KEY, value TEXT NOT NULL) WITHOUT ROWID;';
|
|
75
|
+
|
|
55
76
|
/** ms from one ISO time to another */
|
|
56
77
|
const msBetween = (from, to) => `CAST(ROUND((julianday(${to}) - julianday(${from})) * 86400000) AS INTEGER)`;
|
|
57
78
|
|
|
@@ -83,16 +104,29 @@ export class Ledger {
|
|
|
83
104
|
/** Makes the ledger, or brings an earlier one up to date, in one transaction: another process may be opening it too */
|
|
84
105
|
#migrate() {
|
|
85
106
|
const version = () => this.#db.prepare('PRAGMA user_version').get().user_version;
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
107
|
+
const has = (table) => this.#db.prepare("SELECT 1 FROM sqlite_schema WHERE type = 'table' AND name = ?").get(table);
|
|
108
|
+
if (version() !== VERSION) {
|
|
109
|
+
this.tx(() => {
|
|
110
|
+
if (!has('sessions')) this.#db.exec(SCHEMA);
|
|
111
|
+
else {
|
|
112
|
+
const from = version() || 1;
|
|
113
|
+
if (from > VERSION) throw new Error(`onebrowser: the ledger ${this.#file} was made by a later onebrowser (schema ${from}; this one knows ${VERSION})`);
|
|
114
|
+
for (let v = from; v < VERSION; v++) this.#db.exec(UPGRADES[v]);
|
|
115
|
+
}
|
|
116
|
+
this.#db.exec(`PRAGMA user_version = ${VERSION}`);
|
|
117
|
+
});
|
|
118
|
+
}
|
|
119
|
+
if (!has('meta')) {
|
|
120
|
+
this.tx(() => {
|
|
121
|
+
this.#db.exec(META);
|
|
122
|
+
this.#db.prepare("INSERT OR IGNORE INTO meta (key, value) VALUES ('id', ?)").run(randomUUID());
|
|
123
|
+
});
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
/** This ledger's id, given when it was made or first opened by a onebrowser that has ids, and kept for life */
|
|
128
|
+
get id() {
|
|
129
|
+
return this.#q("SELECT value FROM meta WHERE key = 'id'").get().value;
|
|
96
130
|
}
|
|
97
131
|
|
|
98
132
|
#q(sql) {
|
|
@@ -101,37 +135,40 @@ export class Ledger {
|
|
|
101
135
|
return s;
|
|
102
136
|
}
|
|
103
137
|
|
|
104
|
-
opened({ provider, id, client, openedAt, endsBy }) {
|
|
105
|
-
this.#q('INSERT INTO sessions (provider, id, client, opened_at, ends_by) VALUES (
|
|
138
|
+
opened({ provider, account, id, client, openedAt, endsBy }) {
|
|
139
|
+
this.#q('INSERT INTO sessions (provider, account, id, client, opened_at, ends_by) VALUES (?,?,?,?,?,?)').run(provider, account, id, client, iso(openedAt), iso(endsBy));
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
closed({ account, id, closedAt }) {
|
|
143
|
+
this.#q('UPDATE sessions SET closed_at=? WHERE account=? AND id=? AND closed_at IS NULL').run(iso(closedAt), account, id);
|
|
106
144
|
}
|
|
107
145
|
|
|
108
|
-
|
|
109
|
-
|
|
146
|
+
/** The provider's sessions without an account (recorded before schema 3) are `account`'s: the provider's only one */
|
|
147
|
+
claim(provider, account) {
|
|
148
|
+
this.#q('UPDATE sessions SET account=? WHERE provider=? AND account IS NULL').run(account, provider);
|
|
110
149
|
}
|
|
111
150
|
|
|
112
|
-
/** The
|
|
113
|
-
history(
|
|
151
|
+
/** The account's sessions that a quota reading shows, those that ended before `billedBefore` (ms); openSince() has the rest */
|
|
152
|
+
history(account, billedBefore) {
|
|
114
153
|
return {
|
|
115
154
|
/** Their session time (ms) since `since` (ms), each as long as it can have lasted while it has no timing */
|
|
116
155
|
endedSessionMsSince: (since) =>
|
|
117
|
-
this.#q(`SELECT COALESCE(SUM(${SESSION_MS_AT_MOST}), 0) ms FROM sessions WHERE
|
|
156
|
+
this.#q(`SELECT COALESCE(SUM(${SESSION_MS_AT_MOST}), 0) ms FROM sessions WHERE account=? AND opened_at >= ? AND ${ENDED_AT} < ?`).get(account, iso(since), iso(billedBefore)).ms,
|
|
118
157
|
};
|
|
119
158
|
}
|
|
120
159
|
|
|
121
|
-
/** The
|
|
122
|
-
openSince(
|
|
123
|
-
return this.#q(`SELECT ${SESSION_MS_AT_MOST} durationMs FROM sessions WHERE
|
|
160
|
+
/** The account's sessions open at any time since `at` (ms): [{ durationMs }], as long as they can last while untimed */
|
|
161
|
+
openSince(account, at) {
|
|
162
|
+
return this.#q(`SELECT ${SESSION_MS_AT_MOST} durationMs FROM sessions WHERE account=? AND ${ENDED_AT} >= ?`).all(account, iso(at));
|
|
124
163
|
}
|
|
125
164
|
|
|
126
|
-
/**
|
|
127
|
-
awaitingUsage(
|
|
128
|
-
return this.#q(
|
|
129
|
-
'SELECT provider, id, opened_at openedAt, closed_at closedAt FROM sessions WHERE synced_at IS NULL AND provider IN (SELECT value FROM json_each(?)) ORDER BY opened_at',
|
|
130
|
-
).all(JSON.stringify(providers));
|
|
165
|
+
/** The account's sessions without their provider's record yet: [{ id, openedAt, closedAt }] (ISO times) */
|
|
166
|
+
awaitingUsage(account) {
|
|
167
|
+
return this.#q('SELECT id, opened_at openedAt, closed_at closedAt FROM sessions WHERE synced_at IS NULL AND account=? ORDER BY opened_at').all(account);
|
|
131
168
|
}
|
|
132
169
|
|
|
133
|
-
recordUsage(
|
|
134
|
-
this.#q('UPDATE sessions SET duration_ms=?, proxy_bytes=?, synced_at=? WHERE
|
|
170
|
+
recordUsage(account, { id, durationMs, proxyBytes }) {
|
|
171
|
+
this.#q('UPDATE sessions SET duration_ms=?, proxy_bytes=?, synced_at=? WHERE account=? AND id=?').run(int(durationMs), int(proxyBytes), iso(Date.now()), account, id);
|
|
135
172
|
}
|
|
136
173
|
|
|
137
174
|
/** Runs fn in a transaction that takes the write lock at once: another process may be writing too */
|
|
@@ -150,13 +187,24 @@ export class Ledger {
|
|
|
150
187
|
/** See OneBrowser.usage() */
|
|
151
188
|
usage(since) {
|
|
152
189
|
return this.#q(
|
|
153
|
-
`SELECT substr(opened_at, 1, 10) day, provider, client, COUNT(*) sessions,
|
|
190
|
+
`SELECT substr(opened_at, 1, 10) day, provider, COALESCE(account, provider) account, client, COUNT(*) sessions,
|
|
154
191
|
COALESCE(SUM(${SESSION_MS}), 0) / 60000.0 minutes, SUM(${SESSION_MS} IS NULL) untimed,
|
|
155
192
|
SUM(synced_at IS NULL) unsynced, COALESCE(SUM(proxy_bytes), 0) proxyBytes
|
|
156
|
-
FROM sessions WHERE opened_at >= ? GROUP BY day, provider, client ORDER BY day, provider, client`,
|
|
193
|
+
FROM sessions WHERE opened_at >= ? GROUP BY day, provider, account, client ORDER BY day, provider, account, client`,
|
|
157
194
|
).all(iso(since));
|
|
158
195
|
}
|
|
159
196
|
|
|
197
|
+
/** See OneBrowser.sessions() */
|
|
198
|
+
sessions(since) {
|
|
199
|
+
return this.#q(
|
|
200
|
+
`SELECT provider, COALESCE(account, provider) account, id, client, opened_at openedAt, ends_by endsBy, closed_at closedAt,
|
|
201
|
+
duration_ms durationMs, proxy_bytes proxyBytes, synced_at syncedAt
|
|
202
|
+
FROM sessions WHERE opened_at >= ? ORDER BY opened_at, provider, id`,
|
|
203
|
+
)
|
|
204
|
+
.all(iso(since))
|
|
205
|
+
.map((r) => ({ ...r, openedAt: Date.parse(r.openedAt), endsBy: Date.parse(r.endsBy), closedAt: r.closedAt && Date.parse(r.closedAt), syncedAt: r.syncedAt && Date.parse(r.syncedAt) }));
|
|
206
|
+
}
|
|
207
|
+
|
|
160
208
|
close() {
|
|
161
209
|
this.#db.close();
|
|
162
210
|
}
|
package/src/options.js
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
import { KEEP_ALIVE_MS } from './session.js';
|
|
2
|
+
|
|
3
|
+
/** Session options when openSession() is not given them */
|
|
4
|
+
const SESSION_DEFAULTS = {
|
|
5
|
+
proxyCountry: null,
|
|
6
|
+
idleTimeoutSec: 60,
|
|
7
|
+
maxDurationMs: 15 * 60_000,
|
|
8
|
+
};
|
|
9
|
+
|
|
10
|
+
/**
|
|
11
|
+
* The session options of openSession() with the defaults filled in; throws when one is unknown or out of range,
|
|
12
|
+
* so that a program can check its configuration before it opens any session:
|
|
13
|
+
* proxyCountry country code: the provider's proxy there; null: its default egress
|
|
14
|
+
* idleTimeoutSec a session whose client is gone ends this long after at the latest, or after the provider's
|
|
15
|
+
* minIdleTimeoutSec if longer (≥ 2 × KEEP_ALIVE_MS)
|
|
16
|
+
* maxDurationMs a session ends this long after it was opened at the latest
|
|
17
|
+
*/
|
|
18
|
+
export function sessionOptions(options = {}) {
|
|
19
|
+
const unknown = Object.keys(options).filter((k) => !Object.hasOwn(SESSION_DEFAULTS, k));
|
|
20
|
+
if (unknown.length) throw new Error(`unknown session option ${unknown.join(', ')} (expected ${Object.keys(SESSION_DEFAULTS).join(', ')})`);
|
|
21
|
+
const o = { ...SESSION_DEFAULTS, ...options };
|
|
22
|
+
if (!(Number.isFinite(o.idleTimeoutSec) && o.idleTimeoutSec * 1000 >= 2 * KEEP_ALIVE_MS)) {
|
|
23
|
+
throw new Error(`idleTimeoutSec must be at least ${(2 * KEEP_ALIVE_MS) / 1000} (twice the keep-alive interval), not ${o.idleTimeoutSec}`);
|
|
24
|
+
}
|
|
25
|
+
if (!(Number.isFinite(o.maxDurationMs) && o.maxDurationMs > 0)) throw new Error(`maxDurationMs must be a number > 0, not ${o.maxDurationMs}`);
|
|
26
|
+
return o;
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
/** How long a session lasts at most, whatever becomes of its process: closed at maxDurationMs, else ended by its provider once idle */
|
|
30
|
+
export const sessionMsAtMost = (o, settings) => o.maxDurationMs + Math.max(o.idleTimeoutSec, settings.minIdleTimeoutSec) * 1000;
|
package/src/session.js
CHANGED
|
@@ -5,11 +5,12 @@ export const KEEP_ALIVE_MS = 15_000;
|
|
|
5
5
|
const firstLine = (e) => String(e.message).split('\n')[0];
|
|
6
6
|
|
|
7
7
|
/**
|
|
8
|
-
* A remote browser session (
|
|
8
|
+
* A remote browser session (Account.openSession), used over CDP: cdpUrl with any client (it may carry the API
|
|
9
9
|
* key: never log it), or connect() / newPage() with Playwright. This process closes it maxDurationMs after it was
|
|
10
10
|
* opened; one whose process died is ended by its provider idleTimeoutSec (at least its minIdleTimeoutSec) after
|
|
11
11
|
* its client left.
|
|
12
|
-
* openedAt: when it was asked for; closedAt: when its provider released
|
|
12
|
+
* account: the Account it is billed to; openedAt: when it was asked for; closedAt: when its provider released
|
|
13
|
+
* it, null until then.
|
|
13
14
|
*/
|
|
14
15
|
export class Session {
|
|
15
16
|
#release;
|
|
@@ -18,9 +19,9 @@ export class Session {
|
|
|
18
19
|
#keepAlive = null;
|
|
19
20
|
#deadline;
|
|
20
21
|
|
|
21
|
-
/** release(session) → closedAt: releases it at its provider and records it closed (
|
|
22
|
-
constructor({
|
|
23
|
-
this.
|
|
22
|
+
/** release(session) → closedAt: releases it at its provider and records it closed (Account) */
|
|
23
|
+
constructor({ account, remote, openedAt, maxDurationMs, release }) {
|
|
24
|
+
this.account = account;
|
|
24
25
|
this.id = remote.id;
|
|
25
26
|
this.egress = remote.egress;
|
|
26
27
|
this.cdpUrl = remote.cdpUrl;
|
|
@@ -33,7 +34,7 @@ export class Session {
|
|
|
33
34
|
|
|
34
35
|
/** Playwright's Browser over this session's CDP connection, made once; a keep-alive runs while it is connected */
|
|
35
36
|
connect() {
|
|
36
|
-
if (this.#closing) return Promise.reject(new Error(`${this.
|
|
37
|
+
if (this.#closing) return Promise.reject(new Error(`${this.account.id} session ${this.id} is closed`));
|
|
37
38
|
this.#browser ??= this.#connect();
|
|
38
39
|
return this.#browser;
|
|
39
40
|
}
|
|
@@ -44,11 +45,11 @@ export class Session {
|
|
|
44
45
|
try {
|
|
45
46
|
browser = await chromium.connectOverCDP(this.cdpUrl, { timeout: 60_000 });
|
|
46
47
|
} catch (e) {
|
|
47
|
-
throw new Error(`failed to connect to the ${this.provider} browser: ${firstLine(e)}`);
|
|
48
|
+
throw new Error(`failed to connect to the ${this.account.provider} browser: ${firstLine(e)}`);
|
|
48
49
|
}
|
|
49
50
|
try {
|
|
50
51
|
const cdp = await browser.newBrowserCDPSession();
|
|
51
|
-
if (this.#closing) throw new Error(`${this.
|
|
52
|
+
if (this.#closing) throw new Error(`${this.account.id} session ${this.id} is closed`);
|
|
52
53
|
this.#keepAlive = setInterval(() => cdp.send('Browser.getVersion').catch(() => {}), KEEP_ALIVE_MS);
|
|
53
54
|
this.#keepAlive.unref();
|
|
54
55
|
} catch (e) {
|