onebrowser 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +35 -27
- package/package.json +2 -2
- package/src/account.js +139 -0
- package/src/cli.js +29 -24
- package/src/credentials.js +17 -12
- package/src/index.js +51 -172
- package/src/ledger.js +41 -25
- package/src/options.js +30 -0
- package/src/session.js +9 -8
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# onebrowser
|
|
2
2
|
|
|
3
|
-
One interface to remote browser providers. A
|
|
3
|
+
One interface to remote browser providers. A provider is an adapter; an account is one API key at a provider, and sessions are opened with an account. Every session is recorded in a ledger, which each account's usage and quota are read from.
|
|
4
4
|
|
|
5
5
|
```sh
|
|
6
6
|
npm install onebrowser
|
|
@@ -10,20 +10,21 @@ npm install onebrowser
|
|
|
10
10
|
import { OneBrowser, QuotaError } from 'onebrowser';
|
|
11
11
|
|
|
12
12
|
const browsers = new OneBrowser({ client: 'my-crawler' });
|
|
13
|
-
const
|
|
13
|
+
const [steel] = browsers.accounts('steel'); // one account per key in STEEL_API_KEY
|
|
14
|
+
const session = await steel.openSession({ proxyCountry: 'DE' });
|
|
14
15
|
const page = await session.newPage(); // Playwright; or any CDP client on session.cdpUrl
|
|
15
16
|
await page.goto('https://example.com');
|
|
16
17
|
await session.close();
|
|
17
18
|
|
|
18
|
-
await
|
|
19
|
-
|
|
20
|
-
|
|
19
|
+
await steel.readQuota(); // { remaining, unbilled, unit: 'USD', renewsAt: null }
|
|
20
|
+
steel.quota(); // the same now, from that reading: sessions opened since count at once
|
|
21
|
+
steel.maxSessionCost(); // what a session with the default options costs at most, in USD
|
|
21
22
|
await browsers.syncUsage(); // stores the providers' records of ended sessions in the ledger
|
|
22
|
-
browsers.usage({ since: Date.parse('2026-11-01') }); // [{ day, provider, client, sessions, minutes, … }]
|
|
23
|
+
browsers.usage({ since: Date.parse('2026-11-01') }); // [{ day, provider, account, client, sessions, minutes, … }]
|
|
23
24
|
await browsers.close();
|
|
24
25
|
```
|
|
25
26
|
|
|
26
|
-
Choosing
|
|
27
|
+
Choosing an account, capping sessions and stopping before a quota runs out are the program's policy. onebrowser opens the sessions it is asked for and gives the program what it needs to decide: the accounts it has keys for, what each quota has left, what a session costs at most, and `QuotaError` when a provider refuses a session for the account's quota.
|
|
27
28
|
|
|
28
29
|
Built in: [Steel](https://steel.dev), [TinyFish](https://tinyfish.ai), [Hyperbrowser](https://hyperbrowser.ai), [Lightpanda](https://lightpanda.io), [Kernel](https://www.kernel.sh), [Anchor Browser](https://anchorbrowser.io) and [Firecrawl](https://www.firecrawl.dev); a program can add its own (see [Adding a provider](#adding-a-provider)). Node 22.13 or later.
|
|
29
30
|
|
|
@@ -35,28 +36,33 @@ Built in: [Steel](https://steel.dev), [TinyFish](https://tinyfish.ai), [Hyperbro
|
|
|
35
36
|
|---|---|
|
|
36
37
|
| `client` | The program using it, recorded with each session; required |
|
|
37
38
|
| `ledgerPath` | SQLite file of the ledger (default below; `':memory:'` for tests). Opened when first needed |
|
|
38
|
-
| `providers` | Settings per provider name, on top of its adapter's `defaults`: `
|
|
39
|
+
| `providers` | Settings per provider name, on top of its adapter's `defaults`: `apiKeys` (see [Accounts](#accounts)), `price`, `billingDelayMs`, `minIdleTimeoutSec` (see [Quota](#quota)) and the adapter's own (API base URLs, plan facts such as `monthlyHours`) |
|
|
39
40
|
| `adapters` | Further adapters by provider name, or replacements of built-in ones |
|
|
40
41
|
|
|
41
42
|
| Method | |
|
|
42
43
|
|---|---|
|
|
43
|
-
| `providers` | Names of the providers it
|
|
44
|
-
| `
|
|
45
|
-
| `
|
|
46
|
-
| `
|
|
47
|
-
| `quota(provider)` | What is left now, from the last reading: `{ remaining, unbilled, unit, renewsAt }` (see [Quota](#quota)). Throws until the provider has been read, and while its last reading failed |
|
|
48
|
-
| `maxSessionCost(provider, options)` | What a session with these options costs at most, in the unit of the provider's quota |
|
|
49
|
-
| `syncUsage()` | Stores the providers' records (duration, proxy traffic) of ended sessions: `{ reported, failed: [{ provider, error }] }` |
|
|
50
|
-
| `usage({ since })` | Sessions opened since `since` (ms) per UTC day, provider and client: `sessions`, `minutes` (the provider's record, else the open-to-close timing), `untimed`, `unsynced`, `proxyBytes` |
|
|
44
|
+
| `providers` | Names of the providers it has adapters for |
|
|
45
|
+
| `accounts(provider?)` | The accounts with an API key, in the providers' order and then the keys': all of them, or the provider's (throws for an unknown provider) |
|
|
46
|
+
| `syncUsage()` | Stores the providers' records (duration, proxy traffic) of ended sessions: `{ reported, failed: [{ account, error }] }` |
|
|
47
|
+
| `usage({ since })` | Sessions opened since `since` (ms) per UTC day, account and client: `sessions`, `minutes` (the provider's record, else the open-to-close timing), `untimed`, `unsynced`, `proxyBytes` |
|
|
51
48
|
| `close()` | Closes the sessions still open and the ledger |
|
|
52
49
|
|
|
53
|
-
|
|
50
|
+
An account has `id` (`steel#3f9a1c`: the provider's name and the key's fingerprint), `provider`, `fingerprint`, and:
|
|
51
|
+
|
|
52
|
+
| Method | |
|
|
53
|
+
|---|---|
|
|
54
|
+
| `openSession({ proxyCountry, idleTimeoutSec, maxDurationMs })` | A new session, in the ledger from when the provider has created it. Throws `QuotaError` when the provider refuses it for the account's quota |
|
|
55
|
+
| `readQuota({ maxAgeMs })` | Reads the account's quota from the provider, unless a reading in progress or started at most `maxAgeMs` ago (default 0) can be used, a failed one too; then answers as `quota()` |
|
|
56
|
+
| `quota()` | What is left now, from the last reading: `{ remaining, unbilled, unit, renewsAt }` (see [Quota](#quota)). Throws until the account has been read, and while its last reading failed |
|
|
57
|
+
| `maxSessionCost(options)` | What a session with these options costs at most, in the unit of the quota |
|
|
58
|
+
|
|
59
|
+
A session has `id`, `account`, `egress`, `cdpUrl` (it may carry the API key: never log it), `openedAt` (when it was asked for), `closedAt` (when its provider released it), and:
|
|
54
60
|
|
|
55
61
|
- `connect()`: Playwright's `Browser` over its CDP connection, made once. A keep-alive command every 15 s keeps the provider's idle timeout from ending a session in use.
|
|
56
62
|
- `newPage()`: a new page in the default context: the page a remote browser starts with cannot always be navigated.
|
|
57
63
|
- `close()`: disconnects and releases it; every call answers as the first. Throws when the provider could not release it: the provider ends it by itself, and the ledger counts it as lasting as long as it can.
|
|
58
64
|
|
|
59
|
-
A session and a `OneBrowser` are async-disposable: `await using session = await
|
|
65
|
+
A session and a `OneBrowser` are async-disposable: `await using session = await steel.openSession()` closes the session at the end of its block (Node 24 or later; on Node 22 call `close()`).
|
|
60
66
|
|
|
61
67
|
### Session options: one meaning on every provider
|
|
62
68
|
|
|
@@ -82,7 +88,7 @@ How each provider meets them:
|
|
|
82
88
|
|
|
83
89
|
### Quota
|
|
84
90
|
|
|
85
|
-
`quota()` is the last reading less `unbilled`: what the ledger's sessions that the reading does not show cost at the provider's `price`. A reading shows the sessions that ended `billingDelayMs` before it; the others are open ones, counted as lasting as long as they can (`maxDurationMs`, then `idleTimeoutSec`, at least the provider's `minIdleTimeoutSec`), and those ended since, counted as long as they lasted. So each session counts once from when its provider has created it, whichever program on the machine opened it, and a program can check before each session that the account pays for it (`maxSessionCost`). A provider that bills a session while it runs has part of it counted twice, on the safe side. Not counted: sessions opened through another machine's ledger, until the provider bills them, and proxy traffic billed by the GB.
|
|
91
|
+
`quota()` is the last reading less `unbilled`: what the ledger's sessions of the account that the reading does not show cost at the provider's `price`. A reading shows the sessions that ended `billingDelayMs` before it; the others are open ones, counted as lasting as long as they can (`maxDurationMs`, then `idleTimeoutSec`, at least the provider's `minIdleTimeoutSec`), and those ended since, counted as long as they lasted. So each session counts once from when its provider has created it, whichever program on the machine opened it, and a program can check before each session that the account pays for it (`maxSessionCost`). A provider that bills a session while it runs has part of it counted twice, on the safe side. Not counted: sessions opened through another machine's ledger, until the provider bills them, and proxy traffic billed by the GB.
|
|
86
92
|
|
|
87
93
|
`price` is in the quota's unit: `perHour`, `perSession` (default 0), the session time billed in whole `stepSec`, rounded up (default 1), and at least `minSec` (default 0). Where a provider does not document its rounding, the adapter takes the minute. `renewsAt` (ms) is when the quota renews by itself; `null` when only a top-up renews it. `price` and `billingDelayMs` are settings: a program on another plan sets its own.
|
|
88
94
|
|
|
@@ -98,26 +104,28 @@ How each provider meets them:
|
|
|
98
104
|
|
|
99
105
|
Every provider read from its API showed a session's cost within about a minute of its end (2026-11); `billingDelayMs` leaves room for more.
|
|
100
106
|
|
|
101
|
-
###
|
|
107
|
+
### Accounts
|
|
108
|
+
|
|
109
|
+
An account is one API key at a provider: the quota, the plan's limits and the bill are the key's. The keys of a provider are `apiKeys` in its settings, else those in the environment variable `<PROVIDER>_API_KEY` (`STEEL_API_KEY`, …; `apiKeyEnv(provider)` names it), else those in the file named by `<PROVIDER>_API_KEY_FILE`, separated by commas, spaces or line breaks; each key makes one account, in that order, and a key given twice makes one. A provider without a key has no account.
|
|
102
110
|
|
|
103
|
-
`
|
|
111
|
+
An account is known by its `id`, the provider's name and the first 6 hex digits of the key's SHA-256 (`steel#3f9a1c`): the same for the same key in every program and run, without telling the key. The ledger records sessions under it, so an account's usage is whole however many programs open sessions with its key.
|
|
104
112
|
|
|
105
113
|
### The ledger
|
|
106
114
|
|
|
107
|
-
One SQLite file per machine by default: `$ONEBROWSER_DB`, else `onebrowser/sessions.db` in `$XDG_DATA_HOME` (default `~/.local/share`), shared by every program on the machine under its own `client`. A session is recorded when its provider has created it (it is billed from then on), with when it ends at the latest; closed when its provider has released it; and completed with the provider's own record by `syncUsage()`. A ledger made by an earlier onebrowser is brought up to date when opened; one made by a later onebrowser is refused.
|
|
115
|
+
One SQLite file per machine by default: `$ONEBROWSER_DB`, else `onebrowser/sessions.db` in `$XDG_DATA_HOME` (default `~/.local/share`), shared by every program on the machine under its own `client`. A session is recorded when its provider has created it (it is billed from then on), with its account and when it ends at the latest; closed when its provider has released it; and completed with the provider's own record by `syncUsage()`. A ledger made by an earlier onebrowser is brought up to date when opened; one made by a later onebrowser is refused. Sessions recorded by onebrowser 0.5 or earlier have no account: they become the provider's only account's when it has one here, and stay without one while it has several.
|
|
108
116
|
|
|
109
117
|
## Command line
|
|
110
118
|
|
|
111
|
-
The
|
|
119
|
+
The accounts and the ledger belong to the machine rather than to one program, so the `onebrowser` command reads them without a program of your own. It opens no sessions.
|
|
112
120
|
|
|
113
121
|
```sh
|
|
114
|
-
npx onebrowser quota # what each
|
|
115
|
-
npx onebrowser quota steel kernel
|
|
116
|
-
npx onebrowser usage --days 7 # the ledger's sessions per UTC day,
|
|
122
|
+
npx onebrowser quota # what each account has left, once its sessions not billed yet are paid for
|
|
123
|
+
npx onebrowser quota steel kernel#a81f2c # only these: a provider's accounts, or one account by id
|
|
124
|
+
npx onebrowser usage --days 7 # the ledger's sessions per UTC day, account and client, today included
|
|
117
125
|
npx onebrowser usage --json --env-file .env
|
|
118
126
|
```
|
|
119
127
|
|
|
120
|
-
`usage` first fetches the providers' records of the sessions that have ended, as `syncUsage()` does. With `--json`, `quota` prints an array of what `readQuota()` returns, with the
|
|
128
|
+
`usage` first fetches the providers' records of the sessions that have ended, as `syncUsage()` does. With `--json`, `quota` prints an array of what `readQuota()` returns, with the account: `{ account, provider, remaining, unbilled, unit, renewsAt }`, or `{ account, provider, error }` for a failed reading; `usage` prints `{ ledger, since, sessions, failed }`, where `sessions` is what `usage()` returns and `failed` what `syncUsage()` could not fetch. API keys are read as by the library, from the environment (`--env-file PATH` loads a file into it first). Exit status: 0 done, 1 a reading failed, 2 the command line is wrong.
|
|
121
129
|
|
|
122
130
|
## Adding a provider
|
|
123
131
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "onebrowser",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "One interface to remote browser providers: sessions by provider
|
|
3
|
+
"version": "0.6.0",
|
|
4
|
+
"description": "One interface to remote browser providers: sessions by account (an API key at a provider), a usage ledger, account quotas",
|
|
5
5
|
"keywords": ["browser", "remote-browser", "cdp", "playwright", "steel", "hyperbrowser", "lightpanda", "kernel", "tinyfish", "anchorbrowser", "firecrawl"],
|
|
6
6
|
"license": "MIT",
|
|
7
7
|
"author": "Richard Wang",
|
package/src/account.js
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
import { sessionCost } from './cost.js';
|
|
2
|
+
import { fingerprint } from './credentials.js';
|
|
3
|
+
import { QuotaError } from './errors.js';
|
|
4
|
+
import { sessionMsAtMost, sessionOptions } from './options.js';
|
|
5
|
+
import { Session } from './session.js';
|
|
6
|
+
|
|
7
|
+
/**
|
|
8
|
+
* One API key at one provider: the account its sessions are billed to, with its own quota and its own record in
|
|
9
|
+
* the ledger. id, the provider's name and the key's fingerprint (`steel#3f9a1c`), names it in the ledger, in
|
|
10
|
+
* logs and on the command line; the key itself stays inside.
|
|
11
|
+
*
|
|
12
|
+
* ledger(): the shared Ledger (ledger.js), opened when first needed; client: the program, recorded with each
|
|
13
|
+
* session.
|
|
14
|
+
*/
|
|
15
|
+
export class Account {
|
|
16
|
+
#adapter;
|
|
17
|
+
#settings;
|
|
18
|
+
#apiKey;
|
|
19
|
+
#client;
|
|
20
|
+
#ledger;
|
|
21
|
+
#open = new Set();
|
|
22
|
+
#latest = null; // the latest quota reading, in progress or done
|
|
23
|
+
#done = null; // the latest reading done
|
|
24
|
+
|
|
25
|
+
constructor({ provider, adapter, settings, apiKey, client, ledger }) {
|
|
26
|
+
this.provider = provider;
|
|
27
|
+
this.fingerprint = fingerprint(apiKey);
|
|
28
|
+
this.id = `${provider}#${this.fingerprint}`;
|
|
29
|
+
this.#adapter = adapter;
|
|
30
|
+
this.#settings = settings;
|
|
31
|
+
this.#apiKey = apiKey;
|
|
32
|
+
this.#client = client;
|
|
33
|
+
this.#ledger = ledger;
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
/**
|
|
37
|
+
* A new session (options: sessionOptions()), in the ledger from when the provider has created it. Throws when
|
|
38
|
+
* the provider creates none; QuotaError when it refuses for the account's quota.
|
|
39
|
+
*/
|
|
40
|
+
async openSession(options = {}) {
|
|
41
|
+
const o = sessionOptions(options);
|
|
42
|
+
const openedAt = Date.now();
|
|
43
|
+
let remote;
|
|
44
|
+
try {
|
|
45
|
+
remote = await this.#adapter.createSession(this.#settings, this.#apiKey, o);
|
|
46
|
+
} catch (e) {
|
|
47
|
+
if (e instanceof QuotaError) throw e;
|
|
48
|
+
throw new Error(`failed to create a ${this.provider} session: ${e.message}`);
|
|
49
|
+
}
|
|
50
|
+
const session = new Session({ account: this, remote, openedAt, maxDurationMs: o.maxDurationMs, release: (s) => this.#release(s) });
|
|
51
|
+
try {
|
|
52
|
+
this.#ledger().opened({ provider: this.provider, account: this.id, id: session.id, client: this.#client, openedAt, endsBy: openedAt + sessionMsAtMost(o, this.#settings) });
|
|
53
|
+
} catch (e) {
|
|
54
|
+
await session.close().catch(() => {}); // a session the ledger does not know is not handed out
|
|
55
|
+
throw e;
|
|
56
|
+
}
|
|
57
|
+
this.#open.add(session);
|
|
58
|
+
return session;
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
/** Releases `session` at the provider and records it closed: when it was released, so its whole billed time counts */
|
|
62
|
+
async #release(session) {
|
|
63
|
+
try {
|
|
64
|
+
await this.#adapter.releaseSession(this.#settings, this.#apiKey, session);
|
|
65
|
+
const closedAt = Date.now();
|
|
66
|
+
this.#ledger().closed({ account: this.id, id: session.id, closedAt });
|
|
67
|
+
return closedAt;
|
|
68
|
+
} finally {
|
|
69
|
+
this.#open.delete(session);
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
/**
|
|
74
|
+
* Reads the quota from the provider, unless a reading in progress or started at most maxAgeMs ago can be used
|
|
75
|
+
* (a failed one throws again); then answers as quota().
|
|
76
|
+
*/
|
|
77
|
+
async readQuota({ maxAgeMs = 0 } = {}) {
|
|
78
|
+
const reading = this.#reading(maxAgeMs);
|
|
79
|
+
await reading.settled;
|
|
80
|
+
if (reading.error) throw reading.error;
|
|
81
|
+
return this.quota();
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
/**
|
|
85
|
+
* What is left of the quota now, in the provider's unit: { remaining, unbilled, unit, renewsAt }. The last
|
|
86
|
+
* reading shows the sessions that ended billingDelayMs before it; `unbilled` is what the ledger's others cost
|
|
87
|
+
* at the provider's price, open ones as if they lasted as long as they can, and remaining the reading less
|
|
88
|
+
* that. renewsAt (ms): when the quota renews by itself; null: only a top-up renews it. Throws until the
|
|
89
|
+
* provider has been read, and while the last reading failed.
|
|
90
|
+
*/
|
|
91
|
+
quota() {
|
|
92
|
+
const reading = this.#done;
|
|
93
|
+
if (!reading) throw new Error(`the quota of ${this.id} has not been read yet (readQuota)`);
|
|
94
|
+
if (reading.error) throw reading.error;
|
|
95
|
+
const unbilled = this.#ledger().openSince(this.id, reading.billedBefore).reduce((cost, s) => cost + sessionCost(this.#settings.price, s.durationMs), 0);
|
|
96
|
+
const { remaining, unit, renewsAt = null } = reading.value;
|
|
97
|
+
return { remaining: remaining - unbilled, unbilled, unit, renewsAt };
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
/** The latest reading if it is in progress or at most maxAgeMs old, else a new one */
|
|
101
|
+
#reading(maxAgeMs) {
|
|
102
|
+
const latest = this.#latest;
|
|
103
|
+
if (latest && (latest !== this.#done || Date.now() - latest.startedAt <= maxAgeMs)) return latest;
|
|
104
|
+
const startedAt = Date.now();
|
|
105
|
+
const reading = { startedAt, billedBefore: startedAt - this.#settings.billingDelayMs, value: null, error: null };
|
|
106
|
+
this.#latest = reading;
|
|
107
|
+
reading.settled = (async () => {
|
|
108
|
+
try {
|
|
109
|
+
const q = await this.#adapter.quota(this.#settings, this.#apiKey, this.#ledger().history(this.id, reading.billedBefore));
|
|
110
|
+
if (!(Number.isFinite(q?.remaining) && typeof q.unit === 'string')) throw new Error(`${this.provider} answered no quota: ${JSON.stringify(q)}`);
|
|
111
|
+
reading.value = q;
|
|
112
|
+
} catch (e) {
|
|
113
|
+
reading.error = e;
|
|
114
|
+
}
|
|
115
|
+
this.#done = reading;
|
|
116
|
+
})();
|
|
117
|
+
return reading;
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
/** What a session with these options costs at most, in the unit of the quota */
|
|
121
|
+
maxSessionCost(options) {
|
|
122
|
+
return sessionCost(this.#settings.price, sessionMsAtMost(sessionOptions(options), this.#settings));
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
/** Stores the provider's records (duration, proxy traffic) of this account's ended sessions that have none yet; how many */
|
|
126
|
+
async syncUsage() {
|
|
127
|
+
const ledger = this.#ledger();
|
|
128
|
+
const rows = ledger.awaitingUsage(this.id);
|
|
129
|
+
if (!rows.length) return 0;
|
|
130
|
+
const usage = await this.#adapter.sessionUsage(this.#settings, this.#apiKey, rows);
|
|
131
|
+
ledger.tx(() => usage.forEach((u) => ledger.recordUsage(this.id, u)));
|
|
132
|
+
return usage.length;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
/** Closes its sessions still open; their release errors are ignored: the provider ends those */
|
|
136
|
+
async close() {
|
|
137
|
+
await Promise.all([...this.#open].map((s) => s.close().catch(() => {})));
|
|
138
|
+
}
|
|
139
|
+
}
|
package/src/cli.js
CHANGED
|
@@ -18,6 +18,18 @@ const GLOBAL_OPTIONS = {
|
|
|
18
18
|
help: { type: 'boolean', short: 'h' },
|
|
19
19
|
};
|
|
20
20
|
|
|
21
|
+
/** The accounts `names` ask for: each name a provider's (it must have an API key) or one account's id */
|
|
22
|
+
function askedAccounts(browsers, names) {
|
|
23
|
+
return names.flatMap((name) => {
|
|
24
|
+
const own = browsers.accounts().filter((a) => a.id === name);
|
|
25
|
+
if (own.length) return own;
|
|
26
|
+
if (!browsers.providers.includes(name)) throw new CommandLineError(`unknown provider or account ${name} (expected ${browsers.providers.join(', ')}, or an account id)`);
|
|
27
|
+
const accounts = browsers.accounts(name);
|
|
28
|
+
if (!accounts.length) throw new Error(`${name} has no API key: set ${apiKeyEnv(name)} or ${apiKeyEnv(name)}_FILE`);
|
|
29
|
+
return accounts;
|
|
30
|
+
});
|
|
31
|
+
}
|
|
32
|
+
|
|
21
33
|
/**
|
|
22
34
|
* The commands: their own options, whether they take arguments, and run(browsers, values, args, { out, err })
|
|
23
35
|
* → exit status
|
|
@@ -26,16 +38,16 @@ const COMMANDS = {
|
|
|
26
38
|
quota: {
|
|
27
39
|
positionals: true,
|
|
28
40
|
async run(browsers, { json }, names, { out }) {
|
|
29
|
-
const
|
|
30
|
-
if (
|
|
31
|
-
const
|
|
32
|
-
|
|
33
|
-
|
|
41
|
+
const accounts = names.length ? askedAccounts(browsers, names) : browsers.accounts();
|
|
42
|
+
if (!accounts.length) throw new Error(`no provider has an API key: set ${browsers.providers.map(apiKeyEnv).join(', ')}`);
|
|
43
|
+
const quotas = await Promise.all(
|
|
44
|
+
accounts.map((a) => a.readQuota().then((q) => ({ account: a.id, provider: a.provider, ...q }), (e) => ({ account: a.id, provider: a.provider, error: e.message }))),
|
|
45
|
+
);
|
|
34
46
|
if (json) out.write(`${JSON.stringify(quotas, null, 2)}\n`);
|
|
35
47
|
else {
|
|
36
48
|
for (const q of quotas) {
|
|
37
49
|
const text = q.error ? `unknown (${q.error})` : `${round(q.remaining)} ${q.unit} left${q.unbilled ? ` (after ${round(q.unbilled)} not billed yet)` : ''}${q.renewsAt ? `, renews ${day(q.renewsAt)}` : ''}`;
|
|
38
|
-
out.write(`${q.
|
|
50
|
+
out.write(`${q.account.padEnd(21)} ${text}\n`);
|
|
39
51
|
}
|
|
40
52
|
}
|
|
41
53
|
return quotas.some((q) => q.error) ? 1 : 0;
|
|
@@ -49,15 +61,15 @@ const COMMANDS = {
|
|
|
49
61
|
if (!(Number.isInteger(n) && n >= 1)) throw new CommandLineError(`--days must be a whole number ≥ 1, not ${days}`);
|
|
50
62
|
const since = new Date().setUTCHours(0, 0, 0, 0) - (n - 1) * DAY_MS;
|
|
51
63
|
const { failed } = await browsers.syncUsage();
|
|
52
|
-
for (const f of failed) err.write(`${f.
|
|
64
|
+
for (const f of failed) err.write(`${f.account}: the provider's records were not fetched (${f.error})\n`);
|
|
53
65
|
const sessions = browsers.usage({ since });
|
|
54
66
|
if (json) out.write(`${JSON.stringify({ ledger: browsers.ledgerPath, since: day(since), sessions, failed }, null, 2)}\n`);
|
|
55
67
|
else {
|
|
56
68
|
out.write(`sessions since ${day(since)} in ${browsers.ledgerPath}:\n`);
|
|
57
69
|
if (!sessions.length) out.write(' none\n');
|
|
58
|
-
for (const r of sessions) out.write(usageLine(`${r.day} ${r.
|
|
59
|
-
for (const [
|
|
60
|
-
if (rows.length > 1) out.write(usageLine(`total ${
|
|
70
|
+
for (const r of sessions) out.write(usageLine(`${r.day} ${r.account} ${r.client}`, r));
|
|
71
|
+
for (const [account, rows] of Map.groupBy(sessions, (r) => r.account)) {
|
|
72
|
+
if (rows.length > 1) out.write(usageLine(`total ${account}`, sumUsage(rows)));
|
|
61
73
|
}
|
|
62
74
|
}
|
|
63
75
|
return failed.length ? 1 : 0;
|
|
@@ -68,9 +80,10 @@ const COMMANDS = {
|
|
|
68
80
|
const help = (providers) => `onebrowser: the quotas of remote browser provider accounts, and the sessions in this machine's ledger
|
|
69
81
|
|
|
70
82
|
Usage:
|
|
71
|
-
onebrowser quota [
|
|
72
|
-
|
|
73
|
-
|
|
83
|
+
onebrowser quota [NAME...] what each account has left, less what the sessions in the ledger that it has
|
|
84
|
+
not billed yet cost; NAME: a provider (every account with its API key) or an
|
|
85
|
+
account id such as steel#3f9a1c; default: every account
|
|
86
|
+
onebrowser usage [--days N] sessions per UTC day, account and client in the last N UTC days, today
|
|
74
87
|
included (default 30), after fetching the providers' records of ended sessions
|
|
75
88
|
|
|
76
89
|
Options:
|
|
@@ -79,27 +92,19 @@ Options:
|
|
|
79
92
|
-h, --help
|
|
80
93
|
|
|
81
94
|
Environment:
|
|
82
|
-
${providers.map((p) => ` ${apiKeyEnv(p).padEnd(22)} API
|
|
95
|
+
${providers.map((p) => ` ${apiKeyEnv(p).padEnd(22)} API keys of ${p}, one per account, or ${apiKeyEnv(p)}_FILE naming a file that holds them`).join('\n')}
|
|
96
|
+
(keys are separated by commas, spaces or line breaks)
|
|
83
97
|
ONEBROWSER_DB the ledger file (default: onebrowser/sessions.db in $XDG_DATA_HOME, else ~/.local/share)
|
|
84
98
|
|
|
85
99
|
Exit status: 0 done, 1 a reading failed, 2 the command line is wrong.
|
|
86
100
|
`;
|
|
87
101
|
|
|
88
|
-
function isReady(browsers, provider) {
|
|
89
|
-
try {
|
|
90
|
-
browsers.checkReady(provider);
|
|
91
|
-
return true;
|
|
92
|
-
} catch {
|
|
93
|
-
return false;
|
|
94
|
-
}
|
|
95
|
-
}
|
|
96
|
-
|
|
97
102
|
const USAGE_FIELDS = ['sessions', 'minutes', 'untimed', 'unsynced', 'proxyBytes'];
|
|
98
103
|
|
|
99
104
|
const sumUsage = (rows) => Object.fromEntries(USAGE_FIELDS.map((k) => [k, rows.reduce((t, r) => t + r[k], 0)]));
|
|
100
105
|
|
|
101
106
|
const usageLine = (label, r) =>
|
|
102
|
-
` ${label.padEnd(
|
|
107
|
+
` ${label.padEnd(40)} ${String(r.sessions).padStart(5)} sessions ${r.minutes.toFixed(1).padStart(8)} min` +
|
|
103
108
|
` ${(r.proxyBytes / 1e6).toFixed(1).padStart(7)} MB proxy` +
|
|
104
109
|
(r.untimed ? ` (${r.untimed} untimed)` : '') +
|
|
105
110
|
(r.unsynced ? ` (${r.unsynced} not reported by the provider yet)` : '') +
|
package/src/credentials.js
CHANGED
|
@@ -1,22 +1,27 @@
|
|
|
1
|
+
import crypto from 'node:crypto';
|
|
1
2
|
import fs from 'node:fs';
|
|
2
3
|
|
|
3
|
-
/** The environment variable holding a provider's API
|
|
4
|
+
/** The environment variable holding a provider's API keys: STEEL_API_KEY for `steel` */
|
|
4
5
|
export const apiKeyEnv = (provider) => `${provider.toUpperCase()}_API_KEY`;
|
|
5
6
|
|
|
7
|
+
/** What names the account an API key belongs to without telling the key: the first 6 hex digits of its SHA-256 */
|
|
8
|
+
export const fingerprint = (apiKey) => crypto.createHash('sha256').update(apiKey).digest('hex').slice(0, 6);
|
|
9
|
+
|
|
10
|
+
/** Keys written one after another, separated by commas, spaces or line breaks */
|
|
11
|
+
const split = (text) => text.split(/[\s,]+/).filter(Boolean);
|
|
12
|
+
|
|
6
13
|
/**
|
|
7
|
-
* API
|
|
8
|
-
* else the file named by that variable with _FILE appended
|
|
9
|
-
*
|
|
14
|
+
* A provider's API keys, one per account: `apiKeys` as given in its settings, else those in the environment
|
|
15
|
+
* variable apiKeyEnv(provider), else those in the file named by that variable with _FILE appended, in their
|
|
16
|
+
* order and each once. Separators: commas, spaces or line breaks. None is fine; the file must exist.
|
|
10
17
|
*/
|
|
11
|
-
export function
|
|
12
|
-
if (apiKey?.trim()) return apiKey.trim();
|
|
18
|
+
export function readApiKeys(provider, apiKeys) {
|
|
13
19
|
const env = apiKeyEnv(provider);
|
|
14
20
|
const file = process.env[`${env}_FILE`];
|
|
15
|
-
let
|
|
16
|
-
if (!
|
|
17
|
-
if (!fs.existsSync(file)) throw new Error(`${provider}
|
|
18
|
-
|
|
21
|
+
let keys = apiKeys ?? split(process.env[env] ?? '');
|
|
22
|
+
if (!keys.length && file) {
|
|
23
|
+
if (!fs.existsSync(file)) throw new Error(`${provider} keys: ${env}_FILE names no file (${file})`);
|
|
24
|
+
keys = split(fs.readFileSync(file, 'utf8'));
|
|
19
25
|
}
|
|
20
|
-
|
|
21
|
-
return key.trim();
|
|
26
|
+
return [...new Set(keys.map((k) => k.trim()).filter(Boolean))];
|
|
22
27
|
}
|
package/src/index.js
CHANGED
|
@@ -1,225 +1,104 @@
|
|
|
1
|
-
import {
|
|
2
|
-
import {
|
|
1
|
+
import { Account } from './account.js';
|
|
2
|
+
import { readApiKeys } from './credentials.js';
|
|
3
3
|
import { QuotaError } from './errors.js';
|
|
4
4
|
import { defaultLedgerPath, Ledger } from './ledger.js';
|
|
5
5
|
import { ADAPTERS, checkAdapter, fullSettings } from './providers/index.js';
|
|
6
|
-
import { KEEP_ALIVE_MS, Session } from './session.js';
|
|
7
6
|
|
|
8
7
|
export { apiKeyEnv } from './credentials.js';
|
|
8
|
+
export { sessionOptions } from './options.js';
|
|
9
9
|
export { QuotaError };
|
|
10
10
|
|
|
11
|
-
/** Session options when openSession() is not given them */
|
|
12
|
-
const SESSION_DEFAULTS = {
|
|
13
|
-
proxyCountry: null,
|
|
14
|
-
idleTimeoutSec: 60,
|
|
15
|
-
maxDurationMs: 15 * 60_000,
|
|
16
|
-
};
|
|
17
|
-
|
|
18
|
-
/**
|
|
19
|
-
* The session options of openSession() with the defaults filled in; throws when one is unknown or out of range,
|
|
20
|
-
* so that a program can check its configuration before it opens any session:
|
|
21
|
-
* proxyCountry country code: the provider's proxy there; null: its default egress
|
|
22
|
-
* idleTimeoutSec a session whose client is gone ends this long after at the latest, or after the provider's
|
|
23
|
-
* minIdleTimeoutSec if longer (≥ 2 × KEEP_ALIVE_MS)
|
|
24
|
-
* maxDurationMs a session ends this long after it was opened at the latest
|
|
25
|
-
*/
|
|
26
|
-
export function sessionOptions(options = {}) {
|
|
27
|
-
const unknown = Object.keys(options).filter((k) => !Object.hasOwn(SESSION_DEFAULTS, k));
|
|
28
|
-
if (unknown.length) throw new Error(`unknown session option ${unknown.join(', ')} (expected ${Object.keys(SESSION_DEFAULTS).join(', ')})`);
|
|
29
|
-
const o = { ...SESSION_DEFAULTS, ...options };
|
|
30
|
-
if (!(Number.isFinite(o.idleTimeoutSec) && o.idleTimeoutSec * 1000 >= 2 * KEEP_ALIVE_MS)) {
|
|
31
|
-
throw new Error(`idleTimeoutSec must be at least ${(2 * KEEP_ALIVE_MS) / 1000} (twice the keep-alive interval), not ${o.idleTimeoutSec}`);
|
|
32
|
-
}
|
|
33
|
-
if (!(Number.isFinite(o.maxDurationMs) && o.maxDurationMs > 0)) throw new Error(`maxDurationMs must be a number > 0, not ${o.maxDurationMs}`);
|
|
34
|
-
return o;
|
|
35
|
-
}
|
|
36
|
-
|
|
37
|
-
/** How long a session lasts at most, whatever becomes of its process: closed at maxDurationMs, else ended by its provider once idle */
|
|
38
|
-
const sessionMsAtMost = (o, settings) => o.maxDurationMs + Math.max(o.idleTimeoutSec, settings.minIdleTimeoutSec) * 1000;
|
|
39
|
-
|
|
40
11
|
/**
|
|
41
|
-
* Remote browsers from several providers behind one interface
|
|
42
|
-
*
|
|
43
|
-
*
|
|
44
|
-
* stop using one, are the
|
|
12
|
+
* Remote browsers from several providers behind one interface. A provider is an adapter (providers/index.js);
|
|
13
|
+
* an account is one API key at a provider (account.js), which sessions are opened with. Every session is
|
|
14
|
+
* recorded in a ledger (ledger.js) shared by the programs on the machine, so that each account's usage, and
|
|
15
|
+
* what its quota has left at any moment, are known. Which account to use, and when to stop using one, are the
|
|
16
|
+
* caller's decisions.
|
|
45
17
|
*
|
|
46
18
|
* client: the program using it, recorded with each session; ledgerPath: the ledger's SQLite file, opened when
|
|
47
|
-
* first needed; providers: settings per provider name on top of its adapter's defaults
|
|
48
|
-
*
|
|
19
|
+
* first needed; providers: settings per provider name on top of its adapter's defaults, apiKeys among them
|
|
20
|
+
* (else credentials.js); adapters: further or replacement adapters by provider name.
|
|
49
21
|
*/
|
|
50
22
|
export class OneBrowser {
|
|
51
|
-
#providers =
|
|
23
|
+
#providers = []; // names, in the adapters' order
|
|
24
|
+
#accounts = [];
|
|
52
25
|
#ledger = null;
|
|
53
|
-
#open = new Set();
|
|
54
|
-
#readings = new Map(); // provider → { latest, done }: its latest quota reading, and the latest one done
|
|
55
26
|
|
|
56
27
|
constructor({ client, ledgerPath = defaultLedgerPath(), providers = {}, adapters = {} }) {
|
|
57
28
|
if (!client?.trim()) throw new Error('onebrowser: client must name the program using it');
|
|
58
29
|
this.client = client;
|
|
59
30
|
this.ledgerPath = ledgerPath;
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
const { apiKey, ...given } = providers[name] ?? {};
|
|
63
|
-
this.#providers.set(name, { adapter, settings: fullSettings(name, { ...adapter.defaults, ...given }), apiKey });
|
|
64
|
-
}
|
|
65
|
-
const unknown = Object.keys(providers).filter((name) => !this.#providers.has(name));
|
|
31
|
+
const all = { ...ADAPTERS, ...adapters };
|
|
32
|
+
const unknown = Object.keys(providers).filter((name) => !all[name]);
|
|
66
33
|
if (unknown.length) throw new Error(`onebrowser: no adapter for provider ${unknown.join(', ')}`);
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
}
|
|
76
|
-
|
|
77
|
-
#known(provider) {
|
|
78
|
-
const p = this.#providers.get(provider);
|
|
79
|
-
if (!p) throw new Error(`unknown provider ${provider} (expected ${this.providers.join(', ')})`);
|
|
80
|
-
return p;
|
|
81
|
-
}
|
|
82
|
-
|
|
83
|
-
#ready(provider) {
|
|
84
|
-
const p = this.#known(provider);
|
|
85
|
-
return { ...p, apiKey: readApiKey(provider, p.apiKey) };
|
|
86
|
-
}
|
|
87
|
-
|
|
88
|
-
/** Throws when sessions cannot be opened with `provider`: unknown, or no API key */
|
|
89
|
-
checkReady(provider) {
|
|
90
|
-
this.#ready(provider);
|
|
91
|
-
}
|
|
92
|
-
|
|
93
|
-
/**
|
|
94
|
-
* A new session with `provider` (options: sessionOptions()), in the ledger from when its provider has created
|
|
95
|
-
* it. Throws when the provider creates none; QuotaError when it refuses for the account's quota.
|
|
96
|
-
*/
|
|
97
|
-
async openSession({ provider, ...options }) {
|
|
98
|
-
const { adapter, settings, apiKey } = this.#ready(provider);
|
|
99
|
-
const o = sessionOptions(options);
|
|
100
|
-
const openedAt = Date.now();
|
|
101
|
-
let remote;
|
|
102
|
-
try {
|
|
103
|
-
remote = await adapter.createSession(settings, apiKey, o);
|
|
104
|
-
} catch (e) {
|
|
105
|
-
if (e instanceof QuotaError) throw e;
|
|
106
|
-
throw new Error(`failed to create a ${provider} session: ${e.message}`);
|
|
107
|
-
}
|
|
108
|
-
const session = new Session({ provider, remote, openedAt, maxDurationMs: o.maxDurationMs, release: (s) => this.#release(adapter, settings, apiKey, s) });
|
|
109
|
-
try {
|
|
110
|
-
this.#store.opened({ provider, id: session.id, client: this.client, openedAt, endsBy: openedAt + sessionMsAtMost(o, settings) });
|
|
111
|
-
} catch (e) {
|
|
112
|
-
await session.close().catch(() => {}); // a session the ledger does not know is not handed out
|
|
113
|
-
throw e;
|
|
114
|
-
}
|
|
115
|
-
this.#open.add(session);
|
|
116
|
-
return session;
|
|
117
|
-
}
|
|
118
|
-
|
|
119
|
-
/** Releases `session` at its provider and records it closed: when it was released, so its whole billed time counts */
|
|
120
|
-
async #release(adapter, settings, apiKey, session) {
|
|
121
|
-
try {
|
|
122
|
-
await adapter.releaseSession(settings, apiKey, session);
|
|
123
|
-
const closedAt = Date.now();
|
|
124
|
-
this.#store.closed({ provider: session.provider, id: session.id, closedAt });
|
|
125
|
-
return closedAt;
|
|
126
|
-
} finally {
|
|
127
|
-
this.#open.delete(session);
|
|
34
|
+
for (const [provider, adapter] of Object.entries(all)) {
|
|
35
|
+
checkAdapter(provider, adapter);
|
|
36
|
+
const { apiKeys, ...given } = providers[provider] ?? {};
|
|
37
|
+
const settings = fullSettings(provider, { ...adapter.defaults, ...given });
|
|
38
|
+
this.#providers.push(provider);
|
|
39
|
+
for (const apiKey of readApiKeys(provider, apiKeys)) {
|
|
40
|
+
this.#accounts.push(new Account({ provider, adapter, settings, apiKey, client, ledger: () => this.#store }));
|
|
41
|
+
}
|
|
128
42
|
}
|
|
129
43
|
}
|
|
130
44
|
|
|
131
45
|
/**
|
|
132
|
-
*
|
|
133
|
-
*
|
|
134
|
-
*/
|
|
135
|
-
async readQuota(provider, { maxAgeMs = 0 } = {}) {
|
|
136
|
-
const reading = this.#reading(provider, maxAgeMs);
|
|
137
|
-
await reading.settled;
|
|
138
|
-
if (reading.error) throw reading.error;
|
|
139
|
-
return this.quota(provider);
|
|
140
|
-
}
|
|
141
|
-
|
|
142
|
-
/**
|
|
143
|
-
* What is left of the account's quota now, in the provider's unit: { remaining, unbilled, unit, renewsAt }. The
|
|
144
|
-
* last reading shows the sessions that ended billingDelayMs before it; `unbilled` is what the ledger's others
|
|
145
|
-
* cost at the provider's price, open ones as if they lasted as long as they can, and remaining the reading less
|
|
146
|
-
* that. renewsAt (ms): when the quota renews by itself; null: only a top-up renews it. Throws until the
|
|
147
|
-
* provider has been read, and while its last reading failed.
|
|
46
|
+
* The ledger, opened on first use. Sessions recorded before accounts were (onebrowser 0.5 and earlier) are
|
|
47
|
+
* the provider's only account's, when it has one here; with several, no one knows which they belong to.
|
|
148
48
|
*/
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
return
|
|
49
|
+
get #store() {
|
|
50
|
+
if (!this.#ledger) {
|
|
51
|
+
this.#ledger = new Ledger(this.ledgerPath);
|
|
52
|
+
for (const [provider, accounts] of Map.groupBy(this.#accounts, (a) => a.provider)) {
|
|
53
|
+
if (accounts.length === 1) this.#ledger.claim(provider, accounts[0].id);
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
return this.#ledger;
|
|
157
57
|
}
|
|
158
58
|
|
|
159
|
-
/**
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
this.#readings.set(provider, readings);
|
|
163
|
-
const { latest } = readings;
|
|
164
|
-
const inProgress = latest && latest !== readings.done;
|
|
165
|
-
if (inProgress || (latest && Date.now() - latest.startedAt <= maxAgeMs)) return latest;
|
|
166
|
-
const { adapter, settings, apiKey } = this.#ready(provider);
|
|
167
|
-
const startedAt = Date.now();
|
|
168
|
-
const reading = { startedAt, billedBefore: startedAt - settings.billingDelayMs, value: null, error: null };
|
|
169
|
-
readings.latest = reading;
|
|
170
|
-
reading.settled = (async () => {
|
|
171
|
-
try {
|
|
172
|
-
const q = await adapter.quota(settings, apiKey, this.#store.history(provider, reading.billedBefore));
|
|
173
|
-
if (!(Number.isFinite(q?.remaining) && typeof q.unit === 'string')) throw new Error(`${provider} answered no quota: ${JSON.stringify(q)}`);
|
|
174
|
-
reading.value = q;
|
|
175
|
-
} catch (e) {
|
|
176
|
-
reading.error = e;
|
|
177
|
-
}
|
|
178
|
-
readings.done = reading;
|
|
179
|
-
})();
|
|
180
|
-
return reading;
|
|
59
|
+
/** Names of the providers it has adapters for */
|
|
60
|
+
get providers() {
|
|
61
|
+
return [...this.#providers];
|
|
181
62
|
}
|
|
182
63
|
|
|
183
|
-
/**
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
64
|
+
/** The accounts with an API key, in the providers' order and then the keys': all, or `provider`'s (throws for an unknown one) */
|
|
65
|
+
accounts(provider) {
|
|
66
|
+
if (provider === undefined) return [...this.#accounts];
|
|
67
|
+
if (!this.#providers.includes(provider)) throw new Error(`unknown provider ${provider} (expected ${this.#providers.join(', ')})`);
|
|
68
|
+
return this.#accounts.filter((a) => a.provider === provider);
|
|
187
69
|
}
|
|
188
70
|
|
|
189
71
|
/**
|
|
190
72
|
* Stores the providers' records (duration, proxy traffic) of the ledger's ended sessions that have none yet:
|
|
191
|
-
* { reported, failed: [{
|
|
192
|
-
*
|
|
73
|
+
* { reported, failed: [{ account, error }] }. What cannot be fetched now, and the sessions of accounts without
|
|
74
|
+
* a key here, are left for a later call.
|
|
193
75
|
*/
|
|
194
76
|
async syncUsage() {
|
|
195
|
-
const store = this.#store;
|
|
196
77
|
let reported = 0;
|
|
197
78
|
const failed = [];
|
|
198
|
-
for (const
|
|
79
|
+
for (const account of this.#accounts) {
|
|
199
80
|
try {
|
|
200
|
-
|
|
201
|
-
const usage = await adapter.sessionUsage(settings, apiKey, rows);
|
|
202
|
-
store.tx(() => usage.forEach((u) => store.recordUsage(provider, u)));
|
|
203
|
-
reported += usage.length;
|
|
81
|
+
reported += await account.syncUsage();
|
|
204
82
|
} catch (e) {
|
|
205
|
-
failed.push({
|
|
83
|
+
failed.push({ account: account.id, error: e.message });
|
|
206
84
|
}
|
|
207
85
|
}
|
|
208
86
|
return { reported, failed };
|
|
209
87
|
}
|
|
210
88
|
|
|
211
89
|
/**
|
|
212
|
-
* Sessions opened since `since` (ms), per UTC day,
|
|
213
|
-
* [{ day, provider, client, sessions, minutes, untimed, unsynced, proxyBytes }]. minutes: the
|
|
214
|
-
* (syncUsage), else the open-to-close timing; untimed: sessions with neither; unsynced:
|
|
90
|
+
* Sessions opened since `since` (ms), per UTC day, account and client:
|
|
91
|
+
* [{ day, provider, account, client, sessions, minutes, untimed, unsynced, proxyBytes }]. minutes: the
|
|
92
|
+
* provider's record (syncUsage), else the open-to-close timing; untimed: sessions with neither; unsynced:
|
|
93
|
+
* those without the record. account: the provider's name alone for sessions whose account is not known.
|
|
215
94
|
*/
|
|
216
95
|
usage({ since }) {
|
|
217
96
|
return this.#store.usage(since);
|
|
218
97
|
}
|
|
219
98
|
|
|
220
|
-
/** Closes every session still open
|
|
99
|
+
/** Closes every session still open and the ledger */
|
|
221
100
|
async close() {
|
|
222
|
-
await Promise.all(
|
|
101
|
+
await Promise.all(this.#accounts.map((a) => a.close()));
|
|
223
102
|
this.#ledger?.close();
|
|
224
103
|
this.#ledger = null;
|
|
225
104
|
}
|
package/src/ledger.js
CHANGED
|
@@ -6,7 +6,7 @@ import { DatabaseSync } from 'node:sqlite';
|
|
|
6
6
|
/**
|
|
7
7
|
* Where the ledger lives unless a path is given: $ONEBROWSER_DB, else onebrowser/sessions.db in the user's data
|
|
8
8
|
* directory ($XDG_DATA_HOME, default ~/.local/share). One ledger per machine is shared by every project on it, so
|
|
9
|
-
* that the record of
|
|
9
|
+
* that the record of an account's sessions is whole, whichever project opened them.
|
|
10
10
|
*/
|
|
11
11
|
export function defaultLedgerPath() {
|
|
12
12
|
if (process.env.ONEBROWSER_DB) return process.env.ONEBROWSER_DB;
|
|
@@ -18,6 +18,8 @@ export function defaultLedgerPath() {
|
|
|
18
18
|
const SCHEMA = `
|
|
19
19
|
CREATE TABLE sessions (
|
|
20
20
|
provider TEXT NOT NULL,
|
|
21
|
+
account TEXT, -- the account it is billed to (Account.id); empty when not known: recorded
|
|
22
|
+
-- before onebrowser 0.6, by a provider with several accounts here since
|
|
21
23
|
id TEXT NOT NULL, -- the session id its provider gave it
|
|
22
24
|
client TEXT NOT NULL, -- the program that opened it (OneBrowser's client)
|
|
23
25
|
opened_at TEXT NOT NULL, -- when it was asked for; its provider created it right after
|
|
@@ -28,17 +30,17 @@ CREATE TABLE sessions (
|
|
|
28
30
|
synced_at TEXT, -- when the provider's record was stored; empty = not yet
|
|
29
31
|
PRIMARY KEY (provider, id)
|
|
30
32
|
);
|
|
31
|
-
CREATE INDEX idx_sessions_opened ON sessions(
|
|
33
|
+
CREATE INDEX idx_sessions_opened ON sessions(account, opened_at);
|
|
32
34
|
`;
|
|
33
35
|
|
|
34
36
|
/**
|
|
35
37
|
* The version of SCHEMA (PRAGMA user_version), and the step to it from each earlier one: a ledger outlives the
|
|
36
|
-
* onebrowser that made it. onebrowser 0.1–0.3 made version 1 and left user_version 0.
|
|
38
|
+
* onebrowser that made it. onebrowser 0.1–0.3 made version 1 and left user_version 0; 0.4 and 0.5 made 2. Each
|
|
39
|
+
* step makes its version as that version was, whatever SCHEMA has become since.
|
|
37
40
|
*/
|
|
38
|
-
const VERSION =
|
|
41
|
+
const VERSION = 3;
|
|
39
42
|
const UPGRADES = {
|
|
40
|
-
// 1 → 2: ends_by, as if the sessions of 1 had the default session options (15 min, then 60 s idle)
|
|
41
|
-
// makes its version as that version was, whatever SCHEMA has become since.
|
|
43
|
+
// 1 → 2: ends_by, as if the sessions of 1 had the default session options (15 min, then 60 s idle)
|
|
42
44
|
1: `ALTER TABLE sessions RENAME TO sessions_1;
|
|
43
45
|
DROP INDEX idx_sessions_opened;
|
|
44
46
|
CREATE TABLE sessions (
|
|
@@ -50,6 +52,17 @@ const UPGRADES = {
|
|
|
50
52
|
SELECT provider, id, client, opened_at, strftime('%Y-%m-%dT%H:%M:%fZ', opened_at, '+16 minutes'), closed_at, duration_ms, proxy_bytes, synced_at
|
|
51
53
|
FROM sessions_1;
|
|
52
54
|
DROP TABLE sessions_1;`,
|
|
55
|
+
// 2 → 3: account, not known for the sessions so far (claim() settles those of a provider with one account)
|
|
56
|
+
2: `ALTER TABLE sessions RENAME TO sessions_2;
|
|
57
|
+
DROP INDEX idx_sessions_opened;
|
|
58
|
+
CREATE TABLE sessions (
|
|
59
|
+
provider TEXT NOT NULL, account TEXT, id TEXT NOT NULL, client TEXT NOT NULL, opened_at TEXT NOT NULL, ends_by TEXT NOT NULL,
|
|
60
|
+
closed_at TEXT, duration_ms INTEGER, proxy_bytes INTEGER, synced_at TEXT, PRIMARY KEY (provider, id)
|
|
61
|
+
);
|
|
62
|
+
CREATE INDEX idx_sessions_opened ON sessions(account, opened_at);
|
|
63
|
+
INSERT INTO sessions (provider, id, client, opened_at, ends_by, closed_at, duration_ms, proxy_bytes, synced_at)
|
|
64
|
+
SELECT provider, id, client, opened_at, ends_by, closed_at, duration_ms, proxy_bytes, synced_at FROM sessions_2;
|
|
65
|
+
DROP TABLE sessions_2;`,
|
|
53
66
|
};
|
|
54
67
|
|
|
55
68
|
/** ms from one ISO time to another */
|
|
@@ -101,37 +114,40 @@ export class Ledger {
|
|
|
101
114
|
return s;
|
|
102
115
|
}
|
|
103
116
|
|
|
104
|
-
opened({ provider, id, client, openedAt, endsBy }) {
|
|
105
|
-
this.#q('INSERT INTO sessions (provider, id, client, opened_at, ends_by) VALUES (
|
|
117
|
+
opened({ provider, account, id, client, openedAt, endsBy }) {
|
|
118
|
+
this.#q('INSERT INTO sessions (provider, account, id, client, opened_at, ends_by) VALUES (?,?,?,?,?,?)').run(provider, account, id, client, iso(openedAt), iso(endsBy));
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
closed({ account, id, closedAt }) {
|
|
122
|
+
this.#q('UPDATE sessions SET closed_at=? WHERE account=? AND id=? AND closed_at IS NULL').run(iso(closedAt), account, id);
|
|
106
123
|
}
|
|
107
124
|
|
|
108
|
-
|
|
109
|
-
|
|
125
|
+
/** The provider's sessions whose account is not known are `account`'s */
|
|
126
|
+
claim(provider, account) {
|
|
127
|
+
this.#q('UPDATE sessions SET account=? WHERE provider=? AND account IS NULL').run(account, provider);
|
|
110
128
|
}
|
|
111
129
|
|
|
112
|
-
/** The
|
|
113
|
-
history(
|
|
130
|
+
/** The account's sessions that a quota reading shows, those that ended before `billedBefore` (ms); openSince() has the rest */
|
|
131
|
+
history(account, billedBefore) {
|
|
114
132
|
return {
|
|
115
133
|
/** Their session time (ms) since `since` (ms), each as long as it can have lasted while it has no timing */
|
|
116
134
|
endedSessionMsSince: (since) =>
|
|
117
|
-
this.#q(`SELECT COALESCE(SUM(${SESSION_MS_AT_MOST}), 0) ms FROM sessions WHERE
|
|
135
|
+
this.#q(`SELECT COALESCE(SUM(${SESSION_MS_AT_MOST}), 0) ms FROM sessions WHERE account=? AND opened_at >= ? AND ${ENDED_AT} < ?`).get(account, iso(since), iso(billedBefore)).ms,
|
|
118
136
|
};
|
|
119
137
|
}
|
|
120
138
|
|
|
121
|
-
/** The
|
|
122
|
-
openSince(
|
|
123
|
-
return this.#q(`SELECT ${SESSION_MS_AT_MOST} durationMs FROM sessions WHERE
|
|
139
|
+
/** The account's sessions open at any time since `at` (ms): [{ durationMs }], as long as they can last while untimed */
|
|
140
|
+
openSince(account, at) {
|
|
141
|
+
return this.#q(`SELECT ${SESSION_MS_AT_MOST} durationMs FROM sessions WHERE account=? AND ${ENDED_AT} >= ?`).all(account, iso(at));
|
|
124
142
|
}
|
|
125
143
|
|
|
126
|
-
/**
|
|
127
|
-
awaitingUsage(
|
|
128
|
-
return this.#q(
|
|
129
|
-
'SELECT provider, id, opened_at openedAt, closed_at closedAt FROM sessions WHERE synced_at IS NULL AND provider IN (SELECT value FROM json_each(?)) ORDER BY opened_at',
|
|
130
|
-
).all(JSON.stringify(providers));
|
|
144
|
+
/** The account's sessions without their provider's record yet: [{ id, openedAt, closedAt }] (ISO times) */
|
|
145
|
+
awaitingUsage(account) {
|
|
146
|
+
return this.#q('SELECT id, opened_at openedAt, closed_at closedAt FROM sessions WHERE synced_at IS NULL AND account=? ORDER BY opened_at').all(account);
|
|
131
147
|
}
|
|
132
148
|
|
|
133
|
-
recordUsage(
|
|
134
|
-
this.#q('UPDATE sessions SET duration_ms=?, proxy_bytes=?, synced_at=? WHERE
|
|
149
|
+
recordUsage(account, { id, durationMs, proxyBytes }) {
|
|
150
|
+
this.#q('UPDATE sessions SET duration_ms=?, proxy_bytes=?, synced_at=? WHERE account=? AND id=?').run(int(durationMs), int(proxyBytes), iso(Date.now()), account, id);
|
|
135
151
|
}
|
|
136
152
|
|
|
137
153
|
/** Runs fn in a transaction that takes the write lock at once: another process may be writing too */
|
|
@@ -150,10 +166,10 @@ export class Ledger {
|
|
|
150
166
|
/** See OneBrowser.usage() */
|
|
151
167
|
usage(since) {
|
|
152
168
|
return this.#q(
|
|
153
|
-
`SELECT substr(opened_at, 1, 10) day, provider, client, COUNT(*) sessions,
|
|
169
|
+
`SELECT substr(opened_at, 1, 10) day, provider, COALESCE(account, provider) account, client, COUNT(*) sessions,
|
|
154
170
|
COALESCE(SUM(${SESSION_MS}), 0) / 60000.0 minutes, SUM(${SESSION_MS} IS NULL) untimed,
|
|
155
171
|
SUM(synced_at IS NULL) unsynced, COALESCE(SUM(proxy_bytes), 0) proxyBytes
|
|
156
|
-
FROM sessions WHERE opened_at >= ? GROUP BY day, provider, client ORDER BY day, provider, client`,
|
|
172
|
+
FROM sessions WHERE opened_at >= ? GROUP BY day, provider, account, client ORDER BY day, provider, account, client`,
|
|
157
173
|
).all(iso(since));
|
|
158
174
|
}
|
|
159
175
|
|
package/src/options.js
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
import { KEEP_ALIVE_MS } from './session.js';
|
|
2
|
+
|
|
3
|
+
/** Session options when openSession() is not given them */
|
|
4
|
+
const SESSION_DEFAULTS = {
|
|
5
|
+
proxyCountry: null,
|
|
6
|
+
idleTimeoutSec: 60,
|
|
7
|
+
maxDurationMs: 15 * 60_000,
|
|
8
|
+
};
|
|
9
|
+
|
|
10
|
+
/**
|
|
11
|
+
* The session options of openSession() with the defaults filled in; throws when one is unknown or out of range,
|
|
12
|
+
* so that a program can check its configuration before it opens any session:
|
|
13
|
+
* proxyCountry country code: the provider's proxy there; null: its default egress
|
|
14
|
+
* idleTimeoutSec a session whose client is gone ends this long after at the latest, or after the provider's
|
|
15
|
+
* minIdleTimeoutSec if longer (≥ 2 × KEEP_ALIVE_MS)
|
|
16
|
+
* maxDurationMs a session ends this long after it was opened at the latest
|
|
17
|
+
*/
|
|
18
|
+
export function sessionOptions(options = {}) {
|
|
19
|
+
const unknown = Object.keys(options).filter((k) => !Object.hasOwn(SESSION_DEFAULTS, k));
|
|
20
|
+
if (unknown.length) throw new Error(`unknown session option ${unknown.join(', ')} (expected ${Object.keys(SESSION_DEFAULTS).join(', ')})`);
|
|
21
|
+
const o = { ...SESSION_DEFAULTS, ...options };
|
|
22
|
+
if (!(Number.isFinite(o.idleTimeoutSec) && o.idleTimeoutSec * 1000 >= 2 * KEEP_ALIVE_MS)) {
|
|
23
|
+
throw new Error(`idleTimeoutSec must be at least ${(2 * KEEP_ALIVE_MS) / 1000} (twice the keep-alive interval), not ${o.idleTimeoutSec}`);
|
|
24
|
+
}
|
|
25
|
+
if (!(Number.isFinite(o.maxDurationMs) && o.maxDurationMs > 0)) throw new Error(`maxDurationMs must be a number > 0, not ${o.maxDurationMs}`);
|
|
26
|
+
return o;
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
/** How long a session lasts at most, whatever becomes of its process: closed at maxDurationMs, else ended by its provider once idle */
|
|
30
|
+
export const sessionMsAtMost = (o, settings) => o.maxDurationMs + Math.max(o.idleTimeoutSec, settings.minIdleTimeoutSec) * 1000;
|
package/src/session.js
CHANGED
|
@@ -5,11 +5,12 @@ export const KEEP_ALIVE_MS = 15_000;
|
|
|
5
5
|
const firstLine = (e) => String(e.message).split('\n')[0];
|
|
6
6
|
|
|
7
7
|
/**
|
|
8
|
-
* A remote browser session (
|
|
8
|
+
* A remote browser session (Account.openSession), used over CDP: cdpUrl with any client (it may carry the API
|
|
9
9
|
* key: never log it), or connect() / newPage() with Playwright. This process closes it maxDurationMs after it was
|
|
10
10
|
* opened; one whose process died is ended by its provider idleTimeoutSec (at least its minIdleTimeoutSec) after
|
|
11
11
|
* its client left.
|
|
12
|
-
* openedAt: when it was asked for; closedAt: when its provider released
|
|
12
|
+
* account: the Account it is billed to; openedAt: when it was asked for; closedAt: when its provider released
|
|
13
|
+
* it, null until then.
|
|
13
14
|
*/
|
|
14
15
|
export class Session {
|
|
15
16
|
#release;
|
|
@@ -18,9 +19,9 @@ export class Session {
|
|
|
18
19
|
#keepAlive = null;
|
|
19
20
|
#deadline;
|
|
20
21
|
|
|
21
|
-
/** release(session) → closedAt: releases it at its provider and records it closed (
|
|
22
|
-
constructor({
|
|
23
|
-
this.
|
|
22
|
+
/** release(session) → closedAt: releases it at its provider and records it closed (Account) */
|
|
23
|
+
constructor({ account, remote, openedAt, maxDurationMs, release }) {
|
|
24
|
+
this.account = account;
|
|
24
25
|
this.id = remote.id;
|
|
25
26
|
this.egress = remote.egress;
|
|
26
27
|
this.cdpUrl = remote.cdpUrl;
|
|
@@ -33,7 +34,7 @@ export class Session {
|
|
|
33
34
|
|
|
34
35
|
/** Playwright's Browser over this session's CDP connection, made once; a keep-alive runs while it is connected */
|
|
35
36
|
connect() {
|
|
36
|
-
if (this.#closing) return Promise.reject(new Error(`${this.
|
|
37
|
+
if (this.#closing) return Promise.reject(new Error(`${this.account.id} session ${this.id} is closed`));
|
|
37
38
|
this.#browser ??= this.#connect();
|
|
38
39
|
return this.#browser;
|
|
39
40
|
}
|
|
@@ -44,11 +45,11 @@ export class Session {
|
|
|
44
45
|
try {
|
|
45
46
|
browser = await chromium.connectOverCDP(this.cdpUrl, { timeout: 60_000 });
|
|
46
47
|
} catch (e) {
|
|
47
|
-
throw new Error(`failed to connect to the ${this.provider} browser: ${firstLine(e)}`);
|
|
48
|
+
throw new Error(`failed to connect to the ${this.account.provider} browser: ${firstLine(e)}`);
|
|
48
49
|
}
|
|
49
50
|
try {
|
|
50
51
|
const cdp = await browser.newBrowserCDPSession();
|
|
51
|
-
if (this.#closing) throw new Error(`${this.
|
|
52
|
+
if (this.#closing) throw new Error(`${this.account.id} session ${this.id} is closed`);
|
|
52
53
|
this.#keepAlive = setInterval(() => cdp.send('Browser.getVersion').catch(() => {}), KEEP_ALIVE_MS);
|
|
53
54
|
this.#keepAlive.unref();
|
|
54
55
|
} catch (e) {
|