hazo_collect 0.2.7 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGE_LOG.md +93 -0
- package/README.md +70 -3
- package/config/hazo_collect_config.ini.sample +44 -0
- package/db_setup_postgres.sql +189 -0
- package/db_setup_sqlite.sql +87 -0
- package/dist/chunk-TORPW2KF.js +19 -0
- package/dist/ddl/postgres.sql +141 -6
- package/dist/ddl/sqlite.sql +23 -6
- package/dist/index.d.ts +2 -0
- package/dist/index.js +4 -0
- package/dist/lib/index.d.ts +14 -0
- package/dist/lib/index.js +6 -0
- package/dist/locales/en.json +1 -0
- package/dist/server/index.d.ts +108 -1
- package/dist/server/index.js +283 -26
- package/package.json +15 -8
package/CHANGE_LOG.md
CHANGED
|
@@ -1,5 +1,98 @@
|
|
|
1
1
|
# hazo_collect — Change Log
|
|
2
2
|
|
|
3
|
+
## 0.3.0 — 2026-11-21 (faraday data-foundation ticket 05: shared per-host rate limit)
|
|
4
|
+
|
|
5
|
+
### ⚠️ DDL required
|
|
6
|
+
|
|
7
|
+
New table `hazo_collect_rate_limits` (one token-bucket row per host) plus two
|
|
8
|
+
new Postgres RPC functions, `hazo_collect_take_token_rpc` and
|
|
9
|
+
`hazo_collect_apply_backoff_rpc`, and one new column on
|
|
10
|
+
`hazo_collect_plugin_health` (`last_rate_limit_wait_ms`, default 0).
|
|
11
|
+
Re-run `db_setup_postgres.sql` / `db_setup_sqlite.sql` (idempotent) or apply
|
|
12
|
+
`src/ddl/postgres.sql` / `src/ddl/sqlite.sql` directly. SQLite has no
|
|
13
|
+
RPC/plpgsql, so its atomic "take a token" step is an optimistic
|
|
14
|
+
compare-and-swap retry loop over the same table instead of a stored
|
|
15
|
+
function (see `src/server/rate-limit.ts`, `createSqliteRateLimitStore`) —
|
|
16
|
+
same technique hazo_jobs 0.25.0 used for its SQLite group-completion CAS.
|
|
17
|
+
PostgREST consumers must `GRANT EXECUTE` on both new functions to the role
|
|
18
|
+
PostgREST connects as — see README's "PostgREST setup" note.
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- **Shared, DB-backed per-host token-bucket rate limiting inside `ctx.http`.**
|
|
23
|
+
Consumers configure `requestsPerSecond` + `burst` per host via
|
|
24
|
+
`createManager({ rateLimits: { 'api.example.com': { requestsPerSecond: 5,
|
|
25
|
+
burst: 10 } } })`. A host absent from that map keeps today's unthrottled
|
|
26
|
+
behaviour (no limiting), so this is strictly additive/opt-in. The bucket
|
|
27
|
+
state lives in the database (`hazo_collect_rate_limits`) and is shared by
|
|
28
|
+
every worker/process collecting through it — not just this process's view
|
|
29
|
+
of the limit.
|
|
30
|
+
- **Postgres**: taking a token is one atomic statement,
|
|
31
|
+
`hazo_collect_take_token_rpc` — a single `UPDATE ... FROM (SELECT ...
|
|
32
|
+
FOR UPDATE) ... RETURNING`, so no lock is ever held across requests
|
|
33
|
+
(safe to call via PostgREST's `POST /rpc/hazo_collect_take_token_rpc`,
|
|
34
|
+
not just a raw SQL connection).
|
|
35
|
+
- **SQLite**: the same refill-on-read token-bucket math
|
|
36
|
+
(`computeTokenBucket` in `src/server/rate-limit.ts`) runs in an
|
|
37
|
+
optimistic CAS retry loop over the raw adapter, since SQLite has no
|
|
38
|
+
RPC/plpgsql equivalent.
|
|
39
|
+
- **429 + `Retry-After`** on a rate-limited host trigger backoff: the
|
|
40
|
+
shared bucket is drained and `backoff_until` is raised (monotonically)
|
|
41
|
+
via `hazo_collect_apply_backoff_rpc` / the SQLite equivalent, so every
|
|
42
|
+
caller sharing that host's bucket backs off — not just the process that
|
|
43
|
+
saw the 429.
|
|
44
|
+
- **Collector health** (`hazo_collect_plugin_health.last_rate_limit_wait_ms`)
|
|
45
|
+
records how long the most recent run waited on the limit.
|
|
46
|
+
- **`ManagerOptions.fetch`** — an injected fetch implementation used by
|
|
47
|
+
`ctx.http` in every run the manager dispatches. Lets consumers (and this
|
|
48
|
+
package's own tests) point collectors at a fake host while per-host rate
|
|
49
|
+
limiting still applies, without touching real network code.
|
|
50
|
+
`ManagerOptions.rateLimits` is the per-host config described above; both
|
|
51
|
+
are threaded through a new `createNodeRuntime({ fetch, rateLimits })`
|
|
52
|
+
factory (`nodeRuntime` stays the pre-0.3.0 default instance — no
|
|
53
|
+
behaviour change when neither option is passed).
|
|
54
|
+
- New `hazo_collect/server` exports: `createNodeRuntime`,
|
|
55
|
+
`createRateLimitManager`, `createRateLimitStore`, `computeTokenBucket`,
|
|
56
|
+
`parseRetryAfterMs`, and types `HostRateLimitConfig`, `RateLimitManager`,
|
|
57
|
+
`RateLimitStore`, `TakeTokenResult`, `FetchFn`, `WaitTracker`,
|
|
58
|
+
`NodeRuntimeOptions`.
|
|
59
|
+
- `config/hazo_collect_config.ini.sample` documents the per-host rate-limit
|
|
60
|
+
config shape (`[rate_limit:<host>]`) consumers should map into
|
|
61
|
+
`ManagerOptions.rateLimits` — this package has no ini-loading code of its
|
|
62
|
+
own, so the section is documentation, not a parsed config file.
|
|
63
|
+
|
|
64
|
+
### Scope notes
|
|
65
|
+
|
|
66
|
+
- `RuntimeAdapter.run()`'s return type gained a required `rateLimitWaitMs`
|
|
67
|
+
field (present on both the `ok: true` and `ok: false` branches). This is
|
|
68
|
+
additive for any existing `RuntimeAdapter` implementation that already
|
|
69
|
+
destructures `{ ok, outcome }` / `{ ok, error }` — it simply won't read
|
|
70
|
+
the new field.
|
|
71
|
+
|
|
72
|
+
## 0.2.8 — 2026-09-19
|
|
73
|
+
|
|
74
|
+
### Added
|
|
75
|
+
|
|
76
|
+
Future-proofing i18n plumbing only — no behavior change. hazo_collect is a "zero-string" package
|
|
77
|
+
(headless collector-manager engine, no rendered UI; the string literals in the codebase are
|
|
78
|
+
internal error messages for plugin/collector authors, not end-user copy), so this wires the
|
|
79
|
+
canonical `hazo_core/i18n` bridge without adding any real strings yet:
|
|
80
|
+
|
|
81
|
+
- `src/locales/en.json` — empty flat catalog (`{}`).
|
|
82
|
+
- `src/lib/index.ts` now exports `t = createPackageT('hazo_collect', en)`, re-exported from the
|
|
83
|
+
main entry point. Loads the catalog via `createRequire` rather than
|
|
84
|
+
`import en from '../locales/en.json' with { type: 'json' }` — this package's `tsconfig.json` uses
|
|
85
|
+
`"module": "Node16"`, which throws TS2823 on import attributes syntax (a known incompatibility
|
|
86
|
+
found during the workspace's i18n adoption pass). `tsup.config.ts` gained a `lib/index` build
|
|
87
|
+
entry and copies `src/locales/en.json` to `dist/locales/en.json`. The `createRequire` lookup
|
|
88
|
+
tries a couple of relative depths (`../locales/en.json`, `./locales/en.json`,
|
|
89
|
+
`../../locales/en.json`) rather than one hardcoded path, since tsup's code-splitting extracts
|
|
90
|
+
this module into a shared chunk whose on-disk depth doesn't reliably match the source file's —
|
|
91
|
+
confirmed empirically against a real build. Falls back to `{}` rather than throwing.
|
|
92
|
+
- `hazo_i18n` added to `peerDependencies` (`^0.5.0`) and marked optional in
|
|
93
|
+
`peerDependenciesMeta` — `hazo_core` was already a required peer.
|
|
94
|
+
- Added an `hazo_collect.i18n` autotest scenario group (one trivial case) to the test-app.
|
|
95
|
+
|
|
3
96
|
## Unreleased — test tooling
|
|
4
97
|
|
|
5
98
|
### Fixed
|
package/README.md
CHANGED
|
@@ -75,12 +75,17 @@ const result = await manager.runNow({ plugin: 'my_source' });
|
|
|
75
75
|
|
|
76
76
|
| Export | Description |
|
|
77
77
|
|---|---|
|
|
78
|
-
| `createManager(opts)` | Main orchestration pipeline — resolve → validate → run → persist |
|
|
79
|
-
| `nodeRuntime` |
|
|
78
|
+
| `createManager(opts)` | Main orchestration pipeline — resolve → validate → run → persist. Accepts `fetch` + `rateLimits` (see "Per-host rate limiting" above) |
|
|
79
|
+
| `nodeRuntime` | Default execution adapter for in-process Node.js collectors (no fetch override, no rate limits) |
|
|
80
|
+
| `createNodeRuntime({ fetch, rateLimits })` | Build a `RuntimeAdapter` with an injected fetch and/or per-host rate limits |
|
|
80
81
|
| `createInMemoryRegistry()` | In-memory registry backed by the SDK Map |
|
|
81
82
|
| `persistRegistry(adapter, manifests)` | Upsert manifests into `hazo_collect_plugin_registry`; quarantines invalid entries |
|
|
82
83
|
| `discover(opts)` | Scan directories for `manifest.json` files and return parsed entries |
|
|
83
84
|
| `createWriteAdapter(adapter, runId)` | Create a `ctx.write` function bound to a specific run |
|
|
85
|
+
| `createRateLimitManager(opts)` | Build the `ctx.http` rate-limiting wrapper directly (advanced — `createManager`'s `rateLimits`/`fetch` cover the common case) |
|
|
86
|
+
| `createRateLimitStore(adapter)` | Pick the dialect-appropriate (`pg` / `postgrest` / `sqlite`) atomic token-bucket store for an adapter |
|
|
87
|
+
| `computeTokenBucket(state, cfg, cost, nowMs)` | Pure refill-on-read token-bucket math, shared by the SQLite CAS loop |
|
|
88
|
+
| `parseRetryAfterMs(header, nowMs)` | Parse a `Retry-After` header (seconds or HTTP-date) into a millisecond delay |
|
|
84
89
|
|
|
85
90
|
### `hazo_collect/sdk`
|
|
86
91
|
|
|
@@ -109,8 +114,70 @@ Run migrations from `hazo_collect/ddl/sqlite.sql` (SQLite) or `hazo_collect/ddl/
|
|
|
109
114
|
|
|
110
115
|
- `hazo_collect_plugin_registry` — discovered manifest catalogue with validity + quarantine info
|
|
111
116
|
- `hazo_collect_plugin_runs` — per-run scorecard (status, records, contract_version, watermark)
|
|
112
|
-
- `hazo_collect_plugin_health` — rolling health state (consecutive failures, last status)
|
|
117
|
+
- `hazo_collect_plugin_health` — rolling health state (consecutive failures, last status, last rate-limit wait)
|
|
113
118
|
- `hazo_collect_landing` — idempotent landing zone for collector output rows
|
|
119
|
+
- `hazo_collect_rate_limits` — shared per-host token-bucket state (0.3.0+; see below)
|
|
120
|
+
|
|
121
|
+
## Per-host rate limiting (`ctx.http`)
|
|
122
|
+
|
|
123
|
+
Every collector's `ctx.http` can enforce a shared, database-backed per-host
|
|
124
|
+
token bucket. Configure `requestsPerSecond` + `burst` per host on the
|
|
125
|
+
manager; a host with no entry keeps today's unthrottled behaviour:
|
|
126
|
+
|
|
127
|
+
```ts
|
|
128
|
+
import { createManager } from 'hazo_collect/server';
|
|
129
|
+
|
|
130
|
+
const manager = createManager({
|
|
131
|
+
adapter,
|
|
132
|
+
registry,
|
|
133
|
+
rateLimits: {
|
|
134
|
+
'api.example.com': { requestsPerSecond: 5, burst: 10 },
|
|
135
|
+
},
|
|
136
|
+
});
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Because the bucket lives in `hazo_collect_rate_limits` (not in process
|
|
140
|
+
memory), the limit is shared across every worker/process collecting through
|
|
141
|
+
the same database — not just this process's view of it. A `429` response
|
|
142
|
+
(honouring `Retry-After`, in seconds or as an HTTP-date) drains the shared
|
|
143
|
+
bucket and raises its backoff window, so every caller sharing that host
|
|
144
|
+
backs off, not just the one that saw the 429. `hazo_collect_plugin_health`
|
|
145
|
+
records how long the most recent run waited on the limit
|
|
146
|
+
(`last_rate_limit_wait_ms`).
|
|
147
|
+
|
|
148
|
+
### Injected fetch
|
|
149
|
+
|
|
150
|
+
`createManager({ fetch })` lets you point `ctx.http` at a different fetch
|
|
151
|
+
implementation for every run the manager dispatches — the rate limiter still
|
|
152
|
+
applies on top of it. This is how both this package's own tests and a
|
|
153
|
+
consumer's tests can exercise `ctx.http` against a fake host without any
|
|
154
|
+
real network traffic:
|
|
155
|
+
|
|
156
|
+
```ts
|
|
157
|
+
const manager = createManager({
|
|
158
|
+
adapter,
|
|
159
|
+
registry,
|
|
160
|
+
fetch: fakeFetch, // e.g. a jest.fn() resolving a canned Response
|
|
161
|
+
rateLimits: { 'fake-host.test': { requestsPerSecond: 10, burst: 10 } },
|
|
162
|
+
});
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
`fetch` and `rateLimits` are ignored if you also pass your own `runtime`
|
|
166
|
+
(that runtime owns its own fetch wiring) — use `createNodeRuntime({ fetch,
|
|
167
|
+
rateLimits })` directly in that case.
|
|
168
|
+
|
|
169
|
+
### PostgREST setup
|
|
170
|
+
|
|
171
|
+
If you run `createHazoConnect({ type: 'postgrest', ... })`, the rate limiter
|
|
172
|
+
calls the two new functions via `POST /rpc/<fn>`. Grant execute to the role
|
|
173
|
+
PostgREST connects as:
|
|
174
|
+
|
|
175
|
+
```sql
|
|
176
|
+
GRANT EXECUTE ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) TO api_user;
|
|
177
|
+
GRANT EXECUTE ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) TO api_user;
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
(substitute the role PostgREST actually connects as for `api_user`).
|
|
114
181
|
|
|
115
182
|
## Tailwind v4 (`@source` required)
|
|
116
183
|
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
; hazo_collect configuration
|
|
2
|
+
; Copy to hazo_collect_config.ini and fill in values for your environment.
|
|
3
|
+
|
|
4
|
+
; [env]
|
|
5
|
+
; Uncomment to hard-code the environment name (overrides HAZO_ENV / NODE_ENV).
|
|
6
|
+
; env = development
|
|
7
|
+
|
|
8
|
+
; [secrets]
|
|
9
|
+
; Prefix for environment variables that hold plugin secrets.
|
|
10
|
+
; A manifest declaring secrets = ["API_KEY"] will read env var ${env_prefix}_API_KEY.
|
|
11
|
+
; env_prefix = HAZO_COLLECT_SECRET
|
|
12
|
+
|
|
13
|
+
; [plugins]
|
|
14
|
+
; Directory to scan for plugin manifests (future folder-scan discovery provider).
|
|
15
|
+
; Each sub-directory should contain a manifest.json.
|
|
16
|
+
; dir = ./plugins
|
|
17
|
+
|
|
18
|
+
; [general]
|
|
19
|
+
; Example setting
|
|
20
|
+
; some_key = some_value
|
|
21
|
+
|
|
22
|
+
; [rate_limit:<host>]
|
|
23
|
+
; Per-host token-bucket rate limit enforced inside ctx.http (0.3.0+). One
|
|
24
|
+
; section per host you want throttled — a host with no section here keeps
|
|
25
|
+
; today's unthrottled behaviour. The bucket itself lives in the database
|
|
26
|
+
; (hazo_collect_rate_limits), shared by every worker/process collecting
|
|
27
|
+
; through it, so this config only needs to be loaded wherever createManager()
|
|
28
|
+
; is called — it does not need to agree across processes.
|
|
29
|
+
;
|
|
30
|
+
; This file isn't parsed by hazo_collect itself (there's no ini-loading code
|
|
31
|
+
; in this package yet) — map whatever ini/env/JSON config your app already
|
|
32
|
+
; loads into the shape createManager() expects:
|
|
33
|
+
;
|
|
34
|
+
; createManager({
|
|
35
|
+
; adapter,
|
|
36
|
+
; registry,
|
|
37
|
+
; rateLimits: {
|
|
38
|
+
; 'api.example.com': { requestsPerSecond: 5, burst: 10 },
|
|
39
|
+
; },
|
|
40
|
+
; });
|
|
41
|
+
;
|
|
42
|
+
; [rate_limit:api.example.com]
|
|
43
|
+
; requests_per_second = 5
|
|
44
|
+
; burst = 10
|
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
-- hazo_collect — PostgreSQL DDL
|
|
2
|
+
-- Consumers MAY copy this verbatim or wrap with their own role grants.
|
|
3
|
+
-- Idempotent — safe to re-run.
|
|
4
|
+
--
|
|
5
|
+
-- Tables:
|
|
6
|
+
-- hazo_collect_plugin_runs — one row per plugin run (RunResult scorecard)
|
|
7
|
+
-- hazo_collect_plugin_health — latest health snapshot per plugin
|
|
8
|
+
-- hazo_collect_plugin_registry — discovered manifest catalogue
|
|
9
|
+
-- hazo_collect_watermarks — per-plugin incremental watermark
|
|
10
|
+
-- hazo_collect_landing — raw as-fetched payloads (idempotent on plugin+key)
|
|
11
|
+
-- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+)
|
|
12
|
+
|
|
13
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
|
|
14
|
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
15
|
+
plugin TEXT NOT NULL,
|
|
16
|
+
run_id TEXT NOT NULL UNIQUE,
|
|
17
|
+
kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
|
|
18
|
+
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
19
|
+
completed_at TIMESTAMPTZ,
|
|
20
|
+
status TEXT NOT NULL CHECK (status IN ('success', 'partial', 'failed', 'running')),
|
|
21
|
+
records_fetched INT NOT NULL DEFAULT 0,
|
|
22
|
+
records_written INT NOT NULL DEFAULT 0,
|
|
23
|
+
watermark TEXT,
|
|
24
|
+
errors JSONB NOT NULL DEFAULT '[]'::jsonb,
|
|
25
|
+
metrics JSONB,
|
|
26
|
+
correlation_id TEXT,
|
|
27
|
+
env TEXT
|
|
28
|
+
);
|
|
29
|
+
|
|
30
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
|
|
31
|
+
ON hazo_collect_plugin_runs (plugin, started_at DESC);
|
|
32
|
+
|
|
33
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_run_id_idx
|
|
34
|
+
ON hazo_collect_plugin_runs (run_id);
|
|
35
|
+
|
|
36
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
|
|
37
|
+
plugin TEXT PRIMARY KEY,
|
|
38
|
+
last_success_at TIMESTAMPTZ,
|
|
39
|
+
last_run_at TIMESTAMPTZ,
|
|
40
|
+
last_status TEXT,
|
|
41
|
+
state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
|
|
42
|
+
consecutive_failures INT NOT NULL DEFAULT 0,
|
|
43
|
+
last_rate_limit_wait_ms INT NOT NULL DEFAULT 0
|
|
44
|
+
);
|
|
45
|
+
|
|
46
|
+
-- 0.3.0: existing deployments — add the new column if upgrading in place.
|
|
47
|
+
ALTER TABLE hazo_collect_plugin_health
|
|
48
|
+
ADD COLUMN IF NOT EXISTS last_rate_limit_wait_ms INT NOT NULL DEFAULT 0;
|
|
49
|
+
|
|
50
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
|
|
51
|
+
name TEXT PRIMARY KEY,
|
|
52
|
+
kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
|
|
53
|
+
version TEXT NOT NULL,
|
|
54
|
+
runtime TEXT NOT NULL CHECK (runtime IN ('node', 'python')),
|
|
55
|
+
manifest JSONB NOT NULL,
|
|
56
|
+
source TEXT NOT NULL DEFAULT 'folder' CHECK (source IN ('folder', 'db')),
|
|
57
|
+
discovered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
58
|
+
valid BOOLEAN NOT NULL DEFAULT TRUE,
|
|
59
|
+
quarantine_reason TEXT
|
|
60
|
+
);
|
|
61
|
+
|
|
62
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_watermarks (
|
|
63
|
+
plugin TEXT PRIMARY KEY,
|
|
64
|
+
watermark TEXT NOT NULL,
|
|
65
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
|
66
|
+
);
|
|
67
|
+
|
|
68
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_landing (
|
|
69
|
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
70
|
+
plugin TEXT NOT NULL,
|
|
71
|
+
run_id TEXT NOT NULL,
|
|
72
|
+
idempotency_key TEXT NOT NULL,
|
|
73
|
+
payload JSONB NOT NULL,
|
|
74
|
+
fetched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
75
|
+
window_since TEXT,
|
|
76
|
+
window_until TEXT,
|
|
77
|
+
CONSTRAINT hazo_collect_landing_plugin_key UNIQUE (plugin, idempotency_key)
|
|
78
|
+
);
|
|
79
|
+
|
|
80
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
|
|
81
|
+
ON hazo_collect_landing (plugin, fetched_at DESC);
|
|
82
|
+
|
|
83
|
+
-- ===========================================================================
|
|
84
|
+
-- Shared per-host rate limiting (0.3.0+) — see src/ddl/postgres.sql for the
|
|
85
|
+
-- full design notes. Mirrored here per repo convention.
|
|
86
|
+
-- ===========================================================================
|
|
87
|
+
|
|
88
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
|
|
89
|
+
host TEXT PRIMARY KEY,
|
|
90
|
+
requests_per_second DOUBLE PRECISION NOT NULL,
|
|
91
|
+
burst INT NOT NULL,
|
|
92
|
+
tokens_available DOUBLE PRECISION NOT NULL,
|
|
93
|
+
last_refill_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
94
|
+
backoff_until TIMESTAMPTZ,
|
|
95
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
|
96
|
+
);
|
|
97
|
+
|
|
98
|
+
DROP FUNCTION IF EXISTS hazo_collect_take_token_rpc(text, double precision, int, int);
|
|
99
|
+
|
|
100
|
+
CREATE OR REPLACE FUNCTION hazo_collect_take_token_rpc(
|
|
101
|
+
p_host text,
|
|
102
|
+
p_requests_per_second double precision,
|
|
103
|
+
p_burst int,
|
|
104
|
+
p_cost int DEFAULT 1
|
|
105
|
+
) RETURNS TABLE(allowed boolean, wait_ms int)
|
|
106
|
+
LANGUAGE plpgsql
|
|
107
|
+
SECURITY DEFINER
|
|
108
|
+
AS $$
|
|
109
|
+
DECLARE
|
|
110
|
+
now_ts timestamptz := clock_timestamp();
|
|
111
|
+
BEGIN
|
|
112
|
+
INSERT INTO hazo_collect_rate_limits
|
|
113
|
+
(host, requests_per_second, burst, tokens_available, last_refill_at, updated_at)
|
|
114
|
+
VALUES (p_host, p_requests_per_second, p_burst, p_burst, now_ts, now_ts)
|
|
115
|
+
ON CONFLICT (host) DO NOTHING;
|
|
116
|
+
|
|
117
|
+
RETURN QUERY
|
|
118
|
+
UPDATE hazo_collect_rate_limits r
|
|
119
|
+
SET tokens_available = CASE
|
|
120
|
+
WHEN old.is_allowed THEN old.refilled - p_cost
|
|
121
|
+
ELSE old.refilled
|
|
122
|
+
END,
|
|
123
|
+
last_refill_at = now_ts,
|
|
124
|
+
requests_per_second = p_requests_per_second,
|
|
125
|
+
burst = p_burst,
|
|
126
|
+
updated_at = now_ts
|
|
127
|
+
FROM (
|
|
128
|
+
SELECT
|
|
129
|
+
host,
|
|
130
|
+
LEAST(
|
|
131
|
+
p_burst::double precision,
|
|
132
|
+
tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
|
|
133
|
+
) AS refilled,
|
|
134
|
+
backoff_until,
|
|
135
|
+
(backoff_until IS NULL OR backoff_until <= now_ts)
|
|
136
|
+
AND LEAST(
|
|
137
|
+
p_burst::double precision,
|
|
138
|
+
tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
|
|
139
|
+
) >= p_cost AS is_allowed
|
|
140
|
+
FROM hazo_collect_rate_limits
|
|
141
|
+
WHERE host = p_host
|
|
142
|
+
FOR UPDATE
|
|
143
|
+
) AS old
|
|
144
|
+
WHERE r.host = old.host
|
|
145
|
+
RETURNING
|
|
146
|
+
old.is_allowed AS allowed,
|
|
147
|
+
CASE
|
|
148
|
+
WHEN old.is_allowed THEN 0
|
|
149
|
+
WHEN old.backoff_until IS NOT NULL AND old.backoff_until > now_ts
|
|
150
|
+
THEN CEIL(EXTRACT(EPOCH FROM (old.backoff_until - now_ts)) * 1000)::int
|
|
151
|
+
WHEN p_requests_per_second > 0
|
|
152
|
+
THEN CEIL((p_cost - old.refilled) / p_requests_per_second * 1000)::int
|
|
153
|
+
ELSE 2147483647
|
|
154
|
+
END AS wait_ms;
|
|
155
|
+
END;
|
|
156
|
+
$$;
|
|
157
|
+
|
|
158
|
+
COMMENT ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) IS
|
|
159
|
+
'hazo_collect: atomically take a rate-limit token for a host (refill-on-read token bucket). '
|
|
160
|
+
'Callable via PostgREST POST /rpc/hazo_collect_take_token_rpc. GRANT EXECUTE to the PostgREST role.';
|
|
161
|
+
|
|
162
|
+
DROP FUNCTION IF EXISTS hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int);
|
|
163
|
+
|
|
164
|
+
CREATE OR REPLACE FUNCTION hazo_collect_apply_backoff_rpc(
|
|
165
|
+
p_host text,
|
|
166
|
+
p_backoff_until timestamptz,
|
|
167
|
+
p_requests_per_second double precision,
|
|
168
|
+
p_burst int
|
|
169
|
+
) RETURNS void
|
|
170
|
+
LANGUAGE plpgsql
|
|
171
|
+
SECURITY DEFINER
|
|
172
|
+
AS $$
|
|
173
|
+
BEGIN
|
|
174
|
+
INSERT INTO hazo_collect_rate_limits
|
|
175
|
+
(host, requests_per_second, burst, tokens_available, last_refill_at, backoff_until, updated_at)
|
|
176
|
+
VALUES (p_host, p_requests_per_second, p_burst, 0, clock_timestamp(), p_backoff_until, clock_timestamp())
|
|
177
|
+
ON CONFLICT (host) DO UPDATE
|
|
178
|
+
SET tokens_available = 0,
|
|
179
|
+
backoff_until = GREATEST(hazo_collect_rate_limits.backoff_until, EXCLUDED.backoff_until),
|
|
180
|
+
updated_at = clock_timestamp();
|
|
181
|
+
END;
|
|
182
|
+
$$;
|
|
183
|
+
|
|
184
|
+
COMMENT ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) IS
|
|
185
|
+
'hazo_collect: record a 429 + Retry-After for a host — drains the shared bucket and raises '
|
|
186
|
+
'backoff_until so every caller sharing this host backs off. Callable via PostgREST POST '
|
|
187
|
+
'/rpc/hazo_collect_apply_backoff_rpc. GRANT EXECUTE to the PostgREST role.';
|
|
188
|
+
|
|
189
|
+
NOTIFY pgrst, 'reload schema';
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
-- hazo_collect — SQLite DDL
|
|
2
|
+
-- Idempotent — safe to re-run.
|
|
3
|
+
--
|
|
4
|
+
-- SQLite companion to postgres.sql.
|
|
5
|
+
-- Type mapping: UUID→TEXT, BOOLEAN→INTEGER(0/1), TIMESTAMPTZ→TEXT, JSONB→TEXT.
|
|
6
|
+
--
|
|
7
|
+
-- Tables:
|
|
8
|
+
-- hazo_collect_plugin_runs
|
|
9
|
+
-- hazo_collect_plugin_health
|
|
10
|
+
-- hazo_collect_plugin_registry
|
|
11
|
+
-- hazo_collect_watermarks
|
|
12
|
+
-- hazo_collect_landing
|
|
13
|
+
-- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+);
|
|
14
|
+
-- see src/ddl/sqlite.sql for the CAS-loop design note.
|
|
15
|
+
|
|
16
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
|
|
17
|
+
id TEXT PRIMARY KEY,
|
|
18
|
+
plugin TEXT NOT NULL,
|
|
19
|
+
run_id TEXT NOT NULL UNIQUE,
|
|
20
|
+
kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
|
|
21
|
+
started_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
22
|
+
completed_at TEXT,
|
|
23
|
+
status TEXT NOT NULL CHECK (status IN ('success', 'partial', 'failed', 'running')),
|
|
24
|
+
records_fetched INTEGER NOT NULL DEFAULT 0,
|
|
25
|
+
records_written INTEGER NOT NULL DEFAULT 0,
|
|
26
|
+
watermark TEXT,
|
|
27
|
+
errors TEXT NOT NULL DEFAULT '[]',
|
|
28
|
+
metrics TEXT,
|
|
29
|
+
correlation_id TEXT,
|
|
30
|
+
env TEXT
|
|
31
|
+
);
|
|
32
|
+
|
|
33
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
|
|
34
|
+
ON hazo_collect_plugin_runs (plugin, started_at DESC);
|
|
35
|
+
|
|
36
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
|
|
37
|
+
plugin TEXT PRIMARY KEY,
|
|
38
|
+
last_success_at TEXT,
|
|
39
|
+
last_run_at TEXT,
|
|
40
|
+
last_status TEXT,
|
|
41
|
+
state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
|
|
42
|
+
consecutive_failures INTEGER NOT NULL DEFAULT 0,
|
|
43
|
+
last_rate_limit_wait_ms INTEGER NOT NULL DEFAULT 0
|
|
44
|
+
);
|
|
45
|
+
|
|
46
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
|
|
47
|
+
name TEXT PRIMARY KEY,
|
|
48
|
+
kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
|
|
49
|
+
version TEXT NOT NULL,
|
|
50
|
+
runtime TEXT NOT NULL CHECK (runtime IN ('node', 'python')),
|
|
51
|
+
manifest TEXT NOT NULL,
|
|
52
|
+
source TEXT NOT NULL DEFAULT 'folder' CHECK (source IN ('folder', 'db')),
|
|
53
|
+
discovered_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
54
|
+
valid INTEGER NOT NULL DEFAULT 1,
|
|
55
|
+
quarantine_reason TEXT
|
|
56
|
+
);
|
|
57
|
+
|
|
58
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_watermarks (
|
|
59
|
+
plugin TEXT PRIMARY KEY,
|
|
60
|
+
watermark TEXT NOT NULL,
|
|
61
|
+
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
|
62
|
+
);
|
|
63
|
+
|
|
64
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_landing (
|
|
65
|
+
id TEXT PRIMARY KEY,
|
|
66
|
+
plugin TEXT NOT NULL,
|
|
67
|
+
run_id TEXT NOT NULL,
|
|
68
|
+
idempotency_key TEXT NOT NULL,
|
|
69
|
+
payload TEXT NOT NULL,
|
|
70
|
+
fetched_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
71
|
+
window_since TEXT,
|
|
72
|
+
window_until TEXT,
|
|
73
|
+
CONSTRAINT hazo_collect_landing_plugin_key UNIQUE (plugin, idempotency_key)
|
|
74
|
+
);
|
|
75
|
+
|
|
76
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
|
|
77
|
+
ON hazo_collect_landing (plugin, fetched_at DESC);
|
|
78
|
+
|
|
79
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
|
|
80
|
+
host TEXT PRIMARY KEY,
|
|
81
|
+
requests_per_second REAL NOT NULL,
|
|
82
|
+
burst INTEGER NOT NULL,
|
|
83
|
+
tokens_available REAL NOT NULL,
|
|
84
|
+
last_refill_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
85
|
+
backoff_until TEXT,
|
|
86
|
+
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
|
87
|
+
);
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
// src/lib/index.ts
|
|
2
|
+
import { createRequire } from "module";
|
|
3
|
+
import { createPackageT } from "hazo_core/i18n";
|
|
4
|
+
function loadEnCatalog() {
|
|
5
|
+
const req = createRequire(import.meta.url);
|
|
6
|
+
for (const rel of ["../locales/en.json", "./locales/en.json", "../../locales/en.json"]) {
|
|
7
|
+
try {
|
|
8
|
+
return req(rel);
|
|
9
|
+
} catch {
|
|
10
|
+
}
|
|
11
|
+
}
|
|
12
|
+
return {};
|
|
13
|
+
}
|
|
14
|
+
var en = loadEnCatalog();
|
|
15
|
+
var t = createPackageT("hazo_collect", en);
|
|
16
|
+
|
|
17
|
+
export {
|
|
18
|
+
t
|
|
19
|
+
};
|