@pikku/kysely 0.13.7 → 0.13.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +180 -0
- package/dist/src/index.d.ts +4 -1
- package/dist/src/index.js +3 -1
- package/dist/src/kysely-audit-service.d.ts +34 -2
- package/dist/src/kysely-audit-service.js +163 -3
- package/dist/src/kysely-secret-service.d.ts +4 -3
- package/dist/src/kysely-secret-service.js +6 -3
- package/dist/src/kysely-tables.d.ts +31 -0
- package/dist/src/kysely-virtual-user-run-store.d.ts +43 -0
- package/dist/src/kysely-virtual-user-run-store.js +131 -0
- package/dist/src/schema/audit.schema.d.ts +16 -0
- package/dist/src/schema/audit.schema.js +47 -0
- package/dist/src/schema/credential.schema.d.ts +3 -2
- package/dist/src/schema/credential.schema.js +4 -3
- package/dist/src/schema/index.d.ts +17 -4
- package/dist/src/schema/index.js +40 -6
- package/dist/src/schema/pikku-schema.types.d.ts +41 -5
- package/dist/src/schema/pikku-schema.types.js +27 -12
- package/dist/src/schema/virtual-user.schema.d.ts +19 -0
- package/dist/src/schema/virtual-user.schema.js +54 -0
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/package.json +2 -2
- package/src/create-audited-kysely.test.ts +10 -10
- package/src/index.ts +7 -0
- package/src/kysely-audit-service.test.ts +360 -0
- package/src/kysely-audit-service.ts +181 -4
- package/src/kysely-secret-service.ts +11 -10
- package/src/kysely-services.test.ts +3 -3
- package/src/kysely-tables.test.ts +12 -1
- package/src/kysely-tables.ts +32 -0
- package/src/kysely-virtual-user-run-store.test.ts +212 -0
- package/src/kysely-virtual-user-run-store.ts +149 -0
- package/src/schema/audit.schema.ts +55 -0
- package/src/schema/credential.schema.ts +5 -3
- package/src/schema/index.ts +59 -10
- package/src/schema/pikku-schema.types.ts +65 -10
- package/src/schema/schema.test.ts +82 -4
- package/src/schema/virtual-user.schema.ts +65 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,183 @@
|
|
|
1
|
+
## 0.13.10
|
|
2
|
+
|
|
3
|
+
### Patch Changes
|
|
4
|
+
|
|
5
|
+
- 3df4f95: Scaffold virtual user runs as RPCs, backed by a run store.
|
|
6
|
+
|
|
7
|
+
`pikku persona run` could already turn a declared persona loose on a running
|
|
8
|
+
stage, but only from a terminal, and the result only existed in that terminal's
|
|
9
|
+
output. There was no way for CI, a console, or a scheduled job to start a run —
|
|
10
|
+
and nothing kept what a run found, so this week's findings could not be compared
|
|
11
|
+
against last week's.
|
|
12
|
+
|
|
13
|
+
`scaffold.virtualUser` now generates two RPCs and the function behind them:
|
|
14
|
+
- `runVirtualUser({ persona, goals?, memory?, disposition?, budget?, seed? })
|
|
15
|
+
-> { runId }`
|
|
16
|
+
- `getVirtualUserRun({ runId }) -> { status, findings, tally, memory, … }`
|
|
17
|
+
|
|
18
|
+
They are gated on separate scopes — `virtualUser:run` and `virtualUser:read` —
|
|
19
|
+
because an adversarial run's findings are working exploits carrying live ids,
|
|
20
|
+
which makes reading them the more sensitive of the two. Production refuses every
|
|
21
|
+
disposition but `accountable`, checked against the effective one so the
|
|
22
|
+
per-run override cannot smuggle another in.
|
|
23
|
+
|
|
24
|
+
**A run is not a workflow and not a queued job.** It explores, so no two
|
|
25
|
+
attempts take the same steps and there is nothing to replay; and the record
|
|
26
|
+
already carries the progress a queue would only be holding on the way to the
|
|
27
|
+
same place. `runVirtualUser` writes the record, dispatches without awaiting, and
|
|
28
|
+
returns the id. The cost is stated on the type: a restart mid-run strands a
|
|
29
|
+
record at `running`, so a run older than its budget window and still `running`
|
|
30
|
+
is dead rather than working.
|
|
31
|
+
|
|
32
|
+
`@pikku/core` gains `VirtualUserRunStore` (with `virtualUserRunStore` on
|
|
33
|
+
`CoreSingletonServices`), and `@pikku/kysely` ships
|
|
34
|
+
`KyselyVirtualUserRunStore`, which creates its own table on first use like the
|
|
35
|
+
audit sink — the runtime never needs it, so it arrives with the feature that
|
|
36
|
+
fills it rather than in every database.
|
|
37
|
+
|
|
38
|
+
Also in core: `prepareVirtualUserRun`, which derives the catalogue, intents,
|
|
39
|
+
scopes and reachable agents in one place. `pikku persona run` reads the
|
|
40
|
+
inspector state and the generated RPC reads `metaService`, and the two have to
|
|
41
|
+
agree — otherwise the same persona and seed explore a different API depending on
|
|
42
|
+
how the run was started. `personaScopes` moved here from the CLI for the same
|
|
43
|
+
reason and is still re-exported from its old home. `PRODUCTION_DISPOSITION` is
|
|
44
|
+
now exported from `@pikku/core/virtual-user`, which it should always have been.
|
|
45
|
+
|
|
46
|
+
- Updated dependencies [3df4f95]
|
|
47
|
+
- @pikku/core@0.12.77
|
|
48
|
+
|
|
49
|
+
## 0.13.9
|
|
50
|
+
|
|
51
|
+
### Patch Changes
|
|
52
|
+
|
|
53
|
+
- 62ea4cc: The audit trail is now readable — in the generated meta, through an RPC, and as
|
|
54
|
+
a page in the console.
|
|
55
|
+
|
|
56
|
+
`audit: true` reaches `FunctionRuntimeMeta.audit` as its resolved form
|
|
57
|
+
(`{ durability }`), so which functions record anything is answerable without
|
|
58
|
+
running them. It is informational: the runner still resolves audit from the live
|
|
59
|
+
function config, so meta and runtime cannot disagree.
|
|
60
|
+
|
|
61
|
+
`AuditService` grows an optional read side — `query(AuditQuery)` and `facets()`.
|
|
62
|
+
Optional because a sink can legitimately be write-only: a queue producer that
|
|
63
|
+
hands events to another system has nothing to read back, and a reader that finds
|
|
64
|
+
these absent should say the trail is not readable here rather than that it is
|
|
65
|
+
empty. The two are very different answers to give someone auditing a system.
|
|
66
|
+
|
|
67
|
+
`KyselyAuditService` implements both, newest first with offset paging, filtered
|
|
68
|
+
by user, action and time window. Two things it now gets right that are easy to
|
|
69
|
+
get wrong: an empty filter array means "match nothing" rather than "no filter",
|
|
70
|
+
and results are read by physical _and_ camelCase key, because `CamelCasePlugin`
|
|
71
|
+
is on most pikku Kysely instances and renames result keys on the way out — the
|
|
72
|
+
mismatch does not throw, it returns a page of `undefined`. `init()` creates the
|
|
73
|
+
`audit` table for projects that do not migrate it themselves, from a new
|
|
74
|
+
exported `auditSchema` that stays out of `pikkuSchemas` because the runtime does
|
|
75
|
+
not need it.
|
|
76
|
+
|
|
77
|
+
The console addon exposes `console:getAudits` and `console:getAuditFilters`
|
|
78
|
+
behind a new `pikku:audit:read` scope, and forwards the application's `audit`
|
|
79
|
+
service into the addon's own services — without that last part every install
|
|
80
|
+
reported the trail as unreadable, whatever sink it had configured.
|
|
81
|
+
|
|
82
|
+
The console gets an Audit trail page: an infinite list filtered server-side by
|
|
83
|
+
user and action, and a row that opens the whole event, metadata rendered as a
|
|
84
|
+
JSON tree. Refused, unreadable and empty are three different screens, because
|
|
85
|
+
"you may not read this", "nobody can read this" and "nothing happened" are three
|
|
86
|
+
different facts.
|
|
87
|
+
|
|
88
|
+
Events name the person who caused them. The trail records a user id — the only
|
|
89
|
+
thing stable enough to record, since a name can change after the event — so
|
|
90
|
+
`getAudits` resolves those ids against better-auth's user directory at read
|
|
91
|
+
time, and the page shows the name while keeping the recorded id on the event.
|
|
92
|
+
The filter follows: pick a colleague by name, filter by the id. A scenario
|
|
93
|
+
actor is labelled as one, so synthetic traffic is not mistaken for real, and a
|
|
94
|
+
caller who was signed out shows the wire identity pikku resolved for them
|
|
95
|
+
rather than being credited to the system.
|
|
96
|
+
|
|
97
|
+
**Breaking, for anyone already reading `AuditEvent`:** `actor` is now
|
|
98
|
+
`userIdentity`, and its type `AuditActor` is `AuditUserIdentity`; `AuditQuery`
|
|
99
|
+
takes `userIds`/`orgId` in place of `actorUserIds`/`actorOrgId`, and
|
|
100
|
+
`AuditFacets` returns `userIds`. In pikku an _actor_ is a synthetic person a
|
|
101
|
+
scenario drives, flagged on the user row — so naming the causer of an event
|
|
102
|
+
`actor` made the synthetic case unsayable (`actor.actor === true`) and implied
|
|
103
|
+
every recorded action was a test. The overwhelming majority are ordinary
|
|
104
|
+
customers. The `audit` table follows: `actor_user_id` / `actor_org_id`
|
|
105
|
+
are now `user_id` / `org_id`, and a `pikku_user_id` column joins them so the
|
|
106
|
+
wire identity of a caller who never signed in survives the round trip — the
|
|
107
|
+
sink was dropping it, which left the console's Session field permanently
|
|
108
|
+
blank. A project that already migrated the table needs to rename the two
|
|
109
|
+
columns and add the third; `KyselyAuditService.init()` creates the new shape
|
|
110
|
+
for anyone who did not.
|
|
111
|
+
|
|
112
|
+
- 78b29f0: `SecretService` now returns a `SecretValue<T>` rather than the bare value, so a
|
|
113
|
+
vault secret cannot reach a sink by accident.
|
|
114
|
+
|
|
115
|
+
`SecretValue` is nominally typed, which means it is not assignable to `string`
|
|
116
|
+
(or to any other concretely-typed field). Every sink with a real type — a
|
|
117
|
+
database column, an email body, a session payload — rejects it with no lint
|
|
118
|
+
rule involved. The sinks typed `any`, `unknown`, or a free generic — the logger,
|
|
119
|
+
queue payloads, webhook and email inputs, and a function's own output — are
|
|
120
|
+
guarded with `Safe<T>`, which collapses a `SecretValue` found anywhere inside
|
|
121
|
+
`T`, however deeply nested, to `never`.
|
|
122
|
+
|
|
123
|
+
Unwrap deliberately at the point the secret reaches the wire:
|
|
124
|
+
|
|
125
|
+
```ts
|
|
126
|
+
const secret = await secrets.getSecret('BETTER_AUTH_SECRET')
|
|
127
|
+
betterAuth({ secret: secret.reveal() })
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Two behaviours cover what types cannot see. Structured serialization redacts —
|
|
131
|
+
`JSON.stringify` and node's inspect both yield `[secret]`, so an audit or log
|
|
132
|
+
write stays honest without crashing the request. String coercion throws
|
|
133
|
+
`SecretCoercionError`, because a template literal is always a leak.
|
|
134
|
+
|
|
135
|
+
`AuditLog.write` is guarded the same way as the logger, since an audit event
|
|
136
|
+
carries `input` and `metadata` as `unknown` and nominality alone cannot stop a
|
|
137
|
+
secret landing in one.
|
|
138
|
+
|
|
139
|
+
`.reveal()` is the deliberate escape hatch, and what it hands back is an
|
|
140
|
+
ordinary string as far as every sink signature is concerned. **PKU953** closes
|
|
141
|
+
that gap: under `pikku all --security` the inspector reports a revealed secret
|
|
142
|
+
that flows into a logger, an audit, a queue, an email or a webhook — `console` included.
|
|
143
|
+
|
|
144
|
+
This also fixed a real one: `remote-addon-auth.ts` called `String(token)` on an
|
|
145
|
+
`unknown` and wrote the result straight into an `Authorization` header.
|
|
146
|
+
|
|
147
|
+
- Updated dependencies [62ea4cc]
|
|
148
|
+
- Updated dependencies [9dddff8]
|
|
149
|
+
- Updated dependencies [78b29f0]
|
|
150
|
+
- @pikku/core@0.12.76
|
|
151
|
+
|
|
152
|
+
## 0.13.8
|
|
153
|
+
|
|
154
|
+
### Patch Changes
|
|
155
|
+
|
|
156
|
+
- cabd9dc: Add a `db.schema` CLI config option, so `pikku db generate` can write the runtime tables into a named postgres schema.
|
|
157
|
+
|
|
158
|
+
Without it the generator emits unqualified `create table` against the default `search_path` of `"$user", public`. A project that keeps everything in one namespace — `app`, say — gets a second copy of every runtime table in `public` alongside the ones it already has, which is how stray `public.ai_*` tables appear next to the real `app.ai_*` ones.
|
|
159
|
+
|
|
160
|
+
`compilePikkuSchemas` takes the schema and binds only the rendered SQL, never the caller's connection: that connection is the throwaway database the declaration was just applied to, and qualifying it would create tables in a schema the scratch database has never heard of.
|
|
161
|
+
|
|
162
|
+
Raw SQL is not rewritten by `withSchema`, so `rawStatement` now also accepts a builder taking a `SchemaContext` — the expression index on `credentials` uses it to qualify its own table. Statements otherwise pick the context up from whatever connection they are handed, so a schema-bound connection needs nothing said twice.
|
|
163
|
+
|
|
164
|
+
Two fixes fall out of it:
|
|
165
|
+
- The `ALTER TABLE` delta for a partially covered source is written from bare introspected names, so it is qualified explicitly. Unqualified it altered a table in whichever schema `search_path` found.
|
|
166
|
+
- A source was only counted as partially covered on an exact name match, so a project whose migrations already create `app.workflow_step` read as "nothing covered" and had its whole schema re-emitted over tables that were already there. It now matches the schema qualifier the same way the drift diff does.
|
|
167
|
+
|
|
168
|
+
`db.schema` is postgres only, and is rejected with an explanation on sqlite, whose `REFERENCES` clause takes a bare table name.
|
|
169
|
+
|
|
170
|
+
- Updated dependencies [32277d5]
|
|
171
|
+
- Updated dependencies [ea8aabf]
|
|
172
|
+
- Updated dependencies [33e96ab]
|
|
173
|
+
- Updated dependencies [fd72e58]
|
|
174
|
+
- Updated dependencies [fd72e58]
|
|
175
|
+
- Updated dependencies [fd72e58]
|
|
176
|
+
- Updated dependencies [894b2f8]
|
|
177
|
+
- Updated dependencies [dd19aa7]
|
|
178
|
+
- Updated dependencies [50ec500]
|
|
179
|
+
- @pikku/core@0.12.75
|
|
180
|
+
|
|
1
181
|
## 0.13.7
|
|
2
182
|
|
|
3
183
|
### Patch Changes
|
package/dist/src/index.d.ts
CHANGED
|
@@ -16,8 +16,11 @@ export { KyselyScopeService } from './kysely-scope-service.js';
|
|
|
16
16
|
export { KyselyWebhookService } from './kysely-webhook-service.js';
|
|
17
17
|
export { createAuditedKysely, type CreateAuditedKyselyOptions, } from './create-audited-kysely.js';
|
|
18
18
|
export { KyselyAuditService } from './kysely-audit-service.js';
|
|
19
|
+
export { KyselyVirtualUserRunStore } from './kysely-virtual-user-run-store.js';
|
|
19
20
|
export { SerializePlugin, SqliteSerializePlugin, BaseSerializePlugin, defaultSerializer, defaultDeserializer, maybeJson, skipTransform, dateRegex, type Serializer, type Deserializer, type SerializePluginOptions, } from './serialize-plugin.js';
|
|
20
|
-
export {
|
|
21
|
+
export { virtualUserSchema } from './schema/virtual-user.schema.js';
|
|
22
|
+
export { auditSchema, pikkuSchemas, applyPikkuSchemas, compilePikkuSchemas, ensurePikkuSchema, type EnsureOutcome, resolveRequirements, unsatisfiedRequirements, rawStatement, requiredType, type PikkuSchema, type RequiredTypes, type SchemaRequirement, type UnmetRequirement, type SchemaStatement, type SchemaStatementFactory, } from './schema/index.js';
|
|
21
23
|
export type { KyselyPikkuDB } from './kysely-tables.js';
|
|
22
24
|
export type { WorkflowRunService } from '@pikku/core/workflow';
|
|
23
25
|
export type { AgentRunService, AgentRunRow } from '@pikku/core/ai-agent';
|
|
26
|
+
export type { VirtualUserRunStore, VirtualUserRunRecord, } from '@pikku/core/virtual-user';
|
package/dist/src/index.js
CHANGED
|
@@ -14,5 +14,7 @@ export { KyselyScopeService } from './kysely-scope-service.js';
|
|
|
14
14
|
export { KyselyWebhookService } from './kysely-webhook-service.js';
|
|
15
15
|
export { createAuditedKysely, } from './create-audited-kysely.js';
|
|
16
16
|
export { KyselyAuditService } from './kysely-audit-service.js';
|
|
17
|
+
export { KyselyVirtualUserRunStore } from './kysely-virtual-user-run-store.js';
|
|
17
18
|
export { SerializePlugin, SqliteSerializePlugin, BaseSerializePlugin, defaultSerializer, defaultDeserializer, maybeJson, skipTransform, dateRegex, } from './serialize-plugin.js';
|
|
18
|
-
export {
|
|
19
|
+
export { virtualUserSchema } from './schema/virtual-user.schema.js';
|
|
20
|
+
export { auditSchema, pikkuSchemas, applyPikkuSchemas, compilePikkuSchemas, ensurePikkuSchema, resolveRequirements, unsatisfiedRequirements, rawStatement, requiredType, } from './schema/index.js';
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import type { AuditEvent, AuditEventBatch, AuditService } from '@pikku/core';
|
|
1
|
+
import type { AuditEvent, AuditEventBatch, AuditFacets, AuditQuery, AuditQueryResult, AuditService } from '@pikku/core';
|
|
2
2
|
import type { Kysely } from 'kysely';
|
|
3
3
|
/**
|
|
4
4
|
* Durable {@link AuditService} that persists AuditEvents to an `audit` table
|
|
@@ -6,14 +6,46 @@ import type { Kysely } from 'kysely';
|
|
|
6
6
|
*
|
|
7
7
|
* The column mapping matches a platform audit-queue consumer, so a locally-run
|
|
8
8
|
* project and a deployed stage write identical rows (the read side relies only
|
|
9
|
-
* on `type`, `occurred_at`, `
|
|
9
|
+
* on `type`, `occurred_at`, `user_id`, and the `data` JSON that holds
|
|
10
10
|
* `metadata`). The `audit` table is an optional per-project migration (not in
|
|
11
11
|
* the generated schema), so rows are inserted untyped; all columns are TEXT on
|
|
12
12
|
* every engine and ON CONFLICT DO NOTHING keeps writes idempotent on retries.
|
|
13
13
|
*/
|
|
14
14
|
export declare class KyselyAuditService implements AuditService {
|
|
15
15
|
private db;
|
|
16
|
+
private initialized;
|
|
16
17
|
constructor(db: Kysely<any>);
|
|
18
|
+
/**
|
|
19
|
+
* Creates the `audit` table if the database has none.
|
|
20
|
+
*
|
|
21
|
+
* Optional — a project that migrates the table itself can skip it, and the
|
|
22
|
+
* declaration is written to match the documented shape either way. Calling it
|
|
23
|
+
* on a database that already has the table is a no-op, so two instances
|
|
24
|
+
* booting cold do not race each other into a half-applied schema.
|
|
25
|
+
*/
|
|
26
|
+
init(): Promise<void>;
|
|
17
27
|
audit(event: AuditEvent): Promise<void>;
|
|
18
28
|
write(batch: AuditEventBatch): Promise<void>;
|
|
29
|
+
/**
|
|
30
|
+
* A page of the trail, newest first.
|
|
31
|
+
*
|
|
32
|
+
* Offset paging, matching the rest of the console. The trail is append-only
|
|
33
|
+
* and ordered by `occurred_at DESC`, so events written *while* a reader is
|
|
34
|
+
* paging shift the window and can repeat a row across pages. That is visible
|
|
35
|
+
* but harmless for browsing; anything that must not miss a row should page by
|
|
36
|
+
* a bounded `from`/`to` window instead of scrolling.
|
|
37
|
+
*
|
|
38
|
+
* Ties on `occurred_at` are broken by `audit_id` so the order is total —
|
|
39
|
+
* without it a batch flushed in one call shares a timestamp and the engine is
|
|
40
|
+
* free to order it differently per query, which drops or repeats rows at a
|
|
41
|
+
* page boundary.
|
|
42
|
+
*/
|
|
43
|
+
query(query?: AuditQuery): Promise<AuditQueryResult>;
|
|
44
|
+
facets(): Promise<AuditFacets>;
|
|
45
|
+
/**
|
|
46
|
+
* An empty array means "match nothing" and is applied as such. Treating it as
|
|
47
|
+
* "no filter" would turn a reader's deselect-everything into a full-trail
|
|
48
|
+
* read, which is the opposite of what they asked for.
|
|
49
|
+
*/
|
|
50
|
+
private applyFilters;
|
|
19
51
|
}
|
|
@@ -1,4 +1,64 @@
|
|
|
1
|
+
import { ensurePikkuSchema } from './schema/index.js';
|
|
2
|
+
import { auditSchema } from './schema/audit.schema.js';
|
|
1
3
|
const jsonOrNull = (v) => v != null ? JSON.stringify(v) : null;
|
|
4
|
+
const parseJson = (v) => {
|
|
5
|
+
if (typeof v !== 'string')
|
|
6
|
+
return v ?? undefined;
|
|
7
|
+
try {
|
|
8
|
+
return JSON.parse(v);
|
|
9
|
+
}
|
|
10
|
+
catch {
|
|
11
|
+
// A row written by something other than this service, or truncated in
|
|
12
|
+
// transit. Surfacing the raw text beats dropping the event: the row is
|
|
13
|
+
// still evidence that something happened, and swallowing it would make the
|
|
14
|
+
// trail quietly incomplete.
|
|
15
|
+
return v;
|
|
16
|
+
}
|
|
17
|
+
};
|
|
18
|
+
/**
|
|
19
|
+
* A column by its physical name, whatever the connection renamed it to.
|
|
20
|
+
*
|
|
21
|
+
* `CamelCasePlugin` is on most pikku Kysely instances and rewrites result keys
|
|
22
|
+
* on the way out, so the same row arrives keyed `audit_id` or `auditId`
|
|
23
|
+
* depending on a plugin this service is never told about. Reading both is
|
|
24
|
+
* cheaper than demanding a particular connection — and quieter to get wrong,
|
|
25
|
+
* since the mismatch does not throw, it just returns a page of undefined.
|
|
26
|
+
*/
|
|
27
|
+
const column = (row, name) => {
|
|
28
|
+
const value = row[name];
|
|
29
|
+
if (value !== undefined)
|
|
30
|
+
return value;
|
|
31
|
+
return row[name.replace(/_(\w)/g, (_, char) => char.toUpperCase())];
|
|
32
|
+
};
|
|
33
|
+
/** Rows are TEXT on every engine, so the shape is whatever the driver returned. */
|
|
34
|
+
const rowToEvent = (row) => {
|
|
35
|
+
const userId = column(row, 'user_id');
|
|
36
|
+
const orgId = column(row, 'org_id');
|
|
37
|
+
const pikkuUserId = column(row, 'pikku_user_id');
|
|
38
|
+
return {
|
|
39
|
+
eventId: column(row, 'audit_id') ?? undefined,
|
|
40
|
+
occurredAt: column(row, 'occurred_at'),
|
|
41
|
+
type: row.type,
|
|
42
|
+
source: row.source ?? 'auto',
|
|
43
|
+
outcome: row.outcome ?? undefined,
|
|
44
|
+
functionId: column(row, 'function_id') ?? undefined,
|
|
45
|
+
wireType: column(row, 'wire_type') ?? undefined,
|
|
46
|
+
traceId: column(row, 'trace_id') ?? undefined,
|
|
47
|
+
transactionId: column(row, 'transaction_id') ?? undefined,
|
|
48
|
+
queryId: column(row, 'query_id') ?? undefined,
|
|
49
|
+
userIdentity: userId || orgId || pikkuUserId
|
|
50
|
+
? {
|
|
51
|
+
userId: userId ?? undefined,
|
|
52
|
+
orgId: orgId ?? undefined,
|
|
53
|
+
pikkuUserId: pikkuUserId ?? undefined,
|
|
54
|
+
}
|
|
55
|
+
: undefined,
|
|
56
|
+
metadata: parseJson(row.data),
|
|
57
|
+
};
|
|
58
|
+
};
|
|
59
|
+
/** The most a single page may return, whatever the caller asked for. */
|
|
60
|
+
const MAX_LIMIT = 200;
|
|
61
|
+
const DEFAULT_LIMIT = 50;
|
|
2
62
|
// No global `crypto` is guaranteed across every runtime, and audit_id is the
|
|
3
63
|
// PK — any collision is dropped by ON CONFLICT DO NOTHING.
|
|
4
64
|
const fallbackId = () => `evt_${Date.now().toString(36)}${Math.random().toString(36).slice(2, 10)}`;
|
|
@@ -8,16 +68,31 @@ const fallbackId = () => `evt_${Date.now().toString(36)}${Math.random().toString
|
|
|
8
68
|
*
|
|
9
69
|
* The column mapping matches a platform audit-queue consumer, so a locally-run
|
|
10
70
|
* project and a deployed stage write identical rows (the read side relies only
|
|
11
|
-
* on `type`, `occurred_at`, `
|
|
71
|
+
* on `type`, `occurred_at`, `user_id`, and the `data` JSON that holds
|
|
12
72
|
* `metadata`). The `audit` table is an optional per-project migration (not in
|
|
13
73
|
* the generated schema), so rows are inserted untyped; all columns are TEXT on
|
|
14
74
|
* every engine and ON CONFLICT DO NOTHING keeps writes idempotent on retries.
|
|
15
75
|
*/
|
|
16
76
|
export class KyselyAuditService {
|
|
17
77
|
db;
|
|
78
|
+
initialized = false;
|
|
18
79
|
constructor(db) {
|
|
19
80
|
this.db = db;
|
|
20
81
|
}
|
|
82
|
+
/**
|
|
83
|
+
* Creates the `audit` table if the database has none.
|
|
84
|
+
*
|
|
85
|
+
* Optional — a project that migrates the table itself can skip it, and the
|
|
86
|
+
* declaration is written to match the documented shape either way. Calling it
|
|
87
|
+
* on a database that already has the table is a no-op, so two instances
|
|
88
|
+
* booting cold do not race each other into a half-applied schema.
|
|
89
|
+
*/
|
|
90
|
+
async init() {
|
|
91
|
+
if (this.initialized)
|
|
92
|
+
return;
|
|
93
|
+
await ensurePikkuSchema(this.db, auditSchema);
|
|
94
|
+
this.initialized = true;
|
|
95
|
+
}
|
|
21
96
|
async audit(event) {
|
|
22
97
|
await this.write([event]);
|
|
23
98
|
}
|
|
@@ -37,8 +112,9 @@ export class KyselyAuditService {
|
|
|
37
112
|
trace_id: e.traceId ?? null,
|
|
38
113
|
transaction_id: e.transactionId ?? null,
|
|
39
114
|
query_id: e.queryId ?? null,
|
|
40
|
-
|
|
41
|
-
|
|
115
|
+
user_id: e.userIdentity?.userId ?? null,
|
|
116
|
+
org_id: e.userIdentity?.orgId ?? null,
|
|
117
|
+
pikku_user_id: e.userIdentity?.pikkuUserId ?? null,
|
|
42
118
|
tables: jsonOrNull(metadata?.tables),
|
|
43
119
|
changed_cols: jsonOrNull(metadata?.changedColumns),
|
|
44
120
|
event: e.eventId ?? null,
|
|
@@ -52,4 +128,88 @@ export class KyselyAuditService {
|
|
|
52
128
|
.onConflict((oc) => oc.doNothing())
|
|
53
129
|
.execute();
|
|
54
130
|
}
|
|
131
|
+
/**
|
|
132
|
+
* A page of the trail, newest first.
|
|
133
|
+
*
|
|
134
|
+
* Offset paging, matching the rest of the console. The trail is append-only
|
|
135
|
+
* and ordered by `occurred_at DESC`, so events written *while* a reader is
|
|
136
|
+
* paging shift the window and can repeat a row across pages. That is visible
|
|
137
|
+
* but harmless for browsing; anything that must not miss a row should page by
|
|
138
|
+
* a bounded `from`/`to` window instead of scrolling.
|
|
139
|
+
*
|
|
140
|
+
* Ties on `occurred_at` are broken by `audit_id` so the order is total —
|
|
141
|
+
* without it a batch flushed in one call shares a timestamp and the engine is
|
|
142
|
+
* free to order it differently per query, which drops or repeats rows at a
|
|
143
|
+
* page boundary.
|
|
144
|
+
*/
|
|
145
|
+
async query(query = {}) {
|
|
146
|
+
const limit = Math.min(Math.max(query.limit ?? DEFAULT_LIMIT, 1), MAX_LIMIT);
|
|
147
|
+
const offset = Math.max(query.offset ?? 0, 0);
|
|
148
|
+
let builder = this.db.selectFrom('audit').selectAll();
|
|
149
|
+
builder = this.applyFilters(builder, query);
|
|
150
|
+
const rows = await builder
|
|
151
|
+
.orderBy('occurred_at', 'desc')
|
|
152
|
+
.orderBy('audit_id', 'desc')
|
|
153
|
+
// One extra row is the cheapest honest way to know whether another page
|
|
154
|
+
// exists — a COUNT would be a second scan of the same filter.
|
|
155
|
+
.limit(limit + 1)
|
|
156
|
+
.offset(offset)
|
|
157
|
+
.execute();
|
|
158
|
+
const hasMore = rows.length > limit;
|
|
159
|
+
return {
|
|
160
|
+
events: rows.slice(0, limit).map(rowToEvent),
|
|
161
|
+
nextCursor: hasMore ? offset + limit : null,
|
|
162
|
+
};
|
|
163
|
+
}
|
|
164
|
+
async facets() {
|
|
165
|
+
const [users, types] = await Promise.all([
|
|
166
|
+
this.db
|
|
167
|
+
.selectFrom('audit')
|
|
168
|
+
.select('user_id')
|
|
169
|
+
.distinct()
|
|
170
|
+
.where('user_id', 'is not', null)
|
|
171
|
+
.orderBy('user_id', 'asc')
|
|
172
|
+
.execute(),
|
|
173
|
+
this.db
|
|
174
|
+
.selectFrom('audit')
|
|
175
|
+
.select('type')
|
|
176
|
+
.distinct()
|
|
177
|
+
.orderBy('type', 'asc')
|
|
178
|
+
.execute(),
|
|
179
|
+
]);
|
|
180
|
+
return {
|
|
181
|
+
userIds: users.map((r) => column(r, 'user_id')),
|
|
182
|
+
types: types.map((r) => r.type).filter(Boolean),
|
|
183
|
+
};
|
|
184
|
+
}
|
|
185
|
+
/**
|
|
186
|
+
* An empty array means "match nothing" and is applied as such. Treating it as
|
|
187
|
+
* "no filter" would turn a reader's deselect-everything into a full-trail
|
|
188
|
+
* read, which is the opposite of what they asked for.
|
|
189
|
+
*/
|
|
190
|
+
applyFilters(builder, query) {
|
|
191
|
+
// `in ()` is a syntax error on Postgres and SQLite alike, so an empty
|
|
192
|
+
// selection is expressed as a predicate that cannot hold.
|
|
193
|
+
const matchNothing = (b) => b.where('audit_id', 'is', null);
|
|
194
|
+
if (query.userIds) {
|
|
195
|
+
builder = query.userIds.length
|
|
196
|
+
? builder.where('user_id', 'in', query.userIds)
|
|
197
|
+
: matchNothing(builder);
|
|
198
|
+
}
|
|
199
|
+
if (query.types) {
|
|
200
|
+
builder = query.types.length
|
|
201
|
+
? builder.where('type', 'in', query.types)
|
|
202
|
+
: matchNothing(builder);
|
|
203
|
+
}
|
|
204
|
+
if (query.orgId) {
|
|
205
|
+
builder = builder.where('org_id', '=', query.orgId);
|
|
206
|
+
}
|
|
207
|
+
if (query.from) {
|
|
208
|
+
builder = builder.where('occurred_at', '>=', query.from);
|
|
209
|
+
}
|
|
210
|
+
if (query.to) {
|
|
211
|
+
builder = builder.where('occurred_at', '<', query.to);
|
|
212
|
+
}
|
|
213
|
+
return builder;
|
|
214
|
+
}
|
|
55
215
|
}
|
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
import type
|
|
1
|
+
import { type SecretValue } from '@pikku/core';
|
|
2
|
+
import type { SecretService, SecretValues } from '@pikku/core/services';
|
|
2
3
|
import type { Kysely } from 'kysely';
|
|
3
4
|
import type { KyselyPikkuDB } from './kysely-tables.js';
|
|
4
5
|
export interface KyselySecretServiceConfig {
|
|
@@ -23,10 +24,10 @@ export declare class KyselySecretService implements SecretService {
|
|
|
23
24
|
private logAudit;
|
|
24
25
|
private getKEKSalt;
|
|
25
26
|
private getKEK;
|
|
26
|
-
getSecret<T = string>(key: string): Promise<T
|
|
27
|
+
getSecret<T = string>(key: string): Promise<SecretValue<T>>;
|
|
27
28
|
hasSecret(key: string): Promise<boolean>;
|
|
28
29
|
setSecret(key: string, value: unknown): Promise<void>;
|
|
29
30
|
deleteSecret(key: string): Promise<void>;
|
|
30
|
-
getSecrets<T extends Record<string, unknown> = Record<string, unknown>>(keys: (keyof T & string)[]): Promise<T
|
|
31
|
+
getSecrets<T extends Record<string, unknown> = Record<string, unknown>>(keys: (keyof T & string)[]): Promise<Partial<SecretValues<T>>>;
|
|
31
32
|
rotateKEK(): Promise<number>;
|
|
32
33
|
}
|
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { createSecretValue, isSecretValue } from '@pikku/core';
|
|
1
2
|
import { deriveKEK, generateKEKSalt, envelopeEncrypt, envelopeDecrypt, envelopeRewrap, } from '@pikku/core/crypto-utils';
|
|
2
3
|
import { ensurePikkuSchema } from './schema/index.js';
|
|
3
4
|
import { secretSchema } from './schema/secret.schema.js';
|
|
@@ -93,7 +94,7 @@ export class KyselySecretService {
|
|
|
93
94
|
const kek = await this.getKEK(row.keyVersion);
|
|
94
95
|
const result = await envelopeDecrypt(kek, row.ciphertext, row.wrappedDek);
|
|
95
96
|
await this.logAudit(key, 'read');
|
|
96
|
-
return result;
|
|
97
|
+
return createSecretValue(result);
|
|
97
98
|
}
|
|
98
99
|
async hasSecret(key) {
|
|
99
100
|
const row = await this.db
|
|
@@ -104,7 +105,9 @@ export class KyselySecretService {
|
|
|
104
105
|
return !!row;
|
|
105
106
|
}
|
|
106
107
|
async setSecret(key, value) {
|
|
107
|
-
|
|
108
|
+
// Encrypting the wrapper would store its redaction, not the secret —
|
|
109
|
+
// writing a secret back to the vault is exactly what this method is for.
|
|
110
|
+
const { ciphertext, wrappedDEK } = await envelopeEncrypt(await this.getKEK(this.keyVersion), isSecretValue(value) ? value.reveal() : value);
|
|
108
111
|
const now = new Date().toISOString();
|
|
109
112
|
await this.db
|
|
110
113
|
.insertInto('secrets')
|
|
@@ -139,7 +142,7 @@ export class KyselySecretService {
|
|
|
139
142
|
for (const row of rows) {
|
|
140
143
|
try {
|
|
141
144
|
const kek = await this.getKEK(row.keyVersion);
|
|
142
|
-
out[row.key] = await envelopeDecrypt(kek, row.ciphertext, row.wrappedDek);
|
|
145
|
+
out[row.key] = createSecretValue(await envelopeDecrypt(kek, row.ciphertext, row.wrappedDek));
|
|
143
146
|
}
|
|
144
147
|
catch (cause) {
|
|
145
148
|
// A stored secret that fails to decrypt is never expected — it means the
|
|
@@ -248,6 +248,36 @@ export interface WebhookDeliveryAttemptTable {
|
|
|
248
248
|
error: string | null;
|
|
249
249
|
createdAt: Generated<Date>;
|
|
250
250
|
}
|
|
251
|
+
/**
|
|
252
|
+
* One virtual-user run. The JSON columns (`goals`, `memory`, `findings`,
|
|
253
|
+
* `tally`) are text the store serialises itself, so the row is byte-identical
|
|
254
|
+
* on every engine.
|
|
255
|
+
*/
|
|
256
|
+
export interface VirtualUserRunTable {
|
|
257
|
+
runId: string;
|
|
258
|
+
persona: string;
|
|
259
|
+
disposition: string;
|
|
260
|
+
/**
|
|
261
|
+
* Read back as a string by some drivers — a BIGINT does not always fit a JS
|
|
262
|
+
* number — so the store narrows it rather than trusting the column type.
|
|
263
|
+
*/
|
|
264
|
+
seed: string | number | bigint;
|
|
265
|
+
status: Generated<'running' | 'completed' | 'failed'>;
|
|
266
|
+
goals: Generated<string>;
|
|
267
|
+
memory: Generated<string>;
|
|
268
|
+
findings: Generated<string>;
|
|
269
|
+
tally: string | null;
|
|
270
|
+
stoppedBy: string | null;
|
|
271
|
+
error: string | null;
|
|
272
|
+
startedBy: string | null;
|
|
273
|
+
/**
|
|
274
|
+
* Written as an ISO string and read back as whatever the driver returns — a
|
|
275
|
+
* bare SQLite driver cannot bind a `Date` at all, and the store normalises
|
|
276
|
+
* both ends.
|
|
277
|
+
*/
|
|
278
|
+
createdAt: Generated<Date | string>;
|
|
279
|
+
finishedAt: Date | string | null;
|
|
280
|
+
}
|
|
251
281
|
export interface KyselyPikkuDB {
|
|
252
282
|
pikkuScopes: PikkuScopesTable;
|
|
253
283
|
pikkuRoles: PikkuRolesTable;
|
|
@@ -276,4 +306,5 @@ export interface KyselyPikkuDB {
|
|
|
276
306
|
pikkuUserSessions: UserSessionsTable;
|
|
277
307
|
webhookDelivery: WebhookDeliveryTable;
|
|
278
308
|
webhookDeliveryAttempt: WebhookDeliveryAttemptTable;
|
|
309
|
+
virtualUserRun: VirtualUserRunTable;
|
|
279
310
|
}
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
import type { Kysely } from 'kysely';
|
|
2
|
+
import type { VirtualUserRunOutcome, VirtualUserRunRecord, VirtualUserRunStart, VirtualUserRunStore } from '@pikku/core/virtual-user';
|
|
3
|
+
import type { KyselyPikkuDB } from './kysely-tables.js';
|
|
4
|
+
/**
|
|
5
|
+
* Records virtual-user runs in a `virtualUserRun` table.
|
|
6
|
+
*
|
|
7
|
+
* The store is the whole record of a run: `runVirtualUser` returns as soon as
|
|
8
|
+
* the row exists and the engine keeps going without it, so nothing else knows
|
|
9
|
+
* the run happened. That is why `fail()` exists as its own call — a run that
|
|
10
|
+
* crashed and a run that found nothing are different answers, and a row left at
|
|
11
|
+
* `running` is neither.
|
|
12
|
+
*
|
|
13
|
+
* Works with or without `SerializePlugin`, which not every project installs: the
|
|
14
|
+
* JSON columns are written as strings (which the plugin passes through) and read
|
|
15
|
+
* back through `parseJson` (which passes through what the plugin already
|
|
16
|
+
* parsed). Timestamps are written as ISO strings for the same reason — a bare
|
|
17
|
+
* SQLite driver cannot bind a `Date` at all.
|
|
18
|
+
*
|
|
19
|
+
* SECURITY: an `adversarial` run's findings are working exploits carrying live
|
|
20
|
+
* ids. This is a privileged store; the scaffolded reads are scope-gated, and
|
|
21
|
+
* exposing these rows more widely publishes your own exploits.
|
|
22
|
+
*/
|
|
23
|
+
export declare class KyselyVirtualUserRunStore implements VirtualUserRunStore {
|
|
24
|
+
private db;
|
|
25
|
+
private initialized;
|
|
26
|
+
constructor(db: Kysely<KyselyPikkuDB>);
|
|
27
|
+
/**
|
|
28
|
+
* Creates the table on first use, like the audit sink — the runtime does not
|
|
29
|
+
* need it, so it arrives with the feature that fills it rather than in every
|
|
30
|
+
* database.
|
|
31
|
+
*/
|
|
32
|
+
init(): Promise<void>;
|
|
33
|
+
start(run: VirtualUserRunStart): Promise<string>;
|
|
34
|
+
complete(runId: string, outcome: VirtualUserRunOutcome): Promise<void>;
|
|
35
|
+
fail(runId: string, error: string): Promise<void>;
|
|
36
|
+
get(runId: string): Promise<VirtualUserRunRecord | null>;
|
|
37
|
+
list(options?: {
|
|
38
|
+
persona?: string;
|
|
39
|
+
limit?: number;
|
|
40
|
+
offset?: number;
|
|
41
|
+
}): Promise<VirtualUserRunRecord[]>;
|
|
42
|
+
private toRecord;
|
|
43
|
+
}
|