@marinoscar/kvox-eval 0.1.0-next.0 → 0.1.0-next.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/README.md +5 -4
- package/dist/ask-run/replay.d.ts +1 -1
- package/dist/cli/kg-command.d.ts +1 -1
- package/dist/live/adapters.d.ts +3 -3
- package/dist/live/adapters.js +14 -39
- package/dist/live/live-ai.d.ts +1 -1
- package/host/prisma/kvox.lock +1 -1
- package/host/prisma/platform.lock +1 -1
- package/package.json +10 -8
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# @marinoscar/kvox-eval
|
|
2
2
|
|
|
3
|
+
## 0.1.0-next.1
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 38bf270: `@marinoscar/kvox-ai` and `@marinoscar/kvox-eval` no longer require `@marinoscar/platform-api`: they depend on the lean `@marinoscar/platform-ai-providers` (provider adapters and the provider-agnostic AI core, no Nest runtime) as an exact-version peer, and `kvox-eval` gains an `openai` peer for its live runs (#578). `kvox-ai` exports `isAiError()`, which recognises the lean `AiError` and the web host's distinct `AiError` class, so error classification is unchanged in both editions.
|
|
8
|
+
|
|
3
9
|
## 0.1.0-next.0
|
|
4
10
|
|
|
5
11
|
### Minor Changes
|
package/README.md
CHANGED
|
@@ -21,7 +21,7 @@ Entry points:
|
|
|
21
21
|
## Install and peer dependencies
|
|
22
22
|
|
|
23
23
|
```bash
|
|
24
|
-
npm install @marinoscar/kvox-eval@next @marinoscar/kvox-ai@next @marinoscar/kvox-api@next @marinoscar/kvox-contract@next @marinoscar/platform-
|
|
24
|
+
npm install @marinoscar/kvox-eval@next @marinoscar/kvox-ai@next @marinoscar/kvox-api@next @marinoscar/kvox-contract@next @marinoscar/platform-ai-providers@0.1.0-next.6 openai zod \
|
|
25
25
|
@electric-sql/pglite@0.5.10 @electric-sql/pglite-pgvector@0.0.11 @electric-sql/pglite-socket@0.2.13 \
|
|
26
26
|
@prisma/client @prisma/adapter-pg pg @nestjs/common @nestjs/core reflect-metadata rxjs
|
|
27
27
|
```
|
|
@@ -34,7 +34,8 @@ Every slice is universal, so every peer is required, including those only `ask-r
|
|
|
34
34
|
| `@marinoscar/kvox-contract` | `>=0.0.0-0 \|\| >=0.1.0-0` | `fixtures`, `kg` (the ontology registry), `ask` (the Ask wire types) |
|
|
35
35
|
| `@marinoscar/kvox-api` | `>=0.0.0-0 \|\| >=0.1.0-0` | `kg` (the extraction pipeline, `graph-extraction` and `graph`), `live` and `cli` (`KvoxAiPort`, `KvoxAiError`) |
|
|
36
36
|
| `@marinoscar/kvox-ai` | `>=0.0.0-0 \|\| >=0.1.0-0` | `live` (`createKvoxAi`, `createDirectRuntime`) |
|
|
37
|
-
| `@marinoscar/platform-
|
|
37
|
+
| `@marinoscar/platform-ai-providers` | `0.1.0-next.6` (exact) | `live` and `cli`: `createOpenAiAdapter` and `createOpenAiCompatibleAdapter` from the `/openai` subpath; `parity`, `ask-run`: the provider event types. Only the package root and `/openai` are imported; `@marinoscar/platform-api` is not a peer and importing it is a violation. |
|
|
38
|
+
| `openai` | `^7.23.0` | `live`: the OpenAI SDK that the lean `/openai` factories build their client from. Required, because `live` constructs the SDK client. |
|
|
38
39
|
| `@electric-sql/pglite` | `0.5.10` (exact) | `ask-run`: the in-memory PostgreSQL |
|
|
39
40
|
| `@electric-sql/pglite-pgvector` | `0.0.11` (exact) | `ask-run`: the `vector` extension the migrations create |
|
|
40
41
|
| `@electric-sql/pglite-socket` | `0.2.13` (exact) | `ask-run`: the loopback socket the Prisma client connects to |
|
|
@@ -133,11 +134,11 @@ None: no log, metric or span (the Ask run's Nest context logs errors and warning
|
|
|
133
134
|
|
|
134
135
|
## Security notes
|
|
135
136
|
|
|
136
|
-
- The API key is read from the environment only, never a flag. It is handed to `createDirectRuntime`'s `keys()` per call and never logged, printed, put in an error or written into a prediction or a report (a prediction carries `model` alone). A failed live call is reported by its `KvoxAiError` class, the
|
|
137
|
+
- The API key is read from the environment only, never a flag. It is handed to `createDirectRuntime`'s `keys()` per call and never logged, printed, put in an error or written into a prediction or a report (a prediction carries `model` alone). A failed live call is reported by its `KvoxAiError` class, the provider error code and the HTTP status, never a message body.
|
|
137
138
|
- Real meetings never enter the repository: `--real-dir`, `--out` of a real-data run and `KG_EVAL_REAL_DIR` are refused (exit 3) when they resolve inside the git work tree, symlinks included (`assertOutsideRepo`).
|
|
138
139
|
- The golden set is synthetic; its spec rejects a real-looking email address, a phone number and the product's name.
|
|
139
140
|
- The Ask run uses an in-memory PGlite only: it has no connection string to a real database, so it cannot read or write one. The key never enters that database.
|
|
140
|
-
- Apart from the `ask-run` slice (Prisma, Nest, PGlite), the source imports no Prisma, no `@app/shared`, and of the platform only `@marinoscar/platform-
|
|
141
|
+
- Apart from the `ask-run` slice (Prisma, Nest, PGlite), the source imports no Prisma, no `@app/shared`, and of the platform only `@marinoscar/platform-ai-providers` and its `/openai` subpath (rule `host-neutral` of `npm run check:kvox-slice-peers`).
|
|
141
142
|
|
|
142
143
|
## Conformance suite
|
|
143
144
|
|
package/dist/ask-run/replay.d.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import type { AskMessageResponse } from '@marinoscar/kvox-contract/ask';
|
|
2
|
-
import type { AiStreamEvent } from '@marinoscar/platform-
|
|
2
|
+
import type { AiStreamEvent } from '@marinoscar/platform-ai-providers';
|
|
3
3
|
/**
|
|
4
4
|
* The provider and model a replay names (the scripted runtime answers for them).
|
|
5
5
|
*
|
package/dist/cli/kg-command.d.ts
CHANGED
package/dist/live/adapters.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import type { AiProviderAdapter } from '@marinoscar/platform-
|
|
1
|
+
import type { AiProviderAdapter } from '@marinoscar/platform-ai-providers';
|
|
2
2
|
/**
|
|
3
3
|
* A provider id {@link platformAdapter} can build.
|
|
4
4
|
*
|
|
@@ -6,11 +6,11 @@ import type { AiProviderAdapter } from '@marinoscar/platform-api/ai';
|
|
|
6
6
|
*/
|
|
7
7
|
export type PlatformAdapterProvider = 'openai' | 'openai-compatible';
|
|
8
8
|
/**
|
|
9
|
-
* The
|
|
9
|
+
* The lean provider package's adapter for a provider, constructed outside Nest.
|
|
10
10
|
*
|
|
11
11
|
* @param provider - `openai` or `openai-compatible`.
|
|
12
12
|
* @returns the adapter (`responses`, `embeddings`, ...).
|
|
13
|
-
* @throws Error when the
|
|
13
|
+
* @throws Error when the adapter built does not carry the id asked for.
|
|
14
14
|
* @stability experimental
|
|
15
15
|
*/
|
|
16
16
|
export declare function platformAdapter(provider: PlatformAdapterProvider): AiProviderAdapter;
|
package/dist/live/adapters.js
CHANGED
|
@@ -3,56 +3,31 @@
|
|
|
3
3
|
// The provider adapters a live run calls (issue #580)
|
|
4
4
|
// =============================================================================
|
|
5
5
|
//
|
|
6
|
-
// The OpenAI and OpenAI-compatible adapters are the
|
|
7
|
-
// (`@marinoscar/platform-
|
|
8
|
-
//
|
|
9
|
-
//
|
|
10
|
-
//
|
|
11
|
-
// factory (no Nest application is booted: the adapter registers itself only in
|
|
12
|
-
// `onModuleInit`, which is never called here).
|
|
6
|
+
// The OpenAI and OpenAI-compatible adapters are the lean provider package's
|
|
7
|
+
// (`@marinoscar/platform-ai-providers/openai`): plain factories, no Nest, no
|
|
8
|
+
// database. `platformAdapter` calls the factory for the provider with no options
|
|
9
|
+
// (no logger, no tracer: a live run is a CLI); the key and base URL travel per call
|
|
10
|
+
// in the adapter's call context, never here.
|
|
13
11
|
//
|
|
14
|
-
//
|
|
15
|
-
//
|
|
16
|
-
// `new` expressions.
|
|
12
|
+
// `openai` (the SDK) is an OPTIONAL peer of this package: it is loaded only when a
|
|
13
|
+
// live run asks for one of these two providers, never by a scoring run.
|
|
17
14
|
// =============================================================================
|
|
18
15
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
19
16
|
exports.platformAdapter = platformAdapter;
|
|
20
|
-
/** The platform module that carries each provider's adapter. */
|
|
21
|
-
const ADAPTER_MODULES = {
|
|
22
|
-
openai: 'OpenAiProviderModule',
|
|
23
|
-
'openai-compatible': 'OpenAiCompatibleProviderModule',
|
|
24
|
-
};
|
|
25
|
-
/** `Reflect.getMetadata`, installed by `reflect-metadata` (the platform's Nest import loads it). */
|
|
26
|
-
function metadata(key, target) {
|
|
27
|
-
const reflect = Reflect;
|
|
28
|
-
if (typeof reflect.getMetadata !== 'function')
|
|
29
|
-
throw new Error('reflect-metadata is not loaded');
|
|
30
|
-
return reflect.getMetadata(key, target);
|
|
31
|
-
}
|
|
32
17
|
/**
|
|
33
|
-
* The
|
|
18
|
+
* The lean provider package's adapter for a provider, constructed outside Nest.
|
|
34
19
|
*
|
|
35
20
|
* @param provider - `openai` or `openai-compatible`.
|
|
36
21
|
* @returns the adapter (`responses`, `embeddings`, ...).
|
|
37
|
-
* @throws Error when the
|
|
22
|
+
* @throws Error when the adapter built does not carry the id asked for.
|
|
38
23
|
* @stability experimental
|
|
39
24
|
*/
|
|
40
25
|
function platformAdapter(provider) {
|
|
41
|
-
// Required lazily: the
|
|
42
|
-
//
|
|
43
|
-
const
|
|
44
|
-
const
|
|
45
|
-
if (moduleClass === undefined)
|
|
46
|
-
throw new Error(`@marinoscar/platform-api/ai has no ${ADAPTER_MODULES[provider]}`);
|
|
47
|
-
const exported = metadata('exports', moduleClass) ?? [];
|
|
48
|
-
const providers = metadata('providers', moduleClass) ?? [];
|
|
49
|
-
const Adapter = exported.find((c) => typeof c === 'function');
|
|
50
|
-
const Factory = providers.find((c) => typeof c === 'function' && c !== Adapter);
|
|
51
|
-
if (Adapter === undefined || Factory === undefined) {
|
|
52
|
-
throw new Error(`${ADAPTER_MODULES[provider]} does not carry an adapter and a client factory`);
|
|
53
|
-
}
|
|
54
|
-
const adapter = new Adapter(new ai.AiProviderRegistry(), new Factory());
|
|
26
|
+
// Required lazily: the subpath loads the OpenAI SDK, which a scoring run (gold, a
|
|
27
|
+
// predictions directory) has no reason to load.
|
|
28
|
+
const openai = require('@marinoscar/platform-ai-providers/openai');
|
|
29
|
+
const adapter = provider === 'openai' ? openai.createOpenAiAdapter() : openai.createOpenAiCompatibleAdapter();
|
|
55
30
|
if (adapter.id !== provider)
|
|
56
|
-
throw new Error(
|
|
31
|
+
throw new Error(`The ${provider} adapter factory built '${adapter.id}', expected '${provider}'`);
|
|
57
32
|
return adapter;
|
|
58
33
|
}
|
package/dist/live/live-ai.d.ts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { type KvoxTaskModelSource } from '@marinoscar/kvox-ai';
|
|
2
2
|
import type { KvoxAiPort } from '@marinoscar/kvox-api/ports';
|
|
3
|
-
import type { AiProviderAdapter } from '@marinoscar/platform-
|
|
3
|
+
import type { AiProviderAdapter } from '@marinoscar/platform-ai-providers';
|
|
4
4
|
import { type PlatformAdapterProvider } from './adapters.js';
|
|
5
5
|
/**
|
|
6
6
|
* The providers a live run may name. `copilot` is listed so it can be refused
|
package/host/prisma/kvox.lock
CHANGED
|
@@ -245,7 +245,7 @@
|
|
|
245
245
|
"since": "1.0.0"
|
|
246
246
|
}
|
|
247
247
|
],
|
|
248
|
-
"platformVersion": "0.1.0-next.
|
|
248
|
+
"platformVersion": "0.1.0-next.6",
|
|
249
249
|
"rawSqlIndexes": [
|
|
250
250
|
{
|
|
251
251
|
"definition": "CREATE UNIQUE INDEX ask_messages_one_running_turn_uniq_idx ON public.ask_messages USING btree (conversation_id) WHERE ((role = 'assistant'::\"AskMessageRole\") AND (status = ANY (ARRAY['pending'::\"AskMessageStatus\", 'streaming'::\"AskMessageStatus\"])))",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@marinoscar/kvox-eval",
|
|
3
|
-
"version": "0.1.0-next.
|
|
3
|
+
"version": "0.1.0-next.1",
|
|
4
4
|
"description": "The KVox knowledge-graph and Ask evaluation: the synthetic golden set, its schemas, the scorers, the report and the live runners, over the model port only (CommonJS, with the kvox-eval CLI).",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -73,11 +73,12 @@
|
|
|
73
73
|
"@marinoscar/kvox-ai": ">=0.0.0-0 || >=0.1.0-0",
|
|
74
74
|
"@marinoscar/kvox-api": ">=0.0.0-0 || >=0.1.0-0",
|
|
75
75
|
"@marinoscar/kvox-contract": ">=0.0.0-0 || >=0.1.0-0",
|
|
76
|
-
"@marinoscar/platform-
|
|
76
|
+
"@marinoscar/platform-ai-providers": "0.1.0-next.6",
|
|
77
77
|
"@nestjs/common": "^11.1.12",
|
|
78
78
|
"@nestjs/core": "^11.1.12",
|
|
79
79
|
"@prisma/adapter-pg": "^7.8.0",
|
|
80
80
|
"@prisma/client": "^7.8.0",
|
|
81
|
+
"openai": "^7.23.0",
|
|
81
82
|
"pg": "^8.23.0",
|
|
82
83
|
"reflect-metadata": "^0.2.2",
|
|
83
84
|
"rxjs": "^7.8.1",
|
|
@@ -87,12 +88,12 @@
|
|
|
87
88
|
"@electric-sql/pglite": "0.5.10",
|
|
88
89
|
"@electric-sql/pglite-pgvector": "0.0.11",
|
|
89
90
|
"@electric-sql/pglite-socket": "0.2.13",
|
|
90
|
-
"@marinoscar/kvox-ai": "0.1.0-next.
|
|
91
|
-
"@marinoscar/kvox-api": "0.1.0-next.
|
|
92
|
-
"@marinoscar/kvox-contract": "0.1.0-next.
|
|
93
|
-
"@marinoscar/kvox-db": "0.1.0-next.
|
|
94
|
-
"@marinoscar/platform-
|
|
95
|
-
"@marinoscar/platform-db": "0.1.0-next.
|
|
91
|
+
"@marinoscar/kvox-ai": "0.1.0-next.1",
|
|
92
|
+
"@marinoscar/kvox-api": "0.1.0-next.1",
|
|
93
|
+
"@marinoscar/kvox-contract": "0.1.0-next.1",
|
|
94
|
+
"@marinoscar/kvox-db": "0.1.0-next.1",
|
|
95
|
+
"@marinoscar/platform-ai-providers": "0.1.0-next.6",
|
|
96
|
+
"@marinoscar/platform-db": "0.1.0-next.6",
|
|
96
97
|
"@nestjs/common": "^11.1.12",
|
|
97
98
|
"@nestjs/core": "^11.1.12",
|
|
98
99
|
"@prisma/adapter-pg": "^7.8.0",
|
|
@@ -101,6 +102,7 @@
|
|
|
101
102
|
"@types/jest": "^30.0.0",
|
|
102
103
|
"@types/node": "^22.15.32",
|
|
103
104
|
"jest": "^30.4.2",
|
|
105
|
+
"openai": "^7.23.0",
|
|
104
106
|
"pg": "^8.23.0",
|
|
105
107
|
"prisma": "^7.8.0",
|
|
106
108
|
"reflect-metadata": "^0.2.2",
|