@exellix/ai-tasks 9.0.2 → 9.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/DOWNSTREAM_ENV.md +1 -1
- package/.docs/prefer-openrouter-routing-policy.md +8 -26
- package/CHANGELOG.md +7 -0
- package/dist/aiSkillsUpstreamExports.d.ts +2 -2
- package/dist/aiSkillsUpstreamExports.d.ts.map +1 -1
- package/dist/aiSkillsUpstreamExports.js +1 -1
- package/dist/aiSkillsUpstreamExports.js.map +1 -1
- package/dist/internal/runPostStepLlmCall.js +1 -1
- package/dist/internal/runPostStepLlmCall.js.map +1 -1
- package/dist/internal/setAiToolsBundledOnlyEnv.d.ts.map +1 -0
- package/dist/internal/setAiToolsBundledOnlyEnv.js.map +1 -0
- package/dist/invocation/defaultAiProfilesResolveOptions.d.ts +2 -2
- package/dist/invocation/defaultAiProfilesResolveOptions.d.ts.map +1 -1
- package/dist/invocation/defaultAiProfilesResolveOptions.js +2 -2
- package/dist/invocation/defaultAiProfilesResolveOptions.js.map +1 -1
- package/dist/invocation/preferOpenRouterPolicy.d.ts +4 -6
- package/dist/invocation/preferOpenRouterPolicy.d.ts.map +1 -1
- package/dist/invocation/preferOpenRouterPolicy.js +7 -12
- package/dist/invocation/preferOpenRouterPolicy.js.map +1 -1
- package/dist/invocation/resolveInvocationPlan.js +1 -1
- package/dist/invocation/resolveInvocationPlan.js.map +1 -1
- package/dist/invocation/resolveProfileInvocationRouting.d.ts +2 -2
- package/dist/invocation/resolveProfileInvocationRouting.d.ts.map +1 -1
- package/dist/invocation/resolveProfileInvocationRouting.js +1 -1
- package/dist/invocation/resolveProfileInvocationRouting.js.map +1 -1
- package/dist/invocation/types.d.ts +0 -2
- package/dist/invocation/types.d.ts.map +1 -1
- package/dist/strategies/direct-execution-strategy.js +1 -1
- package/dist/strategies/direct-execution-strategy.js.map +1 -1
- package/dist/utils/resolveAiProfileModel.d.ts +0 -2
- package/dist/utils/resolveAiProfileModel.d.ts.map +1 -1
- package/dist/utils/resolveAiProfileModel.js +1 -1
- package/dist/utils/resolveAiProfileModel.js.map +1 -1
- package/documenations/sessions/2026-06-08-subnets-model-resolution/CR-1-no-concrete-wire-in-graph-plans.md +93 -0
- package/documenations/sessions/2026-06-08-subnets-model-resolution/CR-2-skillModel-profile-only-at-storage.md +88 -0
- package/documenations/sessions/2026-06-08-subnets-model-resolution/CR-3-reject-concrete-models-in-catalog-rows.md +76 -0
- package/documenations/sessions/2026-06-08-subnets-model-resolution/FR-1-suggested-profile-in-catalogs.md +96 -0
- package/documenations/sessions/2026-06-08-subnets-model-resolution/FR-2-graph-engine-failure-phase-attribution.md +92 -0
- package/documenations/sessions/2026-06-08-subnets-model-resolution/INVESTIGATION-original-bug.md +182 -0
- package/documenations/sessions/2026-06-08-subnets-model-resolution/PROBLEM.md +236 -0
- package/documenations/sessions/2026-06-08-subnets-model-resolution/README.md +11 -0
- package/documenations/sessions/2026-06-08-subnets-model-resolution/funcx-test-resolveModel.cheapDefaultWireSlug.test.ts +117 -0
- package/package.json +10 -7
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# CR-1 — No concrete wire models in compiled graph plans
|
|
2
|
+
|
|
3
|
+
Status: **proposed**
|
|
4
|
+
Type: Change request
|
|
5
|
+
Owners: `@x12i/graphenix-plan-compiler`, `@exellix/graph-engine`, `@exellix/ai-tasks`
|
|
6
|
+
Filed: 2026-06-08
|
|
7
|
+
Session: [README](./README.md)
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Summary
|
|
12
|
+
|
|
13
|
+
Compiled executable node plans must store **profile/choice selections only**. They must **not** freeze concrete OpenRouter or gateway wire slugs in `resolvedInvocationSnapshot.modelId` or `AiConcreteModelSelection`.
|
|
14
|
+
|
|
15
|
+
Concrete resolution happens **once at invoke time** inside ai-tasks / xynthesis / ai-skills — same boundary as xynthesis PRE/POST today.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Problem
|
|
20
|
+
|
|
21
|
+
`overlayUnitModelOnRequest` prefers a pre-resolved snapshot over the authored selection:
|
|
22
|
+
|
|
23
|
+
```typescript
|
|
24
|
+
// ai-tasks/src/node-execution/resolveUnitModelSelection.ts
|
|
25
|
+
if (snapshot && typeof snapshot.modelId === "string" && snapshot.modelId.trim()) {
|
|
26
|
+
wireValue = snapshot.modelId; // e.g. "google/gemini-2.5-flash-lite"
|
|
27
|
+
} else if (selection) {
|
|
28
|
+
wireValue = selectionToWireValue(selection); // e.g. "cheap/default"
|
|
29
|
+
}
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
When the compiler writes `resolvedInvocationSnapshot.modelId: "google/gemini-2.5-flash-lite"`:
|
|
33
|
+
|
|
34
|
+
- Authoring shows `cheap/default` (correct).
|
|
35
|
+
- Runtime overlay injects a **concrete slug** into `preActionModel` / `postActionModel` / `skillModel`.
|
|
36
|
+
- Operators cannot tell whether failure is alias resolution or catalog lookup.
|
|
37
|
+
- Catalog drift (retired slugs) breaks frozen plans even when profiles are still valid.
|
|
38
|
+
|
|
39
|
+
This session's error slug may appear in the plan **even when input was `cheap/default`**.
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## Requirements
|
|
44
|
+
|
|
45
|
+
### CR-1.1 — Compiler output
|
|
46
|
+
|
|
47
|
+
- `executionUnits[].modelSelection` MUST be one of:
|
|
48
|
+
- `{ kind: "profileChoice", key: "cheap/default" }`
|
|
49
|
+
- `{ kind: "profile", profile: "cheap", choice: "default" }`
|
|
50
|
+
- MUST NOT emit `{ kind: "concrete", ... }` / `AiConcreteModelSelection` in new plans (version gate).
|
|
51
|
+
- MUST NOT emit `resolvedInvocationSnapshot.modelId` as a concrete provider/OpenRouter slug.
|
|
52
|
+
|
|
53
|
+
### CR-1.2 — Optional compile-time preview (read-only)
|
|
54
|
+
|
|
55
|
+
If compile needs a preview for UI, use a separate non-authoritative field, e.g.:
|
|
56
|
+
|
|
57
|
+
```json
|
|
58
|
+
"invocationPreview": {
|
|
59
|
+
"profileChoiceKey": "cheap/default",
|
|
60
|
+
"resolvedWireModelId": "google/gemini-2.5-flash-lite",
|
|
61
|
+
"phase": "preAction",
|
|
62
|
+
"resolverVersion": "ai-tasks/9.0.3"
|
|
63
|
+
}
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Preview MUST NOT be consumed by `overlayUnitModelOnRequest`.
|
|
67
|
+
|
|
68
|
+
### CR-1.3 — ai-tasks runtime
|
|
69
|
+
|
|
70
|
+
- `overlayUnitModelOnRequest`: ignore `resolvedInvocationSnapshot.modelId` when it looks like a concrete wire id (`isConcreteModelId`).
|
|
71
|
+
- Prefer `modelSelection` profile/choice keys only.
|
|
72
|
+
- Reject plans that only have concrete snapshot and no profile selection (validation error with path to unit).
|
|
73
|
+
|
|
74
|
+
### CR-1.4 — Migration
|
|
75
|
+
|
|
76
|
+
- Graphenix plan version bump (e.g. v2.2) documenting the rule.
|
|
77
|
+
- One release: warn on concrete snapshot; next release: reject.
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
## Acceptance criteria
|
|
82
|
+
|
|
83
|
+
- [ ] Compiled plan for subnet PRE unit contains `modelSelection: { kind: "profileChoice", key: "cheap/default" }` and no concrete `modelId` in authoritative fields.
|
|
84
|
+
- [ ] `overlayUnitModelOnRequest` passes `cheap/default` to `runTask`, not `google/gemini-2.5-flash-lite`.
|
|
85
|
+
- [ ] `resolveInvocationPlan` at compile preview may still show resolved wire for UI — not stored on executable plan.
|
|
86
|
+
- [ ] Existing graphs with concrete snapshots: migration script or re-compile from authoring.
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## Out of scope
|
|
91
|
+
|
|
92
|
+
- Changing ai-profiles registry contents.
|
|
93
|
+
- Banning concrete ids on ephemeral `RunTaskRequest.modelConfig` from direct API callers (see CR-2 for `skillModel` storage policy).
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
# CR-2 — `skillModel` profile/choice only at storage (align with PRE/POST)
|
|
2
|
+
|
|
3
|
+
Status: **proposed**
|
|
4
|
+
Type: Change request
|
|
5
|
+
Owners: `@exellix/ai-tasks`, `@exellix/graph-engine`, `@x12i/graphenix-authoring-format`
|
|
6
|
+
Filed: 2026-06-08
|
|
7
|
+
Session: [README](./README.md)
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Summary
|
|
12
|
+
|
|
13
|
+
Today:
|
|
14
|
+
|
|
15
|
+
| Slot | Storage rule |
|
|
16
|
+
|------|----------------|
|
|
17
|
+
| `preActionModel` | profile/choice only ✓ |
|
|
18
|
+
| `postActionModel` | profile/choice only ✓ |
|
|
19
|
+
| `skillModel` | profile/choice **or** concrete wire id ✗ |
|
|
20
|
+
|
|
21
|
+
There is no catalog field called "main model" (skill catalog stores no model — verified). The issue is **runtime/graph storage**: allowing **concrete** `skillModel` in graph JSON, fixtures, and plans freezes vendor slugs in orchestration data, separate from Catalox skill rows.
|
|
22
|
+
|
|
23
|
+
**Change:** `skillModel` and MAIN `llmCall.model` MUST be profile/choice keys at storage and in graph plans. Concrete wire ids are produced only at invoke resolution (ai-tasks → ai-skills), identical to how xynthesis resolves PRE/POST.
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## Problem
|
|
28
|
+
|
|
29
|
+
- `BREAKING-CHANGES.md` (8.4+) still documents concrete ids on `skillModel` as accepted.
|
|
30
|
+
- Fixtures use concrete MAIN models, e.g. `skillModel: "openrouter/x-ai/grok-4.1-fast"`.
|
|
31
|
+
- Operators conflate "MAIN model" with a durable catalog property; it is only the **`skillModel` runtime slot** for the `mainSkill` execution unit.
|
|
32
|
+
|
|
33
|
+
Desired model:
|
|
34
|
+
|
|
35
|
+
- **Suggested profile** on skill catalog row (FR-1) → default hint for graph authoring.
|
|
36
|
+
- **Per-node override** → `skillModel: "pro/default"` in plan.
|
|
37
|
+
- **Invoke** → ai-tasks resolves to gateway wire id once before `runSkill`.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Requirements
|
|
42
|
+
|
|
43
|
+
### CR-2.1 — Validation
|
|
44
|
+
|
|
45
|
+
- `validateRunTaskModelReferences` / `resolveRunTaskModelReferences`: reject concrete ids on `skillModel` and MAIN `llmCall.model` with `XYNTHESIS_CONCRETE_MODEL_REJECTED` equivalent: `SKILL_MODEL_CONCRETE_REJECTED`.
|
|
46
|
+
- Message: use profile/choice (e.g. `pro/default`, `cyber/default`).
|
|
47
|
+
|
|
48
|
+
### CR-2.2 — Resolution unchanged at invoke
|
|
49
|
+
|
|
50
|
+
- `prepareMainSkillModelConfigForInvoke` continues to resolve profile → gateway wire (`openrouter/...`) immediately before ai-skills.
|
|
51
|
+
- No behavior change for correctly authored graphs.
|
|
52
|
+
|
|
53
|
+
### CR-2.3 — Graph authoring
|
|
54
|
+
|
|
55
|
+
- Node model picker for MAIN unit: profile/choice only (same UX as PRE/POST).
|
|
56
|
+
- Remove concrete model picker from persisted graph JSON.
|
|
57
|
+
|
|
58
|
+
### CR-2.4 — Docs and fixtures
|
|
59
|
+
|
|
60
|
+
- Update `ai-tasks/README.md`, `BREAKING-CHANGES.md`, test fixtures to profile-only `skillModel`.
|
|
61
|
+
- Remove "concrete ids on MAIN only" exception language.
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## Acceptance criteria
|
|
66
|
+
|
|
67
|
+
- [ ] `skillModel: "openrouter/x-ai/grok-4.1-fast"` fails validation at `runTask` entry.
|
|
68
|
+
- [ ] `skillModel: "pro/default"` resolves and invokes successfully.
|
|
69
|
+
- [ ] `professional-answer` Catalox row still has **no** model field (only optional `suggestedProfile` per FR-1).
|
|
70
|
+
- [ ] Graph-engine passes profile keys on `mainSkill` units.
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## Migration
|
|
75
|
+
|
|
76
|
+
| From | To |
|
|
77
|
+
|------|-----|
|
|
78
|
+
| `skillModel: "anthropic/claude-sonnet-4"` | `skillModel: "pro/default"` (or appropriate profile) |
|
|
79
|
+
| `skillModel: "openrouter/x-ai/grok-4.1-fast"` | `skillModel: "cyber/default"` or explicit profile choice |
|
|
80
|
+
|
|
81
|
+
Provide `resolveInvocationPlan` mapping table in migration notes for common concrete → profile mappings.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Non-goals
|
|
86
|
+
|
|
87
|
+
- Removing invoke-time resolution to concrete ids (still required for ai-gateway).
|
|
88
|
+
- Changing Optimixer catalog fields on skill rows.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# CR-3 — Reject concrete model ids in skill / funcx / xynthesis catalog rows
|
|
2
|
+
|
|
3
|
+
Status: **proposed (guardrail)** — verified compliant today
|
|
4
|
+
Type: Change request
|
|
5
|
+
Owners: `@exellix/ai-skills`, `@x12i/funcx`, `@exellix/xynthesis`
|
|
6
|
+
Filed: 2026-06-08
|
|
7
|
+
Session: [README](./README.md)
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Summary
|
|
12
|
+
|
|
13
|
+
**Verified 2026-06-08:** skill / funcx **content** catalogs do **not** store models today. This CR is a **preventive guardrail** (provision lint + schema), not remediation of a current data bug.
|
|
14
|
+
|
|
15
|
+
Do not confuse with:
|
|
16
|
+
|
|
17
|
+
- **Runtime** `modelConfig` / graph `modelSelection` (orchestrator data — see CR-1, CR-2).
|
|
18
|
+
- **`@x12i/ai-tools` models catalog** (npm package used by funcx for slug validation — unrelated to Catalox skill rows).
|
|
19
|
+
|
|
20
|
+
Catalog rows (Catalox `ai-skills`, funcx function defaults, xynthesis sidekick seeds) must **never** store a concrete LLM wire model as the authoritative model for a skill or action.
|
|
21
|
+
|
|
22
|
+
Allowed:
|
|
23
|
+
|
|
24
|
+
- `suggestedProfile` / `modelPick` as **`profile/choice`** (e.g. `pro/default`, `balanced/default`).
|
|
25
|
+
- Optimixer sizing fields (output intent, reasoning effort catalog default, task type).
|
|
26
|
+
|
|
27
|
+
Forbidden:
|
|
28
|
+
|
|
29
|
+
- `model`, `modelId`, `mainModel`, `defaultModel` with values like `google/gemini-2.5-flash-lite`, `gpt-4o-mini`, `openrouter/...`.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Current state (mono-repo audit 2026-06-08)
|
|
34
|
+
|
|
35
|
+
| Catalog | Concrete model stored? | Profile suggestion? |
|
|
36
|
+
|---------|------------------------|---------------------|
|
|
37
|
+
| `ai-skills` Catalox (`AI_SKILLS_CATALOG_ITEMS`) | **No** ✓ | **No** (gap → FR-1) |
|
|
38
|
+
| `xynthesis` `model-routing.json` | **No** ✓ (output sizing only) | N/A |
|
|
39
|
+
| `xynthesis` Catalox seed (`modelPick`) | **No** ✓ | `balanced/default`, etc. |
|
|
40
|
+
| funcx built-in / content catalog | Audit consumer + funcx repo | TBD |
|
|
41
|
+
|
|
42
|
+
`professional-answer` row contains only Optimixer metadata — **compliant** with CR-3.
|
|
43
|
+
|
|
44
|
+
---
|
|
45
|
+
|
|
46
|
+
## Requirements
|
|
47
|
+
|
|
48
|
+
### CR-3.1 — Schema validation at provision time
|
|
49
|
+
|
|
50
|
+
- ai-skills `provision-ai-skills-catalog`: reject rows with `model`, `mainModel`, `defaultModel`, or concrete-pattern `modelId`.
|
|
51
|
+
- xynthesis seed validators: allow only `modelPick` matching `profile/choice` regex.
|
|
52
|
+
- funcx catalog linter: same rule for function `defaults.model`.
|
|
53
|
+
|
|
54
|
+
### CR-3.2 — Runtime must not read catalog model for invoke
|
|
55
|
+
|
|
56
|
+
- ai-skills `runSkill` model comes **only** from `RunSkillRequest.modelConfig.model` (set by orchestrator).
|
|
57
|
+
- funcx `run()` / sidekick: model from request or `modelPick` profile resolved at invoke — never a frozen slug from catalog body.
|
|
58
|
+
|
|
59
|
+
### CR-3.3 — Naming ban
|
|
60
|
+
|
|
61
|
+
- Disallow field names `mainModel`, `defaultModel`, `skillModel` on catalog `data` objects (reserved for runtime request shape).
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## Acceptance criteria
|
|
66
|
+
|
|
67
|
+
- [ ] Provision script fails if a skill row includes `model: "google/..."`.
|
|
68
|
+
- [ ] `professional-answer` provision unchanged (Optimixer-only).
|
|
69
|
+
- [ ] Documentation lists allowed catalog model fields: `suggestedProfile` only (see FR-1).
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## Related
|
|
74
|
+
|
|
75
|
+
- FR-1 adds the **positive** allowed field.
|
|
76
|
+
- CR-2 ensures graph/runtime `skillModel` is also profile-only at storage.
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# FR-1 — Suggested profile on skill and funcx catalog rows
|
|
2
|
+
|
|
3
|
+
Status: **proposed**
|
|
4
|
+
Type: Feature request
|
|
5
|
+
Owners: `@exellix/ai-skills`, `@x12i/funcx`, `@exellix/xynthesis`
|
|
6
|
+
Filed: 2026-06-08
|
|
7
|
+
Session: [README](./README.md)
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Summary
|
|
12
|
+
|
|
13
|
+
Add an optional **`suggestedProfile`** (or reuse xynthesis **`modelPick`**) on catalog rows: a single **`profile/choice`** string that recommends which ai-profiles tier fits this skill or sidekick action.
|
|
14
|
+
|
|
15
|
+
This is a **hint for authoring and studio UIs** — not an invoke-time default that bypasses the orchestrator.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Motivation
|
|
20
|
+
|
|
21
|
+
- Operators want "what model tier fits `professional-answer`?" without storing concrete slugs.
|
|
22
|
+
- Graph nodes should start from `suggestedProfile` when the author does not override `skillModel`.
|
|
23
|
+
- Aligns with CR-3 (no concrete models in catalog) while supporting discoverability.
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## Proposed shape
|
|
28
|
+
|
|
29
|
+
### ai-skills Catalox row
|
|
30
|
+
|
|
31
|
+
```json
|
|
32
|
+
{
|
|
33
|
+
"skillKey": "professional-answer",
|
|
34
|
+
"suggestedProfile": "pro/default",
|
|
35
|
+
"optimixerReasoningEffort": "low",
|
|
36
|
+
"optimixerOutputIntent": { "mode": "relative", "expectedVisibleTokens": 1200 }
|
|
37
|
+
}
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
### xynthesis sidekick seed (already similar)
|
|
41
|
+
|
|
42
|
+
```json
|
|
43
|
+
{
|
|
44
|
+
"defaults": {
|
|
45
|
+
"modelPick": "balanced/default",
|
|
46
|
+
"outputMode": "structured"
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Standardize naming in docs: **`suggestedProfile`** for skills, **`modelPick`** for xynthesis actions (alias in UI).
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## Behavior
|
|
56
|
+
|
|
57
|
+
| Layer | Behavior |
|
|
58
|
+
|-------|----------|
|
|
59
|
+
| **Catalog** | Store `suggestedProfile` only; validate `profile/choice` via `@x12i/ai-profiles`. |
|
|
60
|
+
| **Studio / graph authoring** | New node MAIN unit pre-fills `modelSelection` from skill's `suggestedProfile`. |
|
|
61
|
+
| **runTask / runSkill** | Orchestrator MUST still send `modelConfig.skillModel` (or plan unit selection). Catalog hint is **not** a silent default at invoke. |
|
|
62
|
+
| **diagnoseSkillInCatalog** | Return `suggestedProfile` in diagnostic payload for tooling. |
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## API sketch (ai-skills)
|
|
67
|
+
|
|
68
|
+
```typescript
|
|
69
|
+
type AiSkillsCatalogItemSpec = {
|
|
70
|
+
skillKey: string;
|
|
71
|
+
// ...
|
|
72
|
+
suggestedProfile?: string; // e.g. "pro/default"
|
|
73
|
+
};
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
Export helper:
|
|
77
|
+
|
|
78
|
+
```typescript
|
|
79
|
+
function assertSuggestedProfile(value: string): void;
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
---
|
|
83
|
+
|
|
84
|
+
## Acceptance criteria
|
|
85
|
+
|
|
86
|
+
- [ ] `professional-answer` can ship with `suggestedProfile: "pro/default"` (product choice).
|
|
87
|
+
- [ ] Provision rejects invalid profile keys and concrete slugs.
|
|
88
|
+
- [ ] Invoke without `skillModel` still fails fast (no hidden catalog default).
|
|
89
|
+
- [ ] Studio can list skills with suggested tier for graph palette.
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
93
|
+
## Non-goals
|
|
94
|
+
|
|
95
|
+
- Auto-picking model at `runSkill` without orchestrator `modelConfig`.
|
|
96
|
+
- Storing multiple ranked profiles (future: `suggestedProfiles[]` if needed).
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
# FR-2 — Graph-engine failure phase attribution (PRE vs MAIN vs POST)
|
|
2
|
+
|
|
3
|
+
Status: **proposed**
|
|
4
|
+
Type: Feature request
|
|
5
|
+
Owners: `@exellix/graph-engine`
|
|
6
|
+
Filed: 2026-06-08
|
|
7
|
+
Session: [README](./README.md)
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Summary
|
|
12
|
+
|
|
13
|
+
When `runTask` fails, graph-engine should surface **`phase`** and **`stage`** from `@exellix/ai-tasks` `RunTaskExecutionError` (or root cause), not only `skillKey`.
|
|
14
|
+
|
|
15
|
+
This session's confusion: error message referenced `skillKey=professional-answer` while the root cause was likely **PRE synthesis** (`cheap/default` → funcx catalog), not MAIN skill invoke.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Problem
|
|
20
|
+
|
|
21
|
+
Current wrapper shape (observed):
|
|
22
|
+
|
|
23
|
+
```text
|
|
24
|
+
runTask failed (skillKey=professional-answer nodeId=node:inbound-reachability graphId=graph:subnets-analysis): Unknown model "google/gemini-2.5-flash-lite" ...
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
ai-tasks native format includes phase when known:
|
|
28
|
+
|
|
29
|
+
```text
|
|
30
|
+
runTask failed (phase=pipeline_pre skillKey=...): ...
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Graph-engine omits `phase=`, so operators assume **MAIN / professional-answer model** is wrong.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Requirements
|
|
38
|
+
|
|
39
|
+
### FR-2.1 — Error envelope
|
|
40
|
+
|
|
41
|
+
Node execution error SHOULD include:
|
|
42
|
+
|
|
43
|
+
```json
|
|
44
|
+
{
|
|
45
|
+
"skillKey": "professional-answer",
|
|
46
|
+
"graphId": "graph:subnets-analysis",
|
|
47
|
+
"nodeId": "node:inbound-reachability",
|
|
48
|
+
"phase": "pipeline_pre",
|
|
49
|
+
"stage": "pre-synthesis",
|
|
50
|
+
"rootMessage": "Unknown model \"google/gemini-2.5-flash-lite\" ...",
|
|
51
|
+
"rootCode": "MISSING_ENV",
|
|
52
|
+
"modelConfig": {
|
|
53
|
+
"preActionModel": "cheap/default",
|
|
54
|
+
"skillModel": "pro/default",
|
|
55
|
+
"postActionModel": "cheap/default"
|
|
56
|
+
},
|
|
57
|
+
"failureClassification": { "role": "cause", "event": "provider_invoke" }
|
|
58
|
+
}
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### FR-2.2 — Propagation
|
|
62
|
+
|
|
63
|
+
- Parse `RunTaskExecutionError.details.phase` when present.
|
|
64
|
+
- Fall back to heuristic on `rootMessage` (funcx catalog → `pipeline_pre` or `pipeline_post`; gateway → `main_skill`).
|
|
65
|
+
|
|
66
|
+
### FR-2.3 — UI copy
|
|
67
|
+
|
|
68
|
+
- Display: **"PRE synthesis failed"** vs **"Skill invoke failed"** based on `phase`.
|
|
69
|
+
- Do not show "professional-answer model error" when `phase !== main_skill`.
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## Acceptance criteria
|
|
74
|
+
|
|
75
|
+
- [ ] PRE funcx catalog failure shows `phase: "pipeline_pre"` in graph-engine node error JSON.
|
|
76
|
+
- [ ] MAIN gateway failure shows `phase: "main_skill"`.
|
|
77
|
+
- [ ] Same `skillKey` on wrapper; phase disambiguates.
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
## Reference (ai-tasks)
|
|
82
|
+
|
|
83
|
+
- `RunTaskFailurePhase`: `pipeline_pre`, `main_skill`, `pipeline_post`, `model-resolution`, …
|
|
84
|
+
- `formatRunTaskExecutionMessage` in `ai-tasks/src/errors/runTaskExecutionError.ts`
|
|
85
|
+
- `classifyRunTaskFailure` in `ai-tasks/src/observability/classifyRunTaskFailure.ts`
|
|
86
|
+
|
|
87
|
+
---
|
|
88
|
+
|
|
89
|
+
## Non-goals
|
|
90
|
+
|
|
91
|
+
- Changing ai-tasks error types (consume as-is).
|
|
92
|
+
- Resolving catalog issues (see [INVESTIGATION-original-bug.md](./INVESTIGATION-original-bug.md)).
|
package/documenations/sessions/2026-06-08-subnets-model-resolution/INVESTIGATION-original-bug.md
ADDED
|
@@ -0,0 +1,182 @@
|
|
|
1
|
+
# Investigation: `Unknown model "google/gemini-2.5-flash-lite"`
|
|
2
|
+
|
|
3
|
+
Status: **runtime / consumer deps** (skill catalog ruled out)
|
|
4
|
+
Filed: 2026-06-08
|
|
5
|
+
Repro: `worox-graphs-playground` — `graph:subnets-analysis`, `node:inbound-reachability`, `skillKey=professional-answer`
|
|
6
|
+
Verified: 2026-06-08 — **no models in skill/funcx content catalog**; runtime `cheap/default` → wire slug is correct.
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## Executive summary
|
|
11
|
+
|
|
12
|
+
| Question | Answer |
|
|
13
|
+
|----------|--------|
|
|
14
|
+
| Was `google/gemini-2.5-flash-lite` the authored input? | **No** — runtime uses `cheap/default`. |
|
|
15
|
+
| Is `cheap/default` → `google/gemini-2.5-flash-lite` correct? | **Yes** — expected OpenRouter slug from `@x12i/ai-profiles` at **invoke time**. |
|
|
16
|
+
| Did it come from the skill catalog? | **No** — Catalox skill rows store no model; verified. |
|
|
17
|
+
| Which phase failed? | Almost certainly **PRE or POST xynthesis** (funcx), **not** MAIN `professional-answer` (ai-gateway). |
|
|
18
|
+
| Why does the error say `skillKey=professional-answer`? | Graph-engine wraps the whole `runTask` failure under the node task key. |
|
|
19
|
+
|
|
20
|
+
**Conclusion:** Runtime profile resolution is fine. If the error persists, it is **funcx + `@x12i/ai-tools` package catalog lookup** at invoke (or misleading phase in the error wrapper) — **not** skill catalog data and not wrong input from `cheap/default`.
|
|
21
|
+
|
|
22
|
+
### “Catalog” disambiguation
|
|
23
|
+
|
|
24
|
+
| Name | Role in this incident |
|
|
25
|
+
|------|----------------------|
|
|
26
|
+
| Catalox **ai-skills** catalog | Templates + Optimixer only — **not involved** |
|
|
27
|
+
| **Runtime** `modelConfig` / graph plan | Source of `cheap/default` — **correct** |
|
|
28
|
+
| `@x12i/ai-tools` **models catalog** (inside funcx) | Validates `google/gemini-2.5-flash-lite` at `ask()` — **only place “catalog-resolvable” applies** |
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## Observed error
|
|
33
|
+
|
|
34
|
+
```json
|
|
35
|
+
{
|
|
36
|
+
"nodeId": "node:inbound-reachability",
|
|
37
|
+
"error": {
|
|
38
|
+
"message": "runTask failed (skillKey=professional-answer nodeId=node:inbound-reachability graphId=graph:subnets-analysis): Unknown model \"google/gemini-2.5-flash-lite\". Pass a catalog-resolvable model slug (e.g. \"deepseek/deepseek-v4-flash\")."
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Stack: `@exellix/graph-engine` → `runTask` → root error from `@x12i/funcx` `requireCatalogModel()`.
|
|
44
|
+
|
|
45
|
+
Exact funcx message shape (from `@x12i/funcx` ≥4.3):
|
|
46
|
+
|
|
47
|
+
```text
|
|
48
|
+
Unknown model "<slug>". Pass a catalog-resolvable model slug (e.g. "deepseek/deepseek-v4-flash").
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
ai-gateway / ai-tools `ModelResolutionError` uses a **different** message (`Could not resolve model "..." (provider: "...")`). So this failure is **not** the MAIN gateway path.
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## Resolution chain (verified in mono-repo)
|
|
56
|
+
|
|
57
|
+
### 1. ai-profiles: `cheap/default`
|
|
58
|
+
|
|
59
|
+
Bundled registry (`@x12i/ai-profiles`):
|
|
60
|
+
|
|
61
|
+
- Profile `cheap`, choice `default`
|
|
62
|
+
- `provider: google`, `modelId: gemini-2.5-flash-lite`
|
|
63
|
+
- OpenRouter invocation: `google/gemini-2.5-flash-lite`
|
|
64
|
+
|
|
65
|
+
### 2. xynthesis: alias → bare OpenRouter slug
|
|
66
|
+
|
|
67
|
+
`resolveXynthesisModel("cheap/default")` → `wireModelId: "google/gemini-2.5-flash-lite"`
|
|
68
|
+
Used by `FuncxInvoker.client.ask({ model: wireModelId })`.
|
|
69
|
+
|
|
70
|
+
### 3. ai-tasks PRE step
|
|
71
|
+
|
|
72
|
+
When `taskConfiguration.aiTaskStrategies.pre` includes synthesis, `_runSynthesizedContextPreStep` reads `modelConfig.preActionModel` (typically `cheap/default` for subnet-style graphs).
|
|
73
|
+
|
|
74
|
+
Fixture pattern in this repo:
|
|
75
|
+
|
|
76
|
+
```json
|
|
77
|
+
"modelConfig": {
|
|
78
|
+
"preActionModel": "cheap/default",
|
|
79
|
+
"postActionModel": "cheap/default",
|
|
80
|
+
"skillModel": "<MAIN profile or concrete — separate from PRE>"
|
|
81
|
+
}
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### 4. MAIN professional-answer (different path)
|
|
85
|
+
|
|
86
|
+
`skillModel` resolves to **gateway** wire id, e.g. `openrouter/google/gemini-2.5-flash-lite` — not the bare slug in the error.
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## Failure hypothesis (ranked)
|
|
91
|
+
|
|
92
|
+
### H1 — Stale `@x12i/ai-tools` / `@x12i/funcx` in `worox-graphs-playground` (most likely)
|
|
93
|
+
|
|
94
|
+
funcx `requireCatalogModel()` calls ai-tools `createModelNameResolver(catalog).resolve()`. If the installed catalog predates `gemini-2.5-flash-lite`, or OpenRouter lane index is incomplete, `found: false` → this exact error.
|
|
95
|
+
|
|
96
|
+
**Mono-repo reference versions (known good):**
|
|
97
|
+
|
|
98
|
+
| Package | Version in `ai-tasks` |
|
|
99
|
+
|---------|----------------------|
|
|
100
|
+
| `@exellix/ai-tasks` | 9.0.3 |
|
|
101
|
+
| `@exellix/xynthesis` | 4.6.4 |
|
|
102
|
+
| `@x12i/funcx` (override) | ^4.6.0 |
|
|
103
|
+
| `@x12i/ai-profiles` | ^3.3.0 |
|
|
104
|
+
|
|
105
|
+
### H2 — funcx ai-tools client disabled or throws in catalog load
|
|
106
|
+
|
|
107
|
+
funcx `resolveModelSlug` catches all errors and returns `found: false` — surfaces as "Unknown model" even when the slug is valid.
|
|
108
|
+
|
|
109
|
+
Check env / config: FuncX ai-tools `enabled=false`, broken catalog fetch, etc.
|
|
110
|
+
|
|
111
|
+
### H3 — Compile-time concrete slug injection (secondary)
|
|
112
|
+
|
|
113
|
+
If graph plan has `resolvedInvocationSnapshot.modelId: "google/gemini-2.5-flash-lite"` on a PRE unit, behavior is the same at funcx — but **authoring** was still `cheap/default`. See CR-1.
|
|
114
|
+
|
|
115
|
+
### H4 — Wrong phase in operator UI (symptom only)
|
|
116
|
+
|
|
117
|
+
Operator reads "professional-answer failed" and assumes MAIN model is wrong. PRE failed before MAIN ran. See FR-2.
|
|
118
|
+
|
|
119
|
+
---
|
|
120
|
+
|
|
121
|
+
## Investigation checklist (`worox-graphs-playground`)
|
|
122
|
+
|
|
123
|
+
Run in the consumer repo (not this mono-repo unless linked):
|
|
124
|
+
|
|
125
|
+
- [ ] **1. Confirm runtime `modelConfig` on the failing node**
|
|
126
|
+
- `preActionModel`, `skillModel`, `postActionModel` on the `RunTaskRequest` graph-engine builds.
|
|
127
|
+
- Expect: `cheap/default` on PRE; MAIN slot may differ.
|
|
128
|
+
|
|
129
|
+
- [ ] **2. Confirm compiled `nodePlan` for `node:inbound-reachability`**
|
|
130
|
+
- Per-unit `modelSelection` (`profileChoice` vs concrete).
|
|
131
|
+
- `resolvedInvocationSnapshot.modelId` — if present with concrete slug, note for CR-1.
|
|
132
|
+
|
|
133
|
+
- [ ] **3. Lockfile versions**
|
|
134
|
+
```bash
|
|
135
|
+
npm ls @x12i/funcx @x12i/ai-tools @exellix/xynthesis @exellix/ai-tasks @x12i/ai-profiles
|
|
136
|
+
```
|
|
137
|
+
Compare to mono-repo `ai-tasks/package.json`.
|
|
138
|
+
|
|
139
|
+
- [ ] **4. Catalog spot-check in installed tree**
|
|
140
|
+
```bash
|
|
141
|
+
rg "gemini-2.5-flash-lite" node_modules/@x12i/ai-tools/dist/
|
|
142
|
+
```
|
|
143
|
+
Should appear in bundled `models-catalog.json` chunk.
|
|
144
|
+
|
|
145
|
+
- [ ] **5. Reproduce with trace**
|
|
146
|
+
- `executionMode: "trace"` on `runTask`.
|
|
147
|
+
- Look for xynthesis `wireModelId` / `profileChoiceKey: "cheap/default"` **before** any MAIN skill observation.
|
|
148
|
+
- If only PRE logs appear → confirms H4.
|
|
149
|
+
|
|
150
|
+
- [ ] **6. Minimal isolated PRE call**
|
|
151
|
+
- xynthesis `executeXynthesisAction` with `model: "cheap/default"` and same env as playground.
|
|
152
|
+
- If fails → H1/H2. If passes → graph wiring or plan overlay issue.
|
|
153
|
+
|
|
154
|
+
- [ ] **7. OPENROUTER_API_KEY**
|
|
155
|
+
- Required for OpenRouter backend; missing key causes other errors, but verify present.
|
|
156
|
+
|
|
157
|
+
---
|
|
158
|
+
|
|
159
|
+
## What is **not** the bug
|
|
160
|
+
|
|
161
|
+
- Choosing `cheap/default` for PRE/POST.
|
|
162
|
+
- ai-profiles mapping `cheap/default` → `google/gemini-2.5-flash-lite`.
|
|
163
|
+
- xynthesis emitting bare OpenRouter slug to funcx (by design since xynthesis ≥4.1.8).
|
|
164
|
+
- Skill / funcx **content** catalog storing or supplying a model (verified: **no models there**).
|
|
165
|
+
- Confusing runtime resolved wire with catalog-authored model (there is no latter).
|
|
166
|
+
|
|
167
|
+
---
|
|
168
|
+
|
|
169
|
+
## Recommended fix path (consumer)
|
|
170
|
+
|
|
171
|
+
1. Align `@x12i/funcx`, `@x12i/ai-tools`, `@exellix/xynthesis`, `@exellix/ai-tasks` to mono-repo versions (or newer).
|
|
172
|
+
2. Re-run graph; if still failing, run checklist §5–6.
|
|
173
|
+
3. If catalog is current and resolve still fails, file upstream bug on funcx/ai-tools with `resolveModelSlug` diagnostics (`attemptedStrategies`, `bestRejectedCandidate`).
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## Related mono-repo references
|
|
178
|
+
|
|
179
|
+
- PRE fixture: `ai-tasks/test/fixtures/run-task/graph-engine-pre-synthesis.json`
|
|
180
|
+
- xynthesis wire: `xynthesis/src/funcxInvoker.ts`, `xynthesis/src/resolveAiProfileModel.ts`
|
|
181
|
+
- funcx validation: `node_modules/@x12i/funcx/dist/src/serve.js` → `requireCatalogModel`
|
|
182
|
+
- Error wrap: `ai-tasks/src/errors/runTaskExecutionError.ts`
|