@kindgi/cli 0.2.0-rc.1 → 0.3.0-rc.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +12 -5
- package/dist/cli.js +4 -3
- package/dist/cli.js.map +1 -1
- package/dist/commands/agents.d.ts.map +1 -1
- package/dist/commands/agents.js +75 -19
- package/dist/commands/agents.js.map +1 -1
- package/dist/commands/approval-wait.d.ts +90 -0
- package/dist/commands/approval-wait.d.ts.map +1 -0
- package/dist/commands/approval-wait.js +154 -0
- package/dist/commands/approval-wait.js.map +1 -0
- package/dist/commands/artifacts.d.ts.map +1 -1
- package/dist/commands/artifacts.js +131 -14
- package/dist/commands/artifacts.js.map +1 -1
- package/dist/commands/auth.d.ts.map +1 -1
- package/dist/commands/auth.js +48 -5
- package/dist/commands/auth.js.map +1 -1
- package/dist/commands/deploy.js +7 -3
- package/dist/commands/deploy.js.map +1 -1
- package/dist/commands/deployments.d.ts.map +1 -1
- package/dist/commands/deployments.js +36 -13
- package/dist/commands/deployments.js.map +1 -1
- package/dist/commands/doctor.d.ts +3 -2
- package/dist/commands/doctor.d.ts.map +1 -1
- package/dist/commands/doctor.js +47 -14
- package/dist/commands/doctor.js.map +1 -1
- package/dist/commands/eval-runs.d.ts.map +1 -1
- package/dist/commands/eval-runs.js +21 -9
- package/dist/commands/eval-runs.js.map +1 -1
- package/dist/commands/helpers.d.ts +31 -1
- package/dist/commands/helpers.d.ts.map +1 -1
- package/dist/commands/helpers.js +45 -2
- package/dist/commands/helpers.js.map +1 -1
- package/dist/commands/init.js +5 -4
- package/dist/commands/init.js.map +1 -1
- package/dist/commands/mail-triggers.d.ts.map +1 -1
- package/dist/commands/mail-triggers.js +27 -3
- package/dist/commands/mail-triggers.js.map +1 -1
- package/dist/commands/promote.d.ts.map +1 -1
- package/dist/commands/promote.js +133 -11
- package/dist/commands/promote.js.map +1 -1
- package/dist/commands/provenance.d.ts.map +1 -1
- package/dist/commands/provenance.js +13 -5
- package/dist/commands/provenance.js.map +1 -1
- package/dist/commands/providers.d.ts.map +1 -1
- package/dist/commands/providers.js +47 -4
- package/dist/commands/providers.js.map +1 -1
- package/dist/commands/schedules.d.ts.map +1 -1
- package/dist/commands/schedules.js +25 -9
- package/dist/commands/schedules.js.map +1 -1
- package/dist/commands/skills.d.ts +16 -0
- package/dist/commands/skills.d.ts.map +1 -1
- package/dist/commands/skills.js +122 -36
- package/dist/commands/skills.js.map +1 -1
- package/dist/commands/webhooks.d.ts.map +1 -1
- package/dist/commands/webhooks.js +21 -5
- package/dist/commands/webhooks.js.map +1 -1
- package/dist/config.d.ts +8 -0
- package/dist/config.d.ts.map +1 -1
- package/dist/config.js +11 -0
- package/dist/config.js.map +1 -1
- package/dist/context.d.ts +7 -3
- package/dist/context.d.ts.map +1 -1
- package/dist/context.js +1 -0
- package/dist/context.js.map +1 -1
- package/dist/dev/runtime-image.d.ts +1 -1
- package/dist/dev/runtime-image.js +1 -1
- package/dist/errors.d.ts.map +1 -1
- package/dist/errors.js +14 -1
- package/dist/errors.js.map +1 -1
- package/dist/exports/plan.d.ts.map +1 -1
- package/dist/exports/plan.js +50 -0
- package/dist/exports/plan.js.map +1 -1
- package/dist/init/augment-scaffolder.d.ts +1 -1
- package/dist/init/augment-scaffolder.js +9 -8
- package/dist/init/augment-scaffolder.js.map +1 -1
- package/dist/init/java-augment.d.ts.map +1 -1
- package/dist/init/java-augment.js +7 -6
- package/dist/init/java-augment.js.map +1 -1
- package/dist/init/python-augment.d.ts.map +1 -1
- package/dist/init/python-augment.js +7 -6
- package/dist/init/python-augment.js.map +1 -1
- package/dist/init/scala-augment.js +7 -6
- package/dist/init/scala-augment.js.map +1 -1
- package/dist/main.d.ts +3 -0
- package/dist/main.d.ts.map +1 -1
- package/dist/main.js +1 -0
- package/dist/main.js.map +1 -1
- package/dist/providers/presets/anthropic.json +15 -3
- package/dist/providers/presets/azure-openai.json +10 -2
- package/dist/providers/presets/bedrock.json +6 -2
- package/dist/providers/presets/gemini-api.json +6 -2
- package/dist/providers/presets/openai.json +15 -3
- package/dist/sdk-skills/kindgi-authoring-agents/SKILL.md +21 -9
- package/dist/sdk-skills/kindgi-authoring-flows/SKILL.md +5 -3
- package/dist/sdk-skills/kindgi-authoring-guardrails/SKILL.md +202 -68
- package/dist/sdk-skills/kindgi-authoring-mcp-servers/SKILL.md +12 -188
- package/dist/sdk-skills/kindgi-authoring-mcp-servers/common-mistakes.md +44 -0
- package/dist/sdk-skills/kindgi-authoring-mcp-servers/how-it-works.md +41 -0
- package/dist/sdk-skills/kindgi-authoring-mcp-servers/other-clients.md +19 -0
- package/dist/sdk-skills/kindgi-authoring-mcp-servers/postgres.md +45 -0
- package/dist/sdk-skills/kindgi-authoring-mcp-servers/several-servers.md +32 -0
- package/dist/sdk-skills/kindgi-authoring-providers/SKILL.md +49 -906
- package/dist/sdk-skills/kindgi-authoring-providers/bedrock.md +64 -0
- package/dist/sdk-skills/kindgi-authoring-providers/common-mistakes.md +125 -0
- package/dist/sdk-skills/kindgi-authoring-providers/keys-and-registration.md +58 -0
- package/dist/sdk-skills/kindgi-authoring-providers/kindgi-cloud.md +36 -0
- package/dist/sdk-skills/kindgi-authoring-providers/path-a-anthropic.md +216 -0
- package/dist/sdk-skills/kindgi-authoring-providers/path-b-openai-compat.md +127 -0
- package/dist/sdk-skills/kindgi-authoring-providers/path-c-onnx.md +97 -0
- package/dist/sdk-skills/kindgi-authoring-providers/path-d-vertex.md +95 -0
- package/dist/sdk-skills/kindgi-authoring-providers/router.md +113 -0
- package/dist/sdk-skills/kindgi-authoring-tools/SKILL.md +12 -387
- package/dist/sdk-skills/kindgi-authoring-tools/common-mistakes.md +135 -0
- package/dist/sdk-skills/kindgi-authoring-tools/declarative-http.md +34 -0
- package/dist/sdk-skills/kindgi-authoring-tools/imperative-handler.md +65 -0
- package/dist/sdk-skills/kindgi-authoring-tools/iterating-and-testing.md +45 -0
- package/dist/sdk-skills/kindgi-authoring-tools/secrets-and-connections.md +79 -0
- package/dist/sdk-skills/kindgi-authoring-tools/tool-flags.md +45 -0
- package/dist/sdk-skills/kindgi-fix-corrections/SKILL.md +13 -178
- package/dist/sdk-skills/kindgi-fix-corrections/make-the-change.md +25 -0
- package/dist/sdk-skills/kindgi-fix-corrections/publish-and-promote.md +76 -0
- package/dist/sdk-skills/kindgi-fix-corrections/pull-corrections.md +48 -0
- package/dist/sdk-skills/kindgi-fix-corrections/worked-example.md +27 -0
- package/dist/sdk-skills/kindgi-framework-feedback/SKILL.md +10 -180
- package/dist/sdk-skills/kindgi-framework-feedback/common-mistakes.md +19 -0
- package/dist/sdk-skills/kindgi-framework-feedback/editing-feedback-file.md +33 -0
- package/dist/sdk-skills/kindgi-framework-feedback/fields-and-entry-shape.md +53 -0
- package/dist/sdk-skills/kindgi-framework-feedback/hearing-back.md +24 -0
- package/dist/sdk-skills/kindgi-framework-feedback/sending.md +50 -0
- package/dist/sdk-skills/kindgi-getting-started/SKILL.md +24 -21
- package/dist/sdk-skills/kindgi-java-authoring-agents/SKILL.md +12 -4
- package/dist/sdk-skills/kindgi-java-authoring-flows/SKILL.md +6 -4
- package/dist/sdk-skills/kindgi-java-authoring-guardrails/SKILL.md +33 -23
- package/dist/sdk-skills/kindgi-java-authoring-tools/SKILL.md +19 -12
- package/dist/sdk-skills/kindgi-java-getting-started/SKILL.md +16 -16
- package/dist/sdk-skills/kindgi-python-authoring-agents/SKILL.md +18 -6
- package/dist/sdk-skills/kindgi-python-authoring-flows/SKILL.md +6 -4
- package/dist/sdk-skills/kindgi-python-authoring-guardrails/SKILL.md +39 -30
- package/dist/sdk-skills/kindgi-python-authoring-tools/SKILL.md +19 -9
- package/dist/sdk-skills/kindgi-python-getting-started/SKILL.md +24 -21
- package/dist/sdk-skills/kindgi-scala-authoring-agents/SKILL.md +12 -4
- package/dist/sdk-skills/kindgi-scala-authoring-flows/SKILL.md +6 -4
- package/dist/sdk-skills/kindgi-scala-authoring-guardrails/SKILL.md +34 -23
- package/dist/sdk-skills/kindgi-scala-authoring-tools/SKILL.md +15 -9
- package/dist/sdk-skills/kindgi-scala-getting-started/SKILL.md +16 -16
- package/dist/sdk-skills/kindgi-using/SKILL.md +11 -11
- package/dist/templates/sample/README.md.tmpl +8 -7
- package/dist/templates/sample/agents/echo-agent/index.ts.tmpl +1 -1
- package/dist/templates/sample/kindgi.config.ts.tmpl +1 -1
- package/dist/templates/sample/package.json.tmpl +1 -1
- package/dist/templates/woocommerce/README.md.tmpl +9 -3
- package/package.json +13 -13
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Imperative handler
|
|
2
|
+
|
|
3
|
+
```ts
|
|
4
|
+
// tools/verify-citation/index.ts
|
|
5
|
+
import { defineTool } from '@kindgi/sdk/define';
|
|
6
|
+
import type { ToolId } from '@kindgi/sdk/types';
|
|
7
|
+
import { z } from 'zod';
|
|
8
|
+
|
|
9
|
+
const Input = z.object({
|
|
10
|
+
citation: z.string().min(1),
|
|
11
|
+
jurisdiction: z.enum(['US', 'UK', 'EU']),
|
|
12
|
+
});
|
|
13
|
+
|
|
14
|
+
const Output = z.object({
|
|
15
|
+
found: z.boolean(),
|
|
16
|
+
canonicalCite: z.string().optional(),
|
|
17
|
+
});
|
|
18
|
+
|
|
19
|
+
const defined = defineTool({
|
|
20
|
+
id: 'acme.verify-citation' as ToolId,
|
|
21
|
+
description:
|
|
22
|
+
'Verify a legal citation against the jurisdictional citator. Returns whether the citation resolves and the canonical form.',
|
|
23
|
+
version: '0.1.0',
|
|
24
|
+
input: Input,
|
|
25
|
+
output: Output,
|
|
26
|
+
effects: [],
|
|
27
|
+
mutating: false,
|
|
28
|
+
handler: async (input, ctx) => {
|
|
29
|
+
// ctx.tenantId, ctx.abortSignal, ctx.secrets (what needsSpec declares) available;
|
|
30
|
+
// in a run, ctx.projectId and ctx.orgId (set from the run, never from the input)
|
|
31
|
+
// Return type MUST match Output schema (validated at invoke time)
|
|
32
|
+
return { found: true, canonicalCite: '...' };
|
|
33
|
+
},
|
|
34
|
+
});
|
|
35
|
+
|
|
36
|
+
if (defined.kind === 'err') {
|
|
37
|
+
throw new Error(`acme.verify-citation failed to compile: ${defined.error.message}`);
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
export default defined.value;
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
A handler must return a Promise; one with nothing to `await` can return `Promise.resolve({ … })` instead of being `async`, which keeps lint rules like `require-await` quiet.
|
|
44
|
+
|
|
45
|
+
## What the handler receives
|
|
46
|
+
|
|
47
|
+
The handler gets the **parsed** input, typed `z.infer` of `input` (Zod's output type):
|
|
48
|
+
|
|
49
|
+
- **Defaults.** A `.default()` field is optional to the caller, the model included. The tool's advertised schema doesn't list it as required, and the handler always gets a value.
|
|
50
|
+
- **Transforms and refinements.** `.transform()` results and `.refine()` checks apply before the handler runs. A failed refinement comes back as `input-validation-failed`.
|
|
51
|
+
- **Extra keys.** A plain `z.object` accepts them and strips them. Use `z.strictObject` to reject them.
|
|
52
|
+
- **JSON-Schema-authored tools** get each property's `default` filled in the same way.
|
|
53
|
+
- **Unions of scalars** (`z.union([z.string(), z.number()])`, `type: ['string', 'number']` on the wire) compile and validate. A schema Kindgi's strict schema check still refuses (an open tuple, an unknown keyword) says so at `defineTool`; for a field that may hold any JSON value, `z.json()` compiles.
|
|
54
|
+
|
|
55
|
+
The output side is the reverse: the advertised output schema requires every field, defaulted ones included. Return them all.
|
|
56
|
+
|
|
57
|
+
## Logging from the handler
|
|
58
|
+
|
|
59
|
+
`ctx.log` is a logger bound to the call: its records carry the run's ids, the tool's id and the trace id. It has `info`, `warn`, `error`, `debug` and `trace`, each `(message, fields?)`. The pack service sets it; a context your own code builds (a test's) may not, so write `ctx.log?.`:
|
|
60
|
+
|
|
61
|
+
```ts
|
|
62
|
+
ctx.log?.info('refund issued', { orderId, amountCents });
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Put values in `fields`, never in the message. Log ids, amounts and outcomes, never what a person typed (a refund's reason, a message, an address): the log is read by whoever operates the runtime, not only by the person the data is about. Docs: https://docs.kindgi.com/v0.3/guides/tools/write-a-tool/#log-from-a-tool
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Iterating on a tool, testing it, and shared code
|
|
2
|
+
|
|
3
|
+
## Iterating on a tool
|
|
4
|
+
|
|
5
|
+
Edit the source file (`tools/<tool>/index.ts`), save. The next
|
|
6
|
+
`kindgi runs start` sees the change — new input/output schema, new
|
|
7
|
+
description, new handler behavior. No version bump, no restart, no
|
|
8
|
+
re-registration ceremony. Source is truth in dev.
|
|
9
|
+
|
|
10
|
+
The `version` field is a **semver contract for humans** — it declares
|
|
11
|
+
what callers can rely on. Bump it because the *contract with
|
|
12
|
+
downstream callers* changed (removed a field, tightened a type,
|
|
13
|
+
narrowed enum values), not because you saved the file. If you're
|
|
14
|
+
iterating in dev and the shape isn't finalized, leave `version`
|
|
15
|
+
alone.
|
|
16
|
+
|
|
17
|
+
**When version matters:** `kindgi deploy` publishes to a durable
|
|
18
|
+
production registry that enforces the immutable `(id, version)`
|
|
19
|
+
contract — a re-publish of the same version with different bytes
|
|
20
|
+
gets rejected. That's when semver discipline kicks in. The deploy
|
|
21
|
+
tooling surfaces the check; you don't have to think about it while
|
|
22
|
+
authoring.
|
|
23
|
+
|
|
24
|
+
**Downstream callers still pin ranges.** An agent's `tools: [{id,
|
|
25
|
+
version: '^0.1.0'}]` picks the highest active version matching the
|
|
26
|
+
range at run start. Compatible tool updates (patch, minor) reach the
|
|
27
|
+
agent without editing agent source; breaking updates (major) require
|
|
28
|
+
the agent-author to opt in.
|
|
29
|
+
|
|
30
|
+
## Testing a tool
|
|
31
|
+
|
|
32
|
+
Put a tool's tests beside it, `tools/<tool>/index.test.ts`. Discovery skips `*.test.*` and `*.spec.*` files (`.ts`, `.js`, `.mjs`, `.cjs`), so the indexer never loads a test as a primitive: don't move tests elsewhere to keep them out. `invokeTool(tool, input, ctx)` from `@kindgi/sdk/define` calls the tool the way Kindgi does, schemas included, and returns a `Result`. `toolContextForTest()` builds its `ctx` (tenant `tenant-test`, run `run-test`, an abort signal) and takes what to change: `env`, `secrets`, `projectId`.
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
import { invokeTool, toolContextForTest } from '@kindgi/sdk/define';
|
|
36
|
+
|
|
37
|
+
const result = await invokeTool(lookupOrder, { orderId: 'ord_1001' }, toolContextForTest());
|
|
38
|
+
// a tool with env values: invokeToolForTest(tool, input, { env: { STORE_URL: '…' } })
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
`kindgi test` runs the pack's tests with vitest (`vitest run`; `--watch` keeps watching). In an app Kindgi was added to, `kindgi init` wrote the vitest config (`vitest.config.mts`, or `.ts` in a `"type": "module"` app): it runs `kindgi/**/*.test.ts`, and its `resolve.alias` copies `tsconfig.json`'s `paths`, which vitest doesn't read. A test that can't resolve an alias the app's code uses (`@/…`) means the two differ: update the config's alias, not the import.
|
|
42
|
+
|
|
43
|
+
## Shared code
|
|
44
|
+
|
|
45
|
+
Code several tools share (schemas, a client, helpers) goes outside `tools/`, for example in `lib/` beside it. Discovery loads every `.ts`, `.js` and `.mjs` file under `tools/` (tests aside) as a primitive, so a helper there fails the index. Don't put it in the app's own source either: import the app's functions from where they are (unless their module imports `server-only`: see mistake 12 in `common-mistakes.md`; for a CommonJS package's default import, see mistake 13 in `common-mistakes.md`), and keep what's Kindgi's in the pack's folder.
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Configuration, secrets and connections
|
|
2
|
+
|
|
3
|
+
## Configuration and secrets
|
|
4
|
+
|
|
5
|
+
A secret that belongs to the tenant — an API key a customer gives you — is declared, and read from `ctx.secrets`:
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const defined = defineTool({
|
|
9
|
+
// …id, description, version, input, output, effects…
|
|
10
|
+
needsSpec: { secrets: { CITATOR_KEY: { type: 'string', minLength: 20 } } },
|
|
11
|
+
handler: async (input, ctx) => {
|
|
12
|
+
const key = ctx.secrets?.CITATOR_KEY;
|
|
13
|
+
// …
|
|
14
|
+
},
|
|
15
|
+
});
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
The runtime resolves every declared secret on every call, for the call's tenant, in its env (`KINDGI_ENV`; in `kindgi dev`, `local`: the pack's `.env` and `.env.local`, then Kindgi's own `.kindgi/secrets.env`, where `kindgi secrets set` writes). It checks each value against its schema, and fails the call, naming the secret, when one is missing or doesn't match. A declared secret is required, so in a runtime call `ctx.secrets` holds it; it's optional in the type because a unit test builds its own context and passes `secrets: { CITATOR_KEY: '…' }`. **An optional secret** has a schema that names `null` (`{ type: ['string', 'null'] }`; an unconstrained `{}` stays required): one never stored, or empty, is left out of `ctx.secrets`, and the call goes on, its log line naming it. One revoked, or gone at its provider though mapped, still fails the call. Optional secrets need runtime 0.1.6 or later; an older runtime requires them.
|
|
19
|
+
|
|
20
|
+
A value that differs per tenant, org or project but isn't secret (a base URL, a region, an account id) is an **env value**: declared in `needsSpec.env`, read from `ctx.env`:
|
|
21
|
+
|
|
22
|
+
```ts
|
|
23
|
+
const defined = defineTool({
|
|
24
|
+
// …id, description, version, input, output, effects…
|
|
25
|
+
needsSpec: {
|
|
26
|
+
env: {
|
|
27
|
+
ORDERS_BASE_URL: { type: 'string', pattern: '^https://' },
|
|
28
|
+
ORDERS_REGION: { type: 'string', enum: ['eu', 'us'], default: 'eu' },
|
|
29
|
+
},
|
|
30
|
+
},
|
|
31
|
+
handler: async ({ orderId }, ctx) => ({
|
|
32
|
+
url: `${ctx.env?.ORDERS_BASE_URL}/${ctx.env?.ORDERS_REGION}/orders/${orderId}`,
|
|
33
|
+
}),
|
|
34
|
+
});
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
- **Which value a call gets:** its project's, else its org's, else the tenant's, in the runtime's env; a schema `default` makes a name optional. The values a call used are recorded with it, so a retry or a resume sees the same ones.
|
|
38
|
+
- **Setting them:** `kindgi env set ORDERS_REGION us --scope=project:<project-id> --env=local` (or `--scope=tenant`, for every project). Changing a value that's already set takes `--force`.
|
|
39
|
+
- **A declared value nobody set** stops the call before the tool runs. For a tool `acme-orders.needs-account` that declares `ACME_ACCOUNT_ID`, the message reads:
|
|
40
|
+
```text
|
|
41
|
+
precondition-failed: Tool "acme-orders.needs-account" was not run: env-value-missing: tool "acme-orders.needs-account" needs env value "ACME_ACCOUNT_ID" in env "local", and none is set for project e889c1f5-eae7-45dc-8669-5bd029a5d85c, its org, or the tenant. Set it: kindgi env set ACME_ACCOUNT_ID <value> --scope=project:e889c1f5-eae7-45dc-8669-5bd029a5d85c --env=local (or --scope=tenant, for every project)
|
|
42
|
+
```
|
|
43
|
+
- **Not secret:** env values are recorded with each run that uses them and shown in its journal. A credential is a secret (`needsSpec.secrets`), never an env value.
|
|
44
|
+
- **Schemas compile strictly:** each `needsSpec` schema must compile as the runtime compiles it (an unknown keyword is refused), and an env `default` is a string; otherwise `defineTool` returns `invalid-tool-definition`, and a deploy fails with `deployment-validation-failed`.
|
|
45
|
+
- **In a unit test:** call it with `invokeToolForTest(tool, input, { env: { … } })` (`@kindgi/sdk/define`). It decides the values as a run does: a name you don't set takes its schema's `default`, each value is checked, and the handler sees the declared names only. A missing or invalid one is `precondition-failed` (`env-value-missing`, `env-value-invalid`). Plain `invokeTool` passes `ctx.env` as given (the runtime decides a pack tool's env itself).
|
|
46
|
+
|
|
47
|
+
Everything else comes from the process environment: `process.env.CITATOR_URL`. The pack service runs with the pack's env files in `kindgi dev` (less any secret stored with `kindgi secrets set`, which a tool declares and reads from `ctx.secrets`, and any model provider's key, which no tool gets), and with the container's environment in an image. Declare the names your code reads in `kindgi.config.ts`, `env: { required: ['CITATOR_URL'], optional: [...] }`: a deployment injects exactly those, a pack service missing a required one isn't ready and says which, and `kindgi dev` warns about it. In an image the pack service also drops every variable the pack doesn't declare before your code loads (`kindgi dev` keeps them), so an undeclared name works locally and is unset once deployed: declare every name the code reads. Values per environment go in `environments.<name>.env`, secrets only as references.
|
|
48
|
+
|
|
49
|
+
## Connections to other apps
|
|
50
|
+
|
|
51
|
+
A tool that acts in another app as a person or as the company (sending mail, opening an issue) declares a **connection**, never a secret holding someone's token, and reads the call's short-lived token from `ctx.connections` (runtime 0.2.0 or later):
|
|
52
|
+
|
|
53
|
+
```ts
|
|
54
|
+
import { ConnectionRejectedError, defineTool } from '@kindgi/sdk/define';
|
|
55
|
+
|
|
56
|
+
const defined = defineTool({
|
|
57
|
+
// …id, description, version, input, output, effects…
|
|
58
|
+
needsSpec: {
|
|
59
|
+
connections: { mail: { app: 'google', permissions: ['gmail.send'], as: 'person' } },
|
|
60
|
+
},
|
|
61
|
+
handler: async (input, ctx) => {
|
|
62
|
+
const mail = ctx.connections?.mail;
|
|
63
|
+
const res = await fetch(url, { headers: { authorization: `Bearer ${mail?.accessToken}` } });
|
|
64
|
+
if (res.status === 401) throw new ConnectionRejectedError('mail');
|
|
65
|
+
// …
|
|
66
|
+
},
|
|
67
|
+
});
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
- **Whose connection:** `as: 'person'` is the connection of the person the run acts as (whoever started it, or the owner of the schedule or webhook that did); `as: 'company'` is the company's. The runtime picks it from the run, never from the input or the model. A run that acts as nobody (a service account's, an event's) fails a `'person'` call with `connection-needs-a-person`.
|
|
71
|
+
- **The token:** `accessToken` is current for this call only. Never keep, print, log or return it; it never reaches the journal. `account.label` says whose account it is (`ana@acme.com`).
|
|
72
|
+
- **A 401 from the app:** throw `new ConnectionRejectedError('mail')`, with the connection's name (a second argument can say what the app answered, never the token), or your own error with it as its `cause`. The runtime renews the token once and runs the call again; refused again, the call fails with `connection-rejected`.
|
|
73
|
+
- **An HTTP tool** sends the token with no code: `authorization: { kind: 'connection', connection: 'mail' }`.
|
|
74
|
+
- **Ready-made tools:** for Gmail, Outlook, GitHub and LinkedIn, include one from `@kindgi/connections` instead of writing it. Each is a factory, and the pack names its id; there's no default, because a tool id is the tenant's, filed under the first project that publishes it. A tool folder's `index.ts` is one line, `export default gmailSend({ id: 'acme.gmail.send' })` (`@kindgi/connections/google/tools`, with `gmailMessagesList`, `gmailMessageGet` and `gmailReply`); `githubIssuesList`, `githubLabelsList` and `githubIssuesAddLabels` come from `@kindgi/connections/github/tools`, `outlookSend`, `outlookMessagesList`, `outlookMessageGet` and `outlookReply` from `@kindgi/connections/microsoft/tools`, and `linkedinPost` from `@kindgi/connections/linkedin/tools`. Agents and approval settings use the pack's id. A mail tool acts in the person's own mailbox; `as: 'company'` makes it use the company's. `as` is part of what the tool declares, so an id you've published keeps its `as`: give a company mailbox its own tool id.
|
|
75
|
+
- **Your own calls to an app's API:** `appFetch(ctx, { entry, connection, toolId, url, method, headers, body })` from `@kindgi/connections/app-fetch` sends the connection's token as the only `Authorization`, paces calls by the entry's `rateLimits` (`toolId` is what its `costs` charge; 1 when not listed), tries a busy answer again, and throws a 401 as `ConnectionRejectedError`.
|
|
76
|
+
- **Unattended runs:** in a run no person started, a tool that uses an `as: 'person'` connection waits for that person's yes, reads included: `mutating: false` doesn't lift it. The agent's `conversationPolicy.hitl` can't lift it; only a trigger's `toolsAllowedUnattended` can, which a project admin sets, as for `needsApprovalWhenUnattended` (below). In a run a new mail started, the mail is a stranger's text, so two more calls wait: a call of a tool that isn't `mutating: false`, and, when the trigger reads the company's mailbox, a read with the company's connection to that app. The mail trigger's `toolsAllowedUnattended` lifts these too.
|
|
77
|
+
- **In a unit test:** `ctx.connections` is optional in the type; build a context with `toolContextForTest({ connections: { mail: { accessToken: 'test', account: { id: '1', label: 'ana@acme.com' }, … } } })` (`@kindgi/sdk/define`).
|
|
78
|
+
|
|
79
|
+
Docs: https://docs.kindgi.com/v0.3/guides/connections/use-a-connection/
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Read-only tools, a person's yes, and outside data
|
|
2
|
+
|
|
3
|
+
## Read-only tools: `mutating`
|
|
4
|
+
|
|
5
|
+
`mutating: false` declares that the tool changes nothing outside Kindgi; it only reads. Leaving `mutating` out counts as mutating, the same as `true`. It decides four things, in both authoring modes:
|
|
6
|
+
|
|
7
|
+
- **Dry runs.** `kindgi runs start --dry-run` runs a tool only if it's `mutating: false` and its `effects` declare no `writes`, `deletes`, `spawns-run`, `emits-event` or `external-side-effect`. The first other tool stops the run with `dry-run-effectful-tool`.
|
|
8
|
+
- **Approval gates.** When an agent turns tool gates on (`conversationPolicy.hitl.tools`) and sets neither an override for the tool nor a `default`, a `mutating: false` tool runs straight through, and any other tool asks for approval on its first use.
|
|
9
|
+
- **Running together.** When a model step asks for several calls at once, an agent runs calls to read-only tools (`mutating: false`, no changing effect) that come one after another at the same time, up to 8 of them, and a call to any other tool on its own, in the order the model asked. A call that may wait, for another system's answer or for the person who started the run to connect an app, runs on its own too. A call a `before-tool` guardrail judges doesn't start until the calls before it have finished, so its check sees them as calls that ran; the read-only calls after it can run alongside it. Once a call needs an approval, the step's calls that haven't started run one at a time. The results reach the conversation and the model in the model's order either way. An agent with `parallelToolCalls: false` runs every call on its own.
|
|
10
|
+
- **Runs a new mail started.** A call of a tool that isn't `mutating: false` waits for a person's yes, unless the mail trigger's `toolsAllowedUnattended` lists it.
|
|
11
|
+
|
|
12
|
+
So declare `mutating: false` on every tool that only reads, and never on one that writes.
|
|
13
|
+
|
|
14
|
+
A tool that writes also says what it writes, in `effects`, beside `mutating: true`:
|
|
15
|
+
|
|
16
|
+
```ts
|
|
17
|
+
effects: [{ kind: 'writes', resource: 'acme:refunds' }],
|
|
18
|
+
mutating: true,
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
The kinds are `reads`, `writes`, `deletes`, `network`, `spawns-run`, `emits-event`, `external-side-effect` and `sensitive-data-egress` (`EFFECT_KINDS` in `@kindgi/tools`); `defineTool` refuses any other. `resource` is free text naming what the tool touches. A read-only tool keeps `effects: []`.
|
|
22
|
+
|
|
23
|
+
## A person's yes when no one is watching: `needsApprovalWhenUnattended`
|
|
24
|
+
|
|
25
|
+
`needsApprovalWhenUnattended: true` declares that a person must say yes to every call of the tool in a run no person started: one a schedule, a webhook, a new mail, an event or a service account's key started. In such a run:
|
|
26
|
+
|
|
27
|
+
- **An agent's call waits** for a person's yes, whatever the agent's `conversationPolicy.hitl.tools` says. The agent's rules can't lift it: agents are reused, and the tool's author knows what it does.
|
|
28
|
+
- **A flow's tool step that calls it is refused.** Put the tool behind an agent step instead, whose call waits.
|
|
29
|
+
- **A trigger can lift both, for its own runs only.** A project admin names the tool in a webhook trigger's, a schedule's or a mail trigger's `toolsAllowedUnattended` (`--allow-unattended=<tool-id>` on `kindgi webhooks`, `kindgi schedules` and `kindgi mail-triggers`). In that trigger's runs, the tool is then as in a run a person started. The tenant's approval rules still apply.
|
|
30
|
+
|
|
31
|
+
In a run a person started (signed in, or with their own API key), the agent's rules decide, as for any tool. Absent means `false`. Declare it on a tool that moves money or changes something people rely on:
|
|
32
|
+
|
|
33
|
+
```ts
|
|
34
|
+
effects: [{ kind: 'writes', resource: 'acme:refunds' }],
|
|
35
|
+
mutating: true,
|
|
36
|
+
needsApprovalWhenUnattended: true,
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Docs: https://docs.kindgi.com/v0.3/guides/approvals/ask-before-a-tool-runs/#when-no-person-started-the-run
|
|
40
|
+
|
|
41
|
+
A step can run more than once (resumed after an approval, retried after a failure, run again after a crash), so a tool that writes uses `ctx.idempotencyKey`: the same every time this call runs, different for every other call. Pass it to the system you write to (an `Idempotency-Key` header, a client reference, a unique column) or look for it there first, and a refund never goes out twice. Not `ctx.requestId`: a model's call id is only unique within one of its answers. The key is absent outside a run and from a runtime before 0.1.6. Docs: https://docs.kindgi.com/v0.3/guides/tools/write-a-tool/#make-a-side-effect-happen-once
|
|
42
|
+
|
|
43
|
+
## Results that are text from outside: `resultIsOutsideData`
|
|
44
|
+
|
|
45
|
+
`resultIsOutsideData: true` says the tool's results are text from outside that nobody checked, such as a mail someone sent or a page a search found. An agent reads each result of the tool as data, not instructions, in that turn and in every later turn's history. It changes what the model is told, not what the tool may do: pair it with approvals for anything an agent could do with what it read. Absent means `false`.
|
|
@@ -14,8 +14,8 @@ description: >
|
|
|
14
14
|
not the agent.
|
|
15
15
|
type: core
|
|
16
16
|
library: "@kindgi/sdk"
|
|
17
|
-
version: "0.
|
|
18
|
-
sdk_version: "0.
|
|
17
|
+
version: "0.2.0"
|
|
18
|
+
sdk_version: "0.3.0-rc.0"
|
|
19
19
|
pack_languages: [node, python, java, scala]
|
|
20
20
|
sources:
|
|
21
21
|
- packages/cli/src/commands/judgments.ts
|
|
@@ -71,54 +71,18 @@ corrections and the tools to change each layer.
|
|
|
71
71
|
details. Never paste a whole run, or any customer's data, into the pack,
|
|
72
72
|
a commit, or a message.
|
|
73
73
|
|
|
74
|
-
##
|
|
74
|
+
## The loop
|
|
75
75
|
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
76
|
+
1. **Pull the corrections**: the live version's "no" judgments, each read
|
|
77
|
+
in full with its run. [pull-corrections.md](pull-corrections.md)
|
|
78
|
+
2. **Sort each correction to its layer**: the table below.
|
|
79
|
+
3. **Make the change, with a test per correction.**
|
|
80
|
+
[make-the-change.md](make-the-change.md)
|
|
81
|
+
4. **Publish, and ask a person to make it live**, with the guides to point
|
|
82
|
+
people to. [publish-and-promote.md](publish-and-promote.md)
|
|
79
83
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
```
|
|
83
|
-
|
|
84
|
-
`agents live` shows the version a run would use for the project (and
|
|
85
|
-
`agents live-versions` lists every scope with a live version). List that
|
|
86
|
-
version's "no" judgments — and, as you work through them, newer versions':
|
|
87
|
-
|
|
88
|
-
```sh
|
|
89
|
-
kindgi judgments list --agent=acme-store.refunds --agent-version=1.0.3 --verdict=no
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
- `--agent-version` (it needs `--agent`) scopes to one version — the live one,
|
|
93
|
-
to start. Older versions' corrections may already be fixed; without a version
|
|
94
|
-
you get every version's.
|
|
95
|
-
- `--verdict=no` keeps only the corrections. (Leave it off to see what the
|
|
96
|
-
experts confirmed as right, too.)
|
|
97
|
-
- `--class=<judge-class-id>` keeps only one judge class — use it to take
|
|
98
|
-
only the experts' corrections once the project names its experts.
|
|
99
|
-
|
|
100
|
-
Kindgi doesn't hand out corrections: two people's coding agents can pull the
|
|
101
|
-
same "no" judgments, so agree outside Kindgi who takes which before you start.
|
|
102
|
-
|
|
103
|
-
Read each one in full for the reason the expert gave and the run it's on:
|
|
104
|
-
|
|
105
|
-
```sh
|
|
106
|
-
kindgi judgments show <judgment-id>
|
|
107
|
-
kindgi runs get <run-id>
|
|
108
|
-
kindgi runs journal <run-id>
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
`runs get` is the run's snapshot; `runs journal` is its durable record,
|
|
112
|
-
step by step, where the agent's tool calls and their arguments are. Between
|
|
113
|
-
the correction's reason and the run, you have what the agent did and what it
|
|
114
|
-
should have done.
|
|
115
|
-
|
|
116
|
-
An expert can also have said exactly what the run should have done (runtime
|
|
117
|
-
0.2.0 or later): the judgment then has a `correction`, with `calls` (each
|
|
118
|
-
`{ tool, args }`, with only the argument fields that matter) and `notCalled`
|
|
119
|
-
(the tools it shouldn't have called). Take those as what the pack test
|
|
120
|
-
asserts. It's the expert's: never record or change a correction yourself.
|
|
121
|
-
Test cases built from the run check each new version against it.
|
|
84
|
+
One correction taken through all four steps:
|
|
85
|
+
[worked-example.md](worked-example.md).
|
|
122
86
|
|
|
123
87
|
## Step 2 — Sort each correction to its layer
|
|
124
88
|
|
|
@@ -127,136 +91,7 @@ Test cases built from the run check each new version against it.
|
|
|
127
91
|
| "Never refund shipping", "final sale: no refund", "over 50 on a first order, ask a person" | a rule in **code** — the tool checks it every time; a cap sends the case to a tool that always asks | edit the tool and add a test (your language's `kindgi-authoring-tools`), then `kindgi deploy` |
|
|
128
92
|
| "Our return window is 14 days, not 30" | a **setting** the owner can change | publish the settings block, derive a version that pins it (Step 4) |
|
|
129
93
|
| "Always have a person approve a refund" | an **approval rule** — a tenant policy that makes a tool always ask; it keys on the tool, not its arguments | a person sets it (the owner or the approver); point them to it, don't change it in code |
|
|
130
|
-
| "Store credit before cash", "too stiff", "say when the money arrives" | the agent's **instructions** | leave it to Improve, run on the test cases (`kindgi proposals improve`, the improve guide) |
|
|
94
|
+
| "Store credit before cash", "too stiff", "say when the money arrives" | the agent's **instructions** | leave it to Improve, run on the test cases (`kindgi proposals improve`, the improve guide in `publish-and-promote.md`) |
|
|
131
95
|
|
|
132
96
|
If a correction fits none of these, treat it as a code rule when it must
|
|
133
97
|
hold every time, and as instructions when it's a matter of judgment.
|
|
134
|
-
|
|
135
|
-
## Step 3 — Make the change, with a test per correction
|
|
136
|
-
|
|
137
|
-
Make each change in the layer you sorted it to, and add **one pack test per
|
|
138
|
-
correction** so the mistake can't come back:
|
|
139
|
-
|
|
140
|
-
- **Code (a tool):** add the check in the tool, and a test that the corrected
|
|
141
|
-
case is now refused or handled right (the 20-day-old order is refused, the
|
|
142
|
-
final-sale item isn't refunded). Follow your pack's language:
|
|
143
|
-
`kindgi-authoring-tools` (TypeScript), `kindgi-python-authoring-tools`,
|
|
144
|
-
`kindgi-java-authoring-tools`, or `kindgi-scala-authoring-tools`. Run the
|
|
145
|
-
pack's tests the way that skill shows — `kindgi test` wraps `vitest` for a
|
|
146
|
-
Node pack; a Python, Java or Scala pack runs its own test runner.
|
|
147
|
-
- **A setting:** decide the key, its limits and its default, and publish the
|
|
148
|
-
settings block (Step 4). Add a test that the tool reads the setting and
|
|
149
|
-
acts on it.
|
|
150
|
-
- **Instructions for Improve:** you don't write the prompt — you make sure the
|
|
151
|
-
corrected cases are in the test cases, then let an Improve run try wordings
|
|
152
|
-
on them (`kindgi proposals improve`). It writes a *suggested* change only
|
|
153
|
-
when a wording does better on those cases, and a person
|
|
154
|
-
decides whether to take it; nothing is applied on its own.
|
|
155
|
-
|
|
156
|
-
Write tests with invented data (Step 1, rule 3). A correction that belongs
|
|
157
|
-
to the owner (a setting they'd rather own) or an approver (an approval rule)
|
|
158
|
-
is theirs to make — say so, and give them the one command or the console
|
|
159
|
-
page, don't do it for them.
|
|
160
|
-
|
|
161
|
-
## Step 4 — Publish, and ask to make it live
|
|
162
|
-
|
|
163
|
-
For a **setting** change, publish the block, then derive a new agent version
|
|
164
|
-
that pins it — no code change:
|
|
165
|
-
|
|
166
|
-
```sh
|
|
167
|
-
kindgi blocks publish acme.store-policy \
|
|
168
|
-
--block-version=1.0.1 --project=<project-id> \
|
|
169
|
-
--settings=@store-policy.json --schema=@store-policy.schema.json
|
|
170
|
-
kindgi agents derive acme-store.refunds \
|
|
171
|
-
--from=1.0.3 --setting=acme.store-policy=1.0.1 --label="return window 14 days"
|
|
172
|
-
```
|
|
173
|
-
|
|
174
|
-
- `blocks publish` takes `--block-version` (a semver, not `--version`) and a
|
|
175
|
-
required `--project`. `--settings` is the block's values; `--schema`
|
|
176
|
-
constrains them. (`--prompt` publishes a prompt block instead; the two are
|
|
177
|
-
mutually exclusive.)
|
|
178
|
-
- `agents derive --setting=<block-id>=<version>` re-pins a block the version
|
|
179
|
-
*already* references to its new version (repeat `--setting`/`--prompt` for
|
|
180
|
-
several). It makes the first free patch after the agent's highest version and
|
|
181
|
-
prints it — read that version from its output rather than assuming one. If it
|
|
182
|
-
answers `reused`, an identical version already exists: use that, don't derive
|
|
183
|
-
again.
|
|
184
|
-
|
|
185
|
-
For a **code** change, `kindgi deploy` publishes the new version with the
|
|
186
|
-
changed tool.
|
|
187
|
-
|
|
188
|
-
Deriving needs `publish` on the agent, `blocks publish` needs `write` on the
|
|
189
|
-
project, and promoting needs `promote` — the permission that makes a version
|
|
190
|
-
live, which an agent's admins hold.
|
|
191
|
-
|
|
192
|
-
If the person keeps test cases built from the corrected runs, try the new
|
|
193
|
-
version on them first:
|
|
194
|
-
|
|
195
|
-
```sh
|
|
196
|
-
kindgi eval-runs start <test-set-id> --project=<project-id> \
|
|
197
|
-
--agent=acme-store.refunds --agent-version=1.0.4 --wait
|
|
198
|
-
```
|
|
199
|
-
|
|
200
|
-
Where an expert said what the run should have done, its summary's
|
|
201
|
-
`corrections` says how many of those the version now passes (`passed` of
|
|
202
|
-
`checked`). No release gate policy checks it: quote it in the promote
|
|
203
|
-
`--reason`.
|
|
204
|
-
|
|
205
|
-
Either way, ask a person to make the new version live, naming the
|
|
206
|
-
corrections it fixes. Check first what will happen, then ask:
|
|
207
|
-
|
|
208
|
-
```sh
|
|
209
|
-
kindgi agents promote acme-store.refunds 1.0.4 --project=<project-id> --check
|
|
210
|
-
kindgi agents promote acme-store.refunds 1.0.4 --project=<project-id> \
|
|
211
|
-
--reason="Fixes 1 correction: our return window is 14 days"
|
|
212
|
-
```
|
|
213
|
-
|
|
214
|
-
- The scope is `--tenant` (every project), `--org=<org-id>`, or
|
|
215
|
-
`--project=<project-id> [--segment=<key:value>]…` (one project).
|
|
216
|
-
- `--check` changes nothing and answers whether it *would* make it live,
|
|
217
|
-
needs an approval, or fails the release gate policy.
|
|
218
|
-
- `--reason` is kept on the record — write what the corrections were, so the
|
|
219
|
-
approver reads the change in plain words.
|
|
220
|
-
- The outcome is `promoted`, or `pending-approval` when the scope's release
|
|
221
|
-
gate policy has `approvals` and a second person must approve — the "a
|
|
222
|
-
second person checks it" step, working as it should. (Whoever asked can't
|
|
223
|
-
approve their own request, unless the policy sets `separateApprover: false`.)
|
|
224
|
-
|
|
225
|
-
## A worked example: the return window
|
|
226
|
-
|
|
227
|
-
An expert corrects a refund: "Shouldn't have refunded — delivered 20 days
|
|
228
|
-
ago, our window is 14." You pull it (Step 1), and see the window is a store
|
|
229
|
-
policy, not a one-off.
|
|
230
|
-
|
|
231
|
-
- **If the agent already reads a `returnWindowDays` setting:** publish the
|
|
232
|
-
block with `returnWindowDays: 14` and derive a version that pins it
|
|
233
|
-
(Step 4). Tell the owner they can change this themselves in the console —
|
|
234
|
-
Edit as a new version, then Derive a version.
|
|
235
|
-
- **If it doesn't read one yet:** this is a code change, in two parts. First
|
|
236
|
-
publish the settings block with the key — `returnWindowDays` in its schema
|
|
237
|
-
(0–365, whole days) and `14` in its values — because a deploy that references
|
|
238
|
-
a block with no published version is refused ("publish the block first").
|
|
239
|
-
Then, in code, make the refund tool read `returnWindowDays` and refuse an
|
|
240
|
-
order delivered more than that many days ago (offering a person), add the
|
|
241
|
-
block to what the agent references, and add a test: an order delivered 20
|
|
242
|
-
days ago is refused (`kindgi-authoring-tools`). `kindgi deploy` publishes the
|
|
243
|
-
version with the new reference. (`agents derive` only re-pins a block the
|
|
244
|
-
version already references, so it can't introduce this one — the deploy
|
|
245
|
-
does.) The schema and value live outside the pack and outside code review, so
|
|
246
|
-
show them to the person and name them in the `--reason`.
|
|
247
|
-
|
|
248
|
-
Then ask to make it live: `kindgi agents promote … --reason="Fixes 1
|
|
249
|
-
correction: our return window is 14 days"`. The refund limit — the most a
|
|
250
|
-
refund can be without a person — is never raised here: the cap stays in the
|
|
251
|
-
tool's code, and the large-refund tool still always asks.
|
|
252
|
-
|
|
253
|
-
## References
|
|
254
|
-
|
|
255
|
-
- Experts record corrections in the console, or with
|
|
256
|
-
`kindgi judgments add --run=<run-id> --item=<key> --no --reason=<text>`.
|
|
257
|
-
- The guide for a person:
|
|
258
|
-
<https://docs.kindgi.com/v0.2/guides/evals/fix-what-experts-corrected/>
|
|
259
|
-
- Letting Improve try wordings:
|
|
260
|
-
<https://docs.kindgi.com/v0.2/guides/evals/improve-an-agent/>
|
|
261
|
-
- Changing a setting or a prompt by hand:
|
|
262
|
-
<https://docs.kindgi.com/v0.2/guides/evals/change-a-prompt-or-setting/>
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Step 3 — Make the change, with a test per correction
|
|
2
|
+
|
|
3
|
+
Make each change in the layer you sorted it to, and add **one pack test per
|
|
4
|
+
correction** so the mistake can't come back:
|
|
5
|
+
|
|
6
|
+
- **Code (a tool):** add the check in the tool, and a test that the corrected
|
|
7
|
+
case is now refused or handled right (the 20-day-old order is refused, the
|
|
8
|
+
final-sale item isn't refunded). Follow your pack's language:
|
|
9
|
+
`kindgi-authoring-tools` (TypeScript), `kindgi-python-authoring-tools`,
|
|
10
|
+
`kindgi-java-authoring-tools`, or `kindgi-scala-authoring-tools`. Run the
|
|
11
|
+
pack's tests the way that skill shows — `kindgi test` wraps `vitest` for a
|
|
12
|
+
Node pack; a Python, Java or Scala pack runs its own test runner.
|
|
13
|
+
- **A setting:** decide the key, its limits and its default, and publish the
|
|
14
|
+
settings block (Step 4, `publish-and-promote.md`). Add a test that the tool
|
|
15
|
+
reads the setting and acts on it.
|
|
16
|
+
- **Instructions for Improve:** you don't write the prompt — you make sure the
|
|
17
|
+
corrected cases are in the test cases, then let an Improve run try wordings
|
|
18
|
+
on them (`kindgi proposals improve`). It writes a *suggested* change only
|
|
19
|
+
when a wording does better on those cases, and a person
|
|
20
|
+
decides whether to take it; nothing is applied on its own.
|
|
21
|
+
|
|
22
|
+
Write tests with invented data (rule 3 under "What stays with the person" in `SKILL.md`). A correction that belongs
|
|
23
|
+
to the owner (a setting they'd rather own) or an approver (an approval rule)
|
|
24
|
+
is theirs to make — say so, and give them the one command or the console
|
|
25
|
+
page, don't do it for them.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# Step 4 — Publish, and ask to make it live
|
|
2
|
+
|
|
3
|
+
For a **setting** change, publish the block, then derive a new agent version
|
|
4
|
+
that pins it — no code change:
|
|
5
|
+
|
|
6
|
+
```sh
|
|
7
|
+
kindgi blocks publish acme.store-policy \
|
|
8
|
+
--block-version=1.0.1 --project=<project-id> \
|
|
9
|
+
--settings=@store-policy.json --schema=@store-policy.schema.json
|
|
10
|
+
kindgi agents derive acme-store.refunds \
|
|
11
|
+
--from=1.0.3 --setting=acme.store-policy=1.0.1 --label="return window 14 days"
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
- `blocks publish` takes `--block-version` (a semver, not `--version`) and a
|
|
15
|
+
required `--project`. `--settings` is the block's values; `--schema`
|
|
16
|
+
constrains them. (`--prompt` publishes a prompt block instead; the two are
|
|
17
|
+
mutually exclusive.)
|
|
18
|
+
- `agents derive --setting=<block-id>=<version>` re-pins a block the version
|
|
19
|
+
*already* references to its new version (repeat `--setting`/`--prompt` for
|
|
20
|
+
several). It makes the first free patch after the agent's highest version and
|
|
21
|
+
prints it — read that version from its output rather than assuming one. If it
|
|
22
|
+
answers `reused`, an identical version already exists: use that, don't derive
|
|
23
|
+
again.
|
|
24
|
+
|
|
25
|
+
For a **code** change, `kindgi deploy` publishes the new version with the
|
|
26
|
+
changed tool.
|
|
27
|
+
|
|
28
|
+
Deriving needs `publish` on the agent, `blocks publish` needs `write` on the
|
|
29
|
+
project, and promoting needs `promote` — the permission that makes a version
|
|
30
|
+
live, which an agent's admins hold.
|
|
31
|
+
|
|
32
|
+
If the person keeps test cases built from the corrected runs, try the new
|
|
33
|
+
version on them first:
|
|
34
|
+
|
|
35
|
+
```sh
|
|
36
|
+
kindgi eval-runs start <test-set-id> --project=<project-id> \
|
|
37
|
+
--agent=acme-store.refunds --agent-version=1.0.4 --wait
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Where an expert said what the run should have done, its summary's
|
|
41
|
+
`corrections` says how many of those the version now passes (`passed` of
|
|
42
|
+
`checked`). No release gate policy checks it: quote it in the promote
|
|
43
|
+
`--reason`.
|
|
44
|
+
|
|
45
|
+
Either way, ask a person to make the new version live, naming the
|
|
46
|
+
corrections it fixes. Check first what will happen, then ask:
|
|
47
|
+
|
|
48
|
+
```sh
|
|
49
|
+
kindgi agents promote acme-store.refunds 1.0.4 --project=<project-id> --check
|
|
50
|
+
kindgi agents promote acme-store.refunds 1.0.4 --project=<project-id> \
|
|
51
|
+
--reason="Fixes 1 correction: our return window is 14 days"
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
- The scope is `--tenant` (every project), `--org=<org-id>`, or
|
|
55
|
+
`--project=<project-id> [--segment=<key:value>]…` (one project).
|
|
56
|
+
- `--check` changes nothing and answers whether it *would* make it live,
|
|
57
|
+
needs an approval, or fails the release gate policy.
|
|
58
|
+
- `--reason` is kept on the record — write what the corrections were, so the
|
|
59
|
+
approver reads the change in plain words.
|
|
60
|
+
- The outcome is `promoted`, or `pending-approval` when the scope's release
|
|
61
|
+
gate policy has `approvals` and a second person must approve — the "a
|
|
62
|
+
second person checks it" step, working as it should. (Whoever asked can't
|
|
63
|
+
approve their own request, unless the policy sets `separateApprover: false`.)
|
|
64
|
+
- `--check` and the request exit 3 when the answer is no or the request
|
|
65
|
+
waits for an approval. Read the printed outcome: exit 3 isn't an error.
|
|
66
|
+
|
|
67
|
+
## References
|
|
68
|
+
|
|
69
|
+
- Experts record corrections in the console, or with
|
|
70
|
+
`kindgi judgments add --run=<run-id> --item=<key> --no --reason=<text>`.
|
|
71
|
+
- The guide for a person:
|
|
72
|
+
<https://docs.kindgi.com/v0.3/guides/evals/fix-what-experts-corrected/>
|
|
73
|
+
- Letting Improve try wordings:
|
|
74
|
+
<https://docs.kindgi.com/v0.3/guides/evals/improve-an-agent/>
|
|
75
|
+
- Changing a setting or a prompt by hand:
|
|
76
|
+
<https://docs.kindgi.com/v0.3/guides/evals/change-a-prompt-or-setting/>
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Step 1 — Pull the corrections
|
|
2
|
+
|
|
3
|
+
The corrections are the agent's "no" judgments. Start from the version the
|
|
4
|
+
agent is running now, so you don't re-fix something an earlier version already
|
|
5
|
+
fixed:
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
kindgi agents live acme-store.refunds --project=<project-id>
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
`agents live` shows the version a run would use for the project (and
|
|
12
|
+
`agents live-versions` lists every scope with a live version). List that
|
|
13
|
+
version's "no" judgments — and, as you work through them, newer versions':
|
|
14
|
+
|
|
15
|
+
```sh
|
|
16
|
+
kindgi judgments list --agent=acme-store.refunds --agent-version=1.0.3 --verdict=no
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
- `--agent-version` (it needs `--agent`) scopes to one version — the live one,
|
|
20
|
+
to start. Older versions' corrections may already be fixed; without a version
|
|
21
|
+
you get every version's.
|
|
22
|
+
- `--verdict=no` keeps only the corrections. (Leave it off to see what the
|
|
23
|
+
experts confirmed as right, too.)
|
|
24
|
+
- `--class=<judge-class-id>` keeps only one judge class — use it to take
|
|
25
|
+
only the experts' corrections once the project names its experts.
|
|
26
|
+
|
|
27
|
+
Kindgi doesn't hand out corrections: two people's coding agents can pull the
|
|
28
|
+
same "no" judgments, so agree outside Kindgi who takes which before you start.
|
|
29
|
+
|
|
30
|
+
Read each one in full for the reason the expert gave and the run it's on:
|
|
31
|
+
|
|
32
|
+
```sh
|
|
33
|
+
kindgi judgments show <judgment-id>
|
|
34
|
+
kindgi runs get <run-id>
|
|
35
|
+
kindgi runs journal <run-id>
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
`runs get` is the run's snapshot; `runs journal` is its durable record,
|
|
39
|
+
step by step, where the agent's tool calls and their arguments are. Between
|
|
40
|
+
the correction's reason and the run, you have what the agent did and what it
|
|
41
|
+
should have done.
|
|
42
|
+
|
|
43
|
+
An expert can also have said exactly what the run should have done (runtime
|
|
44
|
+
0.2.0 or later): the judgment then has a `correction`, with `calls` (each
|
|
45
|
+
`{ tool, args }`, with only the argument fields that matter) and `notCalled`
|
|
46
|
+
(the tools it shouldn't have called). Take those as what the pack test
|
|
47
|
+
asserts. It's the expert's: never record or change a correction yourself.
|
|
48
|
+
Test cases built from the run check each new version against it.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# A worked example: the return window
|
|
2
|
+
|
|
3
|
+
An expert corrects a refund: "Shouldn't have refunded — delivered 20 days
|
|
4
|
+
ago, our window is 14." You pull it (Step 1, `pull-corrections.md`), and see the window is a store
|
|
5
|
+
policy, not a one-off.
|
|
6
|
+
|
|
7
|
+
- **If the agent already reads a `returnWindowDays` setting:** publish the
|
|
8
|
+
block with `returnWindowDays: 14` and derive a version that pins it
|
|
9
|
+
(Step 4, `publish-and-promote.md`). Tell the owner they can change this themselves in the console —
|
|
10
|
+
Edit as a new version, then Derive a version.
|
|
11
|
+
- **If it doesn't read one yet:** this is a code change, in two parts. First
|
|
12
|
+
publish the settings block with the key — `returnWindowDays` in its schema
|
|
13
|
+
(0–365, whole days) and `14` in its values — because a deploy that references
|
|
14
|
+
a block with no published version is refused ("publish the block first").
|
|
15
|
+
Then, in code, make the refund tool read `returnWindowDays` and refuse an
|
|
16
|
+
order delivered more than that many days ago (offering a person), add the
|
|
17
|
+
block to what the agent references, and add a test: an order delivered 20
|
|
18
|
+
days ago is refused (`kindgi-authoring-tools`). `kindgi deploy` publishes the
|
|
19
|
+
version with the new reference. (`agents derive` only re-pins a block the
|
|
20
|
+
version already references, so it can't introduce this one — the deploy
|
|
21
|
+
does.) The schema and value live outside the pack and outside code review, so
|
|
22
|
+
show them to the person and name them in the `--reason`.
|
|
23
|
+
|
|
24
|
+
Then ask to make it live: `kindgi agents promote … --reason="Fixes 1
|
|
25
|
+
correction: our return window is 14 days"`. The refund limit — the most a
|
|
26
|
+
refund can be without a person — is never raised here: the cap stays in the
|
|
27
|
+
tool's code, and the large-refund tool still always asks.
|